the new server.dat?

K, since I've given up on it, cba wasting more time. I'll give you what I've done, you can try and finish it if you want.

Code:

int decrypt_server_dot_dat()
{
    FILE* f = fopen("server.dat", "rb");
    FILE* g = fopen("n.dat", "rb");

    fseek(f, 0, SEEK_END);
    int filelen = ftell(f);
    uint8_t* bufferin = new uint8_t[filelen];


    fseek(f, 0, SEEK_SET);
    fread(bufferin, 1, filelen, f);
    fclose(f);

    //rotate bytes in 'n' because BN_bin2bn() requires it...
    uint8_t* reversed_n = new uint8_t[0x100];
    char c;
    for (int i=0;i<0x100;i++)
    {
        fread((void*)&c, 1, 1, g);
        reversed_n[0xFF-i] = c;
    }
    fclose(g);

    RSA* rsa = RSA_new();
    rsa->n = BN_new();
    rsa->e = BN_new();
    BN_bin2bn(reversed_n, 0x100, rsa->n);
    BN_hex2bn(&rsa->e, "01");
    rsa->n->flags = 1;
    rsa->e->flags = 1;
    rsa->flags = 6;
    rsa->references = 1;
    RSA_set_method(rsa, RSA_PKCS1_SSLeay());
    RSA_memory_lock(rsa);

    uint8_t* bufferout = new uint8_t[RSA_size(rsa)];

    int decrypted = RSA_public_decrypt(0x100, bufferin, bufferout, rsa, RSA_PKCS1_PADDING);

    RSA_free(rsa);

    if (decrypted==-1)
    {
        printf("%d", ERR_get_error());
        printf("Error decrypting file");
    }
    else
    {
        FILE* h = fopen("decrypted_server.dat", "wb");
        fwrite(bufferout, 1, decrypted, h);
        fclose(h);
    }
    return 0;
}

where n.dat is just a binary file containing

Code:

09 54 80 D8 E4 A2 87 0B 83 03 C7 B7 18 6E 3B C3 39 6E B6 71 BE 26 49 E5 50 66 A0 82 60 0A 3D CD 26 D6 F5 67 E7 33 0B A6 7D 2C 9F EF 60 85 91 EF 4C 95 B0 C2 11 2C B8 4E EE 97 1B CD 70 9E 62 FC FA F0 24 D7 92 83 C3 94 CC D2 54 81 64 34 9F E9 12 F1 3C 09 A2 78 50 5A B1 0C 45 F4 FA A7 9E BD 51 C2 E4 95 B0 0A 21 99 15 59 37 9B 1C FE 88 9E ED 7E 65 17 65 F5 20 A7 DA DE 47 2D 0C E6 80 97 D3 8D 0A 01 08 D2 26 17 24 7B BE 30 70 DC FE 93 76 A7 80 1A 67 BF DF 38 9B 6B 9B A1 3E 04 6E 29 99 98 29 95 23 5C 25 7B F8 E4 88 EE A5 E6 64 9B 89 3C 84 D7 6D A6 70 7E E0 35 47 44 A7 F2 20 22 16 B2 A3 AE B3 68 F1 44 B9 D8 39 8A 8A C7 9A A9 CA 49 3D 78 14 17 73 A0 40 AE 35 62 C7 3A B0 FA B3 46 E2 44 26 7B 10 0E 4F 08 D2 A9 03 2B 8A C3 D7 A7 10 C0 8E A6 7F 95 0C AC 6B 9B 33 BD F5 BE

Basically, I'm passing exactly the same arguments as the client to RSA_public_decrypt(), but getting a different result to the client. (It's returning -1/ERROR). I've traced this a little further and it appears to be a problem when RSA_eay_public_decrypt() calls BN_mod_exp_mont();. The first BIGNUM argument is returned with twice the length it should have, and I can't figure out what's up with it.

It could possibly be something to do with the latest commits on the file here:
[Only registered and activated users can see links. Click Here To Register...], where someone has tried messing with size_ts. TQ might be using one of these bad file versions perhaps. I've not got any more patience to recompile openssl and shit though.

A couple of pointers if you wanna gget debugging (for 5078)

Code:

User-defined labels, item 6
 Label=<libeay.RSA_eay_public_decrypt()> 00525D70
 Disassembly=MOV EAX,8

User-defined labels, item 8
 Label=<libeay.BN_mod_exp_mont()> 0052CF40
 Disassembly=MOV EAX,9C

Oh, and after decryption the stream needs inflating using zlib, but that's pretty self explanatory.

K, that's me done.