K, since I've given up on it, cba wasting more time. I'll give you what I've done, you can try and finish it if you want.
Code:
int decrypt_server_dot_dat()
{
FILE* f = fopen("server.dat", "rb");
FILE* g = fopen("n.dat", "rb");
fseek(f, 0, SEEK_END);
int filelen = ftell(f);
uint8_t* bufferin = new uint8_t[filelen];
fseek(f, 0, SEEK_SET);
fread(bufferin, 1, filelen, f);
fclose(f);
//rotate bytes in 'n' because BN_bin2bn() requires it...
uint8_t* reversed_n = new uint8_t[0x100];
char c;
for (int i=0;i<0x100;i++)
{
fread((void*)&c, 1, 1, g);
reversed_n[0xFF-i] = c;
}
fclose(g);
RSA* rsa = RSA_new();
rsa->n = BN_new();
rsa->e = BN_new();
BN_bin2bn(reversed_n, 0x100, rsa->n);
BN_hex2bn(&rsa->e, "01");
rsa->n->flags = 1;
rsa->e->flags = 1;
rsa->flags = 6;
rsa->references = 1;
RSA_set_method(rsa, RSA_PKCS1_SSLeay());
RSA_memory_lock(rsa);
uint8_t* bufferout = new uint8_t[RSA_size(rsa)];
int decrypted = RSA_public_decrypt(0x100, bufferin, bufferout, rsa, RSA_PKCS1_PADDING);
RSA_free(rsa);
if (decrypted==-1)
{
printf("%d", ERR_get_error());
printf("Error decrypting file");
}
else
{
FILE* h = fopen("decrypted_server.dat", "wb");
fwrite(bufferout, 1, decrypted, h);
fclose(h);
}
return 0;
}
where n.dat is just a binary file containing
Code:
09 54 80 D8 E4 A2 87 0B 83 03 C7 B7 18 6E 3B C3 39 6E B6 71 BE 26 49 E5 50 66 A0 82 60 0A 3D CD 26 D6 F5 67 E7 33 0B A6 7D 2C 9F EF 60 85 91 EF 4C 95 B0 C2 11 2C B8 4E EE 97 1B CD 70 9E 62 FC FA F0 24 D7 92 83 C3 94 CC D2 54 81 64 34 9F E9 12 F1 3C 09 A2 78 50 5A B1 0C 45 F4 FA A7 9E BD 51 C2 E4 95 B0 0A 21 99 15 59 37 9B 1C FE 88 9E ED 7E 65 17 65 F5 20 A7 DA DE 47 2D 0C E6 80 97 D3 8D 0A 01 08 D2 26 17 24 7B BE 30 70 DC FE 93 76 A7 80 1A 67 BF DF 38 9B 6B 9B A1 3E 04 6E 29 99 98 29 95 23 5C 25 7B F8 E4 88 EE A5 E6 64 9B 89 3C 84 D7 6D A6 70 7E E0 35 47 44 A7 F2 20 22 16 B2 A3 AE B3 68 F1 44 B9 D8 39 8A 8A C7 9A A9 CA 49 3D 78 14 17 73 A0 40 AE 35 62 C7 3A B0 FA B3 46 E2 44 26 7B 10 0E 4F 08 D2 A9 03 2B 8A C3 D7 A7 10 C0 8E A6 7F 95 0C AC 6B 9B 33 BD F5 BE
Basically, I'm passing exactly the same arguments as the client to RSA_public_decrypt(), but getting a different result to the client. (It's returning -1/ERROR). I've traced this a little further and it appears to be a problem when RSA_eay_public_decrypt() calls BN_mod_exp_mont();. The first BIGNUM argument is returned with twice the length it should have, and I can't figure out what's up with it.
It could possibly be something to do with the latest commits on the file here:
, where someone has tried messing with size_ts. TQ might be using one of these bad file versions perhaps. I've not got any more patience to recompile openssl and shit though.
A couple of pointers if you wanna gget debugging (for 5078)
Code:
User-defined labels, item 6
Label=<libeay.RSA_eay_public_decrypt()> 00525D70
Disassembly=MOV EAX,8
User-defined labels, item 8
Label=<libeay.BN_mod_exp_mont()> 0052CF40
Disassembly=MOV EAX,9C
Oh, and after decryption the stream needs inflating using zlib, but that's pretty self explanatory.
K, that's me done.