Well, I used Lenas tutorials to get in that topic:

(I'm not finished, but it's good until now and many people referenced to that)
And of course you need to analyse the Client (disassemble or decompile, if possible) to see which method of encryption it does with the packets it sends. Or which way of decryption is used on packets it receives.
How would you else manage to find the encryption (/ decryption) method? It's unefficient to guess the encryption (and for that the password, too) at more advanced encryptions.