Thats the right attitude DestLuck
If you have the code of the server, you don't need to RE anything but I can admire wanting to learn this I got into server emulation many years ago.
I find spreadsheets helpful to layout the information I know, Packet ID's names of them, description of the action and addresses/sizes.
Coding your own MITM proxy between the game client and real server can also be useful to log and send data to either side to see what happens.
You might be interested in the book Developing Autonomous Bots for Online Games by Nick Cano.
Also I remember this defcon was quite amusing and informative (although they don't do a full dive more a talk about their escapades)
Okay so packet structures!
Most games have a packet id format of some kind eg a byte prefixed to data structure (might be variable in size or encoding data into bits) or perhaps they have a short noting how many bytes of data the other end should process next after that 2 bytes. (or 2 bytes + n) for other header info eg packet id, timestamps, sequence ids (although I don't know why you need those on TCP....) there are many other ways to transmit the data as well. (Look for if they use a known library such as Google ProtoBuffs, Captn Proto).
Once you find their ID system that is used to tell the other end what kind of data it is expecting to process or what kind of request it needs to execute, look for the switch, conditional if branching or array of function pointers that the ID is used against. I've mostly seen an array of functions, I guess its quicker?
If array of functions you can look to see where it is initialized and then you will see possibly an array of sizes and an array of functions at least that has been my experience with a couple of games then you can dump all of them out.
To get the ID's and sizes of packets is a bit more complex.
Look for the call to the API for send eg Win32's send, WSA send or sendTo there are a few others and IOCP? as well but eh. most games use send tbh.
Normally 1 or 2 calls back from the send is a Common "Game Send Method" which often is called from all places one end of the networking code wants to send to the other server. Some games prepare a packet and then send it. (This is not always the case however, some games without encrypt or their own custom headers will simply call send from everywhere)
"Prepare PacketID XX"();
"Game Send method"(data pointer maybe, length);
Look at the calls to the function or call stack, log or bp it and check if one of the arguments or data at the address of a pointer argument (ECX + 4 maybe?) is always a specific one when you write "Hello" into chat. For example BA might be the hex byte for the packet id for chat, you might see something like this in the data pointer.
A bunch of 00's after it and always a fixed length would indicate fixed length packet structures. (For example it might have a total length of 51 meaning the chat message string is always 50 in length.
Or it could be length prefixed for the chat message field. (Pascal string anyone?)
Or they might use an unsigned short (2 bytes)
Look for the Endiness (byte ordering) as well I have noticed that most games use little endian (Intel). I speculate because its easy for Windows based C++ games to simply copy the struct into existing memory.
Or Big Endian (Motorola)
Which is flipped around the other way.
(Look out for exploits if your game server does not check bounds and sends a different length but the client has a hard coded length ohh boy)
The Endianess dictates which way around you read & write your numbers that take up more than 1 byte.
Now if you see weird looking packets, and your sure its not encrypted and you see an x near the start of the data, its probably zlib compressed just run it through offzip and see what it says
To find the encryption/decryption?
Look for common crypts used with IDA's crypt analysis plugin?
Look for the simple for loop doing an XOR (key might be in the first packet, last byte in the packet).
If you are uncertain, some games have a lot of 00 bytes and 00 xored by ?? becomes ??.
Send some data that you KNOW eg the login packet might be a username and a password.
Send all W's for your password normally in game, what do you see in your packet logger/sniffer?
Example, no encryption.
Example Simple XOR encryption over the DATA only, Go on find my key, you know the source data is all W and you can see its the same Key for all the bytes because they are all now FA.
Now as you run your own server I would suggest to remove any encryption if any, because it will make packet sniffing a little easier without the added decryption step or needing to hook the client with a DLL and dump after decrypt on recv and before encrypt on send.
Sorry I'm making a lot of assumptions such as the game uses x86 assembly and runs on windows and probably uses TCP. I haven't actually played this game but I have a friend who is pretty keen on the pserver aspect of it.
It could use other operations like bit shifts, rotating etc. its all just obfuscating do the same things and you have solved it, or in some cases do the same things bu in reverse for encrypt/decrypt. Think of if you have a rubix cube and you have a ClearText message which is the whole Green Side is visible, but then you rotate the cube around or even shift the stickers left right up or down you then have an encrypted message, but if you repeat those same operations backwards you would decrypt it unless you stuffed up somewhere
It could also be a Block Cipher, look for repeating patterns, or with blowfish some developers dont pad their data to multiples of the block size so you see unencrypted data at the end. IDA has a crypto signature scan plugin.
Or they could use XOR with an array of keys, or use the sequence id as the key.
If the data appears completely random you might have ran into a decent encryption.
You can't compress random data very well by the way that can be a good quick test
I wrote on my blog how to find game send/recv and bypass crypt. But I lost my domain name. Please forgive me for the state of the site~ I can attempt to add a backlink if that is required providing I can still login to the admin area...

(I would use x96 dbg or IDA now days)

Useful for if you have a bunch of On this packet ID do this function and you want a way to easily distinguish them.
If using TCP as most MMORPG and other games do, know that the data is a stream, you will need to understand how to receive the data in chunks (buffering it to a point you can process), and send the data completely along the same. Most TCP tutorials will cover this in the form of a loop and checking the return value of the API call and size remaining, then if need be call it again continuing from where it left off.
If UDP then well, every datagram received is generally an entire message although it's not uncommon for games to re-implement some of the features they like from TCP on top of UDP, know that UDP packets can arrive out of order, or even not at all where as TCP has some designs of flow control, sequence and acknowledgement of delivery. This ones tricky.
The structures, might be bitfields or bitpacked this one is trickyer to work out you have to get in there and observe how the game expects to recv the data or what it does with it when sending. Some games will have a description of their on the wire structures that you might be able to RE and dump out
Gafferon Games - Glenn Fiedler talks about compression of structures in this way to use less bytes in good detail but his website seems to be down recently. (Way back machine or google cache might help)
Anyway sorry for going off on a tangent there, network communication is quite a large topic. I haven't even scratched the surface...
I would suggest to look into other Open source bots such as OpenKore and see how they do it for some potential ideas. But give it your best shot you don't have to worry about breaking the server or the admins banning you if you run the server
If the game does many advanced detection techniques look at the history of RS bots they ended up doing some pretty crazy things like moving the "mouse" in human like patterns rather than teleporting it everywhere. RS kept getting crafty every now and again to catch botters. It might not be a good game to focus on/learn on as a beginner if it does advanced protections... But since there are pservers I would say this is not the case yay!
Anti hack? Well you have the server your self so just strip it out if it is present.
One effective method to combat bot developers many years ago that made headless or packet based bots was to use a heartbeat system.
E.g. the server sends a special packet with some info and the client is expected to send back the correct reply within a short time window. The function for generating the heartbeat reply was often in another process and heavily obfuscated e.g. it would unpack the code as it executes then repack it.
Heartbeat emulation is one way around this, you must debug all of the scenarios, this works until they change it. One way around this if the heartbeats are without state was you fired a game client up and had it connect to the bot client on an emulated/faked server to get into world state. Then when the real game server would send the bot client a heart beat challenge (which normally you need to emulate the Anti Cheats heartbeat challenge code and respond within a minute or you get disconnected ~happens within 5 minutes), the bot client would just send the same packet to the game client, receive the response and forward it onto the real server.
Best of all this was possible with running it over the network to be pretty much undetected as far as the AC went only with a simple file mod on the client side.
Once you understand the protocol and some basic structs to get in walk around and chat/sell items etc you would probably want to start building something that has a state, goals to achieve (GOAP? in a bot I would like to see lol) and uses the knowldge you have of the packets to perform actions e.g. walk here attack this pick up item according to some filters, when inventory full, go back to town sell junk to npc. Some games are badly developed and you can simply teleport anywhere or even call NPC actions like selling items from anywhere Oops! Others you can simply say I attack these targets and if it does not distance check you will be able to slay everything without being near it as long as you know its instance id.
Wishing you lots of luck on your trial and error and learning