Tutorial: Introduction to Packets

11/06/2017 08:04 Spirited#1
Introduction
Hello there, this is my third attempt at writing this. I assume you found this tutorial because you have an interest in programming with packets. Before we start, you should know a bit about the basics of programming in C#. If you're relatively new, that's okay. I'll spend a lot of my time in this tutorial reviewing concepts. In general, this is a really good place to start if you want to review on your own time: [Only registered and activated users can see links. Click Here To Register...]. If you have a question for me at any time, feel free to reply below and I'll do my best to answer or redirect you to a good article.

Sockets
Before we talk about packets, let's talk about sockets. Sockets are a way for two applications to talk to one another over a network. You can think of this as a phone line, where each end is a phone (or socket). When the Conquer client is ready to talk to the game server, it dials it up. The game server is listening for that call from the client, and will accept the call once it comes in. How they talk are where packets and protocols come in. Protocols help us define how we talk over a network, while packets contain our actual message. Take a minute to review the video below about the internet, packets, and protocols.


In Conquer Online, we use TCP/IP to define how messages are exchanged between the client and server. IP defines how the packet routes to the server, while TCP defines how packets are broken down and reassembled in the right order (both shown in the video above). To us, however, we only see packets in full (for the most part). We don't have to worry about how it gets to the server; we only need to focus on what they say.

Byte Ordering
To understand how Conquer Online talks, we have to understand how it orders words (both in the figurative and literal sense). Messages are best represented as an array of bytes (bytes being the smallest addressable unit in programming). For example, in the string "Hello World" (assuming ASCII encoding), each character is one byte. So, the message "Hello World" would appear as the following packet:

Code:
 H  e  l  l  o     W  o  r  l  d
[48 65 6C 6C 6F 20 57 6F 72 6C 64]
If you're confused, the above is shown in hexadecimal. See this [Only registered and activated users can see links. Click Here To Register...] for converting characters to bytes. I used hexadecimal because the range of a byte in decimal is from 0 to 255. 255 is an ugly number to work with. In hexadecimal, the range is from 0 to FF. FF is an easier number to work with as a ceiling since it's the highest number it can be for two digits. It means that any number larger than FF or two digits can overflow nicely into the next byte of our message.

For numbers larger than 255, such as a word (ushort), double word (uint), or quad word (ulong), the value of that number overflows into the next bytes. For example, the word 1024 in hexadecimal is 400. This would be split across two bytes as 4 and 00. The problem is, in what order? It can either be written as [04 00] or [00 04]. This is called byte ordering, and is much like a dialect for speaking.

There are two types of byte orders: Little Endian and Big Endian. The goal of Big Endian is to store the most significant byte in the smallest address. This would mean that our hexadecimal number of 400 would appear as [00 00 04 00]. This is easy to read, as it reads from left to right. The goal of Little Endian is to store the least significant byte in the smallest address. This means our hexadecimal number of 400 would appear as [00 04 00 00]. This reads backwards, but the value in doing this is that a number such as 5 converted from a word to a byte wouldn't require changing the address. It would convert simply from [05 00 00 00] to just [05]. We didn't change the starting point of our array to do that, it just worked. Conquer Online uses Little Endian, and that's good because C# also uses Little Endian for bit conversions and streams. For help converting between decimal and hexadecimal, consider using the Windows Calculator in Programmer mode.

Structures
Now that we understand how Conquer Online orders words in messages, let's go over what Conquer Online is talking about. With every packet being sent, the client and server start with a header of two words. This header contains the length of the packet and the type of packet being sent. Here is an example: [18 00 15 27]. The first word is the length: 0018 (or 24 in decimal). The second word is the type: 2715 (10005 in decimal). This is the [Only registered and activated users can see links. Click Here To Register...] packet. Take a minute to look over the structure on the wiki. Each packet type has a different message that it sends. For example, the MsgWalk packet describes a step in the world, performed by a player or monster. Let's read the following packet dump for MsgWalk for patch 5517. The direction diagram below will also help.

[Only registered and activated users can see links. Click Here To Register...]

Code:
18 00 15 27 72 00 00 00 29 20 1A 00 00 00 00 00 
82 2D 3F 05 00 00 00 00 54 51 53 65 72 76 65 72
[Offset 04] 72 00 00 00 = 114: The direction of the player mod possible angles (8). 114 mod 8 = direction 2.
[Offset 08] 29 20 1A 00 = 1712169: The unique character ID making the step.
[Offset 12] 00 00 00 00 = 0: The type of movement being sent (walking).
[Offset 16] 82 2D 3F 05 = 88026498: Timestamp ([Only registered and activated users can see links. Click Here To Register...]).
[Offset 20] 00 00 00 00 = 0: Map ID (not always supplied).
[Offset 24] 54 51 53 65 72 76 65 72 = TQServer: The footer of the packet.

For Conquer Online patches greater than 5017, each packet from the game server ends with either "TQServer" or "TQClient", depending on where it is from. "TQServer" indicates that the packet is from the server, while "TQClient" indicates that the packet is from the client. The length of the footer is not included in offset 0 of the header (which is the length of the packet).

In Practice
Phew! I know that's a lot to take in, but now let's program this structure in C# (a lot easier). In this tutorial, I'll be using [Only registered and activated users can see links. Click Here To Register...] to read the structure into C#. In the example below, I am not handling the packet footer. Instead, I ignore it (as I don't have a need for it - it's really only good for debugging the source of a packet over the network as a packet dump).

Code:
var length = BitConverter.ToUInt16(Packet, 0);
var typeid = BitConverter.ToUInt16(Packet, 2);
var direction = BitConverter.ToUInt32(Packet, 4) % 8;
var character = BitConverter.ToUInt32(Packet, 8);
var typemove = BitConverter.ToUInt32(Packet, 12);
var timestamp = BitConverter.ToUInt32(Packet, 16);
var mapid = BitConverter.ToUInt32(Packet, 20);
If you're using a public source, which I recommend you reference a few anyways to get the hang of programming for Conquer Online, look at how your server handles packet reading and writing. For better examples of a socket system and packet handling, check out my source: [Only registered and activated users can see links. Click Here To Register...]. It has its problems, but is fully documented and easy to learn from. I hope this guide has helped you understand a bit more about how networking and packets work with Conquer Online. If you have any questions, or requests for better explanations, please let me know. Cheers.
11/07/2017 08:58 felixanius#2
glad to see you guys still there lol
nice tutorial
11/18/2017 13:25 zakkwilde_17#3
Congratulations dear, it will be of great help. Thanks for sharing
11/18/2017 13:37 wshbr#4
#sticky
11/18/2017 21:55 Spirited#5
Quote:
Originally Posted by wshbr View Post
#sticky
Thank you o:
05/17/2018 02:04 kiiD2NiCe#6
Could you explain how you get the offset value and what is an offset anyways
05/17/2018 02:12 Spirited#7
Quote:
Originally Posted by kiiD2NiCe View Post
Could you explain how you get the offset value and what is an offset anyways
An offset is a position in the array, otherwise called an index. Arrays are indexed starting from 0. So, for the array { 1, 2, 3 }, offset 0's value is 1, offset 1's value is 2, and offset 2's value is 3. For more information on indexes / offsets for arrays, review arrays for C#: [Only registered and activated users can see links. Click Here To Register...]
05/24/2018 13:17 Super Aids#8
Quote:
Originally Posted by Spirited View Post
An offset is a position in the array, otherwise called an index. Arrays are indexed starting from 0. So, for the array { 1, 2, 3 }, offset 0's value is 1, offset 1's value is 2, and offset 2's value is 3. For more information on indexes / offsets for arrays, review arrays for C#: [Only registered and activated users can see links. Click Here To Register...]
Please don't say array.

An offset is more than just a position in an array.

An offset is the position of data within a data segment such as an array, but it could also be a pointer.

And since this topic is about packets then you really shouldn't, since it may not always be an array or a backed buffer.

Conquer's packet layout can be streamed which will at most only have temporary buffers used by the networking protocol and no further down the chain. They could theoretically be completely converted to a data structure such as a struct, in which case the offset will not even be an index, but the position of the struct's memory the data will remain in.

Ex. for an offset 5

Array:

Code:
[1,      2,      3,      4,      5,      6]
                                             ^5
Result: 6

Pointer:

Code:
0x00000001 a
0x00000002 b
0x00000003 c
0x00000004 d
0x00000005 e
0x00000006 f
0x00000007 g
0x00000008 h
0x00000009 i
0x00000010 j
0x00000011 k


...

{0x00000005} (Pointer to 0x00000005)

0 e
1 f
2 g
3 h
4 i
5 j
Result: j

Struct:

Code:
struct Foo
{
    byte a; // = 1
    byte b; // = 2
    byte c; // = 3
    byte d; // = 4
    byte e; // = 5
    byte f; // = 6
    byte g; // = 7
    byte h; // = 8
    byte i; // = 9
    byte j; // = 10
}


...

{ 1, 2, 3, 4, 5, 6, 7, 8, 9 ,10 }
                      ^5
Result: Foo.f which is 6


---

Indentation is messed up so for some reason it won't point to the correct members.

It should point to 6 for arrays and 6 for struct.

Cba to fix it.
05/24/2018 17:29 Spirited#9
Uuuhhh.... I was more just trying to keep it simple. From my understanding, an offset is the distance from the base address of an array or data structure, and requires that all elements in that structure be the same size. In conquer, we work in terms of bytes since that's the smallest addressable unit. The simplest explanation of that is a byte array. If I added that to my tutorial, would that be sufficient?

Edit: I could go into stream encoding / decoding using their byte ordering rules if you'd like, which would utilize that definition.
07/29/2019 03:01 felipeboladao#10
hello, i know this is not quite the right place to ask this, but i would like to know where i start to set up an online conquer project, what is needed to start sockets first? packets would like to set up a project from the start and would like to take this doubt, thank you very much and sorry for posting in the wrong place.
07/29/2019 04:14 2Explosions#11
Quote:
Originally Posted by felipeboladao View Post
hello, i know this is not quite the right place to ask this, but i would like to know where i start to set up an online conquer project, what is needed to start sockets first? packets would like to set up a project from the start and would like to take this doubt, thank you very much and sorry for posting in the wrong place.
Sockets > Encryption > Packet structure > Packet processing.
07/30/2019 12:18 Super Aids#12
Quote:
Originally Posted by felipeboladao View Post
hello, i know this is not quite the right place to ask this, but i would like to know where i start to set up an online conquer project, what is needed to start sockets first? packets would like to set up a project from the start and would like to take this doubt, thank you very much and sorry for posting in the wrong place.
https://en.wikipedia.org/wiki/Network_socket
02/10/2024 21:18 AlhelalyMahamedd#13
Thank you very much for the explanation
11/16/2024 08:46 wazdzf#14
Thanks for the spirit of this post.I learned some knowledge.
But i meet with a trouble now.
Is there have byte length limit of server send packet for client?
like this picture. string send to client.but it is Incomplete.
11/18/2024 23:27 Spirited#15
Quote:
Originally Posted by wazdzf View Post
Thanks for the spirit of this post.I learned some knowledge.
But i meet with a trouble now.
Is there have byte length limit of server send packet for client?
like this picture. string send to client.but it is Incomplete.
You shouldn't hit any limit under normal circumstances going server -> client. The client has a capped length for its send buffer, though. I think it's around 1024 bytes (or something pretty small). So sometimes, you'll get a fragment of a message rather than the full message. Your socket system needs to read the length of the message from offset zero, check the length of the read from the socket, and make sure it's got the full message. Else, it needs to read more from the socket.

If you'd like an example, then Comet has a socket system that handles this already. It first decrypts the bytes, and then it splits the buffer up into multiple complete messages (leaving incomplete messages in the buffer for the next read):

[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]

This gets more complicated if you have to handle the DH key exchange in patches 5018+, because that sometimes gets cut off as well. You also have to consider the length of the 8-byte footer in the message when calculating the expected length. Comet has an example of handling that, too:

[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]