De/serializing packets

04/16/2020 06:56 2Explosions#1
Started a CO server emulator to spend time (internet down for ~3 weeks here, limited bandwidth on phone) and noticed how painful is to de/serialize packets in pretty much any open source emulator... well, i always thought that tbh., even in my previous projects.

So, as an excuse to learn a bit of System.Linq.Expressions, wich wasn't that hard as i thought, tried to avoid what the whole de/serialization code writing is. Keep in mind that this is still prototype code and before taking it any further, opinions and tips/recommendations would be be appreciated(bold to make it more relevant).

The code is based on two Type's properties iterator (tried to keep the code generic), the first one to access to the properties and the second to assign the values to the properties.

TypeIterator class:

IterableAttribute class:


Usage example:
Usage:

PacketSerializer class

PacketFieldAttribute class


SerializationMethods class, this one exposes what will happen to every property depending of it's type and attribute


Example with MsgAccount packet


Output of console is the expected.

Important things to notice:

Packet class is just a BinaryWriter like class, you can see a similar one in a lot of CO related projects (you can try it, just need to change UInt16 and string methods

There are errors in SerializationMethods class, check strings for example

Type alias are needed for generic types or arrays, the TypeIterator will search for ReadXX where XX is the type name, that is why the StringList alias is used for example.



cya!
04/16/2020 07:17 Spirited#2
So, maybe my opinion is a bit of an old one, but I appreciate good interface design. Golang is one of those languages that relies on good interface design. Their whole standard library can be extended through interfaces. I think in C#, using interfaces and inheritance is a simple and intuitive way to write packet interfaces. It doesn't beat around the bush.

Here's an example from Comet:
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]

A lot of protocol generators such as Google Protocol Buffers uses this design as well, they just generate the message definitions for you based on a proto template file. You could essentially do the same with Conquer Online - make a generator program that compiles packet definitions into message classes that fill that interface. That's what I plan on doing in my project (Golang has a gen command that reads in code comments and runs commands). Maybe I'll open source the tool with template definitions for Chimera and Comet so others can do the same.
04/16/2020 10:10 KraHen#3
This looks nice to the eye but I'm not sure it's really necessary for CO. Lately I tend to use good ol' structs. I became a big fan of this after using CO2_CORE_DLL.

PHP Code:
using CO2_CORE_DLL.Net;
using System;
using System.Runtime.InteropServices;

namespace 
AzurysGame.Packets
{
    public 
unsafe class MsgRegister Msg
    
{
        public const 
Int16 TYPE 1001;

        [
StructLayout(LayoutKind.SequentialPack 1)]
        public 
struct Structure
        
{
            public 
MsgHeader Header;
            public 
fixed byte Account[16];
            public 
fixed byte CharName[16];
            public 
fixed byte Password[16];
            public 
UInt16 Mesh;
            public 
UInt16 Job;
            public 
UInt32 UID;
        }

        public 
string CharName
        
{
            
get
            
{
                return 
Marshal.PtrToStringUTF8((IntPtr)pInfo->CharName);
            }
        }

        private 
StructurepInfo null;

        public 
Structure Payload()
        {
            return *
pInfo;
        }

        public 
MsgRegister(Byte[] packet) : base(packet)
        {
            
pInfo = (Structure*)((Byte*)(pBuffer));
        }
    }

04/18/2020 01:46 2Explosions#4
Quote:
Originally Posted by Spirited View Post
So, maybe my opinion is a bit of an old one, but I appreciate good interface design. Golang is one of those languages that relies on good interface design. Their whole standard library can be extended through interfaces. I think in C#, using interfaces and inheritance is a simple and intuitive way to write packet interfaces. It doesn't beat around the bush.

Here's an example from Comet:
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]

A lot of protocol generators such as Google Protocol Buffers uses this design as well, they just generate the message definitions for you based on a proto template file. You could essentially do the same with Conquer Online - make a generator program that compiles packet definitions into message classes that fill that interface. That's what I plan on doing in my project (Golang has a gen command that reads in code comments and runs commands). Maybe I'll open source the tool with template definitions for Chimera and Comet so others can do the same.
I will probably look for how protocol generators work now, right now, Internet connection down makes hard to research.

Quote:
Originally Posted by KraHen View Post
This looks nice to the eye but I'm not sure it's really necessary for CO. Lately I tend to use good ol' structs. I became a big fan of this after using CO2_CORE_DLL.

PHP Code:
using CO2_CORE_DLL.Net;
using System;
using System.Runtime.InteropServices;

namespace 
AzurysGame.Packets
{
    public 
unsafe class MsgRegister Msg
    
{
        public const 
Int16 TYPE 1001;

        [
StructLayout(LayoutKind.SequentialPack 1)]
        public 
struct Structure
        
{
            public 
MsgHeader Header;
            public 
fixed byte Account[16];
            public 
fixed byte CharName[16];
            public 
fixed byte Password[16];
            public 
UInt16 Mesh;
            public 
UInt16 Job;
            public 
UInt32 UID;
        }

        public 
string CharName
        
{
            
get
            
{
                return 
Marshal.PtrToStringUTF8((IntPtr)pInfo->CharName);
            }
        }

        private 
StructurepInfo null;

        public 
Structure Payload()
        {
            return *
pInfo;
        }

        public 
MsgRegister(Byte[] packet) : base(packet)
        {
            
pInfo = (Structure*)((Byte*)(pBuffer));
        }
    }

Using the same library, but my motivation is different than yours right now (I remember reading in your dev thread saying the motivation was to learn how the features work, using patterns you already know), trying to find and solve problems, from design patterns to performance related issues, doing benchmarks and a bit of documentación about it. Of course im looking for a more flexible and scalable server than what is currently available to force this whole experimental project worth.
Btw. I assumed that dynamic length packets could be a problem for CptSky's Msg class, forcing me to use the Write methods, but didnt really tríed hard.

Maybe I will start a dev thread to publish some results this thread, depending of progress.
04/18/2020 11:10 KraHen#5
Quote:
Originally Posted by 2Explosions View Post
I will probably look for how protocol generators work now, right now, Internet connection down makes hard to research.

Using the same library, but my motivation is different than yours right now (I remember reading in your dev thread saying the motivation was to learn how the features work, using patterns you already know), trying to find and solve problems, from design patterns to performance related issues, doing benchmarks and a bit of documentación about it. Of course im looking for a more flexible and scalable server than what is currently available to force this whole experimental project worth.
Btw. I assumed that dynamic length packets could be a problem for CptSky's Msg class, forcing me to use the Write methods, but didnt really tríed hard.

Maybe I will start a dev thread to publish some results this thread, depending of progress.
By all means, experimenting is the best thing you can do in programming, ever.

Regarding dynamic packets, yes, it can be a bit more complicated, however IMO it's really not an issue and there's really only two cases where that happens :

Packets with strings and the skill packet - just append the stringpacker/target buffer to the serialized packet. It might cost an extra allocation and that might be a performance issue, unless it isn't. I'm a huge fan of "make it work, then make it work better". Implement the simple version, then profile, if it's an issue, improve.
04/18/2020 16:15 CptSky#6
The last way I designed CO packets was done to try to minimize the memory allocations and keep the packed struct (because honestly, I still think it is the cleanest way to represent a packet... I tried other solutions but none is as straightforward).

The Msg base class provide auto-registration of derived classes, as such the factory of messages is created at runtime (the beautiful switch case we have in all our code bases). You create the proper message type after peeking at the headers, and after you can "deserialize" from a buffer into that message (it's just a memory copy). This allow me to queue typed messages and never copy the recv buffer of the socket. Also, the message factory relies on an object pool, so at some point, I should reach a point where all messages are just recycled again and again (which would put less pressure on the GC).

Here's an example for the MsgAccount (note that I wrote to extensions to the Encoding class to be able to safely get a NUL terminated string, but also to be able to get the bytes of a String into a buffer using an in-place algorithm -- instead of generating a temporary byte[] array).

Here's an example of the MsgTalk:

The StringPacker used to manage the string buffer at the end of the MsgTalk packet. It is a "view" on that buffer.

And finally some examples on how you can use these. So first, lets say you want to serialize as much messages in the send buffer:

And for the receive buffer (note there's no fragmentation handling in that code):

As always, feel free to reuse that code as you wish.