Handling TCP Packet Fragmentation.

[Danial Eugen]

Well after long time digging Network programming, one topic remains ambiguous to me. Packet Fragmentation, as far as i know TCP may not give you the message you sent as a whole but might fragment or rebuild it. So there is no guarantee that when i receive for example the Client's Login packet that it would be Full so how in all the source around here From Albetros to Pheonix i can't see any sort of Fragmentation Handling ? does it exists some how and i don't know/understand ?. Please i need a really detailed answer for this topic.

[TheComputerist]

The answer is within the API I believe. In the WinSock API there's a function named recv and it returns the number of bytes received, if the number of bytes don't add up to the number of bytes expected to receive then recv again. Does that help or?

[Spirited]

Project Phoenix does have fragmentation handling. I would encourage that you look over the source again.

[Danial Eugen]

Originally Posted by Spirited

Project Phoenix does have fragmentation handling. I would encourage that you look over the source again.

This part is a fragmentation Handling ? or you are referring to another part ?

    if (length == sizeof(PacketHeader) && passport != null)
    {
       ...
    }

Also i don't understand the part of sizeof(PacketHeader) which seems to be always 0 ?

I really need you to explain this to me in a very detailed way (do me that favor please).

[Spirited]

Originally Posted by Danial Eugen

This part is a fragmentation Handling ? or you are referring to another part ?

    if (length == sizeof(PacketHeader) && passport != null)
    {
       ...
    }

Also i don't understand the part of sizeof(PacketHeader) which seems to be always 0 ?

I really need you to explain this to me in a very detailed way (do me that favor please).

No, that's basic error checking...

[Ultimation]

i use a circular buffer, basically everytime i receive the data gets appened to the buffer, Pushed in if u like.

Then when handling the packets i pop the buffer based on the first 2 bytes being the packet length. If the buffer < than the expected length, i know the packet has not yet been fully received. thus wait for it

[Spirited]

Originally Posted by Ultimation

i use a circular buffer, basically everytime i receive the data gets appened to the buffer, Pushed in if u like.

Then when handling the packets i pop the buffer based on the first 2 bytes being the packet length. If the buffer < than the expected length, i know the packet has not yet been fully received. thus wait for it

That's how my socket system in Phoenix works as well, so he has an example of that available to him. It's not really that complex of a concept.

[Danial Eugen]

Well just a small more thing by waiting for it you mean ? just skip the current Handler ?, and what if i am clearing out the Buffer after each receive (to avoid overlapping ?) ?

[pro4never]

Originally Posted by Danial Eugen

Well just a small more thing by waiting for it you mean ? just skip the current Handler ?, and what if i am clearing out the Buffer after each receive (to avoid overlapping ?) ?

This all happens in your socket system LONG before you handle the actual packet.


When the socket system receives raw data (still encrypted) it decrypts the packets and adds them to a list of waiting bytes (with no concept of how long each packet is or what they represent. It's simply data waiting to be processed by the server)


You can then check the first 2 bytes to check the expected length of the next packet being processed (packet header). If there's not enough bytes to process it (20 packets in queue, packet length of 30) then you simply wait till there are more packets waiting in the buffer.


Clearer?

[SteveRambo]

Originally Posted by pro4never

This all happens in your socket system LONG before you handle the actual packet.


When the socket system receives raw data (still encrypted) it decrypts the packets and adds them to a list of waiting bytes (with no concept of how long each packet is or what they represent. It's simply data waiting to be processed by the server)


You can then check the first 2 bytes to check the expected length of the next packet being processed (packet header). If there's not enough bytes to process it (20 bytes in queue, packet length of 30) then you simply wait till there are more bytes waiting in the buffer.


Clearer?

Np bro!

[Danial Eugen]

Originally Posted by pro4never

This all happens in your socket system LONG before you handle the actual packet.


When the socket system receives raw data (still encrypted) it decrypts the packets and adds them to a list of waiting bytes (with no concept of how long each packet is or what they represent. It's simply data waiting to be processed by the server)


You can then check the first 2 bytes to check the expected length of the next packet being processed (packet header). If there's not enough bytes to process it (20 packets in queue, packet length of 30) then you simply wait till there are more packets waiting in the buffer.


Clearer?

So something like this ?

                Socket socket = asynchronousState.Socket;
                int length = socket.EndReceive(ar);
                if (8 <= length) {
                    if (null != ClientReceive) {
                        var bytes = new byte[length];
                        Array.Copy(asynchronousState.Buffer, bytes, length); //Copy data to our local array
                        var packetLength = BitConverter.ToUInt16(bytes, 0); //Read the packet length
                        if (packetLength > length) {
                            //Received bytes is less than expected so start receiving more ?
                            if (socket.Connected) {
                                socket.BeginReceive(asynchronousState.Buffer, 0, packetLength, SocketFlags.None, HandleAsyncReceive, asynchronousState); //Receive the remaining data
                            }
                            return;
                        }
                        ClientReceive(asynchronousState, bytes); //Announce receive
                        Array.Clear(asynchronousState.Buffer, 0, asynchronousState.Buffer.Length); //Clear the Buffer to begin receive more new data
                    }
                    if (socket.Connected) {
                        socket.BeginReceive(asynchronousState.Buffer, 0, asynchronousState.Buffer.Length, SocketFlags.None, HandleAsyncReceive, asynchronousState);
                        return;
                    }
                }
                DisposeSocket(asynchronousState);

Also do i need to edit the receive offset to avoid overwriting the buffer or the incoming information then will be the old + new bytes ?

[KraHen]

This would only work if the packet is decrypted already. Also, if ClientReceive is null, you should still handle what`s going on, there`s a small chance you`d miss something with the code you posted above.

Disregarding these, yes, the general idea is correct. In a CO context, you`d decrypt the packet, check if the length is corresponding with the length you received in the packet header, and if not, you know that the next packet is still part of the current one.

[Danial Eugen]

Originally Posted by KraHen

This would only work if the packet is decrypted already. Also, if ClientReceive is null, you should still handle what`s going on, there`s a small chance you`d miss something with the code you posted above.

Disregarding these, yes, the general idea is correct. In a CO context, you`d decrypt the packet, check if the length is corresponding with the length you received in the packet header, and if not, you know that the next packet is still part of the current one.

I am just trying to get the whole idea infront of me so here is an updated version. Is this better/correct ?

                Socket socket = asynchronousState.Socket;
                int readLength = socket.EndReceive(ar);
                if (sizeof (PacketHeader) <= readLength) {
                    byte[] bytes = asynchronousState.Buffer.Take(readLength).ToArray();
                    ushort packetLength = BitConverter.ToUInt16(bytes, 0);
                    ushort packetType = BitConverter.ToUInt16(bytes, 2);
                    byte[] packetBody = bytes.Take(bytes.Length - 4).ToArray();
                    if (packetLength > readLength) {
                        //Fragmented Packet, receive more.
                        if (socket.Connected) {
                            socket.BeginReceive(asynchronousState.Buffer, 0, packetLength, SocketFlags.None, HandleAsyncReceive, asynchronousState);
                        }
                        return;
                    }
                    if (null != ClientReceive) {
                        //We have received the whole packet announce it.
                        ClientReceive(asynchronousState, new Packet
                        {
                            Header = new PacketHeader
                            {
                                Length = packetLength,
                                Type = packetType
                            },
                            Body = packetBody
                        });
                    }
                    Array.Clear(asynchronousState.Buffer, 0, asynchronousState.Buffer.Length); //Clear for new packet receiving.
                    if (socket.Connected) {
                        socket.BeginReceive(asynchronousState.Buffer, 0, asynchronousState.Buffer.Length, SocketFlags.None, HandleAsyncReceive, asynchronousState);
                        return;
                    }
                }
                DisposeSocket(asynchronousState); //Some condition didn't met so disconnect.

Do i need to set the receive index to the last write one or the next receive will receive both the old bytes as well as the new ones ?

socket.BeginReceive(asynchronousState.Buffer, 0, packetLength, SocketFlags.None, HandleAsyncReceive, asynchronousState) ?

[Y u k i]

yes

[Danial Eugen]

Originally Posted by Y u k i

yes

yes what ?