Sockets - large data handling

03/06/2017 13:39 XxharCs#1

Hi I would ask you for some help.
I already searched around on google but I always found bad examples or atleast I didn't understand them. (If I worked with sockets then mostly in Java and only chat stuff)

So I want to send large data over network, as I read send/recv has a max length and that I have to send data in chunks. Or don't I have to do that?

The large data is in my case a file, which I read in a char vector and want to send it. The files I want to send are between 100-500KB. (depends on which file)

Scenario:
Read file in a char vector, send that char array(vector.data()), client receives it and write to file.
Read to char vector and writing from char vector to file is no problem, only the part with sending/receiving.

Do you have any tips, or good references or snippets to such a scenario? Do I even have to send in chunks or will the TCP protocol handle it?

Thanks in advice!

03/06/2017 16:15 Jeoni#2

Basically if you use TCP sockets, the dividing of the data to packets is done by the TCP implementation (so you don't have to care about maximum packet sizes). Since send is a blocking function, it could use the specified buffer instead of an internal one. In that case, you may freely choose how much data you pass to send in one call, only restricted by how big you want your buffer to be. Passing a large portion of data should in theory be more efficient since TCP does only need to pack one packet which is not full per send-call. More send-calls means more half-full packets which causes slightly more overhead. But with all those TCP flow and congestion control mechanisms that won't matter that much, I guess.
With best regards
Jeoni

03/06/2017 16:22 XxharCs#3

Quote:

Originally Posted by Jeoni

Basically if you use TCP sockets, the dividing of the data to packets is done by the TCP implementation (so you don't have to care about maximum packet sizes). Since send is a blocking function, it could use the specified buffer instead of an internal one. In that case, you may freely choose how much data you pass to send in one call, only restricted by how big you want your buffer to be. Passing a large portion of data should in theory be more efficient since TCP does only need to pack one packet which is not full per send-call. More send-calls means more half-full packets which causes slightly more overhead. But with all those TCP flow and congestion control mechanisms that won't matter that much, I guess.
With best regards
Jeoni

So it would be enough to first send a packet with the buffer length, so the client can allocate the needed buffer with the size and then I receive on the client side as long the received length isn't null?

03/06/2017 18:48 Jeoni#4

Quote:

Originally Posted by XxharCs

So it would be enough to first send a packet with the buffer length, so the client can allocate the needed buffer with the size and then I receive on the client side as long the received length isn't null?

Yes, you may first send the message length / data size as a single value. One call to recv to receive the data size and then you just need to call recv once more with the data size and you buffer as params to receive the whole data. However, depending on the size of the data, that may block the thread for some time.
With best regards
Jeoni

03/06/2017 19:03 th0rex#5

Quote:

Originally Posted by XxharCs

So it would be enough to first send a packet with the buffer length, so the client can allocate the needed buffer with the size and then I receive on the client side as long the received length isn't null?

But depending on what program you write malicious people may just send really large data sizes, without the data and if you then allocate all the RAM up front your RAM may be gone rather fast. You actually don't need to allocate a really big buffer at first. You just keep sending the size at first, but don't allocate the whole buffer just yet. Rather receive chunks of some size (you don't need to send chunks, just calling read with some size like 4096 and a buffer with 4096 bytes will work; the OS has to buffer the data) and then merge them together. That way the attacker would have to waste bandwidth as well. If attacks like that are not a concern then just allocate the whole buffer instantly.

03/06/2017 21:02 warfley#6

As already stated TCP takes care of all the packetizing, you don't need (and even can't) make assumptions about how it is packetized. All you need to know is, if you write data into the socket stream it will be received on the other side. You only need to check the return values of recv and send, especially receive can only read as many bytes as already been received by the NIC, so if you try to read 1k bytes it might happen that the function recv might return with a smaller value, and you need to read the rest of the data again.

Also if the network buffer is full the system can't receive more, and it requires a recv call to clear the buffer to continue reading.

An example to read large amount of data could be:

Code:

size_t sz = getPacketLen(); // some dummy function for receiving the size of the data
byte *buffer = malloc(sz);
for (size_t i = 0; i < sz; i = recv(sock, &buff[i], sz-i, 0);

of course this is neither a safe nor a complete implementation, its just how you could do it

But you should first think about if you really need to implement your own protocol, you should rather search for a protocol which fits your needs. I don't know what you want to do, but as i think from reading your post you might as well be fine with just using existing FTP libraries, or using HTTP/2 (not HTTP/1.1 it sucks). Because developing your own protocol is hard (especially the testing is pretty difficult). So the first step in developing a network application is first to check what is already existing.

Quote:

Originally Posted by Jeoni

Passing a large portion of data should in theory be more efficient since TCP does only need to pack one packet which is not full per send-call. More send-calls means more half-full packets which causes slightly more overhead. But with all those TCP flow and congestion control mechanisms that won't matter that much, I guess.

Well this should be prevented by nagels which is the default packetizing behavior. If you want your own packetizing behavior you could use the CORK option, but i wouldn't recommend this to OP
Also the overhead through flow control and congestion control is processing overhead on the end hosts. The real problem with many TCP packets is the Overhead for the routers, and that really gets critical and is a big problem, but TCP isn't the big problem here, users who use/develop unfitted protocols are the real problem like those who use HTTP/1.1 for an messaging app or something similar.
With TCP it might get some overhead (20 Bytes per TCP header), but because you can't assume anything about the packetizing behavior of TCP you can't optimize this (Except for using CORK), so you don't really have to care about this.

03/08/2017 10:07 XxharCs#7

Quote:
Originally Posted by warfley
As already stated TCP takes care of all the packetizing, you don't need (and even can't) make assumptions about how it is packetized. All you need to know is, if you write data into the socket stream it will be received on the other side. You only need to check the return values of recv and send, especially receive can only read as many bytes as already been received by the NIC, so if you try to read 1k bytes it might happen that the function recv might return with a smaller value, and you need to read the rest of the data again.

Also if the network buffer is full the system can't receive more, and it requires a recv call to clear the buffer to continue reading.

An example to read large amount of data could be:
Code:
size_t sz = getPacketLen(); // some dummy function for receiving the size of the data
byte *buffer = malloc(sz);
for (size_t i = 0; i < sz; i = recv(sock, &buff[i], sz-i, 0);
of course this is neither a safe nor a complete implementation, its just how you could do it

But you should first think about if you really need to implement your own protocol, you should rather search for a protocol which fits your needs. I don't know what you want to do, but as i think from reading your post you might as well be fine with just using existing FTP libraries, or using HTTP/2 (not HTTP/1.1 it sucks). Because developing your own protocol is hard (especially the testing is pretty difficult). So the first step in developing a network application is first to check what is already existing.

Well this should be prevented by nagels which is the default packetizing behavior. If you want your own packetizing behavior you could use the CORK option, but i wouldn't recommend this to OP
Also the overhead through flow control and congestion control is processing overhead on the end hosts. The real problem with many TCP packets is the Overhead for the routers, and that really gets critical and is a big problem, but TCP isn't the big problem here, users who use/develop unfitted protocols are the real problem like those who use HTTP/1.1 for an messaging app or something similar.
With TCP it might get some overhead (20 Bytes per TCP header), but because you can't assume anything about the packetizing behavior of TCP you can't optimize this (Except for using CORK), so you don't really have to care about this.

How would I go about encoding? The files get corrupted when sending them from linux to windows

03/08/2017 14:43 florian0#8

Quote:

Originally Posted by XxharCs

How would I go about encoding? The files get corrupted when sending them from linux to windows

TCP does it for you. The protocol assures that all bytes are transmitted in the correct order and without any errors.
If you still wanna add some doublechecking, just add MD5 or CRC32 checksum at the end of the transfer.

Or are you talking about Little and Big Endian? Because that shouldn't be a problem between Linux and Windows ...

03/08/2017 15:45 XxharCs#9

Quote:

Originally Posted by florian0

TCP does it for you. The protocol assures that all bytes are transmitted in the correct order and without any errors.
If you still wanna add some doublechecking, just add MD5 or CRC32 checksum at the end of the transfer.

Yeah I know that.

Quote:

Originally Posted by florian0

Or are you talking about Little and Big Endian? Because that shouldn't be a problem between Linux and Windows ...

Yeah, was talking about that, but fixed it.
I didn't receive it correctly.

Code:

while (((len = recv(connectSocket, buf, file_size, 0)) > 0) && (remain_data > 0))
	{
		bufHolder.write(buf, len);
		remain_data -= len;
		std::cout << "Receive "<< len << "bytes and remaining: "<< remain_data << "bytes\n";
	}

03/08/2017 15:59 warfley#10

Quote:
Originally Posted by XxharCs
Yeah I know that.

Yeah, was talking about that, but fixed it.
I didn't receive it correctly.
Code:
while (((len = recv(connectSocket, buf, file_size, 0)) > 0) && (remain_data > 0))
	{
		bufHolder.write(buf, len);
		remain_data -= len;
		std::cout << "Receive "<< len << "bytes and remaining: "<< remain_data << "bytes\n";
	}

the endianness only matters for primitive Types larger than 1 byte, so for integers of size 16, 32 or 64 bit, and for double and single precision floating point numbers. Because you are transmitting the data as a byte array, the file data should not differ from little to big endian (because arrays as compound types are not effected by the endianess). So you only need to take care about the length information you are sending. Big endian is as network byte order defined, so everything you send should be in that byte order. therefore you can use the functions htons (Host to network short) htonl (host to network long) and for Network to host ntohs and ntohl. These functions will convert from and to network byte order (Big endian) to whatever byte order your system uses

So if you say the data you receive is corrupted, can you post an example file (a text file so we can read it) and what is received by the other Host? Because anything else would be wild guessing, and this wont help you

03/08/2017 16:45 XxharCs#11

Quote:

Originally Posted by warfley

So if you say the data you receive is corrupted, can you post an example file (a text file so we can read it) and what is received by the other Host? Because anything else would be wild guessing, and this wont help you

As I said above, fixed it, and the snippet was the solution how I did it^^
I tried your solution, then one more and then I came to the solution above which worked for me.

The data was corrupted, because not everything was received lel

03/08/2017 17:43 warfley#12

Quote:

Originally Posted by XxharCs

As I said above, fixed it, and the snippet was the solution how I did it^^
I tried your solution, then one more and then I came to the solution above which worked for me.

The data was corrupted, because not everything was received lel

Yeah I made a mistake in my code above, instead of i = recv... it should be i+=recv...