Re: Named pipes vs. Unix sockets

On 11/22/2024 03:38, vallor wrote:

On 22 Nov 2024 07:29:16 GMT, vallor <vallor@cultnix.org> wrote in
<lqaq6cF8btnU3@mid.individual.net>:
 

However, the speed appears to be limited by dd in my examples -- setting
a block size to fill the pipe/packets seems to increase throughput:
>
$ nc -l -U -N /tmp/socket > /dev/null & time -p ( dd if=/dev/zero
count=$[1024*1024*4] bs=1024 | nc -U -N /tmp/socket > /dev/null )

 Realized the bottleneck would be the pipe between dd and nc, so wrote
a program to connect to /tmp/socket and spew data at it -- it sends
46950 212992-byte buffers (9999974400 bytes) in 2.41 seconds.
(4149366971 bytes/second, or 4.1GB/s).
 (The default "MTU" for a Linux Unix socket connection
is 212992 bytes.  Default pipe size is 8*512 bytes.)
 

Yeah, this has been the experience of our testing as well. We can also bump the MTU up a bit to get sockets to do better and we've also modified some of the socket code to our environment so it's able to do around 12GB/s. We think we can tune it further to get about 15GB/s from sockets. At the moment we have to do 4.6GB per 0.5sec to keep up, and 7GB per 0.5sec to stay ahead. We've done everything we can to squeeze out as much performance from pipes as we could but there just isn't much we can do anymore. But yeah, sockets are way more capable of large data intake vs pipes, and this shows as well in a lot of research that's been done. We've been testing sockets for about the last year and the results have shown we are keeping up better then we are with pipes right now.
And for tribute to pipes, we've been using pipes in this project for the last 30 years. That's a pretty great track record for something to have been useful for such a long time and not be replaced with something else until now. Sadly sockets will likely bottom out in the next 10-15 years and we'll have to switch to something else. But for now, it's working.
--
Phillip Frabott
----------
- Adam: Is a void really a void if it returns?
- Jack: No, it's just nullspace at that point.
----------