Re: xxd -i vs DIY Was: C23 thoughts and opinions

On Wed, 29 May 2024 12:23:51 +0100
bart <bc@freeuk.com> wrote:

On 29/05/2024 10:38, Michael S wrote:

On Wed, 29 May 2024 00:54:23 +0100
bart <bc@freeuk.com> wrote:
 

>
I suspect that your system just has a much faster fgetc
implementation. How long does an fgetc() loop over a 100MB input
take on your machine?
>
On mine it's about 2 seconds on Windows, and 3.7 seconds on WSL.
Using DMC, it's 0.65 seconds.
 

 
Your suspicion proved incorrect, but it turned out to be pretty good
question!
 
 

 

$ time ../quick_xxd/getc_test.exe uu.txt
193426754 byte. xor sum 1.
 
real    0m3.604s
user    0m0.000s
sys     0m0.000s
 
52 MB/s. Very very slow! 

 
I got these results for a 100MB input. All are optimised where
possible:
 
mcc      1.9  seconds
gcc      1.9
tcc      1.95
lccwin32 0.7
DMC      0.7
 
The first three likely just use fgetc from msvcrt.dll. The other two
probably use their own libraries.
 

So, may be, fgetc() is not at fault? May be, its OS and the crap
that the corporate IT adds on top of the OS? 

 
 
 

Let's test this hipothesys. 

 

$ time ../quick_xxd/fread_test.exe uu.txt
193426754 byte. xor sum 1.
 
real    0m0.094s
user    0m0.000s
sys     0m0.000s 

 
I get these results:
 
mcc      0.25 seconds
gcc      0.25
tcc      0.35
lccwin32 0.35
DMC      0.3
 
All are repeated runs of the same file, so all timings likely used
cached version of the data file.
 
Most of my tests assume that since (1) I don't know how to to do a
'cold' load without restarting my machines; (2) in real applications
such as compilers the same files are repeatedly processed anyway, eg.
you're compiling the file you've just edited, or just downloaded, or
just copied...
 

So, let's rewrite our tiny app with fread(). 

 

real    0m0.577s
user    0m0.000s
sys     0m0.000s
 
152.8 MB/s. That's much better. Some people would even say that it
is good enough. 

 
I now get:
 
mcc      2.3 seconds
gcc      1.6
tcc      2.3
lccwin32 2.9
DMC      2.9
>

Mine was with MSVC from VS2019. gcc on msys2 (ucrt64 variant) should be
identical.
I wonder why your results are so much slower than mine.
Slow write speed of SSD or slow CPU?

You might remember that the last revised version of your test,
compiled with gcc, took 3.6 seconds, of which 2 seconds was reading
the file a byte at a time took 2 seconds.
 
By using a 128KB buffer, you get most of the benefits of reading the
whole file at once

I hope so.

(it just lacks the simplicity).

The simplicity in your case is due to complexity of figuring out the
size of the file and of memory allocation and of handling potential
failure of memory allocation all hidden within run-time library of your
language.
And even despite all the upfront work that went into your
infrastructure, it probably would not be able to deal with big files on
32-bit system.
Yes, I know, 32-bit compiler would not be able to compile the resulting
big include anyway. But still...

So nearly all of that 2 seconds is saved.
 
3.6 - 2.0 is 1.6, pretty much the timing here.
 
 
 >Two hours later it turned out to be completely incorrect. That is,
 >the 
time was spent in routine related to I/O, but in the 'soft' part of it
rather than in the I/O itself.
 
You don't count time spent within file-functions as I/O? To me 'I/O'
is whatever happens the other side of those f* functions, including
whatever poor buffering strategies they could be using.
 
Because 'fgetc' could also have been implemented using a 128KB buffer
instead of 512 bytes or whatever it uses.
 
I discovered the poor qualities of fgetc many years ago and generally
avoid it; it seems you've only just realised its problems.
 

Octet-by-octet processing of big files is not how I earn butter to put
on my bread.
When I write this sort of utilities for real work, the size of the
input is not arbitrary. Typically I deal with small files (small
relatively to memory capacity of target machines, so 100 MB is still
considered small). So for real work more often than not I use the same
strategy as your did in the program in your language.

BTW I also tweaked the code in my own-language version of the
benchmark. (I also ported it to C, but that version got accidentally
deleted). The fastest timing of this test is now 1.65 seconds.
 
If I comment out the 'fwrite' call, the timing becomes 0.7 seconds,
of which 50ms is reading in the file, leaving 0.65 seconds.
 
So the I/O in this case accounts for 1.0 seconds of the 1.65 seconds
runtime, so when I said:
 
 >I think runtime is still primarily spent in I/O. 
 
That was actually correct.
 

That was incorrect for the previous variant of my program.
Quite likely, it was correct for the program in your language that was
loading a full file before processing.
Quite likely it is correct for my latest variant.
Pay attention how I remember to avoid categorical statements ;-)

If I comment out the 'fwrite' calls in your program, the runtime
reduces to 0.2 seconds, so it is even more correct in that case. Or
is 'fwrite' a 'soft' I/O call too?
 

I think not (on my test gear), but I no longer know.
However I suspect that on Linux with plenty of memory the program could
return before even a single byte of output data was sent to SSD.