Re: xxd -i vs DIY Was: C23 thoughts and opinions

On 29/05/2024 10:38, Michael S wrote:

On Wed, 29 May 2024 00:54:23 +0100
bart <bc@freeuk.com> wrote:
 

>
I suspect that your system just has a much faster fgetc
implementation. How long does an fgetc() loop over a 100MB input take
on your machine?
>
On mine it's about 2 seconds on Windows, and 3.7 seconds on WSL.
Using DMC, it's 0.65 seconds.
>

 Your suspicion proved incorrect, but it turned out to be pretty good
question!
 

$ time ../quick_xxd/getc_test.exe uu.txt
193426754 byte. xor sum 1.
 real    0m3.604s
user    0m0.000s
sys     0m0.000s
 52 MB/s. Very very slow!

I got these results for a 100MB input. All are optimised where possible:
mcc      1.9  seconds
gcc      1.9
tcc      1.95
lccwin32 0.7
DMC      0.7
The first three likely just use fgetc from msvcrt.dll. The other two probably use their own libraries.

So, may be, fgetc() is not at fault? May be, its OS and the crap that
the corporate IT adds on top of the OS?

Let's test this hipothesys.

$ time ../quick_xxd/fread_test.exe uu.txt
193426754 byte. xor sum 1.
 real    0m0.094s
user    0m0.000s
sys     0m0.000s

I get these results:
mcc      0.25 seconds
gcc      0.25
tcc      0.35
lccwin32 0.35
DMC      0.3
All are repeated runs of the same file, so all timings likely used cached version of the data file.
Most of my tests assume that since (1) I don't know how to to do a 'cold' load without restarting my machines; (2) in real applications such as compilers the same files are repeatedly processed anyway, eg. you're compiling the file you've just edited, or just downloaded, or just copied...

So, let's rewrite our tiny app with fread().

real    0m0.577s
user    0m0.000s
sys     0m0.000s
 152.8 MB/s. That's much better. Some people would even say that it is
good enough.

I now get:
mcc      2.3 seconds
gcc      1.6
tcc      2.3
lccwin32 2.9
DMC      2.9
You might remember that the last revised version of your test, compiled with gcc, took 3.6 seconds, of which 2 seconds was reading the file a byte at a time took 2 seconds.
By using a 128KB buffer, you get most of the benefits of reading the whole file at once (it just lacks the simplicity). So nearly all of that 2 seconds is saved.
3.6 - 2.0 is 1.6, pretty much the timing here.
 >Two hours later it turned out to be completely incorrect. That is, the
time was spent in routine related to I/O, but in the 'soft' part of it
rather than in the I/O itself.
You don't count time spent within file-functions as I/O? To me 'I/O' is whatever happens the other side of those f* functions, including whatever poor buffering strategies they could be using.
Because 'fgetc' could also have been implemented using a 128KB buffer instead of 512 bytes or whatever it uses.
I discovered the poor qualities of fgetc many years ago and generally avoid it; it seems you've only just realised its problems.
BTW I also tweaked the code in my own-language version of the benchmark. (I also ported it to C, but that version got accidentally deleted). The fastest timing of this test is now 1.65 seconds.
If I comment out the 'fwrite' call, the timing becomes 0.7 seconds, of which 50ms is reading in the file, leaving 0.65 seconds.
So the I/O in this case accounts for 1.0 seconds of the 1.65 seconds runtime, so when I said:
 >I think runtime is still primarily spent in I/O.
That was actually correct.
If I comment out the 'fwrite' calls in your program, the runtime reduces to 0.2 seconds, so it is even more correct in that case. Or is 'fwrite' a 'soft' I/O call too?