Re: xxd -i vs DIY Was: C23 thoughts and opinions

On Wed, 29 May 2024 15:16:06 +0100
bart <bc@freeuk.com> wrote:

On 29/05/2024 13:23, Michael S wrote:

On Wed, 29 May 2024 12:23:51 +0100
bart <bc@freeuk.com> wrote: 

 

So, let's rewrite our tiny app with fread(). 

 

real    0m0.577s
user    0m0.000s
sys     0m0.000s
>
152.8 MB/s. That's much better. Some people would even say that it
is good enough. 

>
I now get:
>
mcc      2.3 seconds
gcc      1.6
tcc      2.3
lccwin32 2.9
DMC      2.9
 

 
Mine was with MSVC from VS2019. gcc on msys2 (ucrt64 variant)
should be identical.
I wonder why your results are so much slower than mine.
Slow write speed of SSD or slow CPU? 

 
You'd need to isolate i/o from the data processing to determine that.
 
However, the fastest timing on my machine is 1.4 seconds to read
100MB and write 360MB.
 
Your timing is 0.6 seconds to read 88MB and write, what, 300MB of
text?
>

Much less. Only 193 MB. It seems, this DLL I was textualizing is stuffed
with small numbers. That explains big part of the difference.

I did another test with big 7z archive as an input:
Input size: 116255887
Output size: 425944020
$ time ../quick_xxd/bin_to_listmb /d/bin/tmp.7z uu.txt

real    0m1.170s
user    0m0.000s
sys     0m0.000s

Almost exactly 100 MB/s which is only 1.4-1.6 times faster than your
measurements.

The difference is about 2:1, which is not that unusual given two
different processors, two kinds of storage device, two kinds of OS
(?) and two different compilers.
 
But remember that a day or two ago, your original program took over 4
seconds, and it now takes 1.6 seconds (some timings are 1.4 seconds,
but I think that's the C port of my code).
 
(BTW I guess that superimposing your own faster buffer is not
consdered cheating any more!)
>

No, moderately sized buffer with size that does not depend on the size
of the input or output is not a cheating. It's just additional work that
I wanted to avoid, but did not succeed.

I likely could have majority of the benefit with setvbuf() +
replacement of fgetc() by getc(). But by now it does not make sense to
go back to original variants.

You might remember that the last revised version of your test,
compiled with gcc, took 3.6 seconds, of which 2 seconds was reading
the file a byte at a time took 2 seconds.
>
By using a 128KB buffer, you get most of the benefits of reading
the whole file at once 

 
I hope so.
 

(it just lacks the simplicity). 

 
The simplicity in your case is due to complexity of figuring out the
size of the file and of memory allocation and of handling potential
failure of memory allocation all hidden within run-time library of
you > language. 

 
Yes, that moves those details out of the way to keep the main body of
the code clean.
 
Your C code looks chaotic (sorry), and I had quite a few problems in
understanding and trying to modify or refactor parts of it.
 
Below is the main body of my C code. Below that is the main body of
your latest program, not including the special handling for the last
line, that mine doesn't need.
 
------------------------------------------
   while (n) {
       m = n;
       if (m > perline) m = perline;
       n -= m;
       p = str;
 
       for (int i = 0; i < m; ++i) {
           bb = *data++;
           s = numtable[bb];
           slen = numlengths[bb];
 
           *p++ = *s;
           if (slen > 1)
               *p++ = *(s+1);
               if (slen > 2)
                   *p++ = *(s+2);
 
           *p++ = ',';
       }
       *p++ = '\n';
 
       fwrite(str, 1, p-str, f);
   }
 
------------------------------------------
 
   for (;;) {
     enum { BUF_SZ = 128*1024 };
     unsigned char inpbuf[BUF_SZ];
     size_t len = fread(inpbuf, 1, BUF_SZ, fpin);
     for (int i = 0; i < (int)len; ++i) {
       unsigned char* dec = bin2dec[inpbuf[i] & 255];
       memcpy(outptr, dec, MAX_CHAR_PER_NUM);
       outptr += dec[MAX_CHAR_PER_NUM];
       if (outptr > &outbuf[ALMOST_FULL_THR]) { // spill output buffer
         *outptr++ = '\n';
         ptrdiff_t wrlen = fwrite(outbuf, 1, outptr-outbuf, fpout);
         if (wrlen != outptr-outbuf) {
           err = 2;
           break;
         }
         outptr = outbuf;
       }
     }
     if (err || len != BUF_SZ)
       break;
   }
------------------------------------------
 
 

Each to his own.
For me your code is unreadable, mostly due to very short names of
variables that give no hint of usage, absence of declarations (I'd
guess, you have them at the top of the function, for me it's no better
than not having them at all) and zero comments.

Besides, our snippets are not functionally identical. Yours don't handle
write failures. Practically, on "big" computer it's a reasonable choice,
because real I/O problems are unlikely to be detected at fwrite. They
tend to manifest themselves much later. But on comp.lang.c we like to
pretend that life is simpler and more black&white than it is in reality.