Re: xxd -i vs DIY Was: C23 thoughts and opinions

On 28/05/2024 21:23, Michael S wrote:

On Tue, 28 May 2024 19:57:38 +0100
bart <bc@freeuk.com> wrote:
 

OK, I had go with your program. I used a random data file of exactly
100M bytes.
>
Runtimes varied from 4.1 to 5 seconds depending on compiler. The
fastest time was with gcc -O3.

 It sounds like your mass storage device is much slower than aging SSD
on my test machine and ALOT slower than SSD of David Brown.

My machine uses an SSD.
However the tests were run on Windows, so I ran your program again under WSL; now it took 14 seconds (using both gcc-O3 and gcc-O2).

I then tried a simple program in my language, which took 10 seconds.
>
I looked more closely at yours, and saw you used a clever method of a
table of precalculated stringified numbers.
>
Using a similar table, plus more direct string handling, the fastest
timing on mine was 3.1 seconds, with 21 numbers per line. (The 21 was
supposed to match your layout, but that turned out to be variable.)
>

 Yes, I try to get line length almost fixed (77 to 80 characters) and
make no attempts to control number of entries per line.
Since you used random generator, a density advantage of my approach is
smaller than in more typical situations, where 2-digit numbers are more
common than 3-digit numbers.
 Also, I think that random numbers are close to worst case for branch
predictor / loop length predictor in my inner loop.
Were I thinking about random case upfront, I'd code an inner loop
differently. I'd always copy 4 octets (comma would be stored in the same
table). After that I would update outptr by length taken from
additional table, similarly, but not identically to your method below.

The difference in file sizes for N bytes will be a factor of 2:1 maximum (all "1," or all "123," for example).

There exist files that have near-random distribution, e.g. anything
zipped or anything encrypted, but I would think that we rarely want
them embedded.

This hardcodes the input filename. 'readfile' is a function in my
library.
>
--------------------------------
>
[0:256]ichar numtable
[0:256]int numlengths
>
proc main=
      ref byte data
      [256]char str
      const perline=21
      int m, n, slen
      byte bb
      ichar s, p
>
      for i in 0..255 do
          numtable[i] := strdup(strint(i))
          numlengths[i] := strlen(numtable[i])
      od
>
      data := readfile("/c/data100")

Reading whole file upfront is undoubtly faster than interleaving of
reads and writes. But by my set of unwritten rules that I imposed on
myself, it is cheating.

Why not? Isn't the whole point to have a practical tool which is faster than xxd?
I never use buffered file input or request file data a character at a time from a file system API; who knows how inefficient it might be.
I looked at your code again, and saw you're using fwrite to output each line. If I adapt my 'readln' (which ends up calling 'printf') to use fwrite too, then my timing reduces from 3.1 to 2.1 seconds.
Of that 2.1 seconds, the file-loading time is 0.03 seconds. If I switch to using a fgetc loop (after determining the file size; it still loads the whole file), the file-loading takes nearly 2 seconds. Overall the timing becomes only a little faster than the gcc-compiled C code.
(My compiler doesn't have an equivalent optimiser, but this is mostly about I/O and algorithm.)
My view is that my approach leads to a simpler program.
Maybe processing very large files that won't fit into memory might be a problem, eg. a 2GB binary. But remember that the output will be a text file of 4-8GB, which for now has to be processed by a compiler which has to build data structures in memory to represent that, taking even more space.
So they would be unviable anyway.
A proper 'embed' feature would most likely have to load the entire binary file into memory too.