Re: xxd -i vs DIY Was: C23 thoughts and opinions

On 28/05/2024 22:23, Michael S wrote:

On Tue, 28 May 2024 19:57:38 +0100
bart <bc@freeuk.com> wrote:
 

On 28/05/2024 16:56, Michael S wrote:

On Tue, 28 May 2024 15:06:40 +0100
bart <bc@freeuk.com> wrote:
  

On 28/05/2024 12:41, Michael S wrote:

On Sun, 26 May 2024 13:09:36 +0200
David Brown <david.brown@hesbynett.no> wrote:
     

>
I think you might be missing the point here.
 

>
I don't think so.
I understand your points and agree with just about everything. My
post was off topic, intentionally so.
>
If we talk about practicalities, the problems with xxd, if there are
problems at all, are not its speed, but the size of the text file
it produces (~6x the size of original binary) and its availability.
I don't know to which package it belongs in typical Linux or BSD
distributions, but at least on Windows/msys2 it is part of Vim -
rather big package for which, apart from xxd, I have no use at all.
>
  

>
>
OK, I had go with your program. I used a random data file of exactly
100M bytes.
>
Runtimes varied from 4.1 to 5 seconds depending on compiler. The
fastest time was with gcc -O3.
>

 It sounds like your mass storage device is much slower than aging SSD
on my test machine and ALOT slower than SSD of David Brown.

I didn't use an SSD - I used a tmpfs filesystem, so no disk at all. There are still limits from memory bandwidth, of course.

 

I then tried a simple program in my language, which took 10 seconds.
>
I looked more closely at yours, and saw you used a clever method of a
table of precalculated stringified numbers.
>
Using a similar table, plus more direct string handling, the fastest
timing on mine was 3.1 seconds, with 21 numbers per line. (The 21 was
supposed to match your layout, but that turned out to be variable.)
>

 Yes, I try to get line length almost fixed (77 to 80 characters) and
make no attempts to control number of entries per line.
Since you used random generator, a density advantage of my approach is
smaller than in more typical situations, where 2-digit numbers are more
common than 3-digit numbers.
 Also, I think that random numbers are close to worst case for branch
predictor / loop length predictor in my inner loop.

That makes them a good test here.

Were I thinking about random case upfront, I'd code an inner loop
differently. I'd always copy 4 octets (comma would be stored in the same
table). After that I would update outptr by length taken from
additional table, similarly, but not identically to your method below.
 There exist files that have near-random distribution, e.g. anything
zipped or anything encrypted, but I would think that we rarely want
them embedded.
 

I'd say these are actually quite common cases.

Both programs have a trailing comma on the last number, which may be
problematical, but also not hard to fix.
>

 I don't see where (in C) it could be a problem. On the other hand, I can
imagine situations where absence of trailing comma is inconvinient.
Now, if your language borrow its array initialization syntax from
Pascal then trailing commas are indeed undesirable.
 

For initialising arrays, an extra comma at the end is no issue for C. But for more general use, #embed can also be used for things like function call or macro parameters, and the extra comma is then a problem.  So your program is not a direct alternative to "xxd -i" or #embed in those cases.  (I don't think such uses would be common in practice.)

I then tried xxd under WSL, and that took 28 seconds, real time, with
a much larger output (616KB instead of 366KB).

 616 MB, I suppose.
Timing is very similar to my measurements. It is obvious that in case
of xxd, unlike in the rest of our cases, the bottleneck is in CPU rather
than in HD.
 

But it's using fixed
width columns of hex, complete with a '0x' prefix.
>
Below is that program but in my language. I tried transpiling to C,
hoping it might be even faster, but it got slower (4.5 seconds with
gcc-O3). I don't know why. It would need manual porting to C.
>

 Why do you measure with gcc -O3 instead of more robust and more popular
-O2 ? Not that it matters in this particular case, but in general I
don't think that it is a good idea.
 

Bart always likes to use "gcc -O3", no matter how often people tell him that it should not be assumed to give faster results than "gcc -O2".  It often results in a few extra percent of speed, but sometimes works against speed (depending on things like cache sizes, branch prediction, and other hard to predict factors).  Alternatively, he uses gcc without any optimisation and still thinks the speed results are relevant.

 Reading whole file upfront is undoubtly faster than interleaving of
reads and writes. But by my set of unwritten rules that I imposed on
myself, it is cheating.
 

I would expect that for big files, mmap would be the most efficient method.  You might also want to use cache pre-loads of later parts of the file as you work through it.  But I don't know portable that would be, and it would be more effort to write.