Re: xxd -i vs DIY Was: C23 thoughts and opinions

On 28/05/2024 13:41, Michael S wrote:

On Sun, 26 May 2024 13:09:36 +0200
David Brown <david.brown@hesbynett.no> wrote:
 

>
No, it does /not/.  That's the /whole/ point of #embed, and the main
motivation for its existence.  People have always managed to embed
binary source files into their binary output files - using linker
tricks, or using xxd or other tools (common or specialised) to turn
binary files into initialisers for constant arrays (or structs).
I've done so myself on many projects, all integrated together in
makefiles.
>

 Let's start another round of private parts' measurements turnament!
'xxd -i' vs DIY
 

I used 100 MB of random data:
dd if=/dev/urandom bs=1M count=100 of=100MB
I compiled your code with "gcc-11 -O2 -march=native".
I ran everything in a tmpfs filesystem, completely in ram.
xxd took 5.4 seconds - that's the baseline.
Your simple C code took 4.35 seconds.  Your second program took 0.9 seconds - a big improvement.
One line of Python code took 8 seconds :
print(", ".join([hex(b) for b in open("100MB", "rb").read()]))
A slightly nicer Python program took 14.3 seconds :
import sys
bs = open(sys.argv[1], "rb").read()
xs = "".join([" 0x%02x," % b for b in bs])
ln = len(xs)
print("\n".join([xs[i : i + 72] for i in range(0, ln, 72)]))
Like "xxd -i", that one split the output into lines of 12 bytes.  Some compilers might not like a single 300-600 MB line !
I didn't try compiling a test file from the 100 MB source data, but gcc took about 16 seconds for an include file generated from 20 MB of random data.  It didn't make a significant difference if the data was in decimal or hex, one line or multiple lines.
But since compilation took about ten times as long as the single line of Python code, my conclusion is that the speed of generating the include file is pretty much irrelevant.  Compared to the one-line Python code and considering the generation and compilation combined, using xxd saves 5% of the time and your best code saves 9% - out of possible 10% cost saving.
Thus if you want to save build time when including large arrays of data in the generated executable, time spent on beating xxd is wasted - implementing optimised #embed is the only way to make an impact.
(I have had reason to include a 0.5 MB file in a statically linked single binary - I'm not sure when you'd need very fast handling of multi-megabyte embeds.)