Re: xxd -i vs DIY Was: C23 thoughts and opinions

On 28/05/2024 12:41, Michael S wrote:

On Sun, 26 May 2024 13:09:36 +0200
David Brown <david.brown@hesbynett.no> wrote:
 

>
No, it does /not/.  That's the /whole/ point of #embed, and the main
motivation for its existence.  People have always managed to embed
binary source files into their binary output files - using linker
tricks, or using xxd or other tools (common or specialised) to turn
binary files into initialisers for constant arrays (or structs).
I've done so myself on many projects, all integrated together in
makefiles.
>

 Let's start another round of private parts' measurements turnament!
'xxd -i' vs DIY
 /c/altera/13.0sp1/quartus/bin64/db_wys.dll is 52 MB file
 $ time xxd -i < /c/altera/13.0sp1/quartus/bin64/db_wys.dll > xxd.txt
 real    0m15.288s
user    0m15.054s
sys     0m0.187s
 $ time ../quick_xxd/bin_to_list1
/c/altera/13.0sp1/quartus/bin64/db_wys.dll > bin_to_list1.txt
 real    0m8.502s
user    0m0.000s
sys     0m0.000s
 $ time ../quick_xxd/bin_to_list
/c/altera/13.0sp1/quartus/bin64/db_wys.dll > bin_to_list.txt
 real    0m1.326s
user    0m0.000s
sys     0m0.000s
 bin_to_list probably limited by write speed of SSD that in this
particular case is ~9 y.o. and was used rather intensively during these
years.
 bin_to_list1 is DYI written in ~5 min.
bin_to_list is DYI written in ~55 min.
In post above David Brown mentioned 'other tools (common or
specialised)'. I'd like to know what they are and how fast they are.
 

I think you might be missing the point here.
The start point is a possibly large binary data file.
The end point is to end up with an application whose binary code has embedded that data file. (And which makes that data available inside the C program as a C data structure.)
Without #embed, one technique (which I've only learnt about this week) is to use a tool called 'xxd' to turn that binary file into C source code which contains an initialised array or whatever.
But, that isn't the bottleneck. You run that conversion once (or whenever the binary changes), and use the same resulting C code time you build the application. And quite likely, the makefile recognises you don't need to compile it anyway.
It is that building process that can be slow if that C source describing the data is large.
That is what #embed helps to address. At least, if it takes the fast path that has been discussed. But implemented naively, or the fast path is not viable, then it can be just as slow as compiling that xxd-generated C.
It will at least however have eliminated that xxd step.
The only translation going on here might be:
* Expanding a binary file to text, or tokens (if #embed is done poorly)
* Parsing that text or tokens into the compiler's internal rep
But all that is happening inside the compiler.
It might be that when xxd /is/ used, there might be a faster program to do the same thing, but I've not heard anyone say xxd's speed is a problem, only that it's a nuisance to do.