Re: xxd -i vs DIY Was: C23 thoughts and opinions

On 28/05/2024 16:34, David Brown wrote:

On 28/05/2024 13:41, Michael S wrote:

Let's start another round of private parts' measurements turnament!
'xxd -i' vs DIY
>

 I used 100 MB of random data:
 dd if=/dev/urandom bs=1M count=100 of=100MB
 I compiled your code with "gcc-11 -O2 -march=native".
 I ran everything in a tmpfs filesystem, completely in ram.
  xxd took 5.4 seconds - that's the baseline.
 Your simple C code took 4.35 seconds.  Your second program took 0.9 seconds - a big improvement.
 One line of Python code took 8 seconds :
 print(", ".join([hex(b) for b in open("100MB", "rb").read()]))

That one took 90 seconds on my machine (CPython 3.11).

A slightly nicer Python program took 14.3 seconds :
 import sys
bs = open(sys.argv[1], "rb").read()
xs = "".join([" 0x%02x," % b for b in bs])
ln = len(xs)
print("\n".join([xs[i : i + 72] for i in range(0, ln, 72)]))

This one was 104 seconds (128 seconds with PyPy).
This can't be blamed on the slowness of my storage devices, or moans about Windows, because I know that amount of data (the output is 65% bigger because of using hex format) could be processed in a couple of a seconds using a fast native code program.
It's just Python being Python.

(I have had reason to include a 0.5 MB file in a statically linked single binary - I'm not sure when you'd need very fast handling of multi-megabyte embeds.)

I have played with generating custom executable formats (they can be portable between OSes, and I believe less visible to AV software), but they require a normal small executable to launch them and fix them up.
To give the illusion of a conventional single executable,  the program needs to be part of that stub file.
There are a few ways of doing it, like simply concatenating the files, but extracting is slightly awkward. Embedding as data is one way.