Re: xxd -i vs DIY Was: C23 thoughts and opinions

On 5/30/2024 9:05 AM, David Brown wrote:

On 29/05/2024 23:08, bart wrote:

On 28/05/2024 16:34, David Brown wrote:

On 28/05/2024 13:41, Michael S wrote:

>

Let's start another round of private parts' measurements turnament!
'xxd -i' vs DIY
>

>
I used 100 MB of random data:
>
dd if=/dev/urandom bs=1M count=100 of=100MB
>
I compiled your code with "gcc-11 -O2 -march=native".
>
I ran everything in a tmpfs filesystem, completely in ram.
>
>
xxd took 5.4 seconds - that's the baseline.
>
Your simple C code took 4.35 seconds.  Your second program took 0.9 seconds - a big improvement.
>
One line of Python code took 8 seconds :
>
print(", ".join([hex(b) for b in open("100MB", "rb").read()]))

>
That one took 90 seconds on my machine (CPython 3.11).
>

A slightly nicer Python program took 14.3 seconds :
>
import sys
bs = open(sys.argv[1], "rb").read()
xs = "".join([" 0x%02x," % b for b in bs])
ln = len(xs)
print("\n".join([xs[i : i + 72] for i in range(0, ln, 72)]))

>
This one was 104 seconds (128 seconds with PyPy).
>
This can't be blamed on the slowness of my storage devices, or moans about Windows, because I know that amount of data (the output is 65% bigger because of using hex format) could be processed in a couple of a seconds using a fast native code program.
>
It's just Python being Python.

 
I have two systems at work with close to identical hardware, both about 10 years old.  The Windows one has a little bit faster disk, the Linux one has more memory, but the processor is the same.  The Windows system is Win7 and as old as the machine, while the Linux system was installed about 6 years ago.  Both machines have a number of other programs open (the Linux machine has vastly more), but none of these are particularly demanding when not in direct use.
 
On the Linux machine, that program took 25 seconds (with python 3.7). On the Windows machine, it took 48 seconds (with python 3.8).  In both cases, the source binary file was recently written and therefore should be in cache, and both the source and destination were on the disk (ssd for Windows, hd for Linux).
 
Python throws all this kind of stuff over to the C code - it is pretty good at optimising such list comprehensions.  (But they are obviously still slower than carefully written native C code.)  If it were running through these loops with the interpreter, it would be orders of magnitude slower.
 
So what I see from this is that my new Linux PC took 14 seconds while my old Linux PC took 25 seconds - it makes sense that the new processor is something like to 80% faster than the old one for a single-threaded calculation.  And Windows (noting that this is Windows 7, not a recent version of Windows) doubles that time for some reason.
 

>

(I have had reason to include a 0.5 MB file in a statically linked single binary - I'm not sure when you'd need very fast handling of multi-megabyte embeds.)

>
I have played with generating custom executable formats (they can be portable between OSes, and I believe less visible to AV software), but they require a normal small executable to launch them and fix them up.
>
To give the illusion of a conventional single executable,  the program needs to be part of that stub file.
>
There are a few ways of doing it, like simply concatenating the files, but extracting is slightly awkward. Embedding as data is one way.
>

 
Sure.
 
The typical use I have is for embedded systems where there is a network with a master card and a collection of slave devices (or perhaps multiple microcontrollers on the same board).  A software update will typically involve updating the master board and have that pass on updates to the other devices.  So the firmware for the other devices will be built into the executable for the master board.
 
Another use-case is small web servers in program, often for installation, monitoring or fault-finding.  There are fixed files such as index.html, perhaps a logo, and maybe jquery or other javascript library file.

Did you turn off Windows Defender while benching ?

   [Picture]

    https://i.postimg.cc/QCgLJLHQ/windows11-AV-off-control.gif

Benching on Windows is an art, because of all the crap going
on under the hood.

I've had programs slowed to 1/8th normal speed to 1/20th normal
speed, by forgetting to turn off a series of things. Once all
that is done, now you're getting into the same ballpark as Linux.

I also have to turn off the crap salad in Windows, when Windows Update
is running!!! The OS is too stupid to optimize conditions for its
own activity. My laptop for example, ran out of RAM, because "SearchApp"
was eating a three-course meal while I was working. Attempting to kill
that mother, caused the incoming Update to install at closer to normal
speed.

It takes practice to get good at benching modern Windows. On
an OS like Windows 2000, it was always ready to bench. It came
with no AV. It had no secret agenda. It just worked. Each succeeding
version is more of a nightmare.

Imagine when the local AI is running on the machine, and the power
consumption is 200W while it "listens to your voice". At least they're
staying true to their design principles.

   Paul