Re: Are We Back to the "Wars" Now ?

186282@ud0s4.net <186283@ud0s4.net> wrote:

On 11/23/24 4:25 PM, Lawrence D'Oliveiro wrote:

That’s irrelevant.  Pipes originated on the earliest Unix machine,
which was a PDP-11 with only a 64kiB address space.  They work great
for pumping around gigabytes of data, but you don’t need
gigabyte-sized memory buffers to do that.

 
  It all has to be SOMEWHERE ... if not in RAM then
  on a mass storage device.

Nope, at least not with pipes.

  Of course 'gigabytes' ALL AT ONCE -vs- "a little at a time, added
  up" are entirely different things.

"A little at a time, added up" is exactly what pipes give you, and is
what allows processing gigabytes of total data, without also needing
gigabytes of memory (or disk) to do the processing.

I regularly re-encode video files using a pipeline of several image
filters from the mjpegtools package, ending up with a pipe to x264 for
final compression (for size reduction purposes).

The source files tend to be around 2-5G.  The pipes between the image
filters, and the final pipe to the x264 compressor transfers raw
uncompressed YUV data.

The raw uncompressed YUV data for an hours worth of 1080p video is
about 336GiB of data (way more data than I have RAM to store it all,
and slightly more than the total free space available right now on the
1.9T disk I run the encodes from).  Not one byte of that 336GiB of raw
YUV data that passes through the pipes ever touches disk storage.  The
only data in disk files are the source, compressed, video file, and the
final result, also compressed, video file.