Re: DMA is obsolete

On Sun, 27 Apr 2025 19:13:47 +0000, Theo wrote:

MitchAlsup1 <mitchalsup@aol.com> wrote:

On Sat, 26 Apr 2025 17:29:06 +0000, Scott Lurndal wrote:
--------------

One concern that arises from the paper are the security
implications of device access to the cache coherency
protocol.   Not an issue for a well-behaved device, but
potentially problematic in a secure environment with
third-party CXL-mem devices.

>
Citation please !?!

>
CXL's protection model isn't very good:
https://dl.acm.org/doi/pdf/10.1145/3617580
(declaration: I'm a coauthor)

Thanks for the URL !!

Secondarily, using 1-few cores to perform PIO is not going to
have the data land in the cache of the core that will run when
the data has been transferred. The data lands in the cache doing
PIO and not in the one to receive control after I/O is done.
{{It may still be "closer than" memory--but several cache
coherence protocols take longer cache-cache than dram-cache.}}

>
I think this is an 'it depends'.  If you're doing RPC type operations,
it
takes more work to warm up the DMA than it does to just do PIO.

Yes, it takes more cycles for a CPU to tell a device to move memory
from here to there than it takes CPU to just move the memory from
here to there. I was, instead, referring to a CPU where it has an
MM (move memory to memory) instruction, where the instruction is
allowed to be sent over the interconnect (say CRM controller) and
have DRC perform the M2M movement locally.
That is:: all major blocks in the system have their own DMA sequencer.

  If you're
an SSD pulling a large file from flash, DMA is more efficient.  If
you're
moving network packets, which involve multiple scatter-gathers per
packet,
then maybe some heavy lifting is useful for the address handling.
>

It was also the only ARM64 processor chip we built with a cache-coherent
interconnect until the recent CXL based products.
>
Overall, a very interesting paper.

>
Reminds me of trying to sell a micro x86-64 to AMD as a project.
The µ86 is a small x86-64 core made available as IP in Verilog
where it has/runs the same ISA as main GBOoO x86, but is placed
"out in the PCIe" interconnect--performing I/O services topo-
logically adjacent to the device itself. This allows 1ns access
latencies to DCRs and performing OS queueing of DPCs,... without
bothering the GBOoO cores.
>
AMD didn't buy the arguments.

>
Intel tried that with the Quark line of 'microcontrollers', which
appeared
to be a warmed over P54 Pentium (whether it shared microarchitecture or
RTL I'm not sure).

In my case, the LBIO core ran exactly the same ISA as the big central
cores. If the remote cores were present, they would field the interrupt
access the device, schedule further cleanup work, and then kick the main
cores in their side.
In order to be viable, the same OS SW has to run whether the LBIO cores
are present or not.

                They were too power hungry and unwieldy to be
microcontrollers - they also couldn't run Debian/x86 despite having an
MMU because they were too old for the LOCK CMPXCHG instruction Debian
used (P54 didn't need to worry about concurrency, but we do now).

It is not generally known, but back in ~2006 when Opteron was in full
swing, the HT fabric to the SouthBridge was actually coherent, we just
did not publish the coherence spec and thus devices could not use it.
But it was present, and if someone happened to know the protocol they
could have used the coherent nature of it.
My LBIO core would have made use of that.
And unlike P54, I was being designed for low power operations, and
it was going to use Opteron building blocks to simplify bug-for-bug
compatibilities.