Re: DMA is obsolete

[Apologies for the excessively long time to respond; it's been a
very busy few weeks!]

In article <BfwRP.15340$qm51.4765@fx12.iad>,
Scott Lurndal <slp53@pacbell.net> wrote:

cross@spitfire.i.gajendra.net (Dan Cross) writes:

[snip]
These are issues solveable with the software architecture and
build system for the host OS.   The important characteristic is
that the software coupling makes architectural sense, and that
simply does not require using the same ISA across IPs.

>
I think there are good reasons to have specialized (or low cost,
e.g. riscv) ancilliary cores in a processor package.   Having
been on both sides of the keep them proprietary vs. fully document
them for the OS folks argument, I remain ambivilent.
>
There are good reasons for both positions.  The same reasons behind
the MP1.5 spec and UEFI apply in many cases - widening the
ecosystem and 'you-fix-it' capabilities.   On the other hand,
there may be trade secrets, or system security implications that
might preclude full disclosure.   Once a capability is documented
in the PC world, it tends to live forever good or bad (ISA anyone?)
which may limit future choices in the product line (or discommode
customers).

The idea that one introduces interfaces to facilitate change
across multiple independent dimensions is a good one.  But the
interfaces that we have are awful.

At work, our service processor (granted, outside of the SoC but
tightly coupled at the board level) is a Cortex-M7, but we wrote
the OS for that,

>
What, not Zephyr?

lol nope. https://hubris.oxide.computer

and we control the host OS that runs on x86,
so the SP and big CPUs can be mutually aware.  Our hardware RoT
is a smaller Cortex-M.  We don't have a BMC on our boards;
everything that it does is either done by the SP or built into
the host OS, both of which are measured by the RoT.
>
The problem is when such service cores are hidden (as they are
in the case of the PSP, SMU, MPIO, and similar components, to
use AMD as the example) and treated like black boxes by
software.  It's really cool that I can configure the IO crossbar
in useful way tailored to specific configurations, but it's much
less cool that I have to do what amounts to an RPC over the SMN
to some totally undocumented entity somewhere in the SoC to do
it.  Bluntly, as an OS person, I do not want random bits of code
running anywhere on my machine that I am not at least aware of
(yes, this includes firmware blobs on devices).

>
As a hardware (and long-time OS) person (not necessarily in that order),
I sympathize, but, yet, see above.
>

And if I already have to modify or configure the OS to
accommodate the existence of these things in the first place,
then accommodating an ISA difference really isn't that much
extra work.  The critical observation is that a typical SMP view
of the world no longer makes sense for the system architecture,
and trying to shoehorn that model onto the hardware reality is
just going to cause frustration.  Better to acknowledge that the

>
Most of the hardware should be standardized through ACPI calls,
allowing the underlying implementation to vary over time.

I strongly disagree.  ACPI is a disaster; it may be the diaster
we know, but it's a disaster nonetheless.  Plus, with its tight
entanglement with UEFI, it forces you into that world.

Taking a step back, the real desideratea here might be some
well-defined interface that creates a seam between the hardware
and the OS, but UEFI+ACPI ain't it.

Granted, we have the massive luxury of developing the hardware
and software together and in concert at my job.  I recognize
that that is not the common case.

<big snip>
>

Also, on AMD machines, again considering EPYC, it's up to system
software running on x86 to direct either the SMU or MPIO to
configure DXIO and the rest of the fabric before PCIe link
training even begins (releasing PCIe from PERST is done by
either the SMU or MPIO, depending on the specific
microarchitecture).  Where are these cores, again?  If they're
close to the devices, are they in the root complex or on the far
side of a bridge?  Can they even talk to the rest of the board?

>
It's not the core that's proprietary in this case, it's the
intimate knowledge of the mainboard that is required for that
operation - consider address routing - once a function BAR
is programmed, the SoC fabric needs to be configured to route
that range of addresses to the correct PCI controller to
be converted to PCIe TLPs to the target function.

Fortunately, we designed and built the board ourselves.  :-D

But this is an issue that the OS must contend with anyway;
consider the case of PCIe hotplug: a device newly inserted in a
system must be programmed with a useful BAR, which largely means
that the OS must be capable of allocating physical address space
to that device and setting up those routes.  One might argue
that, perhaps, the OS ought to use some sort of existing
interface (like an AML flow or whatever) to do so, but
that doesn't change the fact that ultimately the responsibility
belongs to the OS.

I suppose another argument might be that system firwmare just
sets up all the root bridge ports with a hunk of address space
and routes to those, and then all the OS has to do is plop a
number into a BAR when a new device shows up, but you're already
half way there, and again, the OS has to contend with
understanding the routing that firmware has set up, which in
turn introduces non-trivial complexity.

In any event, we found that setting up the routes isn't that
onerous; we defined some structs that make up effectively a DSL
that lets us specify these in a pretty declarative way;
comparing with e.g. AGESA, or OpenSIL, they do things in a
pretty similar way.  In either case, it's up to the board
manufacturers to provide all of that data.

That routing can be incredibly complicated, requiring
very substantial and rather tricky configuration
steps in several related IP blocks as well as the
inter-cpu routing fabric/mesh.   Getting it right
is difficult, and there's really no reason for the
OS level to be aware of it (and given it is highly
mainboard/SoC dependent, just complicates the
operating software when not behind  a standard
configuration mechanism like ACPI.)

Again, I disagree.  What is the OS there for, if not to control
and configurethe hardware and provide useful abstractions to
application software?  Hiding the actual hardware configuration
from the OS means that the OS is not, in fact, the final entity
in charge of the hardware.  Having a second operating system, in
parallel, gives me all of the same problems I had with BIOSes.

This opens all sorts of issues with respect to safety and
security, provenance of software, fault management, and
ultimately, control.  It's my machine, I want to run it as I
see fit.

- Dan C.