Re: PCIe MSI-X interrupts

Scott Lurndal wrote:

mitchalsup@aol.com (MitchAlsup1) writes:

Scott,

Note that the ECAM must support 8, 16, 32-bit (optionally 64-bit)
single-copy atomic accesses to all configuration space registers.

MMI/O is sequentially consistent while Config is Strongly ordered.
Exactly what are you intending to mean from "single-copy atomic
accesses" ??
Load Multiple (LM) instruction provides ATOMIC access to a series
of sequentially ordered bytes in MMI/O (or Config, or DRAM). So,
if desired, one could read 0x0-0x40 configuration header in a
single LM. This is completely ATOMIC without SW doing anything
other than supplying an address and a count.
Memory to Memory Move (MM) is similar but reads from one place and
writes to another in a single interconnect transaction--that is
ATOMICally. {Basically as long as no page boundaries are crossed,
each memory reference instruction is ATOMIC with respect to
interested 3rd party observations.}
Likewise, LDB, LDH, LDW, LDD and their ST counterparts are unit
ATOMIC.
But I don't know what you mean by "single-copy atomic accesses" ??
<snip>

That depends on how your interrupt controller is designed.  If
you have a multi-socket/multi-chiplet configuration where all
chiplets are identical, and each has its own interrupt controller
(allowing single chiplet implementations), then you'll probably
want to use your CHIP bits in the address to route to the interrupt
controller on the closest chiplet just to reduce interrupt
latency.

Yes, that was the concern. One might expect that the Guest OS
would send an I/O request to a Guest OS device driver on the
chip local to the PCIe tree the device is on, to minimize all
the latency, not just interrupt delivery. Reading and writing
of MMI/O space is <as they say> slow.

         The interrupt controllers will likely need to cooperate
at the hardware level to maintain a single OS-visible "interrupt
space" where each controller handles a subset of the interrupt
number space.

My model has multiple interrupt tables from the get go.
For a start, I assume that each Guest OS has an interrupt table
shared across however many number of virtual of physical cores
the system manages. A HyperVisor has its own Interrupt Table,
and the Secure Monitor has its own table.
A core control register points at this table, and is used when
negotiating for an interrupt, and used to detect Interrupt
table priority escalation (when a priority bit is turned on).
If SW wants a different Interrupt Table (or none) a simple
write to the control register switches the table.*
An interrupt table has interrupts raised when an MSI or MSI-X
interrupt arrives at the Interrupt service provider port.
I have configured the I/O MMU to translate DMA acceses through
one set of MMU tables, and translate Interrupt access through
<a conceptually> different MMU tables, and have access to the
priority of the interrupt without taking MSI-X message bits.
So, DMA can read or write directly through application MMU
tables while the associated Interrupt goes to Guest OS at
priority of SW's choice using the Interrupt table in charge
when the I/O was setup.
-----------------------------------------------------------
(*) So a virtual machine with 17 virtual cores and accepting
interrupts on only 5 of them will have 5 with IT set and 12
with IT set invalid.