Re: PCIe MSI-X interrupts

Scott Lurndal wrote:

mitchalsup@aol.com (MitchAlsup1) writes:

Scott Lurndal wrote:
>
>
Question:: Down below you talk of the various interrupt control-
lers routing an interrupt <finally> to a core. What happens if the core has changed its priority by the time the interrupt signal arrives, but before it can change the state of the tables in the
interrupt controller that routed said interrupt here ?

Speaking for the ARM64 systems that I'm most recently
familiar with, the concept of priority is associated with
an interrupt (up to 8-bits worth of priority - an implementation
of the GIC is allowed to support as few as three bits).

The interrupt controller is distributed logic;  there is a
component called the 'distributor' and another component called
the 'redistributor'.   The former is global to the system and
the latter is a per-CPU component.  The distributor also contains
a subsystem called the interrupt translation subsystem (ITS) which
supports interrupt virtualization.

The redistributor, being part of the core, handles the delivery
of an interrupt to the core (specifically asserting either the FIQ
or IRQ signals that cause entry to the IRQ or FIQ exception
handlers).   The redistributor tracks the current running priority
(which is directly associated with the priority of the current
active interrupt, when not processing an interrupt, the current
running priority is called the IDLE priority and doesn't block
delivery of any interrupts). The redistributor communicates changes to
the RPR to the distributor, which will hold any interrupt that
is not eligable for delivery (for any reason, including lack
of priority).   There is no way for software to change the
RPR - it only tracks the priority of the currently executing
interrupt.

Thank you for the wonderful ASCII art::

+-------------------------+
|    PCI Device           |
+-------------------------+
           |   MSI-X message (address:  GITS_TRANSLATER control
register)
           |                 (payload:  Interrupt number (0 to N))
           v                 (sideband:  streamid)
+-------------------------+
|   Interrupt Translation |  (DRAM tables:  Device, Collection)
|   Service               |  Lookup streamid in device table.
|                         |  DT refers to Interrupt Translation Table
|                         |  Translate inbound payload based on ITT to
an LPI
|                         |  Collection table identifies target core
+-------------------------+
           |    Internal message from ITS to redistributor for target
           v
+-------------------------+
|   Redistributor         | (DRAM table:   LPI properties)
|                         |  Lookup LPI properties, contains priority
and enable bit
|                         |  If not enabled or priority too low,
|                         |    store in LPI pending table (also DRAM)
[*]
|                         |  If enabled, unmasked at the CPU interface
|                         |  and priority higher than RPR, assert FIQ
|                         |  or IRQ signals to core.
+-------------------------+
           |     IRQ/FIQ signals
           v
+-------------------------+
|   Core                  |  Check PSTATE IRQ and FIQ mask bits
|                         |  IRQ/FIQ can be routed to EL1, EL2 or EL3
|                         |  by per-core control bits.   Update
|                         |  core state appropriately and enter ISR
+-------------------------+

[*] as core RPR and signal masks change, the ITS re-evaluates pending
[**] LPI properties and pending bits are generally cached in the
     redistributor for performance.

My concept looks like:: Based on a new understanding of where things want to be due to CXL:: I am assuming that the Last Level Cache (LLC) is placed side by side
with HostBridge on Chip. This facilitates using PCIe for DRAM access
and for CXL caches. LLC also provides the access services to the I/O MMU (with lots of caching) and maintains the interrupt tables.

+-------------------------+
|    PCI Device           |
+-------------------------+
           |   MSI-X message (address:  GITS_TRANSLATER control
register)
           |                 (payload:  Interrupt number (0 to N))
           v                 (sideband:  streamid)
+-------------------------+
|  HostBridge Translation |  IOMMU tables:  B:D,F->Originating Context
|       Service           |  Originating Context supplies Root pointers
|                         |  and interrupt table address
|        In LLC           |  HostBridge DRAM accesses are performed
through LLC
|                         |  HostBridge MMI/O accesses routed out into
Chip
+-------------------------+
           |     MMI/O message from HTS to virtual context stack DRAM
address
           |
           |     If core interrupt table matches MMI/O address ? SNARF
message
           |        the message contains pending priority interrupt
bits.
           v
+-------------------------+
|   Core                  |  If there is a interrupt at higher priority
|                         |  than I am currently running ? begin
interrupt
|                         |  negotiation (core continues to run
instructions)
|                         |  If negotiation is successful ? Claim
interrupt
|                         |  and context switch to Interrupt
Dispatcher.
+-------------------------+

There is no IRQ-like signal to the core, it is all done by a SNARF of
data to an address cores are watching. When a virtual core gets a new
time slice, as the core is fetching instructions, it also fetches its
pending priority interrupts from its interrupt table (maintained by
LLC),
and will "take" an higher pending interrupt prior to executing any instructions at lower priority or lower privilege. Hereinafter, core
monitors its interrupt table address to SRARF updates.
Context stack contains pointers to the Thread headers of the 4
privilege levels, a pointer to the associated interrupt table,
and some other stuff--it is a cache line in size (8 DoubleWords)
The pointers to the Thread Headers have access the Root pointers
of those levels.
There is a 2-bit indicator in the context stack indicating which Root Pointer is used to translate this I/O request.
All indexed via B:D,F (extended or not) and some rather static tables
placed in unCacheable DRAM. unCacheable DRAM is actually cached in
LLC, just not more local to cores. LLC, in essence, SNARFs the
HostBridge MSI-X message recognizes that this is an update to
the interrupt tables, inserts update, and then provides a message
which cores running that interrupt table will SNARF.
No wires (IRQ), just std messages flying across MMI/O space doing exactly the same things.

>
For what reason are there multiple addresses ?

A system may have multiple interrupt controllers.  In the
case of the ARM64 systems, there may be a case where some
interrupts should be considered level sensitive, in which
case they must use SPI type interrupts which have a different
target register for the MSI-X address field when compared with
LPI type interrupts.

Recall that the PCI spec must accomodate a wide range of system
implementations (including Z-series).

Would you consider that "multiple interrupt tables all being
maintained by a single service port inside LLC which then
spews out updates any/all cores can see" to be multiple
interrupt controllers ??