Re: MSI interrupts

On Thu, 13 Mar 2025 19:59:02 +0000, Robert Finch wrote:

On 2025-03-13 2:42 p.m., Scott Lurndal wrote:

Robert Finch <robfi680@gmail.com> writes:

On 2025-03-13 12:34 p.m., MitchAlsup1 wrote:

On Thu, 13 Mar 2025 4:50:47 +0000, Robert Finch wrote:
>

Planning on having the Q+ CPU respond directly to MSI interrupts. My
thought is to integrate an interrupt controller directly into the CPU
which will have several queues to prioritize interrupts. The controller
will take care of detecting the highest priority interrupt to process,
performing priority inversion occasionally, and detect stuck interrupts.

>
My 66000 went all the way to MSI-X interrupts as its base.
>
There are mapping methods to map INTx and INTk into MSI-X messages.
MSI->MSI-X mapping is "just zero extend the message".

>
MSI interrupts are the way to go when there are transistors available.
Just having discovered them not to long ago, I still think they are
great.
I have the luxury of not needing to support legacy interrupt systems
(yet), all the peripheral cores needing interrupts will have to support
MSI responses.

>
MSI has been deprecated for years.  MSI-X is preferred and used by
most modern devices (and supported by major operating software).
>

>
The Interrupt port lives in the memory controller and takes an
aperture of addresses and maps them to various interrupt tables.
>

I have read that MSI devices target particular I/O addresses to send the
messages to. Not sure what having a dedicated I/O addresses buys. In my
architecture, rather than target an I/O address a specific processing
core number is specified.

>
There are many reasons that MSI-X supports vector specific target
addresses.
  1) It is possible to target DRAM (so software can poll for the
interrupt)

>
This can be done using a dummy core in the system that echos interrupts
messages back to the DRAM. A current issue with the way interrupts in my
system are working, is they always spit out the interrupt on the
response bus, which means they cannot act like the master.
>
To drive messages onto the bus as requests instead of responses every
peripheral device would need to be able to be a bus master. It would add
a lot of logic to the system. Extra cross-bar ports etc.
>
>

  2) It is possible to support level-sensitive semantics by using two
vectors;
     one to assert the interrupt and the second to deassert it.  Each
needs
     to target a different interrupt controller register (address).

>
I think this can be handled with the cause code being returned as part
of the response.
>

  3) A system may have multiple interrupt controllers for different
     security states (e.g. monitor/secure vs. hypervisor vs. direct
virtual
     to a guest).

>
Is this just to get more target vectors for each interrupt? Could there
be one interrupt controller with multiple target vectors depending on
the security state?

Take a AHCI STAT drive, that has multiple GuestOSs and several
HyperVisors using various functions therein.
While ALL ISRs from this device can use the same code, they must
not be in a position where any state is shared across system-
states. It is often easier to not share the code, even though it
is exactly the same set of bit patterns because where the "other
state" resides is necessarily different in GuestOS as it is in
HyperVisor[k] as it is in Secure Monitor[j].

Q+ CPU always switches to machine mode to process interrupts. It can
then redirect to other operating modes. A security state code could be
added to the response so that different vector tables may be used.

Interrupts in My 66000 can be directed at any interrupt table, at
any privilege level, at any priority. So, in practice, one could
allow for a user to handle his own SATA interrupts without ever
taking an excursion through any GuestOS or Hypervisor code/state;
nor is it architecturally necessary to disable interrupts or
exceptions.

>
>

>
My architecture goes with a 3-bit priority field, and an 8-bit field to
help resolve the cause of the interrupt. The bus/device/function is also
part of the interrupt response.

>
That's what ARM64 GIC does, each interrupt is accompanied by a 20+ bit
'device id', which is composed of the segment, bus, dev/func values
for the sending device.   Since each PCIe controller can consume the
entire bus space potentially, the segment widens the device id space
to include a large number of devices.   The 'device id' is used to
select the in-memory table used to map the interrupt to the target
guest and the target guest vector number.

>
Seems like going out to memory tables to fetch the targets of an
interrupt would be costly, perhaps slow. In the SoC there are already a
half dozen devices fighting for access to the DRAM.

Cache it and pay the cost once (or once per context switch).
It is not like the ISR for the SATA drive for GuestOS[7]
changes very often !! And it is (I S) used often.

I think you really want an interrupt controller as an intermediary
between the device and the CPU interrupt signal, allows a great deal
of implementation flexibility (e.g. changing the target CPU in the

Wrong abstraction::
You want an interrupt service device that services interrupts
(logs them into tables) even when no core (Virtual Machine or
Virtual Machine Monitor) knows anything about said interrupt
table. That is an interrupt to a table is serviced even when
there is no recipient at this instant in time. This prevents
loss of interrupts the table is supposed to accumulate.
Only when at least 1 core is processing VM[k] or VMM[j]
associated with that interrupt table are cores running
VM[k] or VMM[j] concerned with any random interrupt from
that table.
----------------
Connecting the missing dots can be done several ways, My
66000 does::
My 66000 has a service port (in the memory controller) which
services the MMI/O requests to the interrupt aperture. I (currently)
call this the interrupt service port (rather than interrupt
controller) because it just services messages to the interrupt
aperture with almost no regard to any consumer. Said consumer
regard is via a broadcast MMI/O message using the DRAM address
of the interrupt table with a 64-bit message of raised interrupts.
Should the Interrupt table CR match broadcast address, core
captures the raised vector...and decided to deal with interrupt
or not.
A core can IPI a VM or VMM by performing a MMI/O write using
the same bit patterns the I/O device would have used. Since
all interrupts carry privilege (SW stack index) and priority
along with 8-bit ISR and 32-bit message, SW has a "big Enough"
interrupt name-space to do anything it likes.
Finally, there is interrupt negotiation where a service port
on the core sees the interrupt table broadcast and decides
to try-to process said interrupt.
Thus, the Interrupt Service Philosophy
a) is not close to any core
b) cores are notified by MMI/O broadcast
c) cores continue to run code while negotiating for interrupts
d) HW prefetches thread state and RF before interrupt acceptance
e) anybody with VAS access can perform IPIs via MMU-PTEs
Thus, the local Pic is a 6-rail-road-track state machine,
with a single DW of state (&interrupt_table).

Absolutely. Be crazy not to. There is an irq controller component
separate from the CPU. (I decided to move the controller outside of the
CPU).

You need 2 concepts:
a) the deposition of interrupts in the table
b) the delivery of interrupts to consuming (cores)
(a) must happen even when no core is looking at the table.
(b) can happen by having SW perform read requests to interrupt
    aperture or by the normal HW broadcast mechanism.

interrupt controller tables rather than requiring the device driver
to change the msix-vector address field).

The MSI-X address is (nominally) set once just after GuestOS device
discovery. And becomes a virtual constant pointer afterwards.

Updating I/O is faster than updating DRAM tables.

Expand:: is this the distinction between flip-flops and a RAM
buried inside a pre-defined pin-boundary ??