Re: MSI interrupts

On 2025-03-15 6:28 p.m., MitchAlsup1 wrote:

On Sat, 15 Mar 2025 20:46:41 +0000, Robert Finch wrote:
 

On 2025-03-15 1:46 p.m., MitchAlsup1 wrote:

On Sat, 15 Mar 2025 2:10:48 +0000, Robert Finch wrote:

------------

Secondly, you want to be in a position where is someone chooses Q+
as their main CPUs, that they are not hindered in building large
systems using it.
>

Yikes. I had not thought of someone choosing Q+. I do not have
experience building large hardware systems. Undoubtedly, there would be
all kinds of pitfalls with it. I have been using Q+ as a learning tool.

 In general, you can achieve this goal by making the index fields big
enough, or the tree/DAG depth adjustable.
 

Q+ SoC has just 32-bits for the IRQ message payload data as many devices
are 32-bit and IRQ messages are passed on the 32-bit response bus. That
means a full I/O address cannot be used as it would be too large. As the
message data, the I/O device supplies the vector number (12-bits) and
interrupt controller number (6 bits). Specific interrupt controllers are
targeted rather than an I/O address as targeting requires only six bits
rather than a 32+ bit I/O address. Currently there will be only one
interrupt controller in the system. Interrupt controllers could be
designed to do other things like log to memory.

>
Under advice of council (EricP), I punted HW vectoring--which removed
a couple of pointers from the device:function control block along with
restrictions on where said table is, and on what alignment it is.
>

I thought the address of the hardware table would have to be specified
in a device control register for MSI-X. The address for triggering the
interrupt would be a location in the table. Are not the low order bits
of this address the same as the interrupt vector number? I had split
this location into two pieces (vector number, plus int controller
number) so that a 32-bit register could specify the (64-bit) memory
address.

 The device knows the virtual address of the interrupt table and has
the ability to send that address a 32-bit message.

Okay, I am going to have a 32-bit or 64-bit data extended interrupt message information in the device itself. The ISR will typically need to examine device registers anyway.

 In My 66000, that virtual address is translated to universal address
in the normal ways, and if the universal address happens to fit in the
aperture of the interrupt service port, said message is logged into
specified table.
 {Who determines which vector or which priority or which layer of the
software stack receives control when this interrupt is taken by a
core} are all system decisions.
 In My 66000, the I/O MMU is associated with a "device"* via its ECAM
address and each "device" has a device control block governing its
activities based on the "device". In this device control block is a
stack of context pointers {Secure, Hyper, Super, User} which point
at Context Headers. The context headers provide {Root pointers, ASIDs,
..} (The Context pointers C O U L D point at the threads of the SW
stack that originated the I/O request.) In addition to the context
headers, there is an interrupt table address, and four supply points.
Each supply point provides a software stack index so the proper trans-
lations can be applied on the device access. Three also supply the
6-bit priority of the request, and an 8-bit service provider index.
 (*) a function within a device--as a device may have several physical
and many virtual functions.
 In My 66000, memory references have priority, so memory accessed from
devices need said piece of information. My 66000 uses this to speed
up higher priority requests over those of lower priority requestors.
Priority is also used when the MSI-X message is logged into the
interrupt table.
 The 8-bit service provider index is used to tell the Interrupt
dispatcher
which ISR should it send the MSI-X message.
 All of the Device control block fields are initialized at GuestOS
activation -- at GuestOS device discovery and configuration; and
remains static in use.
 Thus, Each ISR can have its own 32-bit MSI-X namespace should it choose,
but the method still works fine when the system prescribes the MSI-X
message name-space.
 It is legal to use bits from the universal address or from the MSI-X
message to provide information pertaining to {SW stack index, priority,
ISR[index], device[error]handler, device[pagefault]handler, ...}.
>

Okay.

I added an IMSI controller. This allows a bus master to generate
interrupts easily with software. It basically reflects the interrupt
request onto the slave response bus. The interrupt request is generated
by writing to a memory address.

 In My 66000, since a device can raise an interrupt on a software stack
by performing a MMI/O space write to the interrupt aperture--so can any
core when the MMU sets up a PTE to the interrupt aperture. Thus, a
core sends an interrupt to another core the same way a device sends
an interrupt to the core. {Also useful in debug and bring up}
 

>

Each interrupt controller has a base addresses and length for the vector
tables so they may be located in system RAM (DRAM). The interrupt
controller takes care of fetching the vector and broadcasting the vector
to a specific CPU core.

>
There are situations where you want any 1 from a set of cores to "take"
the interrupt, rather than targeting one core (which may be busy at the
time.
>

I have more work to do on this. I don’t know how to approach it if there
is more than a small number of cores. For the number of cores fitting
into the FPGA, I suppose the six-bit target core number could just be an
index into a table containing a list of cores that are targeted. With 64
CPU cores for instance, each entry in the table would be a 64-bit
vector.

 I need to impress upon you that there are many interrupt tables, and
that the interrupt table remains "capable of receiving interrupts"
even when the GuestOS which "owns" that table is not running on ANY
ore (at this instant).
 The system I cam up with tries to make the number of cores irrelevant
to the architectural documentation of the interrupt architecture.
 

The SoC system is limited to 64 CPU cores by the use of a 6-bit core
number. So, a 64-bit vector containing which cores to send the interrupt
to could be used. A find-first-one responding could be used to mask
things so that only one CPU sees the interrupt.

 Consider that you have a pool of 4 cores setup to receive interrupts.
That those 4 cores are running at differing priorities and the interrupt
is at a still different priority. You want the core operating at the
lowest
priority (with the right software stack) to accept the interrupt !!

Okay, I wrote a naive hardware filter for this which should work okay for small numbers of CPUs but does not scale up very well.
What happens if there is no core ready? Place the IRQ back into the queue (regenerate the IRQ as an IRQ miss IRQ)? Or just wait for an available core. I do not like the idea of waiting as it could stall the system.
To identify the CPU pool, with a max of 63 CPU per interrupt controller, the pool could be represented directly as a 64-bit word in the interrupt vector table. I have been thinking of using a 256-bit vector entry instead of 128-bits. I allocated 96-bit for an instruction / address field.