Re: MSI interrupts

On Mon, 24 Mar 2025 12:37:13 +0000, Dan Cross wrote:

In article <4603ec2d5082f16ab0588b4b9d6f96c7@www.novabbs.org>,
MitchAlsup1 <mitchalsup@aol.com> wrote:

---------------------

My architecture has a mechanism to perform ATOMIC stuff over multiple
instruction time frames that has the property that a higher priority
thread which interferers with a lower priority thread, the HPT wins
and the LPT fails its ATOMIC event. It is for exactly this reason
that I drag priority through the memory hierarchy--so that real-time
things remain closer to real-time (no or limited priority inversions).

>
Without being able to see this in practice, it's difficult to
speculate as to how well it will actually work in real-world
scenarios.  What is the scope of what's covered by this atomic
thing?
>
Consider something as simple as popping an item off of the front
of a queue and inserting it into an ordered singly-linked list:
In this case, I'm probably going to want to take a lock on the
queue, then lock the list, then pop the first element off of the
queue by taking a pointer to the head.  Then I'll set the head
to the next element of that item, then walk the list, finding
the place to insert (make sure I'm keeping track of the "prev"
pointer somehow), then do the dance to insert the item: set the
item's next to the thing after where I'm inserting, set the
prev's next to the item if it's not nil, or set the list head
to the item.  Then I unlock the list and then the queue (suppose
for some reason that it's important that I hold the lock on the
queue until the element is fully added to the list).

I have almost that in the documentation for my ATOMIC stuff::
The following code moves a doubly linked element from one place
in a concurrent data structure to another without any interested
3rd party being able to see that the element is now within the
CDS at any time::
BOOLEAN MoveElement( Element *fr, Element *to )
{
     Element *fn = esmLOCKload( fr->next );
     Element *fp = esmLOCKload( fr->prev );
     Element *tn = esmLOCKload( to->next );
     esmLOCKprefetch( fn );
     esmLOCKprefetch( fp );
     esmLOCKprefetch( tn );
     if( !esmINTERFERENCE() )
     {
                   fp->next = fn;
                   fn->prev = fp;
                   to->next = fr;
                   tn->prev = fr;
                   fr->prev = to;
     esmLOCKstore( fr->next,  tn );
                   return TRUE;
     }
     return FALSE;
}
The ESM macros simply tell the compiler to set a bit in the LD/ST
instructions. esmINTERFERENCE is a query on the state of the event
and whether a 2nd party has performed a memory action that kills
the event.
It is required to touch every cache line participating in the event
prior to attempting the ATOMIC update--this is what esmLOCKprefetch
does. One can use up to 8 cache lines in a single ATOMIC event.
Either everyone in the system sees the update and at the same
instant in time, or no one sees any update at all.
Technically, there is no one holding a lock ...

>
This isn't complicated, and for a relatively small list it's not
expensive.  Depending on the circumstances, I'd feel comfortable
doing this with only a spinlocks protecting both the queue and
the list.  But it's more than a two or three instructions, and
doesn't feel like something that would fit well into the "may
fail atomic" model.

Should be obvious how to change the insertion into a singly linked
list in a different data structure.

Mandating to SW that typ SW must use this mechanism will not fly,
at least in the first several iterations of an OS for the system.
So the std. methods must remain available.
>
AND to a large extend this sub-thread is attempting to get a
usefully large number of them annotated.

>
It sounds like the whole model is fairly different, and would
require software writers to consider interrupts occuring in
places that wouldn't have to in a more conventional design.  I
suspect you'd have a fairly serious porting issue with existing
systems, but maybe I misunderstand.

Which is why ID is available.

A valid response here might be, "don't context switch from the
interrupt handler; use a DPC instead".  That may be valid, but
it puts a constraint on the software designer that may be onerus
in practice: suppose the interrupt happens in the scheduler,
while examining a run queue or something.  A DPC object must be
available, etc.

>
This seems to be onerous on SW mainly because of too many unknowns
to track down and deal with.

>
Yes.
>

Further, software must now consider the complexity of
potentially interruptable critical sections.  From the
standpoint of reasoning about already-complex concurrency issues
it's simpler to be able to assert that (almost) all interrupt
delivery can be cheaply disabled entirely, save for very
special, specific, scenarios like NMIs.  Potentially switching
away from a thread holding a spinlock sort of defeats the
purpose of a spinlock in the first place, which is a mutex
primitive designed to avoid the overhead of switching.

>
Agreed.

>
The idea of priority as described above seems to prize latency
above other considerations, but it's not clear to me that that
is the right tradeoff.  Exactly what problem is being solved?

The instruction path into and out of handlers is shorter. Basically
nobody has to look at the schedule queues because HW will do it for
you. For example: GuestOS interrupt dispatcher is approximately::
-- +-------------+-----+------------------------+
R0 | ----------- | ISR | --- MSI-X message ---- |
-- +-------------+-----+------------------------+
Dispatcher:
// at this point we are reentrant
SR   R7,R0,<39:32>
CMP  R2,R7,MaxTable
PLO R2,TTTF
SR R1,R0,<32:0>
CALX [IP,R7<<3,Table-.]
SVR
// if we get here R0<37:32>
// was out of bounds
This does all of the register file saving and restoring, all of
the DPC/softIRQ stuff, all of the checking for additional ISRs,
IPIs, ... Changes priority, privilege, ROOT pointers and associated
ASIDs from and back to.

- Dan C.