Re: MSI interrupts

On Mon, 24 Mar 2025 15:55:33 +0000, EricP wrote:

MitchAlsup1 wrote:

On Sun, 23 Mar 2025 16:16:16 +0000, EricP wrote:

I may come back to the snipped stuff later

>
But all of this adds to the return from interrupt code path length.

As noted in my response to Dan C, the return from interrupt code path
length is the RET instruction from the ISR back to Dispatcher and the
SVR instruction at the Dispatcher. Is this short enough?

------------------

I, too, have a single instruction (SVR) that delivers control to the
highest priority still pending thread, regardless of whether it is
a new interrupt, a scheduled DPC/softIRQ, a local timer, an IPI, or
the thread the interrupt stack was placed upon.
>
SVR looks at the priority of the thread to receive control, and the
priorities of pending 'interrupts' (of which DPC/softIRQ and IPIs
are part) and transfers control to the highest priority pending
thread--occasionally that thread is the interrupted one.

>
The new idea I'm proposing is CSVR *Conditional* Supervisor Return
which can fall through in the unlikely event something disturbs the
non-interruptible return sequence.

I detected that. My SVR has most of those properties (can do several
different return sequences including take the next interrupt) and does
not necessarily return whence the interrupt was taken.

This can be used to optimize the return path by avoiding returning from
an interrupt only to immediately take a priority-1 software interrupt.

My 66000 SVR exits a current handler, but instead of returning to the
interrupted thread, it checks for any other interrupt disturbances
and if it finds one of higher priority than the interrupted thread, it
does the higher priority one(s) first, leaving the interrupted thread
for later.
SW does not have to look. SW just has to send DPCs/softIRQs as MSI-X
messages to the interrupt table of its choice. So, setting up a DPC/
softIRQ requires storing some data in a SW controlled place and manner,
and performing a single ST to the interrupt aperture to enQueue the
message at chosen priority and privilege, to be serviced by chosen
handler.

Should a higher priority thread become pending while SVR is in progress,
the higher priority thread will receive control before the first inst-
ruction of the mis-receiving thread is performed.
>
As far as you have explained these things to me, I see no race
conditions
in the architected means.

>
Perhaps not for your approach but I'm looking at other optimizations
that might introduce them. One was optimizing the common return path:

It is currently standing at 2 instructions--an RET from ISR back to its
dispatcher, followed immediately by a SVR.

Before a supervisor return starts to reload the saved register set
it tests if a DPC is pending and about to generate a software interrupt,
and if so jumps back into the Dispatcher. This saves reloading the
registers
only to immediately take another interrupt, a pipeline drain and fill,
and resave the same registers.

Yes, yes, my SVR does that.

To do this optimization, before it starts to reload the registers
it tests the Interrupt Control Register (ICR) to check if it is about to
trigger a SWI and if so adjusts the ICR and jumps into the Dispatcher.
Otherwise it restores registers and does a Conditional Supervisor Return
which reperforms the same test and can fall through if the test fails.

Because I do things in HW, I can start the RF reload (bring in the 5
cache lines of the thread) and while waiting for them to arrive, look
at the pending interrupts and abandon the return, ignoring the fetched
thread state; and proceed back to the top of Dispatcher. So, if SVR
finds more stuff to do, it can continue to use the RF it has.
But, because the next interrupt may be Hyper instead of Super (or
even vice versa) architecture states that HW uses the RF of the
current privilege level. This may require doing nothing (super->super)
or swapping RF (hyper->super or super->secure, from != to)
Architecturally, HW is always using the thread-state of core SW-stack
context pointers indexed by the privilege level.

This also requires that kernel/interrupt stacks be switched a certain
way,
which just so happens to be the same as the x86 way. That is, hardware
only
switches stacks if interrupted mode is User. This leaves it up to OS
software to decide which stack to use at what points inside the kernel.

Does x86 HW switch stacks when going from super to hyper or hyper to
Secure ?? like it does from user to super ??
--------
My 66000 has the property that a core runs a SW stack, each element
on the stack points at thread-state. Thread-state has all the control
registers needed to run a thread and a register file for each level
on the stack.
It is possible to program the system such that a (semi-trusted) user
(no privilege) is allowed to "take" his own interrupts. When doing
so, he really wants to use his own (non-privileged) stack.
It is also possible for a single interrupt table to hold interrupts
for any privilege level at any priority.
---------
You may ask why ? One of the goals here is to take the same number of
cycles to context switch between user processes as it does to switch
between different HyperVisors. Currently these are in the 1000 cycle
range (user->user) and 10,000 range for (Guest OS[1]->Guest OS[3]).
I want both of these down in the sub-100 range close; to >20 cycles
plus memory latency.
For example, one can context switch the user process by overwriting
its core software stack CR with the new process' Thread-state pointer
(and then when SVR happens the new process runs instead of the old
process) -- context switch is essentially 1 instruction. This happens
at Guest OS, and Hypervisor levels of the SW stack, too.
Supervisory software has the same path to figure out what to do
(next) and then once decided the code-path to "make it happen"
is tiny.

Also I'd save the interrupt context, the old RIP, old RSP and old
interrupt
status in hardware registers, not on the stack, and the interrupt vector
RIP address is calculated based on the priority. This eliminates all
memory
access from the exception and interrupt entry and exit transition
sequences,
and thereby eliminates page faults and memory error possibility and
cache access latency.

As you remain aware, all thread control registers have a known memory
address from which they are read and written, these do not go on any
stack. Along with the RF they occupy 5 sequential cache lines. HW
treats them (essentially) as a write back cache.
The only things on the supervisor stack is the original value in R0
(which I had to clobber to put the MSI-X message in. And a 2-bit
of the interrupted privilege level and a 4-bit token telling SVR
what kind of stack it is dealing with {Interrupt, exception, SVC}.)
{{Looking at this now, I could probably get rid of R0 on the stack))
One of the reasons I want HW doing these things, is HW strips off
bits SW is not supposed to be looking at. It is not clear is even
supervisory SW can be so trusted--witness we have (essentially
unprivileged) units of SW that do not even want GuetOS to look at
their data (banking applications, video, rented audio, ...). They
want GuestOS to perform the I/O but not look at any of the data.