Re: MSI interrupts

On Tue, 18 Mar 2025 18:51:57 +0000, EricP wrote:

Robert Finch wrote:

On 2025-03-17 2:33 p.m., EricP wrote:

----------------

This also raises a question about what the ICU is doing during this
long latency handoff. One wouldn't want ICU to sit idle so it might
have to manage the handoff of multiple interrupts to multiple cores
at the same time, each as its own little state machine.
>
One should see that this decision on how the core handles the
handoff has a large impact on the design complexity of the ICU.
>
Personally, I would use method 1 first to get something working
then if this is OoO think about getting fancy with method 2.
>
>

Would it not be rare for this to occur? If so, I think the pipeline
could just be flushed from the int disabling instruction on. Then the
interrupted address recorded as the int disabling instruction. <- this
requires identification of an int disabling instruction though. It might
just be a store to a specific I/O address, therefore a special store
opcode to make it easy to detect?

>
It likely is rare but the possibility that the existing instructions,

Which is rare:
executing 200 instructions without an exception ?
OR
executing 200 instructions without a  mispredict ?
The former is fairly rare, the later is not so rare.

a branch mispredict or an exception throw, could retroactively change
the state of the interrupt enable flag to disabled creates this
Window of Uncertainty (WoU) that is the length of the pipeline.

All the more reasons that the very vast majority of interrupts and
exceptions do not have to disable more of their kind or their
opposite.
In My 66000 interrupt architecture, the very first instruction in the
interrupt dispatcher can take a higher priority interrupt-----the
page fault handler in Guest OS can take a page fault in HyperVisor
MMU tables, without any special precautions or code sequences.

Unless one can find a way to make WoU go away the ICU needs to deal
with the possibility that there could be hundreds of clocks between
when ICU asks a core if it can accept an interrupt and when the core
does.

The window will always exist--so, the best one can do is eliminate
the core's sensitivity to the WoU.

I had a few ideas walking in the park yesterday. I had been assuming
that
when an exception is delivered that it would also automatically disable
interrupts. That may not be a good idea as it means that any instruction
that can throw an exception could cause this retroactive disabling.

My 66000 simply defines that this cannot happen::
Exceptions do not alter the behavior of interrupts;
Interrupts do not alter the behavior of exceptions.

If I remove that exception side effect then the only things that can
manipulate the core's master Interrupt Enable flag are the enable INTEN
and disable INTDI instructions, and the actual delivery of an interrupt.

Let's postulate we have a core running at low priority and its
instruction
stream raises an exception. That exception delivers control at that same
low priority to the exception handler.
Do you want any of this to slow down a high priority interrupt ??
even by 20-50 cycles ??
I submit the answer is:: NO !

And those three things can be tracked with a counter in Decode as they
move down the pipeline to Retire. If the counter >0 then there is a
flag change in flight and we don't accept a new interrupt. When count ==
0
and Interrupt Enable flag == 1 then core accepts interrupts, which are
injected at Decode and cause a light weight purge of the Fetch buffer.

Once you start adding mechanisms to track this and that and to deal
with different this and that--you should receive an indication that
you are not on the correct path.
{{Hint: make the problem go away for all time}}

Another pipeline issue is Privileged Control Register (PCR) reads and
writes.
PCR's don't behave like general registers in that they are not renamed,
and in general operations cannot be replayed or performed out of order.

While I have a way to read and write control registers with varying
privilege, a thread at privilege[k] can only read and write his own
CRs and those of the core on which it is currently running.
If you want to read/write control register data for lower privilege
threads, you can access the pointer in core.CR and then access
thread-state directly in memory. By the time control returns to the
low privilege level, all CRs will have been made resident in core.
That is the per thread Program-Status-Line is treated like a write
back cache.

They behave more like atomic stores that must be performed at Retire
and must not be bypassed by any memory and other PCR ops. (This is why
many of the x86/x64 MSR ops are serialized with a pipeline drain.)

By converting them to real memory accesses, the problem solves itself.

This affects what it can do with PCR inside the WoU if it causes a state
change that cannot be undone, for example, reading the interrupt data
FIFO.

Since the device is accessed through MMI/O space, and since My 66000
demands sequential consistency for MMI/O accesses, the LD of the
interrupt data FIFO cannot leave the confines of the core until
all pending memory references from the interrupted thread have
"been performed".