Re: MSI interrupts

In article <kVgEP.1277108$_N6e.605199@fx17.iad>,
EricP  <ThatWouldBeTelling@thevillage.com> wrote:

Dan Cross wrote:

In article <fe9715fa347144df1e584463375107cf@www.novabbs.org>,
MitchAlsup1 <mitchalsup@aol.com> wrote:

On Thu, 20 Mar 2025 12:44:08 +0000, Dan Cross wrote:

 

Sometimes you really don't want to be interrupted.

And sometimes you don't want to be interrupted unless the
"house is on fire"; I cannot see a time when "the house is
on fire" that you would not want to take the interrupt.
>
Is there one ?!?

 
Consider a thread that takes a spinlock; suppose some
high-priority interrupt comes in while the thread is holding
that lock.  In response to the interrupt, software decides to
suspend the thread and switch some other thread; that thread
wants to lock the spin lock that the now-descheduled thread is
holding: a classic deadlock scenario.

>
Terminology: mutexes coordinate mutual exclusion between threads,
spinlocks coordinate mutual exclusion between cpu cores.
Windows "critical sections" are mutexes with a fast path.

A spin lock is simply any lock where you spin trying to acquire
the lock, as opposed to a blocking synchronization protocol.
Here I'm using the terminology of Herlihy and Shavit [Her08].

Traditional Unix "sleep" and "wakeup" are an example of a
blocking protocol, where a thread may "sleep" on some "channel",
yielding the locking thread while the lock cannot be acquired,
presumably scheduling something else until the thread is later
marked runnable by virtual of something else calling "wakeup" on
the sleep channel.

But note that I am deliberately simplifying in order to
construct a particular scenario in which a light-weight
synchronization primitive is being used for mutual exclusion
between concurrent entities, hence mentioning spinlocks
specifically.

Suggesting that spin locks iare only applicable to
multiprocessing scenarios is flatly incorrect.  Software may use
the technique to spin on a "busy" bit on some IP in a device for
example, even on a uniprocessor system.

A valid response here might be, "don't context switch from the
interrupt handler; use a DPC instead".  That may be valid, but
it puts a constraint on the software designer that may be onerus
in practice: suppose the interrupt happens in the scheduler,
while examining a run queue or something.  A DPC object must be
available, etc.

>
That is exactly what DPC/SoftIrq are design to do - allow the bulk of
the kernel, like the scheduler, non-paged heap management, IO pre and
post processing, to be interrupted without causing reentrancy.
The higher level device interrupt enqueues a request to the lower software
interrupt level, which is processed when it will not cause reentrancy.
>
That constraint is by design and any good OS will crash if violated.

Yes, I'm aware of the technique.  That wasn't the point.

Further, software must now consider the complexity of
potentially interruptable critical sections.  From the
standpoint of reasoning about already-complex concurrency issues
it's simpler to be able to assert that (almost) all interrupt
delivery can be cheaply disabled entirely, save for very
special, specific, scenarios like NMIs.  Potentially switching
away from a thread holding a spinlock sort of defeats the
purpose of a spinlock in the first place, which is a mutex
primitive designed to avoid the overhead of switching.

>
This is why I mentioned the terminology thing: threads do not hold
spinlocks, they hold mutexes.

See above.  Threads can certainly "hold" a spin lock, as they
can hold any kind of lock.  To quote from sec 7.6.1 of [Val96],
page 202:

|On a uniprocessor, if a thread tries to acquire a spin lock
|that is already held, it will loop forever.  Multiprocessor
|algorithms, however, must operate correctly regardless of the
|number of processors, which means that they should handle the
|uniprocessor case as well.  This requires strict adherence to
|the rule that threads not relinquish control of the CPU while
|holding a spin lock.

Threads and mutexes are a fiction created
by the OS scheduler, and switching threads while waiting for a mutex
is exactly what they are supposed to do.
>
Spinlocks are used by cores to coordinate access to shared memory by
things like a OS scheduler looking at list of threads waiting for a mutex.

Spinlocks, of the form I was describing, are an optimization for
cases where the critical section guarded by the lock is short,
and the overhead of invoking a scheduler and context switching
to some other thread is higher than just spinning until the lock
becomes available again.  The whole point is to avoid blocking
and switching.

E.g., as [Vaj11] puts it (page 98):

|Spin locks have the advantage of preventing pre-emption by the
|OS (due to blocking while waiting for the lock)

That is, the advantage is because the thread does not block and
yield.

I like to think of it as all having to do with hats.
The cpu is wearing one of three hats: a thread hat when it pretends to be
a time shared execution machine; a core hat when it acts as a single
execution machine running non-reentrant things like a scheduler which
creates the fiction of threads and processes (multiple virtual spaces);
and an interrupt hat when executing nestable reentrant interrupts.
>
The interrupt mechanism coordinates exactly when and how we switch hats.

I suppose you are lumping system calls into the overall bucket
of "interrupts" in this model, regardless of whether one might
use a dedicated supervisor call instruction (e.g., something
like x86 SYSENTER or SYSCALL or ARM SVC or RISC-V's ECALL)?

The hat analogy feels strained to me, and too colloquial to be
useful.

- Dan C.

References:

[Her08] Maurice Herlihy and Nir Shavit. 2008. _The Art of
Multiprocessor Programming_.  Morgan Kaufmann, Burlington, MA.

[Vah96] Uresh Vahalia. 1996. _Unix Internals: The New
Frontiers_. Prentice Hall, Upper Saddle River, NJ.

[Vaj11] Andras Vajda. 2011. _Programming Many-Core Chips_.
Springer, New York, NY.