Re: PCIe MSI-X interrupts

On Wed, 10 Jul 2024 19:02:12 +0000, Scott Lurndal wrote:

kegs@provalid.com (Kent Dickey) writes:

In article <922220c8593353c7ed0fda9e656d359d@www.novabbs.org>,
MitchAlsup1 <mitchalsup@aol.com> wrote:

On page 43 of::
https://www.cs.uml.edu/~bill/cs520/slides_15E_PCI_Express_IOV.pdf
>
it states: "Must not indicate an invalidation has completed
until all outstanding Read Requests that reference the
associated translation have retired"
>
"Must insure that the invalidation completion indication to RC
will arrive at the RC after previously posted writes that use
the stale address."
>
and
>
"...If transactions are in a queue waiting to be sent, It is
not necessary for the device to expunge requests from the
queue even if those transaction[s] use an address that is
being invalidated."
>
The first 2 seem to be PCIe ordering requirements between
EP and RC.
>
The 3rd seems to say if EP used a translation while it was
valid, then its invalidation does not prevent requests
using the now stale translation.
>
So, a SATA device could receive a command to read a page
into memory. SATA EP requests ATS for the translation of
the given virtual address to the physical page. Then the
EP creates a queue of write requests filling in the addr
while waiting on data. Once said queue has been filled,
and before the data comes off the disk, an invalidation
arrives and is ACKed. The data is still allowed to write
into memory.
>
{{But any new command to the SATA device would not be
allowed to use the translation.}}
>
Is this a reasonable interpretation of that page?

>
No, it's saying that the EP can keep using a stale translation UNTIL it
returns the ACK for an invalidation.  It does not need to toss those
requests--it just needs to delay the ACK.  Or it could toss the requests,
and then send the ACK faster, but it's optional if it wants to toss
requests.
>

>
Indeed.  And I'd suggest that the official PCI Express
specification is a better source than a set of slides.
>
From the spec:

I do not have access through the PCIe paywall.

>
a. A Function is required not to indicate the invalidation has completed
until
   all outstanding Read Requests or Translation Requests that reference
the
   associated translated address have been retired or nullified.
b. A Function is required to ensure that the Invalidate Completion
indication
   to the RC will arrive at the RC after any previously posted writes
that use
   the "stale" address.