Re: PCIe MSI-X interrupts

Scott Lurndal wrote:

mitchalsup@aol.com (MitchAlsup1) writes:

Scott Lurndal wrote:
>

mitchalsup@aol.com (MitchAlsup1) writes:

One thing to note::
>
The PCIe root complex may send out an address, but when the
MMI/O receiver of said address then uses bits in the address
for routing purposes or interpretation purposes, it matters not that the sender did not know those bits were being used thusly--those bits actually do carry specific meaning--just not to the sender.

>

Said routing does not necessarily mean that the bits in
the addresses are interpreted in any specific way, if the
routing is done by a set of programmed range registers
(i.e. range X to X+Y is routed to destination Z,e.g. the
interrupt controller).

>
>
>
I was asking the contrapositive::
>
Is a system architecture allowed to define certain bits of
the translated address to be used as either routing or
indexing of a table that provides routing information.
>
Not as seen by request originator or request target, but
by the middle-men of transport ?

From the standpoint of the PCI specification, the host
side is completely unspecified.  You could, for example,
use bits <63:60> to specify the socket, or chiplet that
the address should be routed to.   Other bits may encode
the PCI controller #, interrupt controller, IOMMU, etc.

Yes, I have information contained in PTEs that convert a std LD or ST into a KNOWN configuration space access or Memory
mapped I/O space access--known inside the core*, and understood
by the on-die-interconnect to route said request to addressed
device (or at least HostBridge where it, then, figures out
where the device is after it has been configured.) During
this "figuring out" if bits have to be moved about--well that
is a typical HW problem that HW has various mostly cheap
solution for. {{Like concatenating the fields of a x86
segment register into an linear address.}}
At the time of this discussion, I am working out how all
the middlemen between cores and (effectively endpoints)
use the available bits to send stuff where it needs sent.
I know these will not be like those of other architectures
a) because I have no legacy to match
b) because I am trying out "new stuff"
c) there is no concept of wires (INTx) all of these have
   been mapped into messages in MMI/O space.
New ways of connecting the dots that should be enough like
what other guys are doing that Linux porting is no harder
than necessary, but novel enough to require "even fewer"
excursions through the HyperVisor to get the dots connected
and maintain those connections.
(*) configuration space accesses are known within the core
because they follow strong ordering, while memory mapped I/O
accesses are known within the core because they are sequen-
tially consistent, unlike DRAM accesses which are only
cache consistent (except when ATOMIC stuff is going on
where the core drops back to sequentially consistent.)
This knowledge of which space enables higher performing
memory systems that automagically drop back to SC when it matters and without Fence instructions being needed.
Since the core knows the space of the access, so does the
interconnect and orders things appropriately. From the core
end of looking at things, a configuration request is
properly ordered and delivered reliably to the endpoint;
A MMI/O request is properly ordered on the interconnect
and reliably delivered to endpoint. Likewise, endpoint
requests are reliably delivered to MMI/O or DRAM address
spaces. {{given a "special" device as some endpoint, and
with the already defined facilities, said device could
go out and read/write core control registers without
the data ever passing through memory (security stuff).
Of course one would want said device to be very secure indeed in order to trust it that far....but electrically
is has to be at some "normal place" accessible via normal
protocol and transports.}}
It may seem that I have dumped a lot of requirements on
the last level cache. This may be true--but with the advent of CXL, DRAM may migrate farther away from the
cores to the point where no pins on the chip are dedicated
to DRAM control, instead a PCIe channel to a DRAM control-
ler down the PCIe tree(s) allows the Chip to connect to
any DRAM technology (DDR4,5,6, HBM, RamBus, ...) any size
of DRAM, any position of DRAM,... just by changing the popcorn part at the end of the tree.
Thus, LLC has to be able to read/write that DRAM and
CXL caches and then cache its results as is normally expected. CXL caches extends this to SRAM in addition to DRAM--its just a different popcorn part.
{{There are also no need to build chip-repeaters
like HyperTransport or whatever Intel call their similar
chip-to-chip transport. These will migrate to CXL for
all the right reason................................}}