Re: Stacks, was Segments

On Tue, 11 Feb 2025 20:49:24 +0000, Scott Lurndal wrote:

mitchalsup@aol.com (MitchAlsup1) writes:

On Tue, 11 Feb 2025 14:04:59 +0000, Scott Lurndal wrote:
>

mitchalsup@aol.com (MitchAlsup1) writes:

On Mon, 10 Feb 2025 20:18:04 +0000, Scott Lurndal wrote:
>

mitchalsup@aol.com (MitchAlsup1) writes:

On Fri, 7 Feb 2025 20:32:39 +0000, Scott Lurndal wrote:

-----------------isolating---------------------------------
>

HyperVisor maintains a PTE to map guest physical addresses
within an aperture to the page matching the device's BAR.

>
Standard HV stuff.   Although you may want to consider that
the value in the function's BAR, must be a guest PA, not a
machine PA.

>
I need a "WHY" on this sentence before responding to the rest.

>
The guest OS (or user application) is directly programming the addresses
in the
PCIe function DMA engine[*].   The point is to completely
avoid the hypervisor from being involved in the I/O,
so the guest programs the function DMA engine using guest PA and the
IOMMU translates the guest PA into machine
addresses for all transactions initiated by the
function.
>
(Each PCIe function is treated as an individual device
by the OS/HV.  The entire purpose of SR-IOV is to
present the device directly to the guest to avoid any
hypervisor involvment in the I/O path).

>
This is the missing part:: When a user performs a LD or ST

>
You're missing the point completely.  There no user
performing a LD or ST.   The DMA controller on the device
is initiating the transaction, not the host CPU.
>

the guest Virtual address is translated to Guest Physical
by Guest OS translation tables. Guest Physical is then
interpreted as Host virtual and translated a second time
by {SM,HV} mapping tables. This nested MMU does both
translations in a single access; so core TLB is organized
to associate guest virtual directly with machine physical !
as if there were a level crossing PTE providing all the
"right bits".

>
This is basically how all modern CPUs handle it, yes.
>
But it is not relevent to the inbound traffic initiated
-by the device- which can't be translated by the CPU,
rather must be translated at some point between the
PCIe controller and the internal processor interconnect
(e.g. mesh).

I am tracing the path from user of device (core Guest OS).
If I don't understand this "should be simple" path, I am
too lost to continue from the device side looking towards
memory.
I can see how core can write Guest Physical Address into
device.BAR using config space access (with appropriate
MMU permissions).
But, right now: I can't see how the appropriate bit pattern
from core gets to HostBridge in MMI/O space and is recognized
by matching device.BAR down the PCIe tree.
core executes the following instruction::
       STH    R7,[Rdevice,#controlreg]
Rdevice has a Virtual Address bit pattern which after 1 level of
translation matches the bit pattern put into device.BAR at config.
#controlreg is the offset to the control reg.
core TLB translates GVA to GPA; GPA contains the same bit pattern
that was written into device.BAR.
Since we use nested page tables, GPA is interpreted as Host
virtual address, HVA is translated to machine PA before access
leaves the confines of the core.
Unless a range of bits in MPA == GPA, I can't see how the address
can be routed (over the interconnect, into HostBridge,) and then on
to the device and match the configured BAR bit pattern ?!?
That is I can't see how device.BAR == MPA can match when it
has HPA != GPA !!

>
Now that the device has been configured, Guest OS decides
to write some control registers of the device. Guest OS
has its own translation tables for Guest Virtual to Guest
Physical--but the core MMU then translates guest Physical
to Machine Physical before it gets transported over the
interconnect. So, by the time said address gets to Host-
Bridge it is already in Machine Physical, not the Guest
physical you mention.*

>
my66000 as no insight into the device, you can't know
a priori which 64-bit write to the device contains
a physical address.   Particularly in all modern
devices where there may be only one "control register"[*]
the guest driver writes commands and s/g lists to one of
several hundred queues in local dram, then signals
the device to initiate a DMA operation to read the
entry from the queue in DRAM (using a guest physical
address).     The CPU never sees that read,
nor can you know a priori that a particular write
from a device driver is an guest address that might need
to be translated - the driver is writing the command
to main memory and just poking the device to read the
command directly there is no way to associate that
write with any particular device in the CPU.

I have a touchy feely knowledge of the above paragraph.
And I am trying to process without anyone seeing any
interconnect transaction is should not.
But the core does see its own writes to control registers
which are recognized by/at the Device.BAR set bit-pattern
from config, as meaningful to this device.

[*] The doorbell.   There will generally be a few more
    to set global characteristics, configurations, etc,
    but they'll only be actively used during driver initialization
    and will likely not contain addresses of any form.