Re: Short Vectors Versus Long Vectors

Lawrence D'Oliveiro <ldo@nz.invalid> writes:

On Tue, 23 Apr 2024 02:14:32 +0000, MitchAlsup1 wrote:
>

CRAY machines stayed "in style" as long as memory latency remained
smaller than the length of a vector (64 cycles) and fell out of favor
when the cores got fast enough that memory could no longer keep up.

Mitch Alsup repeatedly makes this claim without giving any
justification.  Your question may shed some light on that.

So why would conventional short vectors work better, then? Surely the
latency discrepancy would be even worse for them.

Thinking about it, they probably don't work better.  They just don't
work worse, so why spend area on 4096-bit vector registers like the
Cray-1 did when 128-512-bit vector registers work just as well?  Plus,
they have 200 or so of these registers, so 4096-bit registers would be
really expensive.  How many vector registers does the Cray-1 (and its
successors) have?

On modern machines OoO machinery bridges the latency gap between the
L2 cache, maybe even the L3 cache and the core for data-parallel code.
For the latency gap to main memory there are the hardware prefetchers,
and they use the L1 or L2 cache as intermediate buffer, while the
Cray-1 and followons use vector registers.

So what's the benefit of using vector/SIMD instructions at all rather
than doing it with scalar code?  A SIMD instruction that replaces n
scalar instructions consumes fewer resources for instruction fetching,
decoding, register renaming, administering the instruction in the OoO
engine, and in retiring the instruction.

So why not use SIMD instructions with longer vector registers?  The
progression from 128-bit SSE through AVX-256 to AVX-512 by Intel
suggests that this is happening, but with every doubling the cost in
area doubles but the returns are diminishing thanks to Amdahl's law.
So at some point you stop.  Intel introduced AVX-512 for Larrabee (a
special-purpose machine), and now is backpedaling with desktop, laptop
and small-server CPUs (even though only the Golden/Raptor Cove cores
are enabled on the small-server CPUs) only supporting AVX, and with
AVX10 only guaranteeing 256-bit vector registers, so maybe 512-bit
vector registers are already too costly for the benefit they give in
general-purpose computing.

Back to old-style vector processors.  There have been machines that
supported longer vector registers and AFAIK also memory-to-memory
machines.  The question is why have they not been the answer of the
vector-processor community to the problem of covering the latency?  Or
maybe they have?  AFAIK NEC SX has been available in some form even in
recent years, maybe still.

Anyway, after thinking about this, the reason behind Mitch Alsup's
statement is that in a

doall(load process store)

computation (like what SIMD is good at), the loads precede the
corresponding processing by the load latency (i.e., memory latency on
the Cray machines).  If your OoO capabilities are limited (and I think
they are on the Cray machines), you cannot start the second iteration
of the doall loop before the processing step of the first iteration
has finished with the register.  You can do a bit of software
pipelining and software register renaming by transforming this into

load1 doall(load2 process1 store1 load1 process2 store2)

but at some point you run out of vector registers.

One thing that comes to mind is tracking individual parts of the
vector registers, which allows to starting the next iteration as soon
as the first part of the vector register no longer has any readers.
However, it's probably not that far off in complexity to tracking
shorter vector registers in an OoO engine.  And if you support
exceptions (the Crays probably don't), this becomes messy, while with
short vector registers it's easier to implement the (ISA)
architecture.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
  Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>