Re: Short Vectors Versus Long Vectors

On 4/24/2024 2:08 AM, Lawrence D'Oliveiro wrote:

On Wed, 24 Apr 2024 00:57:07 -0600, John Savard wrote:
 

But they've managed to get GPUs to multiply matrices - and they're quite
good at it, which is why we're having all this amazing progress in AI
recently.

 Worth noting that this AI stuff requires very low-precision floats: 16-
bit, even 8-bit. And they sacrifice mantissa bits in favour of exponents--
down to something like maybe only a couple of mantissa bits in the 8-bit
format.

Yeah... Something like S.E4.F3 or similar is almost useless, but not entirely useless.
In my case I have a few things to allow for 8-bit formats as storage, but ended up going with Binary16 as the smallest floating-point format used for actual computation.
In some of my own small experiments with NN's (mostly perceptron style), FP8 was an OK format for storing weights and biases, but generally had an adverse effect if used for the accumulators (which need at least slightly higher precision to "actually accumulate stuff" effectively).
Something like FP12 (S.E5.F6) would probably be OK though, and in most other areas one can "throw precision to the wind" with this stuff.
But, this was my own limited experience.
Probably, if doing something like, say, multiplying two FP8 values and then accumulating as a 16-bit fixed-point value or similar, converting the result back to FP8 as part of the activation function, this could probably also work for specialized hardware.
Or, maybe even go smaller for the impulses, say, 3-bit:
   000:  0.0  001:  0.5  010:  1.0  011:  1.5+
   100: -0.0  101: -0.5  110: -1.0  111: -1.5+
Where, say, weights could be FP8 but impulses would be mostly 3-bit (since, following the activation function, the actual values are no longer particularly relevant).
This could be used to further reduce the storage cost, and the cost of multipliers (which are basically a 6-bit lookup and a bit-shift in this case). Though, FP8*FP8 isn't that much more, and would possibly be cheaper if the output is Binary16 (though, Binary16 would be more expensive to accumulate than, say, a 16-bit fixed-point value).
...