Re: Windows-on-ARM Laptop Is A “Frequently-Returned Item” On Amazon

On Tue, 3/25/2025 4:26 AM, Chris wrote:

rbowman <bowman@montana.com> wrote:

On Mon, 24 Mar 2025 17:48:25 -0000 (UTC), Chris wrote:
>

Not quite sure what exactly you mean by "inference", the latest Mac
Studio that can go up to 512GB RAM is certainly heading in the right
direction for local LLM training.

>
Inference is when new data is fed into an existing model to reach
conclusions.

 
Yes, I know. Paul seems to have a different definition, however, as
inference does not need a 4MW supercomputer as he seems to be implying. I
have Ollama running completely locally on mac laptop with 32GB RAM.
 

My definition, is I don't want a hardware solution that delivers
results at one token per second. Somebody did a video showing
his "success" and his hardware setup was doing one token a second.

The person who did an LLM using a spreadsheet, it could handle only
a handful of input tokens, like maybe five or seven or something.

These are pitiful demos.

Sure, they eventually produce an output.

But they suck just as bad as my ZX81 sucked playing chess.
The ZX81 would take 30 minutes per move, and it had a
very shallow ply search (it's not like the time was profitably used).

If you can deliver tokens at a decent rate on output, you can
connect the voice synthesizer and produce voice output.

Also, with a healthy token rate, you can enter "interactive mode"
with your AI. At least one USENETter, uses an interactive
style when asking questions (stepwise refinement). Whereas my style,
is I ask just one question, with maybe five or six sentences spelling
out details. You have to be careful doing that -- the extra sentences
are not "constraints" and you can "taint" your question if you feed
it too many "details". Sometimes it is better to leave out details,
just to see if the machine can deliver those details its own self.

Also, if you have a logging AI, like the Deepseek one, where it
writes down what it was thinking along the way, those have
slower processing rates as a result of the architecture, and
again, you need that horsepower to compensate. It was the Deepseek
log, that told us why the AI can't count the letter R in strawberry.
It's because they "turn up the heat" in the model, which causes
hallucinations, and what is actually happening, is the AI takes
your spelling of "strawberry" and "corrects the spelling" and
now there is a different set of letters internally for the
word, than you fed it in the first place. Those logs are
valuable, because they can tell you pretty quickly, the
thing will never be able to count the letters properly
with such a handicap. They only attempt to do it that way,
because that module isn't actually a math module.

You can actually tell the AI to stop screwing around. If you
correctly identify the mistake it is making, it can stop
making the mistake. I was shocked! :-) For example, if you know
the AI is preparing an itemized list with a dozen items, you
can tell it "and don't lose any of the items!". And such
an instruction makes a difference. You would not think that
would work. Many other "hints" merely serve to taint the output.

   Paul