Re: Byte Addressability And Beyond

According to EricP  <ThatWouldBeTelling@thevillage.com>:

John Levine wrote:

If you mean an array of pointers to sequences of code points, well
sure, but now we're back to variable length encodings. I know that I
have no idea how big these fixed size things would have to be and i
suspect nobody else does either.

>
One could have instructions that make it easier to parse the
variable length UTF-8 sequences into codepoints.

That would be the CU14 instruction on zSeries, to turn UTF-8 into
UTF-32. CU41 goes the other way.

It would still have to look up whether a codepoint was combining or
stand alone. I don't see a firm definition whether combining codepoints
come before or after, after requiring a lookahead parse and so extra
checks to ensure it doesn't look past the buffer end.

I think they come after but I haven't looked in enough detail. And
then you have all of the issues with precomposed characters: do you
normalize as you go or denormaiize, or what?

--
Regards,
John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly