Re: python text, Byte Addressability And Beyond

Lawrence D'Oliveiro <ldo@nz.invalid> writes:

On Sun, 12 May 2024 05:40:45 GMT, Anton Ertl wrote:
>

This is a nice demonstration of the unnecessary complexity that the
codepoint mistake leads to. ...
 
But if they had decided to just store the data as UTF-8 and use byte
indexes and lengths in their API, and adjusted the rest of their API
accordingly, they could have avoided this complexity and
inefficiency ...

>
But UTF-8 is just a representation of code points, not characters. So I
don’t understand why one way leads to “unnecessary complexity” and the
other way does not.

In UTF-32 a character is a sequence of code points.  In UTF-8 it is a
sequence of code units.  In either case, if you have to deal with
characters, you have to deal with sequences (and most of the code does
not have to deal with characters and even less code has to deal with
code points).  So converting to UTF-32 buys you nothing and is
unnecessary complexity.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
  Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>