Re: python text, Byte Addressability And Beyond

John Levine <johnl@taugh.com> writes:

Python3 has a complex internal string format that stores each string
as 1, 2, or 4 byte values, depending on what the contents of the
string are, so ASCII is one byte, UCS-2 is two bytes, and strings that
contain code points beyond UCS-2 are four bytes. It's not clear how
hard they try to shrink stuff down when taking substrings.
>
https://peps.python.org/pep-0393/

This is a nice demonstration of the unnecessary complexity that the
codepoint mistake leads to.  In the general case they can have three
representations of the same string: wstr, utf8, and data; only one of
them needs to be non-NULL, and data is canonical if it is non-NULL
(not sure what is canonical if wstr and utf8 are present but data is
not).  If data is in latin1 format, but not ASCII, outputting both
UTF-8 and UTF-16 needs conversion (it's just 8bit->16bit expansion in
the UTF-16 case, but that means that a fast block copy is
insufficient).  On top of that, they specify both zero termination and
length indicators: length, utf8_length and wstr_length.

Of course Python3 has baked this mistake into their API, and once
software has been written for that API, the complexity becomes
necessary.

But if they had decided to just store the data as UTF-8 and use byte
indexes and lengths in their API, and adjusted the rest of their API
accordingly, they could have avoided this complexity and inefficiency,
and only palindrome and anagram programs that limit themselves to
character=codepoint would have become harder to write.

Python lets you subscript strings either individual items or
substrings, and I have written a fair amount of code that does that. I
realize that if I were doing semantic processing on Greek or Arabic, I
would not be subscripting and expecting it to return straightforwardly
useful results.

I don't doubt that the API works, it just leads to unnecessary
complexity in the implementation.

The string structure has a field for the length of the string in
UTF-8, but they don't seem to use it for anything, at least not yet,

My understanding from the PEP is that they use it for specifying the
length of the utf8 representation; of course, they also use zero
termination, so if the utf8 field is only passed to functions that use
zero-termination, the utf8_length field is not used.  Given that, as
soon as data has been initialized, the contents of the utf8 and wstr
fields are no longer used (they are not canonical), I expect that the
only function that is called for the utf8 field is that for converting
from utf8 to the data form.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
  Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>