Re: python text, Byte Addressability And Beyond

According to Anton Ertl <anton@mips.complang.tuwien.ac.at>:

It turns out that UCS-2 was not enough, and these have all been
suffering from mixed APIs ever since.

>
That's certainly true for Java (first release 1995), Windows NT (first
released 1993) and QT (first released 1995).

Don't forget Javascript, which means every browser if full of UCS-2 and/or UTF-16.

Except Python3.  I am not familiar with Python, but from the
discussions I have read my impression is: Python2 (released 2000)
supported strings of bytes, and people put UTF-8 in there and worked
with that.  Python3 (released 2008) was supposed to be a cleanup and
instead of refining the code-unit-based approach of Python2 they
introduced a code-point-based approach, which supported fast indexing
of code points, a worthless feature.  And they found out how hard it
is to migrate a code base.

It makes somewhat more sense than that.

Python2 had a string type which was an variable length array of 8-bit
characters, and a Unicode type which was an variable length array of
code points. You could use a string to hold either ASCII text or
arbitrary strings of bytes, depending on what operators and functions
you used. Python3 reorganized this so that there is only one string
type used for both ASCII and Unicode and a separate byte type for
arbitrary strings of data.

I can say from experience that the python3 approach is less confusing,
and that in contexts where you know the strings are ASCII, e.g., mail
or http message headers, subscripting makes sense, even though it
mostly doesn't for sequences of Unicode code points. Even with code
points it can make some sense, e.g., if you know you have text in an
alphabetic language, you can find the code points that are white space
to do stuff with words.

In recent years python has been adding type declarations so you can
say that a particular variable or function parameter has to be of a
particular type or one of a union of types, e.g. it can be an int or a
float or a Decimal object. They haven't yet created subtypes to limit
the range of a type but I expect they will. That would let say a
Percent is an integer between 0 and 100 and or an ASCII is a string
with all the code points <= 0x7f. You could write more robust code
that doesn't accidentally try to subscript into random non-ASCII code
points.
--
Regards,
John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly