Re: python text, Byte Addressability And Beyond

On 12/05/2024 07:40, Anton Ertl wrote:

John Levine <johnl@taugh.com> writes:

Python3 has a complex internal string format that stores each string
as 1, 2, or 4 byte values, depending on what the contents of the
string are, so ASCII is one byte, UCS-2 is two bytes, and strings that
contain code points beyond UCS-2 are four bytes. It's not clear how
hard they try to shrink stuff down when taking substrings.
>
https://peps.python.org/pep-0393/

 This is a nice demonstration of the unnecessary complexity that the
codepoint mistake leads to.

A lot of this is, I suspect, for historical reasons.  When Python was young, most software and languages used either plain ASCII or a mess of code pages for 8-bit encodings (or an even bigger mess of 16-bit encodings for CJK languages).  Unicode was the new hope for a unifying 16-bit system that would work for all characters in all languages.  So Python - like Java, Windows NT, QT, and some other systems of that era, chose UCS-2 as the modern, international and future-proof solution to strings and characters.
It turns out that UCS-2 was not enough, and these have all been suffering from mixed APIs ever since.