Re: python text, Byte Addressability And Beyond

According to Anton Ertl <anton@mips.complang.tuwien.ac.at>:

Looking up "splicing strings", I find that this is a term used in
connection with Python for specifying substrings.  Python3 is a
language that lives the codepoint mistake to the extreme (and from
what I read, this was one of the major pain points in the
Python2->Python3 transition), but anyway, with UTF-8 one way to
represent a substring is to use the start index and length in bytes
(aka code units) rather than code points.

Python3 has a complex internal string format that stores each string
as 1, 2, or 4 byte values, depending on what the contents of the
string are, so ASCII is one byte, UCS-2 is two bytes, and strings that
contain code points beyond UCS-2 are four bytes. It's not clear how
hard they try to shrink stuff down when taking substrings.

https://peps.python.org/pep-0393/

Python lets you subscript strings either individual items or
substrings, and I have written a fair amount of code that does that. I
realize that if I were doing semantic processing on Greek or Arabic, I
would not be subscripting and expecting it to return straightforwardly
useful results.

The string structure has a field for the length of the string in
UTF-8, but they don't seem to use it for anything, at least not yet,
--
Regards,
John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly