Re: Byte Addressability And Beyond

On 11/05/2024 17:31, Anton Ertl wrote:

David Brown <david.brown@hesbynett.no> writes:

On 10/05/2024 18:20, Anton Ertl wrote:

1) I only came up with the following use cases where you need to deal
with individual non-ASCII characters: Palindrome checkers and anagram
programs; I remember somebody mentioning a third use (which I promptly
forgot), but anyway, there are few cases.
>
2) But even for those few cases, UTF-32 is not good enough, because a
code point is not a character.
>

>
I agree that it is usually unnecessary to convert to UTF-32 - I am
merely saying that /if/ you feel you want to expand the code points,
then UTF-32 is fine for the purpose and you should not have to worry
about endianness because you should not be moving it off your computer,
thus native endianness is all you need.

 Yes.  The point I wanted to make is that there is the frequent
misconception that dealing with individual arbitrary characters is
something that is relatively common, and that one can do that by using
UTF-32 (or UTF-16); it isn't, and one cannot.  If you stick with UTF-8
and use byte lengths and byte indexes, you can do almost everything as
well or better (with less complication and more efficiently) as by
converting to UTF-32 and back.
 

Agreed.

People sometimes say they want to expand to code points to be able to
see the length of the string in characters, or to index characters, or
for easier splicing or joining strings.  I don't think these are
particularly useful in practice, but UTF-32 is fine for those that want it.

 Looking up "splicing strings", I find that this is a term used in
connection with Python for specifying substrings.  Python3 is a
language that lives the codepoint mistake to the extreme (and from
what I read, this was one of the major pain points in the
Python2->Python3 transition), but anyway, with UTF-8 one way to
represent a substring is to use the start index and length in bytes
(aka code units) rather than code points.
 

I was not thinking of Python in particular, and I don't think the term "splicing" is Python specific.  But Python is generally a good and popular language when you need to do lots of text manipulation, so maybe that's where the association comes from (at least for search engines).
People often think it is easier to do string manipulation - joining, splitting, replacing, etc., - when you have fixed size units per character.  I agree with you that this is not actually true, especially if you want to support arbitrary Unicode characters (such as combining characters) that don't fit in a single code point.  But it is not uncommon to think it is, and if you can make some simplifications to the text you support (specifically, limiting your code to single code point characters) then UTF-32 can be helpful.  (I think everyone will at least agree that it's better than UTF-16!)

Looking up "joining strings" brings up the Python join() method, which
is a variant of string concatenation.  There is certainly no need to
convert UTF-8 to UTF-32 and back for concatenating strings; just
concatenate the UTF-8 strings.
 

Sure.