Re: Byte Addressability And Beyond

David Brown <david.brown@hesbynett.no> writes:

On 10/05/2024 18:20, Anton Ertl wrote:

1) I only came up with the following use cases where you need to deal
with individual non-ASCII characters: Palindrome checkers and anagram
programs; I remember somebody mentioning a third use (which I promptly
forgot), but anyway, there are few cases.
 
2) But even for those few cases, UTF-32 is not good enough, because a
code point is not a character.
 

>
I agree that it is usually unnecessary to convert to UTF-32 - I am
merely saying that /if/ you feel you want to expand the code points,
then UTF-32 is fine for the purpose and you should not have to worry
about endianness because you should not be moving it off your computer,
thus native endianness is all you need.

Yes.  The point I wanted to make is that there is the frequent
misconception that dealing with individual arbitrary characters is
something that is relatively common, and that one can do that by using
UTF-32 (or UTF-16); it isn't, and one cannot.  If you stick with UTF-8
and use byte lengths and byte indexes, you can do almost everything as
well or better (with less complication and more efficiently) as by
converting to UTF-32 and back.

People sometimes say they want to expand to code points to be able to
see the length of the string in characters, or to index characters, or
for easier splicing or joining strings.  I don't think these are
particularly useful in practice, but UTF-32 is fine for those that want it.

Looking up "splicing strings", I find that this is a term used in
connection with Python for specifying substrings.  Python3 is a
language that lives the codepoint mistake to the extreme (and from
what I read, this was one of the major pain points in the
Python2->Python3 transition), but anyway, with UTF-8 one way to
represent a substring is to use the start index and length in bytes
(aka code units) rather than code points.

Looking up "joining strings" brings up the Python join() method, which
is a variant of string concatenation.  There is certainly no need to
convert UTF-8 to UTF-32 and back for concatenating strings; just
concatenate the UTF-8 strings.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
  Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>