Re: Unicode in strings

Thomas Koenig <tkoenig@netcologne.de> writes:

Anton Ertl <anton@mips.complang.tuwien.ac.at> schrieb:

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>

Does Unicode even has the notion of "character", really?

>
AFAIK it does not.  But applications like palindrome checkers care
about characters, not code points.

>
Considering the huge market for palindrome checkers, that is a
real concern, especially if they involve characters for which
UTF-32 is not sufficient, such as smileys.
>
Is there any language whose characters cannot be represented in
UTF-32?

The goal of Unicode is to support all writng systems; AFAIK they are
not yet finished, but they expect that these writing systems will all
fit into the space provided by UTF-16 (i.e., a little over one million
code points), but they found it necessary to introduce the concept of
composing glyphs from multiple code points.

So if your question is: "Is there any language where one character
cannot be represented by a single Unicode code point?"  The answer is
that the Unicode designers certainly expect that there are such
writing systems.

And looking at <https://en.wikipedia.org/wiki/Telugu_script> (just an
example), I see that the table of Unicode code points for Telugu
<https://en.wikipedia.org/wiki/Telugu_script#Unicode> is much smaller
than the tables of glyphs in
<https://en.wikipedia.org/wiki/Telugu_script#Articulation_of_consonants>
and
<https://en.wikipedia.org/wiki/Telugu_script#Consonants_with_vowel_diacritics>,
so the Telugu script seems to be one writing system that cannot be
represented with only precomposed characters.

I don't know if palindromes are a thing in Telugu, though.

But, as your reference to the size of the market for palindrome
checkers indicates, there is actually little code where dealing with
individual characters is relevant.  For code where individual
characters are not relevant and opaque strings are sufficient, there
is no reason to use UTF-32.  And for code where individual characters
are relevant, code points are not sufficient in general, so there is
no reason to use UTF-32 for that, either.

Interestingly, Emacs 27.1 manages to deal with "తెలుగు లిపి" (which
contains 6 characters composed of a total of 11 code points) just
fine, while it fails on König (with a decomposed Umlaut-o).

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
  Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>