Re: architectural goals, Byte Addressability And Beyond

Thomas Koenig <tkoenig@netcologne.de> writes:

I'm not sure that would be the case for text containing some
non-ASCII characters, where you cannot predict branches well
(consider Å, Ø and Æ, which together appear to make up around
a bit more than 2.5% according to a random statistic I just
grabbed off the Internet), or ä, ö and ü which have around 1.5%
occurrence together.
>
In Chinese or Japanese text, I assume the spaces and punctuation
are 7-bit ASCII (are they, actually?) so things would be even
worse for branch prediction.

Branch prediction for what purpose?

A typical usage is processing of csv files containing participant
lists of a course, and the results for the course.  The participant
names contain various non-ASCII characters*.  The participant names
are usually just copied literally from some inputs to some outputs.  I
don't know if the tools I use (awk, join, sort (in the C.utf8 locale)
etc.)  do some code point processing, but if they do, it's totally
unnecessary.

In one case I sort on the names to produce reports, so in that case a
different locale and actual knowledge of the characters for collating
order purposes might be a good idea, but none of the report users has
complained yet about the sorting.  And the question is which locale
one should use when some names are from Turkey, some from Hungary,
some from Austria, some from Croatia etc.; how would
LC_COLLATE=de_AT.UTF-8 deal with all the characters that don't occur
in German?  In any case, sorting in a locale other than C involves
much more (and much more expensive operations) than just code point
recognition.  And the actual lion's share of the CPU time spent on
report processing is the conversion from .md to .pdf using pandoc.  I
am sure that this would not be measurably faster if there were only
ASCII characters in the .md files.

Bottom line: Code point conversion instructions like CU14 solve a
problem that people imagine who have no experience working with UTF-8.

* I have never encountered names containing characters outside the
roman-based alphabets, though, they are probably all romanized at some
earlier administrative step (probably when they register for the
university, or earlier), but Cyrillic, Greek, or CJK characters would
make no difference to the scripts; they would make it harder for
course staff to pronounce the names, though, so it's probably good
that the names are all romanized.

BTW, the biggest problems stem from ASCII characters in names, in
particular " " and "'".

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
  Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>