Re: Byte Addressability And Beyond

EricP <ThatWouldBeTelling@thevillage.com> writes:

Stefan Monnier wrote:

I've not dealt with UTF-8 or code points but that's because I've not
written software that interacts with the non 1-byte character markets.
But even something as simple as sanitizing a character string to feed
into SQL will have to.

AFAIK you can do that by treating the UTF-8 byte sequence as if it were
an ASCII byte-sequence: all the Unicode weirdness is neatly stashed in
bytes >127 which aren't used by SQL itself anyway.
        Stefan

Of course with apologies to Herr Koenig's umlauts. :-)
>
And what of all those new Asian customers your company was hoping
to get by dealing with them in their native written language???
You could always explain to the company president that
you only work in ASCII so they should just get used to it.

 
I think you misunderstand: the code written to sanitize an ASCII string to
feed into SQL will *just work* to sanitize a UTF-8 string to feed
into SQL, no matter how many funny characters and joiners and combiners
and emojis you have in there.
 
That's part of the reason why UTF-8 is so popular: you can surprisingly
often treat it as "good old ASCII".
 
 
        Stefan

>
Ok, you accept international character data, you just don't have to
check >127 characters for "drop table" etc commands.

Actually what you check for is meta-characters like ; " '.  They are
all ASCII, so as long as your code is 8-bit-clean, your SQL string
sanitizer needs to know nothing about UTF-8.

I don't think you are being paranoid enough.
I still think you have to validate or sanitize the >127 string to
ensure the code sequences only contain well formed characters.

Then run your string through a checker/normalizer before or
afterwards.  No need to complicate your SQL sanitizer by trying to do
both at the same time.  But if you want the last bit of performance by
doing both at the same time, then you certainly don't want to convert
to UTF-32 and back.

Random hack thought #1: if the string I send starts with an umlaut as
the first code point, which doesn't display because it is invalid.

I found that hard to understand.  Do you mean that the string starts
with a composing diaresis code point and is invalid because it has no
preceding basis with which to compose?  The string may fail at the
Unicode checking/normalization stage (depending on what it checks).

Then someone edits the first char to a/o/u and magically it changes
to a different character, and deposits now go to a different account.

If someone can edit the string, and that changes where deposits go to,
someone can do that even with no Unicode involved.  E.g., if someone
can change "EricP" to "Ertl".  However, my impression is that banks
use account numbers (pure ASCII) for deposits, names are used only for
validation; so if you provide the wrong name, a money transfer may
fail to go through (not sure what happens if a deposit does not go
through), but won't be to the wrong account.

Random hack thought #2: If a character has multiple combiner code points,
does changing the order create a different character or do they map to
the same display character? Or worse, maybe combiner code point order
sensitivity is character dependent, some are, some are not.
If they do display the same, then I might create two accounts that
look identical but index differently, and redirect deposits.

That's solved by normalization.

Here's a story from work I had to do a while ago: users provided data
through some tools written in Python, that data was somehow aggregated
into one csv file (maybe with cat), and there was a Python3 script I
had to run for processing the data.  Now some users provided the data
as Latin-1 and some as UTF-8, so the csv file contained a mixture of
that.  The Python3 script dutyfully reported an error on reading the
csv file as guidelines recomment.  This was the wrong thing to do in
this application, as continuing to have this mixture was harmless.

I then wrote a small program (in Gforth) that converted such mixed
files to UTF-8, and that was one of the few uses of the Gforth words
for dealing with UTF-8 that I needed (in most other cases strings are
treated just as opaque data).  The principle was to see if the next
bytes were an UTF-8 code point or ASCII; if so, just output them.  If
they were neither, the next byte is a Latin-1 character, and is
converted to UTF-8.  Fortunately, there is no overlap between the
Latin-1 characters that occured in these data and the bytes that start
a non-ASCII UTF-8 code point.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
  Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>