Sujet : Re: Byte Addressability And Beyond
De : terje.mathisen (at) *nospam* tmsw.no (Terje Mathisen)
Groupes : comp.archDate : 30. May 2024, 09:10:55
Autres entêtes
Organisation : A noiseless patient Spider
Message-ID : <v39caf$1jtgk$1@dont-email.me>
References : 1 2 3 4 5 6 7 8
User-Agent : Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0 SeaMonkey/2.53.18.2
EricP wrote:
Thomas Koenig wrote:
Lawrence D'Oliveiro <ldo@nz.invalid> schrieb:
On Mon, 27 May 2024 19:09:51 -0000 (UTC), John Levine wrote:
>
According to EricP <ThatWouldBeTelling@thevillage.com>:
One could have instructions that make it easier to parse the variable
length UTF-8 sequences into codepoints.
That would be the CU14 instruction on zSeries, to turn UTF-8 into
UTF-32. CU41 goes the other way.
What is the point, in this day and age, of having special machine instructions to convert character encodings?
>
Have you looked at decoding algorithms for UTF-8?
It's almost like the perfect application of risc instruction design:
a long sequence of individual instructions of conditional branches,
bit field extracts, inserts, and shifts, is replace in HW by
a small number of muxes that can to the same in one clock.
If that CU14 can also return the number of bytes consumed, along with the resulting 32-bit character, then it would be perfect. Is that what it is doing?
You still have the horrible combining codepoints problem of course, where you have to apply CU14 once more just in order to find out if it was in fact a combining code, and do that without any buffer overruns etc.
Personally I tend to punt on these kinds of algorithms and simply demand that the decoding source buffer have at least enough extra buffer space at the end to avoid the problem.
I.e. my LZ4 decoder is significantly faster than what Google is using, but it will happily grab up to 11 or 27 bytes past the actual end of input.
Terje
-- - <Terje.Mathisen at tmsw.no>"almost all programming can be viewed as an exercise in caching"
Date | Sujet | # | | Auteur |
1 May 24 | Byte Addressability And Beyond | 590 | | Lawrence D'Oliveiro |
1 May 24 | Re: Byte Addressability And Beyond | 431 | | John Levine |
1 May 24 | Re: Byte Addressability And Beyond | 409 | | Lawrence D'Oliveiro |
1 May 24 | Re: Byte Addressability And Beyond | 3 | | John Levine |
1 May 24 | Re: Byte Addressability And Beyond | 1 | | John Levine |
1 May 24 | Re: Byte Addressability And Beyond | 1 | | Lawrence D'Oliveiro |
1 May 24 | Re: Byte Addressability And Beyond | 1 | | Michael S |
1 May 24 | Re: Byte Addressability And Beyond | 404 | | John Levine |
2 May 24 | Re: Byte Addressability And Beyond | 382 | | Lawrence D'Oliveiro |
2 May 24 | Re: Byte Addressability And Beyond | 4 | | John Levine |
2 May 24 | Re: Byte Addressability And Beyond | 3 | | Lawrence D'Oliveiro |
2 May 24 | Re: Byte Addressability And Beyond | 2 | | John Levine |
5 May 24 | Re: Byte Addressability And Beyond | 1 | | Lawrence D'Oliveiro |
2 May 24 | Re: Byte Addressability And Beyond | 367 | | John Savard |
2 May 24 | Re: Byte Addressability And Beyond | 2 | | MitchAlsup1 |
11 May 24 | Re: Byte Addressability And Beyond | 1 | | John Savard |
4 May 24 | Re: Byte Addressability And Beyond | 364 | | Lawrence D'Oliveiro |
8 May 24 | Re: Byte Addressability And Beyond | 363 | | John Savard |
8 May 24 | Re: Byte Addressability And Beyond | 2 | | Lawrence D'Oliveiro |
10 May 24 | Re: Byte Addressability And Beyond | 1 | | David Brown |
8 May 24 | Re: Byte Addressability And Beyond | 360 | | MitchAlsup1 |
8 May 24 | Re: Byte Addressability And Beyond | 359 | | John Levine |
8 May 24 | Re: Byte Addressability And Beyond | 357 | | Lawrence D'Oliveiro |
9 May 24 | Re: Byte Addressability And Beyond | 356 | | John Levine |
10 May 24 | Re: Byte Addressability And Beyond | 354 | | David Brown |
10 May 24 | Re: Byte Addressability And Beyond | 353 | | Anton Ertl |
11 May 24 | Re: Byte Addressability And Beyond | 352 | | David Brown |
11 May 24 | Re: Byte Addressability And Beyond | 351 | | Anton Ertl |
11 May 24 | Re: Byte Addressability And Beyond | 158 | | David Brown |
11 May 24 | Re: Byte Addressability And Beyond | 1 | | Anton Ertl |
27 May 24 | Re: Byte Addressability And Beyond | 156 | | Lawrence D'Oliveiro |
27 May 24 | Re: Byte Addressability And Beyond | 155 | | John Levine |
27 May 24 | Re: Byte Addressability And Beyond | 154 | | Lawrence D'Oliveiro |
27 May 24 | Re: Byte Addressability And Beyond | 153 | | John Levine |
27 May 24 | Re: Byte Addressability And Beyond | 149 | | John Levine |
27 May 24 | Re: Byte Addressability And Beyond | 1 | | MitchAlsup1 |
28 May 24 | Re: Byte Addressability And Beyond | 147 | | Lawrence D'Oliveiro |
28 May 24 | Re: encoding conversion, Byte Addressability And Beyond | 1 | | John Levine |
28 May 24 | Re: Byte Addressability And Beyond | 145 | | Thomas Koenig |
29 May 24 | Re: Byte Addressability And Beyond | 137 | | Lawrence D'Oliveiro |
29 May 24 | Re: Byte Addressability And Beyond | 136 | | Anton Ertl |
29 May 24 | Re: Byte Addressability And Beyond | 12 | | Stefan Monnier |
29 May 24 | Re: Byte Addressability And Beyond | 10 | | Stefan Monnier |
29 May 24 | Re: Byte Addressability And Beyond | 3 | | John Levine |
30 May 24 | Re: Byte Addressability And Beyond | 2 | | George Neuner |
4 Jun 24 | Re: Byte Addressability And Beyond | 1 | | George Neuner |
30 May 24 | Re: Byte Addressability And Beyond | 6 | | Anton Ertl |
4 Jun 24 | Re: Byte Addressability And Beyond | 1 | | Lawrence D'Oliveiro |
4 Jun 24 | Re: Byte Addressability And Beyond | 4 | | Stefan Monnier |
7 Jun 24 | Re: Byte Addressability And Beyond | 1 | | Terje Mathisen |
7 Jun 24 | Re: Character non-equivalence, was Byte Addressability And Beyond | 2 | | John Levine |
9 Jun 24 | Re: Character non-equivalence, was Byte Addressability And Beyond | 1 | | Lawrence D'Oliveiro |
30 May 24 | Re: Byte Addressability And Beyond | 1 | | Lawrence D'Oliveiro |
30 May 24 | Re: Byte Addressability And Beyond | 117 | | Lawrence D'Oliveiro |
30 May 24 | Re: architectural goals, Byte Addressability And Beyond | 66 | | John Levine |
30 May 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | Stephen Fuld |
30 May 24 | Re: architectural goals, Byte Addressability And Beyond | 22 | | Anton Ertl |
30 May 24 | Re: architectural goals, Byte Addressability And Beyond | 21 | | Thomas Koenig |
30 May 24 | Re: architectural goals, Byte Addressability And Beyond | 8 | | Michael S |
30 May 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | Thomas Koenig |
30 May 24 | Re: IBM architectural goals, Byte Addressability And Beyond | 5 | | John Levine |
30 May 24 | Re: IBM architectural goals, Byte Addressability And Beyond | 2 | | Michael S |
30 May 24 | Re: IBM architectural goals, Byte Addressability And Beyond | 1 | | John Levine |
30 May 24 | Re: IBM architectural goals, Byte Addressability And Beyond | 2 | | Thomas Koenig |
30 May 24 | Re: IBM architectural goals, Byte Addressability And Beyond | 1 | | John Levine |
30 May 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | Anton Ertl |
30 May 24 | Re: architectural goals, Byte Addressability And Beyond | 3 | | Anton Ertl |
30 May 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | John Levine |
30 May 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | Thomas Koenig |
31 May 24 | Re: architectural goals, Byte Addressability And Beyond | 5 | | Terje Mathisen |
1 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 4 | | Thomas Koenig |
1 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 3 | | Anton Ertl |
2 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 2 | | John Levine |
4 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | Stefan Monnier |
4 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 4 | | Lawrence D'Oliveiro |
4 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | MitchAlsup1 |
4 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | Lynn Wheeler |
4 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | Stefan Monnier |
31 May 24 | Re: architectural goals, Byte Addressability And Beyond | 42 | | John Savard |
31 May 24 | Re: architectural goals, Byte Addressability And Beyond | 41 | | John Levine |
1 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 31 | | John Savard |
1 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 20 | | Thomas Koenig |
2 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 6 | | John Savard |
2 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 5 | | Thomas Koenig |
2 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 3 | | John Levine |
3 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 2 | | OrangeFish |
3 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | John Levine |
4 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | Lawrence D'Oliveiro |
4 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 13 | | Lawrence D'Oliveiro |
5 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 12 | | Lawrence D'Oliveiro |
5 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | Lawrence D'Oliveiro |
6 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 10 | | George Neuner |
6 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 6 | | John Levine |
7 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 4 | | Lawrence D'Oliveiro |
7 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 3 | | Stephen Fuld |
7 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 2 | | Lawrence D'Oliveiro |
7 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | Stephen Fuld |
7 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | Terje Mathisen |
6 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | Lynn Wheeler |
6 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | OrangeFish |
7 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | Lawrence D'Oliveiro |
2 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 10 | | John Dallman |
2 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | Michael S |
2 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 1 | | John Dallman |
4 Jun 24 | Re: architectural goals, Byte Addressability And Beyond | 7 | | Lawrence D'Oliveiro |
30 May 24 | Re: Byte Addressability And Beyond | 49 | | Stephen Fuld |
30 May 24 | Re: Byte Addressability And Beyond | 1 | | Anton Ertl |
30 May 24 | Re: Byte Addressability And Beyond | 2 | | Lawrence D'Oliveiro |
30 May 24 | Re: Byte Addressability And Beyond | 4 | | Terje Mathisen |
30 May 24 | Re: Byte Addressability And Beyond | 7 | | Terje Mathisen |
28 May 24 | Re: Byte Addressability And Beyond | 3 | | Lawrence D'Oliveiro |
12 May 24 | Re: python text, Byte Addressability And Beyond | 14 | | John Levine |
12 May 24 | Re: Byte Addressability And Beyond | 178 | | Thomas Koenig |
27 May 24 | Re: Byte Addressability And Beyond | 1 | | Lawrence D'Oliveiro |
8 May 24 | Re: Byte Addressability And Beyond | 1 | | Michael S |
2 May 24 | Re: Byte Addressability And Beyond | 10 | | MitchAlsup1 |
2 May 24 | Re: Byte Addressability And Beyond | 3 | | Michael S |
2 May 24 | Re: Byte Addressability And Beyond | 18 | | Anton Ertl |
1 May 24 | Byte Order (was: Byte Addressability And Beyond) | 4 | | Anton Ertl |
1 May 24 | Re: Byte Addressability And Beyond | 17 | | Stefan Monnier |
1 May 24 | Re: Byte Addressability And Beyond | 40 | | MitchAlsup1 |
1 May 24 | Re: Byte Addressability And Beyond | 15 | | Thomas Koenig |
1 May 24 | Re: Byte Addressability And Beyond | 3 | | Michael S |
2 May 24 | Re: Byte Addressability And Beyond | 4 | | Lawrence D'Oliveiro |
3 May 24 | Re: Byte Addressability And Beyond | 75 | | Anton Ertl |
5 May 24 | Re: Byte Addressability And Beyond | 20 | | John Savard |
5 May 24 | Re: Byte Addressability And Beyond | 1 | | John Savard |