Re: multi bytes character - how to make it defined behavior?

On 14/08/2024 13:10, Thiago Adams wrote:

On 14/08/2024 12:34, Bart wrote:

On 14/08/2024 14:31, Thiago Adams wrote:

On 14/08/2024 10:05, Bart wrote:

On 14/08/2024 12:41, Thiago Adams wrote:

On 13/08/2024 21:33, Keith Thompson wrote:

Bart<bc@freeuk.com>  writes:
[...]

What exactly do you mean by multi-byte characters? Is it a literal
such as 'ABCD'?
>
I've no idea what C makes of that,

It's a character constant of type int with an implementation-defined
value.  Read the section on "Character constants" in the C standard
(6.4.4.4 in C17).
>
(With gcc, its value is 0x41424344, but other compilers can and do
behave differently.)
>
We discussed this at some length several years ago.
>
[...]

>
>
"An integer character constant has type int. The value of an integer character constant containing
a single character that maps to a single value in the literal encoding (6.2.9) is the numerical value
of the representation of the mapped character in the literal encoding interpreted as an integer.
The value of an integer character constant containing more than one character (e.g. ’ab’), or
containing a character or escape sequence that does not map to a single value in the literal encoding,
is implementation-defined. If an integer character constant contains a single character or escape
sequence, its value is the one that results when an object with type char whose value is that of the
single character or escape sequence is converted to type int."
>
>
I am suggesting the define this:
>
"The value of an integer character constant containing more than one character (e.g. ’ab’), or containing a character or escape sequence that does not map to a single value in the literal encoding, is implementation-defined."
>
How?
>
First, all source code should be utf8.
>
Then I am suggesting we first decode the bytes.
>
For instance, '×' is encoded with 195 and 151. We consume these 2 bytes and the utf8 decoded value is 215.

>
By that you mean the Unicode index. But you say elsewhere that everything in your source code is UTF8.

>
>
215 is the unicode number of the character '×'.
>

Where then does the 215 appear? Do your char* strings use 215 for ×, or do they use 195 and 215?

>
215 is the result of decoding two utf8 encoded bytes. (195 and 151)
>

I think this is why C requires those prefixes like u8'...'.

>

>
Then this is the defined behavior
>
static_assert('×' == 215)

>
This is where you need to decide whether the integer value within '...', AT RUNTIME, represents the Unicode index or the UTF8 sequence.

>
why runtime? It is compile time. This is why source code must be universally encoded (utf8)

>
>
In that case I don't understand what you are testing for here. Is it an error for '×' to be 215, or an error for it not to be?

  GCC handles this as multibyte. Without decoding.
 The result of GCC is 50071
static_assert('×' == 50071);
 The explanation is that GCC is doing:
 256*195 + 151 = 50071
 (Remember the utf8 bytes were 195 151)
 The way 'ab' is handled is the same of '×' on GCC. Clang have a error for that. The standard just says the value is implementation defined.
 

And what is the test for, to ensure encoding is UTF8 in this ... source file? ... compiler?

 MSVC has some checks, I don't know that is the logic.
 

Where would the 'decoded 215' come into it?

 215 is the value after decoding utf8 and producing the unicode value.
 So my suggestion is decode first.
 The bad part of my suggestion we may have two different ways of producing the same value.
 For instance the number generated by ab is the same of
 'ab' == '𤤰'
 The advantage is to converge to utf8 unicode and make it specified.
  

I use multibyte chars in my code.
For instance:
enum token {TK_EQUAL == '=='}
I prefer to write and read token.type == '==' rather than
token.type = TK_EQUAL.
An alternative for me also could be a macro.
if (token.type = MC('=', '=')) {...}
but then its worst than the type = TK_EQUAL