On 9/12/2024 9:18 AM, David Brown wrote:
On 11/09/2024 20:51, BGB wrote:
On 9/11/2024 5:38 AM, Anton Ertl wrote:
Josh Vanderhoof <x@y.z> writes:
anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:
>
George Neuner <gneuner2@comcast.net> writes:
On Sun, 08 Sep 2024 15:36:39 GMT, anton@mips.complang.tuwien.ac.at
(Anton Ertl) wrote:
1) At first I thought that yes, one could just check whether there is
an overlap of the memory areas. But then I remembered that you cannot
write such a check in standard C without (in the general case)
exercising undefined behaviour; and then the compiler could eliminate
the check or do something else that's unexpected. Do you have such a
check in mind that does not exercise undefined behaviour in the
general case?
...
It is legal to test for equality between pointers to different objects
so you could test for overlap by testing against every element in the
array. It seems like it should be possible for the compiler to figure
out what's happening and optimize those tests away, but unfortunately
no compiler I tested did it.
>
That would be an interesting result of the ATUBDNH lunacy: programmers
would see themselves forced to write workarounds such as the one you
suggest (with terrible performance when not optimized), and then C
compiler maintainers would see themselves forced to optimize this kind
of code. The end result would be that both parties have to put in
more effort to eventually get the same result as if ordered comparison
between different objects had been defined from the start.
>
For now, the ATUBDNH advocates tell programmers that they have to work
around the lack of definition, but there is usually no optimization
for that.
>
One case where things work somewhat along the lines you suggest is
unaligned accesses. Traditionally, if knowing that the hardware
supports unaligned accesses, for a 16-bit load one would write:
>
int16_t foo1(int16_t *p)
{
return *p;
}
>
If one does not know that the hardware supports unaligned accesses,
the traditional way to perform such an access (little-endian) is
something like:
>
int16_t foo2(int16_t *p)
{
unsignedchar *q = p;
return (int16_t)(q[0] + (q[1]>>8));
}
Correcting the typos (in case anyone wants to copy-and-paste to godbolt.org for testing):
int16_t foo2(int16_t *p)
{
unsigned char *q = (unsigned char *) p;
return (int16_t)(q[0] + (q[1] << 8));
}
>
Now, several years ago, somebody told me that the proper way is as
follows:
>
int16_t foo3(int16_t *p)
{
int16_t v;
memcpy(&v,p,2);
return v;
}
>
That way looked horribly inefficient to me, with v having to reside in
memory instead of in a register and then the expensive function call,
and all the decisions that memcpy() has to take depending on the
length argument. But gcc optimizes this idiom into an unaligned load
rather than taking all the steps that I expected (however, I have seen
cases where the code produced on hardware that supports unaligned
accesses is worse than that for foo1()). Of course, if you also want
to support less sophisticated compilers, this idiom may be really slow
on those, although not quite as expensive as your containment check.
>
>
It is a unfortunate truth that code that is correct can be inefficient on some compilers, while code that is efficient on those compilers is not correct (according to the C standards) and can fail on other compilers. I may be a "ATUBDNH advocate", but I can certainly acknowledge that much. The C standard is concerned with the behaviour of the code, not its efficiency, and it has always been a fact of life for C programmers that different compilers give better or worse results for different ways of writing source code. Not all code can be written portably /and/ efficiently, without at least some conditional compilation.
foo1() is defined behaviour if and only if the pointer is correctly aligned. For a stand-alone function,
foo2() above is perfectly correct C and has fully defined behaviour (with the obvious assumptions that CHARBIT is 8 and that int16_t exists), but only gives the correct results for little-endian systems.
foo3() is correct regardless of the endianness (with the same assumptions about the targets), but efficiency can vary.
Testing these on godbolt.org with gcc and MSVC shows these both optimise the memcpy() into a single 16-bit load. MSVC does not recognize the pattern in foo2() and generates poor code for it (it even uses an "imul" instruction!).
Another alternative is:
int16_t foo1v(int16_t *p)
{
volatile int16_t * q = p;
return *q;
}
The C standard does not say exactly what this will do, but you can expect the compiler to generate the load, even if it knows "p" is misaligned, and even if it knows the target does not support misaligned accesses. Of course, this has implications for optimisations as the compiler can't re-order such loads.
Would be nice, say, if there were semi-standard compiler macros for various things:
Ask, and you shall receive! (Well, sometimes you might receive.)
Endianess (macros exist, typically compiler specific);
And, apparently GCC and Clang can't agree on which strategy to use.
#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
...
#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
...
#else
...
#endif
Works in gcc, clang and MSVC.
Technically now also in BGBCC, since I have just recently added it.
And C23 has the <stdbit.h> header with many convenient little "bit and byte" utilities, including endian detection:
#include <stdbit.h>
#if __STDC_ENDIAN_NATIVE__ == __STDC_ENDIAN_LITTLE__
...
#elif __STDC_ENDIAN_NATIVE__ == __STDC_ENDIAN_BIG__
...
#else
...
#endif
This is good at least.
Though, generally takes a few years before new features become usable.
Like, it is only in recent years that it has become "safe" to use most parts of C99.
Whether or not the target/compiler allows misaligned memory access;
If set, one may use misaligned access.
Why would you need that? Any decent compiler will know what is allowed for the target (perhaps partly on the basis of compiler flags), and will generate the best allowed code for accesses like foo3() above.
Imagine you have compilers that are smart enough to turn "memcpy()" into a load and store, but not smart enough to optimize away the memory accesses, or fully optimize away the wrapper functions...
So, for best results, the best case option is to use a pointer cast and dereference.
For some cases, one may also need to know whether or not they can access the pointers in a misaligned way (and whether doing so would be better or worse than something like "memcpy()").
Whether or not memory uses a single address space;
If set, all pointer comparisons are allowed.
Pointer comparisons are always allowed for equality tests if they are pointers to objects of compatible types. (Function pointers cannot be compared at all.)
For other relational tests, the pointers must point to sub-objects of the same aggregate object. (That means they can't be null pointers, misaligned pointers, invalid pointers or pointers going nowhere.) This is independent of how the address space(s) are organised on the target machine.
What you /can/ do, on pretty much any implementation with a single linear address space, is convert pointers to uintptr_t and then compare them. There may be some targets for which there is no uintptr_t, or where the mapping from pointer to integer does not match with the address, but that would be very unusual.
I can't think when you would need to do such comparisons, however, other than to implement memmove - and library functions can use any kind of implementation-specific feature they like.
Yeah.
My "_memlzcpy()" functions do a lot of relative comparisons (more than needed for memmove):
dst<=src: memmove
(dst-src)>=sz: memcpy
(dst-src)>=32: can copy with 32B blocks
(dst-src)>=16: can copy with 16B blocks
(dst-src)>= 8: can copy with 8B blocks
1/2/4: Generate a full-block fill pattern
3/5/6/7: partial fill pattern (16B block with irregular step)
There is a difference here between "_memlzcpy()" and "_memlzcpyf()" in that:
the former will always copy an exact number of bytes;
the latter may write 16-32 bytes over the limit.
Also a "_memcpyf()" which has a similar property (may pad the copy up to the nearest multiple of 32 bytes).
This is because in some cases, the performance overhead of copying the last (sz&31) bytes is significant, say:
rsz=cte-ct;
if(rsz)
{
if(rsz&16)
{
v0=((u64 *)cs)[0]; v1=((u64 *)cs)[1];
((u64 *)ct)[0]=v0; ((u64 *)ct)[1]=v1;
cs+=16; ct+=16;
}
if(rsz&8)
{
v0=((u64 *)cs)[0];
((u64 *)ct)[0]=v0;
cs+=8; ct+=8;
}
if(rsz&4)
{
v0=((u32 *)cs)[0];
((u32 *)ct)[0]=v0;
cs+=4; ct+=4;
}
if(rsz&2)
{
v0=((u16 *)cs)[0];
((u16 *)ct)[0]=v0;
cs+=2; ct+=2;
}
if(rsz&1)
{
v0=((byte *)cs)[0];
((byte *)ct)[0]=v0;
cs++; ct++;
}
}
For small copies with awkward sizes, this tailing part can cost more than the whole rest of the copy.
>
>
Clang:
__LITTLE_ENDIAN__, __BIG_ENDIAN__
One or the other is defined based on endian.
GCC:
__BYTE_ORDER__ which may equal one of:
__ORDER_LITTLE_ENDIAN__
__ORDER_BIG_ENDIAN__
__ORDER_PDP_ENDIAN__
MSVC:
REG_DWORD is one of:
REG_DWORD_LITTLE_ENDIAN
REG_DWORD_BIG_ENDIAN
>
GCC:
__SIZEOF_type__ //gives sizeof various types
>
See above.
>
Possible:
__MINALIGN_type__ //minimum allowed alignment for type
_Alignof(type) has been around since C11.
_Alignof tells the native alignment, not the minimum.
Where, _Alignof(int32_t) will give 4, but __MINALIGN_INT32__ would give 1 if the target supports misaligned pointers.
>
Maybe also alias pointer control:
__POINTER_ALIAS__
__POINTER_ALIAS_CONSERVATIVE__
__POINTER_ALIAS_STRICT__
>
Where, pointer alias can be declared, and:
If conservative, then conservative semantics are being used.
Pointers may be freely cast without concern for pointer aliasing.
Compiler will assume that "non restrict" pointer stores may alias.
If strict, the compiler is using TBAA semantics.
Compiler may assume that aliasing is based on pointer types.
>
Faffing around with pointer types - breaking the "effective type" rules - has been a bad idea and risky behaviour since C was standardised. You never need to do it. (I accept, however, that on some weaker or older compilers "doing the right thing" can be noticeably less efficient than writing bad code.) Just get a half-decent compiler and use memcpy(). For any situation where you might think casting pointer types would be a good idea, your sizes are small and known at compile time, so they are easy for the compiler to optimise.
It depends.
In some things, like my ELF and PE/COFF program loaders, the code can get particularly nasty in these areas...
If you /must/ do such casts, or you are dealing with questionable quality code that uses them, at least add this to your code:
#ifdef __GNUC__
#pragma GCC optimize("-fno-strict-aliasing")
#endif
Ironically, a lot of the code which this comes up in is not usually used in GCC (mostly used in BGBCC).
Though, did recently feed some of it through GCC in the hope that it would maybe catch and warn about some potential bugs I was looking for (warned about a lot of other stuff, mostly unrelated, did clean up most of the warnings).
Annoyingly, it seems virtual memory has broken again in TestKern, and I am not sure the cause. Given it has broken in both the emulator and Verilog simulations (and breaking in a very similar way), with no notable recent changes to either, but a bunch of recent hackery inside TestKern trying to get Linux RV64 ELF binaries to work on it; evidence points towards something being broken in TestKern (it is behaving like there might be uninitialized or dangling pointers somewhere).
But, will not claim TestKern has particularly clean coding practices.
But, yeah, otherwise this sub-project has gotten the binaries to load, but both GLIBC and MUSL crash fairly early, but the matter is complicated by the lack of debug symbols. Currently the shared objects are loaded via a WAD4 image, but my emulator can't see any debug maps inside of a WAD4 (if they were present).
The emulator generally only sees symbols for binaries that are loaded directly into the VM's FAT filesystem image.
But:
I was generally doing this via the command-line;
The command-line is getting excessively long;
I can only see the contents of the relevant "/usr/local/..." from within WSL.
So, the main option was to build a WAD4 image from within WSL, and then load this image into the VM (and then mount it into the VFS).
But, among other tweaks:
Process creation now puts argc/argv on the top of the stack (Linux seems to do it this way); it also seems to pass the environment variables on the stack (for now, the loader leaves an empty list for these).
Have changed RV64 register mappings slightly, such than X4 and X5 now map to R4 and R5 (the RV64 context will now have TP as a separate register from TBR; looks like I will need to figure out how to mimic Linux's version of this structure, if using a Linux C library, as the C libraries appear to be trying to poke around in there).
So, register mapping is now:
X0=ZR, X1=LR, X2=SP, X3=GBR, X4..X13=R4..R13, X14=R2, X15=R3,
X16..X31=R16..R31
This change did effect TestKern, but is mostly invisible for the existing programs.
But, alas, it seems somewhere in this, something has gone amiss, and invalid memory addresses are being accessed (but, thus far, the issue is only manifesting when virtual memory is enabled). It also seems to be unrelated to the change in the register remapping for RV Mode (undoing this change has no effect, and the bugs are manifesting before this should have any effect).
...
It won't make the code correct if you are using a compiler other than gcc or clang, but it's a help.
And as a general rule, if you feel you really want to break the rules of C and still get something useful out at the end, use "volatile" liberally.
I have used "volatile" here to good effect.
Computer architects leaving Intel...
Re: Computer architects leaving Intel...
