Re: transpiling to low level C

On 12/25/2024 3:41 AM, BGB wrote:

On 12/23/2024 3:18 PM, Tim Rentsch wrote:

Michael S <already5chosen@yahoo.com> writes:
>

On Mon, 23 Dec 2024 09:46:46 +0100
David Brown <david.brown@hesbynett.no> wrote:
>

And Tim did not rule out using the standard library,

>
Are you sure?

>
I explicitly called out setjmp and longjmp as being excluded.
Based on that, it's reasonable to infer the rest of the
standard library is allowed.
>
Furthermore I don't think it matters.  Except for a very small
set of functions -- eg, fopen, fgetc, fputc, malloc, free --
everything else in the standard library either isn't important
for Turing Completeness or can be synthesized from the base
set.  The functionality of fprintf(), for example, can be
implemented on top of fputc and non-library language features.

  If I were to choose a set of primitive functions, probably:
   malloc/free and/or realloc
     could define, say:
       malloc(sz) => realloc(NULL, sz)
       free(ptr) => realloc(ptr, 0)
     Maybe _msize and _mtag/..., but this is non-standard.
       With _msize, can implement realloc on top of malloc/free.
 For basic IO:
   fopen, fclose, fseek, fread, fwrite
 printf could be implemented on top of vsnprintf and fputs
   fputs can be implemented on top of fwrite (via strlen).
   With a temporary buffer buffer being used for the printed string.
 ...
  Though, one may still end up with various other stuff over the interface as well. Though, the interface can be made open-ended if one has a GetInterface call or similar, which can request other interfaces given an ID, such as, FOURCC/EIGHTCC pair, a SIXTEENCC, or GUID (*1). IMHO, generally preferable over a "GetProcAddress" mechanism due to lower overheads; tough, with an annoyance that interface vtables generally have a fixed layout (generally can't really add or change anything without creating binary compatibility issues; so a lot of tables/ structures need to be kept semi-frozen).
 Though, APIs like DirectX had dealt with the issue of having version numbers for vtables and then one requests a specific version of the vtable (within the range of versions supported by the major version of DirectX). But, this is crufty.
 *1: Say: QWORD qwMajor, QWORD qwMinor.
   qwMajor:
     Major ID (FOURCC, EIGHTCC)
     Or: First 8 bytes of SIXTEENCC or GUID
   qwMinor:
     SubID/Version (FOURCC or EIGHTCC)
     Second 8 bytes of SIXTEENCC or GUID.
   Where:
     High 32 bits are 0, assume FOURCC.
     Else, look at bits to determine EIGHTCC vs GUID.
     Assume if both are EIGHTCC, value represents a SIXTEENCC.
     Bit patterns for valid SIXTEENCCs vs GUIDs are mutually exclusive.
     Names make more sense for public interfaces.
       Leaving GUIDs mostly for private/internal interfaces.
 Well, unlike Windows, where they use GUIDs for pretty much everything here (and also, I didn't bother with an IDL compiler; generally doing all this directly in C).
 

Clarification:
Though, despite taking influence from COM, it is not COM.
I am not using the COM API, and generally practices regarding vtable structure, etc, are a bit more loose.
There is also not currently any plan to actually implement the OLE or COM APIs. Only that some similar ideas are in use.
Pretty much everything else is different...
COM uses a 16-byte struct to convey a GUID;
I was using pairs of 64-bit integer values.
...
Some ideas from OLE, such as storing object instances from one library in a "document" held by an unrelated program instance, and then saving/reloading them later, are not a thing in my case.
It is possible I could consider doing something similar to OLE, but I don't have an immediate use-case (and, more often, I was using the object interfaces internally for things like OS level APIs).
Note that many core OS APIs are still a bit more mundane, like:
   memory is still managed using pointers;
   files IO is still managed using integer handles;
   ...
Though, within the kernel, open VFS files are implemented via objects with vtable pointers. This detail is not exposed to program instances, where the system calls identify them via integer handles.
Well, and also I am using a Unix style directory tree structure, rather than drive letters.
But, does differ some in things like locating DLLs for a program:
   ELF:
     Either "/lib/", "/usr/lib/",
     or a hard-coded path in the binary.
   Win:
     Check current directory;
     Then search PATH;
   TK:
     Check first in the directory the EXE is found;
     Then search LIBPATH;
     Then search PATH.
Hard coding paths in the binary does mean though that the installation path for any binaries that depends on custom SO's is fixed, which is not ideal. Checking relative to the binary allows more flexible installation paths.

 Well, and some wonk, like the exact contents of structures like BITMAPINFOHEADER being interpreted based on using biSize as a magic number (well, sometimes with other stuff glued onto the end, as understood based the use of the biCompression field), ...
 But, it has held up well, this structure being almost as old as I am...
 

Clarification:
I am towards the older end of the Millennial / Gen Y age range...
Started existence in the IBM Clones and MS-DOS era, but by the time I was using computers, was mostly in the era of Windows, CD-ROM based FMV games, and early/slow internet.

  In a few cases, one might also take the option of using a "DriverProc()" style interface, where one provides a pair of context-dependent pointers and uses magic numbers to identify the desired operation, or, intermediate:
   (*ifvt)->QueryProc(ifvt, iHdl, lParm, pParm1, pParm2);
   (*ifvt)->ModifyProc(ifvt, iHdl, lParm, pParm1, pParm2);
 Where, QueryProc is intended for non-destructive operations, and ModifyProc for destructive operations.
   iHdl: Context-dependent integer handle;
   lParm: Magic command number.
   pParm1/pParm2: Magic pointers, often:
     pParm1: Input data address;
     pParm2: Output data address.
 Where, vtable is usually provided in "VT **" form, hence the need to deref the table before a method can be invoked.
 

Well, theoretically, say:
   First 4 pointers are reserved;
     Used internally for various stuff;
   Methods take the object instance as the first argument;
   ...
The pointer itself points to an object instance, which may often be a dummy.
   Then, the object starts with a pointer to the vtable.

  Actually, some of this overlaps with how I had implemented the C library for DLLs in my project:
Only the main binary has the full C library;
DLL's generally use a C library which calls back to the main C library via a COM style interface (things like malloc/free and stdio calls are routed over this interface).
 

Looking back at it, this may not count as "COM like", as, many of the vtable pointers deviate from the traditional form:
   Don't take the object as a first argument;
     Many are just plain C function pointers.
    Eg: ptr=(*vt)->malloc_fp(sz);
Then again, for marshaling the C library across DLL boundaries, it likely does not matter (and would have complicated the interface; requiring method pointers to take a vtable pointer only to ignore it; as conceptually the C library is global across the whole process instance).

Note that this is partly because in my case:
1, DLLs only allow an acyclic dependency graph;
2, The mechanism does not currently allow sharing global variables;
3, There was a desire to allow dlopen/dlsym to dynamically load libraries.
 1 & 3 mean that if a statically-linked C library is used for the main binary:
One needs to also statically link a C library to each DLL;
The C library needs to operate over a COM interface for shared interfaces.
 Or, alternatively, that only a DLL may be used for the C library, and all DLLs would need to use the same C library DLL.
 

Groan, I had described it as a COM interface, but as noted above, this is not correct in this case...
The vtable in question isn't even close to following COM patterns.
How closely patterns are followed is kinda variable.
But, as noted, if no shared interface were used, then each DLL (and the main program binary) would effectively have their own heap and could not share "FILE *" pointers.
While Windows generally has this limitation (at least with MSVC), I personally didn't want this (better if one can "malloc()" something in one library and "free()" it in another, and not horribly break the C runtime).
Cygwin and MinGW had addressed this issue in different ways (say, in the case of Cygwin, by consolidating all of the core stuff into "cygwin1.dll").
Can note that in my case, each binary image still gets its own native copy of things like memcpy/memset/strlen/...
These don't depend on any external state, and generally one wants a low-overhead interface for these (along with potential of special handling by the compiler).
I had at one point considered writing a new C library where this stuff would have been engineered better, but this fizzled due to inertia. In this case, the library would have been fully split into a "client" and "server" parts:
"client": Has all the recognizable parts of the C library;
"server": Backend where all the magic happens (the core parts of malloc and stdio and similar reside here).
In such a library, a lot the client-side calls would be wrappers, say:
   void *malloc(size_t size)
   {
      __clib_autoinit();  //bring up C library if needed
      return((*__clib_vta)->Malloc(__clib_vta, size));
   }
It is unclear here if the server would still be static linked to the main EXE, or if it would be a component that is dynamically loaded by the kernel as needed during process creation (this could make the main EXE smaller);
or, instead, going the route of having the C library server part inside of a common DLL (like in Cygwin).
At present, there is a pointer in the task context structure than is set by the main binary to allow for the DLL's C libraries to bootstrap themselves (initially a "GetProcAddress" function, that at present only serves to fetch the main instance vtable pointer).
Note that some libraries like TKGDI (used for graphics/sound/user-input) is itself mostly a thin wrapper over a vtable on the client side (though, with some of its own logic, as data needs to be passed over the interface in "GlobalAlloc" memory buffers; as the server is generally running in a different process).
TKRA-GL (my OpenGL implementation) also internally uses a vtable structure, but slightly different:
Most of the normal OpenGL calls are handled on the client side;
The vtable essentially mostly handles things like texture uploads and the backend logic for glDrawArrays / glDrawElements calls.
Note that the "glBegin()"/"glEnd()" interface exists primarily as a wrapper over the "glDrawArrays()" mechanism.
Currently the backend parts run in the kernel, but it is tempting to consider folding it off to a dynamically loadable module as it adds significant bulk (and is not always needed).
Say, for example, only loading the OpenGL DLL kernel-side if a user program tries to create an instance of OpenGL.
Likewise, maybe work towards further separating the client and server parts of the GL implementation, as there was not a split initially. Not really sure how it usually works in other systems, this stuff doesn't seem well documented (though, generally seems like, at least on Windows: "opengl32.dll" wraps GPU vendor provided DLL, which then does whatever, to communicate with the backend driver).

 Note that neither 1 nor 2 traditionally apply with ELF Shared Objects (which usually both shared everything and allow for cyclic dependency graphs). But, traditionally ELF has other drawbacks, like needing to access variables and call functions via a GOT (which has higher overhead than direct calls, or accessing global variables as a fixed offset relative to a known base register, ...).
 Note that having the kernel inject DLLs into a running process wouldn't really mix well with the way glibc approaches shared objects (where, it manages this stuff in userland, rather than having this left up to the kernel's program loader).
 May not matter as much though as if providing an COM-like interface, one doesn't necessarily actually need dlopen/dlsym to be able to see the symbols in the library that the interface came from.
  Where, in this case, COM-like interfaces may be used in ways that deviate from usual dependency ordering; and was more flexible. They are awkward to use directly, so it may make sense to provide C API wrappers (thus far, usually statically linked, but they can fetch the interfaces they need from the main C library or the OS).
 Where, in my case, the OS interface is a mix of conventional syscalls and object-method-calls routed over the syscall interface (the target being either in the kernel or in another process; or the OS might load a DLL into the client process and return a process-local vtable).
 If non-local, generally the method pointers are generic, and serve to forward the call over the syscall mechanism (the syscall interface being used in a somewhat different way from how it would be used in something like Linux; where Linux generally just does not do things this way...).
 

Can note that in trying to get glibc ELF binaries to work on my stuff, effectively there is a separate syscall interface that mimics the Linux syscall interface.
But, likely, these would represent a different "ecosystem" in terms of the binaries (besides just the ELF / PE differences).