Re: Article on new mainframe use

On 8/22/2024 9:36 PM, Lawrence D'Oliveiro wrote:

On Thu, 22 Aug 2024 15:59:34 -0500, BGB wrote:
 

Underused: It is a sensible way of structuring data, but needing to
interface with it though SQL strings is awkward and inefficient.

 Actually, that works fine in any language with decent string handling.
 

String processing adds bulk and inefficiency.
Granted, maybe not enough to matter in the face of a typical database query.
But, I was left thinking, some programs use SQLite, which exists as a single giant C file. I guess it technically works, but has the downside of adding something like IIRC around 900K or so to the size of the binaries' ".text" section.
In my recent projects, I am left being a little more aware of how much code footprint and space everything costs (along with the limitations of trying to keep things performing acceptably at 50 MHz).
But, say, a goal would be to try to fit a semi-usable database engine into say a few kLOC or so (rather than several hundred kLOC).
Things like text parsing and other complex high level features would be on a high priority list of things to jettison.
Some can maybe still be present in the code, but:
If they are API calls, then if they are never called, code reachability can cull them;
If there is a text parser, and they are reachable from said parser, and the program uses said parser, one is SOL, code reachability will need to include the whole thing...
I guess, similar reasons for why one doesn't want to add too many complex string formatting features to "printf()" (since, if they are not used, they uselessly add bulk to any program that uses "printf()" and similar).
Similar related reasons for why TKGDI's built in image loading doesn't support JPG or PNG. Granted, OTOH, QOI is a bit niche, but decodes quickly and doesn't need all that much code (vs a PNG decoder).
Similar reasons for why I am still going with BMP for stuff, and am making use of CRAM (including in non-standard CRAM BMP images). Like, as much as CRAM "kinda sucks", it is fast and needs relatively little code for a decoder.
I still lack a good "cheap" JPEG stand-in. I had defined LCIF, which was essentially "QOI but with some color-cell stuff glued on". OK, but still not an ideal JPEG stand-in (one might want something that can be lossless or near lossless; LCIF's "top end" quality still basically looks like DXT1).
Then LCIF was left in competition with the "cheaper" option of CRAM+RP2 (where, if I already have a CRAM decoder and an RP2 decoder, RP2 encoded CRAM in a BMP doesn't add too much additional code footprint). Though, both top-end quality and Q/bpp are worse with CRAM.
Basic CRAM is ~ 2.0bpp, but with RP2 compression may be ~ 0.25 to 2.0 bpp depending on how LZ compressible the image is (and RP2 tends to beat LZ4 in this case). Works OK for UI graphics.
I have some ideas I could try, if I wanted to do something vaguely JPEG like but at a lower line count. But, in any case, if it goes too much over ~ 1k lines, it becomes moot (at around 2k lines, that is the size of a JPEG decoder).
Though, minimizing line-count would likely mean the "semi-crappy" option of probably using Rice coding and a block Haar transform.
Using 12-bit length limited Huffman would give better compression, but (unlike Rice) would still need around 250 lines for the entropy coder (vs ~ 100 or so for Rice, mostly due to bitstream code, like PeekBits/SkipBits/ReadBits functions).
Well, and my kernel has already gotten fairly large, due to the existing overhead needed for GUI and GL backend stuff (it becomes tempting to try to fold this off into a DLL).

Or, do some weird OO thing where object methods exist for SQL keywords
and each method returns an object that can be used to glue on the next
keyword:
    res=db.select().from("FOOTAB").where().equals("NAME", "John").end();

 ORMs are a waste of time.

Yeah, this is not an API design of honor...
Granted, trying to awkwardly map it to an OpenGL style API design wouldn't be much better, say:
   dbmsBeginSelect();
   dbmsFromTable("FOOTAB");
   dbmsWhere(DBMS_EQUAL, "NAME", "John");
   dbmsEndSelect();
   while(dbmsPollResult())
   {
     dbmsGetResultStr("NAME", dbFirstName);
     dbmsGetResultStr("LAST", dbLastName);
     ...
     dbmsNextResult();
   }
One other possibility might be something like, say:
   DBMS_HTABLE hFootab;
   DBMS_HKEYELEM aSelKey[8];
   DBMS_HVALELEM aSelVal[8];
   DBMS_HENUM    aSelOpr[8];
   DBMS_HKEYELEM aResKey[12];
   DBMS_HVALELEM aResVal[12];
   FBMS_HQUERY   hRes;
   int           nResCol;
   int i, j, k;
   aSelKey[0]=dbmsGetHKey(db, "NAME");
   aSelVal[0]=dbmsGetHString(db, "John");
   aSelOpr[0]=DBMS_EQUAL;
   hFooTab=dbmsOpenTable(db, "FOOTAB");
   hRes=dbmsTableSelect(db,
     hFooTab, aSelKey, aSelOpr, aSelVal, 1, NULL, 0);
   nResCol=12;  //max of 12 columns
   dbmsGetResultRowKeys(db, hRes, aResKey, &nResCol);
   for(i=0; i<nResCol; i++)
   {
     str=dbGetKeyName(db, aResKey[i]);
     printf("%s  ". str);
   }
   printf("\n");
   while(dbmsPollResultNotDone(db, hRes))
   {
     dbmsReadResults(db, hRes, aResKey, aResVal, nResCol);
     // do something with results
     for(i=0; i<nResCol; i++)
     {
       str=dbGetValueAsStringTemp(db, aResVal[i]);  //*1
       printf("%s  ". str);
     }
     printf("\n");
   }
   dbmsCloseResults(db, hRes);
*1: Where the string pointer will be valid until either the next results are read or the results are closed. Similarly, any values read from the results will only be valid until either the next results are read or the results are closed.
Such an API design would still be awkward, but could potentially be lower overhead than some other options.
Skimming SQLite DB format:
Why such large nodes?...
I would have thought maybe 4K, not 32K or 64K...
Then again, maybe bigger might be better for less row-related fragmentation.
Big Endian for some reason.
It is a bit more variable-length than I had imagined.
   My guess would have been to have a 4-bit tag, say:
     0: Int32
     1: Float32
     2: Int64
     3: Float64
     4: String, 0..8 B (Inline)
     5: String, 9..16 B (Inline)
     6: String/Blob, Stored at end of node
     7, String/Blob, Stored elsewhere
     ...
Maybe a case could be made for including Int16 and Binary16.
The Null value need not be present, since any absent column would be assumed Null.
Likely end-of-node would be a 32-bit format, say:
   (31:30): String or Blob
   (29:18): Size
   (17: 0): Offset
The "Elsewhere" format would be 64-bit, also encoding the Node for where to look for the data:
   0/1: String/Blob, Direct position within node.
   2/3: Big String/Blob, (29:0) giving total size.
   (63:32): Destination Node
Looks like SQLite relies on the table-layout to identify the columns, rather than what I had assumed of encoding a key's name ID with each item (and more built-in tables to describe the contents of the user-defined tables).
But, in a high-level sense, it seems that it isn't that far removed from what I had imagined either.
...
Skim: I had not considered the possibility of supporting journals or rollback. Seems like a needless expense for a "light duty" database.