Re: Unicode in strings

On 5/31/2024 12:21 PM, MitchAlsup1 wrote:

BGB wrote:
 

On 5/30/2024 11:25 AM, Anton Ertl wrote:

Stefan Monnier <monnier@iro.umontreal.ca> writes:

I'm not sure the codepoint-oriented API is the best option, but it's
not
completely clear what *is* the best option.  You mention a
byte-oriented
API and you might be right that it's a better option, but in the case
of
Emacs that's what we used in Emacs-20.1 but it worked really poorly
because of backward compatibility issues.  I think if we started from
scratch now (i.e. without having to contend with backward
compatibility,
and with a better understanding of Unicode (which barely existed back
then)) it might work better, indeed, but that's not been an option

>
Plus, editors are among the very few uses where you have to deal with
individual characters, so the "treat it as opaque string" approach
that works so well for most other code is not good enough there.  The
command-line editor of Gforth is one case where we use the xchar words
(those for dealing with code points of UTF-8).
>

 

Yeah.

 

For text editors, this is one of the few cases it makes sense to use 32
>
or 64 bit characters (say, combining the 'character' with some additional metadata such as formatting).

 

Though, one thing that makes sense for text editors is if only the "currently being edited" lines are fully unpacked, whereas the others can remain in a more compact form (such as UTF-8), and are then
unpacked
>
as they come into view (say, treating the editor window as a 32-entry modulo cache or similar).

 

For the rest, say, one can have, say, a big buffer, with an array of lines giving the location and size of the line's text in the buffer.

 In a modern text editor, one can paste in {*.xls tables, *.jpg, *.gif,
..}
along with text from different fonts and different backgrounds on a per
character basis.
 

Errm, I think we call this a word processor, not a text editor.
Granted, text editors don't usually store font or formatting information in the text itself, but rather it exists temporarily for things like "syntax highlighting".

If a line is modified, it can be reallocated at the end of the buffer, and if the buffer gets full, it can be "repacked" and/or expanded as needed. When written back to a file, the buffer lines can be emitted in-order to the text file.

 

Not entirely sure how other text editors manage things here, not really
>
looked into it.

 If you think about it with the above features, you quickly realize it
is not just text anymore.
 

But, word processors are their own category...
Typically, they also have their own specialized formats (though, "big blob of XML inside a ZIP package" seems to have become popular).
Whereas text-editors typically use plain ASCII/UTF-8/UTF-16 files...
The great "feature creep" in text editors is mostly that modern ones support syntax highlighting and emojis.
An intermediate option would be a wysiwyg editor that does MediaWiki or Markdown. Though, annoyingly, there don't seem to be any that exist as standalone desktop programs (seemingly invariably they are written in JavaScript or similar and intended to operate inside a browser).
I might eventually need to get around to writing something like this (mostly because I use MediaWiki notation for some of my own documentation). Also arguably mode advanced than the system used by "info" and "man", though a tool along these lines could make sense (but possibly as an intermediate, with an interface more like "man" but able to jump between documents more like "info").
Also, bug hunt is annoying. Find/fix one bug, but more bugs remain...
My project is seemingly in a rather buggy state right at the moment.
But, I guess, did add things like file redirection and similar, along with a few more standard commands.
So, in the working version, technically things like "cat file1 > file2" or "program > file" and similar are now technically possible...
But, also, everything has turned into a crapstorm of crashes...

 

- anton