Re: Small web protocols / alternative protocols

Here is a list of some other protocols and file formats. It is like the
"SMALL INTERNET PROTOCOL ROUNDUP" document posted to the mailing list for
Gemini previously, but with many differences.
 
This document is public domain. You are free to copy this document and to
make changes, e.g. to add things which are missing, change things that are
incorrect, make your own comments about any parts of it, etc.
 
**** Gopher
Specification: RFC 1436
Scheme: gopher
Port number: 70
File format: Gopher menu, Plain text
TLS: None (but see below)
Request encoding: ASCII
Client implementations: (many)
Server implementations: (many)
 
The old one. It uses a menu format, which lacks any support for relative
links; only absolute links are possible. It also does not use MIME types;
you have to use a fixed set of item types, some of which are uncommon, but
some new ones have been added since the original specification (such as "i"
for text lines that do not link to anything).
 
It also does not use URLs; to link to something that is not one of the
Gopher item types, usually a HTML link (type "h") is used with the target
selector string that starts with "URL:" and then the client should
recognize that as a URL link (if the client does not recognize it, the
server should serve a HTML page that links and redirects to the specified
URL, but some servers don't do that).
 
The Gopher protocol lacks any kind of response headers, so you cannot
specify a "file not found" error or anything else like that. The file
type must be known by the client ahead of time, usually by the item type
specified in the menu.)
 
Gopher has the "+" item type for an alternate service link (e.g. mirrors
and load balancing), although I read somewhere that it does not seem to
be used by any server nor by any client, as far as I know.
 
Gopher and Gopher+ are the only protocols in this list that do not
support virtual hosting.
 
**** Gopher+
Specification: gopher://gopher.floodgap.com/0/gopher/tech/gopherplus.txt
Scheme: gopher
Port number: 70
File format: Gopher menu, Plain text
TLS: None
Request encoding: ASCII
Client implementations: (not widely implemented)
Server implementations: (not widely implemented)
 
Adds several features to Gopher, such as:
 
- Response header (either the length of the file, a code to specify that
the length is unspecified (as in the original Gopher) or an error code).
(It still does not specify the file type.)
 
- Both temporary and permanent errors, and redirects.
 
- Item attributes, such as modification date, email address of owner of
file, abstract text, list of MIME types which it is available in (together
with file sizes), languages, geographic coordinates, time zone, etc.
 
- Forms.
 
**** Gopher with TLS
Specification: (?)
Scheme: gophers
Port number: 70
File format: Gopher menu, Plain text
TLS: Yes
Request encoding: ASCII
Client implementations: hurl
Server implementations: (?)
 
Use of Gopher protocol with TLS. I think that it isn't very good, because
Gopher does not have relative links, and the menu format of Gopher does
not allow to specify the use of TLS or not, making it difficult to know if
you should use TLS with a specific server.
 
(Presumably, virtual hosting is possible by the use of SNI.)
 
**** Gemini
Specification: gemini://geminiprotocol.net/docs/specification.gmi
Scheme: gemini
Port number: 1965
File format: Gemini
TLS: Mandatory
Request encoding: UTF-8
Client implementations: (many)
Server implementations: (many)
 
A protocol and file format which has been worked on for a while, and some
minor changes to specifications have been made. (There have also been many
suggestions, most of which have been rejected. Scorpion (see below) does
some of the things that were suggested, but deliberately not all of them;
most of them have been rejected for Scorpion as well. If you merely want
non-TLS, you might also consider Spartan.)
 
The file format is limited: only normal paragraphs, block quotes, list
items (nesting is not possible), preformatted blocks, and links. The MIME
type specifies the character encoding (it recommends use of UTF-8, but
allows any character encoding, as long as it is consistent within one
file) and the language that it is written in (although mixed languages in
one file is not implemented).
 
Some Gemini files have SGR codes, which are not a part of the
specification. Some clients implement SGR codes, but some don't.
 
There is a response status line, which specifies a temporary or permanent
error and the specific error, or can be an input request (meaning that the
client must repeat the request with a user-specified query string), or a
redirect, or specifying requiring a client certificate.
 
The response header can specify the MIME type of the file, but the file
size is not specified.
 
It recommends use of TOFU (rather than certificate authorities) to verify
server certificates.
 
**** Mercury
Specification:
gemini://gemini.circumlunar.space/users/solderpunk/gemlog/the-mercury-protocol.gmi
Scheme: mercury
Port number: (Unspecified)
File format: Links and plain text only
TLS: None
Request encoding: UTF-8 (?)
Client implementations: (?)
Server implementations: (?)
 
The specification is incomplete, but it is intended to be like Gemini but
simpler. The only character encoding is UTF-8, most status codes are
removed, and the language code used in Gemini is also removed.
 
**** Spartan
Specification: gemini://spartan.mozz.us
Scheme: spartan
Port number: 300
File format: Gemini
TLS: None
Request encoding: ASCII
Client implementations: spartan_client.py
Server implementations: spartan_server.py
 
A protocol similar than Gemini, but without TLS, and has some other
differences as well. The number of status codes is much less.
 
The file format is the same as Gemini but with an additional link type
added for a link requesting input (similar than Gopher item type "7",
but different in some ways). (However, the same MIME type is used.)
 
The request is the host name and the file path, and the upload size
(zero if no upload). There is no way to distinguish the query string
from upload data, and no way to distinguish a GET request from an empty
upload. Although the user name and password are allowed in the URL, there
is no way to use the user name and password (why?).
 
**** Titan
Specification: (?)
Scheme: titan
Port number: 1965
File format: (N/A)
TLS: Mandatory
Request encoding: UTF-8
Client implementations: Lagrange
Server implementations: (?)
 
An upload protocol for use with Gemini. The security token, file size,
and MIME type are added on the end of the URL.
 
**** Scorpion
Specification: scorpion://zzo38computer.org/specification.txt
Scheme: scorpion, scorpions
Port number: 1517
File format: Scorpion
TLS: Optional
Request encoding: ASCII
Client implementations: astroget
Server implementations: scorpiond
 
Another protocol and file format. The protocol is similar than Gemini
in some ways, but there is a request method, although not request headers.
 
Scorpion has most status codes of Gemini, but omits the 11 status code,
and changes some of them (e.g. 20 now specifies the file size (which is
optional; you can use ? if it is unknown) as well as the MIME type, and
can optionally use ULFI instead of MIME). There are also some additional
features such as range requests, uploads, etc. The username and password
in the URL are allowed, unlike Gemini (client certificates are also
possible, if you are using TLS).
 
Scorpion has its own file format, which is a binary format, although some
of its features are vaguely like Gemini (others are different; e.g. mixed
character encodings are possible within one file, unlike Gemini). Like
Spartan and Gopher, there is a link type for requesting input from the
user, although the 10 status code from Gemini is also available.
 
The specification document also defines some X.509 extensions, which can be
used with other protocols as well. The "hashed:" scheme is also defined in
this document, which is also independent of the protocol.
 
**** Text
Specification: https://textprotocol.org/
Scheme: text
Port number: 1961 (plain), 1965 (TLS), 1968 (Noise)
File format: Plain text, with links as in Gemini
TLS: Optional
Request encoding: UTF-8 (NFC)
Client implementations: PUBLIC
Server implementations: PUBLICTEXT
 
Seems to use a similar file format than Mercury. There is only three
status codes (OK, redirect, and error) and there is no input requests.
It seems to use DNS Service Discovery though, and it supports Noise as
well as TLS, although all of the encryption types seem to use the same
scheme ("text"), so you cannot distinguish them by the scheme. The
specification also says that C escape codes can be used.
 
Note that the port number with TLS is the same as the port number of
Gemini. This is mostly not a problem; the only problem that I can see
with this is that the error code 40 means a temporary error in Gemini
but means any error in Text Protocol, which can potentially cause a
problem when it is treated as a Gemini-based proxy server.
 
**** SuperTXT
Specification: https://supertxt.net/00-intro.html
Scheme: (N/A)
Port number: 22
File format: SuperTXT
TLS: N/A; uses SSH instead
Client implementations: brsh, cats
Server implementations: Conserv
 
Another protocol and file format, which uses SSH for connection and
for authentication (and apparently a SSH key is required). There are
no URLs; you have to send the command including the file name, to the
SSH server. Instead of a response header, it uses exit codes and stderr
to indicate error conditions.
 
The file format supports tables, using a format similar than GNU long
options, e.g.   --lat=82 --lon=28 "Buried Treasure"
 
The file format has links on lines by themself (like Gemini), but a link
to another part of the same document can be included inside of the text
in a paragraph.
 
There is also a comment mechanism for users to provide feedback comments
to any file, by the use of the "ccmnt" command.
 
It is possible to run WebAssembly programs on SuperTXT, by the use of a
system interface called "WA-Nine", which is like a combination of WASI
with Plan9, and with a GUI layer called "Shiny Wire GUI".
 
**** Molerat
Specification: https://molerat.trinket.icu/
Scheme: molerat
Port number: 2693
File format: Molerat
TLS: Mandatory
Client implementations: Molehole (incomplete)
Server implementations: Moleserv (incomplete)
 
Another proposed design, more complicated than Gemini and others; it
has request and response headers like HTTP, and request methods like
HTTP, and the file format seems complicated to parse.
 
I think it is badly designed in many ways, as well as being complicated.
The documentation also has many mistakes in it. (I would recommend
avoiding Molerat.)
 
It has forms, which seems to be the only possible upload format, and
each field is required to be non-empty and cannot contain the sequence
<tab><crlf> which is used to delimit form fields, so files cannot be
uploaded that might contain such sequence.
 
It specifies the file size in the response headers, and apparently it
is mandatory to specify the file size, which can be a problem. And,
apparently the fragment part of the URL is supposed to be sent in the
request, even though that is for the client to use and not for the server.
 
Like Gemini, the language can be specified in the MIME type.
 
**** TerseNet
Specification: https://github.com/runvnc/tersenet/blob/master/README.md
Scheme: terse
Port number: (?)
File format: reStructuredText
TLS: (?)
Client implementations: (none)
Server implementations: (none)
 
This is not actually implemented yet, and is not designed much yet either,
so it cannot actually be used at this time.
 
It defines a maximum file size of 5K for text, and other media and
applications are not loaded automatically, but must be explicitly selected
by the user from the Media tab or Application tab, respectively.
 
There is three kind of browsers, those only for viewing text, those for
media applications, and those for extended media applications. Applications
can use WebAssembly, which can communicate with the TerseNet P2P backbone
and is able to use some cryptocurrencies (Bitcoin and Etherium).
 
TerseNet also uses P2P, and automatically adds everything to a full-text
search system, which is powered by the peers in the network.