Re: Improving trn: LISTGROUP with no arguments vs. ...?

[Please do not mail me a copy of your followup]

=?UTF-8?Q?Julien_=C3=89LIE?= <iulius@nom-de-mon-site.com.invalid> spake the secret code
<vgdu4f$16ru$1@news.trigofacile.com> thusly:

Hi Richard,
>

It's not the algorithm that is slow per se, but asking inn for all the
article numbers in a group with thousands of articles takes quite a
long time.

>
How much time does it take?

On my server (admittedly, nntp isn't as important to my ISP as it was
in the 90s, so they probably have it running on an old machine for the
last few customers that actually read news -- like me), it takes
multiple minutes to get all the articles for, say, comp.arch.

When I watch it with strace it's about one article per second and the
read buffer is so empty that it's literally reading just one response
line per call to read.

LISTGROUP returns the 1,365,681 articles in 2,6 seconds.  Not that slow...

Yes, it all varies, depending on your server, etc.

My long-term goal for trn is to move more towards asynchronous I/O
instead of synchronous I/O.  So I'm thinking LISTGROUP with low and
high used from your newsrc in order to fetch the likely never seen
before article numbers on the blocking path and then asynchronously
fetch whatever is needed in the background once you've entered the
group.  There's nothing wrong with doing a LISTGROUP for all the
remaining article numbers asynchronously in the background.

The current algorithm has you blocked waiting for every article number
in the group before it shows you anything about new articles.

I suspect that inn doesn't have any sort of in-memory
cache for this information and instead is directory scanning the spool
directory and reporting back the article numbers one by one.

>
INN has the available articles and its overview database, and depending
on the configuration of the server (nnrpdcheckart and groupexactcount
parameters), either directly returns the information from overview or
checks the article is still here.
The 2,6 seconds above include the check.

Well, I'm not certain why it's so slow on my ISP.  Watching it in
strace reveals that reading all these article numbers back for a
newsgroup with many articles is indeed the thing that takes a long
time.
--
"The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline>
            The Terminals Wiki <http://terminals-wiki.org>
     The Computer Graphics Museum <http://computergraphicsmuseum.org>
  Legalize Adulthood! (my blog) <http://legalizeadulthood.wordpress.com>