Re: Compacting CNFS buffers

Hi Nigel,

Has anyone investigated the feasibility of compacting or compressing
the cnfs buffer files?

Some people use ZFS to compress CNFS buffers (cancelled articles are still present though).  I am not aware of a compaction feature like the one you want.

If you can find where an expired article is on disk and then find
the next article, you can just move it on disk and update the
pointers to the file. This could be a process that you just kick off
or, preferably, something that runs when innd isn't fully occupied
using spare cycles or something.

I understand your point; I can add it to the wish list.
FWIW, though technically this is not what you are asking for, some mechanisms may be used to mitigate your problems:

1. You are sent a bunch of articles but discover you've left some
binary newsgroups in your active file. You put this groups in your
expire list and delete rmgroup but you're left with a lot of empty
space, never to be used again unless the buffer recycles.

You may want to configure Cleanfeed to reject binaries (including in binary groups) so as not to store them and waste space.  Since a few weeks, NoCeM notices have also been sent for misplaced binaries (in non-binary groups).

2. You receive a bunch of googlegroup spam articles that are deleted
via NOCEM, however considering there are so many, that leaves a lot of
unused space.

Christoph Biedl implemented a new feature for INN 2.7.2 to store articles by their Path header field.  It is a new "path" option in storage.conf.  A typical use case is to store articles from a spammy site in a small CNFS buffer to avoid overall retention impacts.
There's also the delayer program (in the contrib directory before INN 2.7.2) that you can use to delay articles, and give cancel control articles and NoCeM messages time to arrive.  For instance, by having a frontend instance of innd receiving the articles from all your peers and another local instance of innd fed by your frontend with a delay except for cancels and NoCeM articles.  The CNFS buffers of that second instance will be spam free.
   https://www.eyrie.org/~eagle/software/inn/docs/delayer.html
--
Julien ÉLIE
« Aequum est ut cuius participauit lucrum, participet et damnun. »