Re: Compacting CNFS buffers

Hi Nigel,

I'll hold out hope someone with more knowledge than I also sees the
issue and decides to look into compacting CNFS buffers.

It may as well be a new type of storage method, mixing the best of cnfs and timecaf.
As far as I understand, the use case is to have large compacted buffers without wrapping (articles do not expire but cancelled articles should not be kept).  It would correspond to timecaf except that a new CAF file is created when it is full instead of every 256 seconds.  Expiring CAF files just compacts them if articles have been cancelled, releasing disk space.
The feature may be implemented as an evolution of the current timecaf method with options to parameterize it in storage.conf (like cnfs has options).  For instance with a maxart and a maxtime option to specify the number of articles per CAF file (currently hard-coded to 262144) and the number of seconds before creating a new CAF file (currently hard-coded to 256 seconds but it may easily be a multiple of 256 seconds so as to keep the current file naming).  With maxtime set to 0, a new file is created when maxart is reached.
Naturally, though it is more work, a totally new storage method could also be created as timecaf is inherently linked to time and suffers from the limitation that you cannot store more than maxart articles received during maxtime seconds.  They will just be dropped until a new CAF file is created.  It is not what you expect from the storage method you're asking for.  And re-using CNFS buffers may be tricky (to find and refill holes, or to totally rewrite them - changing the storage tokens of all articles).
--
Julien ÉLIE
« Vinum bonum laetificat cor hominis. »