Re: Emigration from Usenet [was: Re: PTD was the most-respected of the AUE regulars ...]

On Fri, 26 Jul 2024, Computer Nerd Kev wrote:

Anton Shepelev <anton.txt@gmail.moc> wrote:

Computer Nerd Kev:
>

I must get around to trying to build a web forum to NNTP
gateway one day, although It'll probably send me crazy
trying to keep up with scraping different forum platform
page layouts.

>
In case you are building it for the heck of it, there is one
already:
                <https://news.novabbs.org>
>
If, on the other hand, your ambition is a bridge between
real web forums and NNTP, there have been working
implementations in englishforums.com (many years ago) and
the Microsoft forums (via NNTPBridge, not so many years
ago). Both were abandoned, partly because forums do not
agree with NNTP either ideologically (centralisation)
physically (different message format).

>
Perhaps I was too vague, I'm talking about something run
separately from the forum operators, like Gwene does with RSS.
I described my ambitions better last year in news.software.nntp
when I asked about any existing projects (without receiving any
relevent suggestions). See below.
>
Later I found this project that might help as a basis for the forum
scraping aspect:
https://github.com/mikwielgus/forum-dl
>
Though I don't think I could rely on that project to be maintained
and keep up with forum platform changes, since a recent comment
from the author is:
"Sadly, I haven't had much time to continue this project as I'm
currently very busy with another one."
So I'd probably have to maintain something like it myself, which
might not be worth the pain.
>
Message-ID: <651b4c54@news.ausics.net>
Subject: Client-Side Bridge to Web Forums?
Newsgroups: news.software.nntp
>
Web forums keep annoying me more and more with bloated interfaces,
so I'm using them less and less, yet there's less and less to read
on Usenet too. So lately I've been considering, not entirely
seriously, writing a program to scrape specific web forums and
generate a news spool with the content of the forum's latest posts
for me to read (either locally, or perhaps remotely via NNTP).
>
Has anyone done this before? I know there are various web forum
platforms that support NNTP server-side, but I'm talking about web
forums hosted by other people who I have no association or
influence with. Has anyone done something that's purely a
client-side implementation?
>
Simple Machines Forum and Discourse are prime targets for me, maybe
phpBB too. Most don't have RSS enabled, or the feed only shows the
start of new posts. The ideal would be a system supporting scrapers
for multiple forum platforms which can be easily extended.
>
Support for posting would be nice, but read-only access in a news
reader (Tin) would be better than nothing.
>
>

Read only sounds very simple. I usually scrape in python with the requests library and the beautiful soup library. A simple scraping loop could look like this (modify per web board of course):
for page in range(100, 150):
     html = requests.get("https://www.svt.se/text-tv/" + str(page))
     soup = BeautifulSoup(html.text, 'html.parser')
     div_bs4 = soup.find('div', {"class": "Content_screenreaderOnly__3Cnkp"})
     try:
         email_body += div_bs4.string + "\n"
     except AttributeError:
         None
So basically a range of pages, then loop over those pages, clean it up with beautifulsoup, and then generate messages for nntp. In my case, the code generates an email message, so I can read the content in my mail client instead of online.
What you could do then, is to generate nntp format files and dump them into leafnode, and then read locally with a nntp reader by connencting to leafnode.
That would, very broadly and high level, be how I would do it.
https://www.leafnode.org/ .