Michael MD @ SPRACI

9 months ago

nntp/nnrp connector

having some fun playing with it

http://spraci.info/profile/sci.space.news
http://spraci.info/profile/rec.audio.pro

needs work though ...

those two are usenet groups

(nntp server is just a local inn2 install and is not peered with anything so posts don't to go out to usenet ..... but they do make it to inn .. and I can see them in a newsreader)

this connector will only forward public posts for obvious reasons

9 comments show more

Mr. X

9 months ago

Nice...

Michael MD

9 months ago

its still in its infancy .. and pretty hacky

I need to work out good places to store data to associate items with the news server's article numbers
to make it more efficient and prevent duplicate posting or fetching items it already has.

I'm trying to only use the Friendica database and not any custom stuff so that any Friendica site could use it.

any suggestions?

I also need to decide what to do about text formatting - eq quoting, etc
such as a good way to convert between usenet-style > quoting and bbcode
(maybe I need to look at the markdown code?)

also still trying to work out all those crazy config files for the news server I'm testing it with (inn)

Mr. X

9 months ago

If it's a generic message id - you can store it directly as uri/parent-uri.

Or use item['extid'] to store the ID you get from USENET. Think "external-id".

If that's not sufficient we also have an "item_id" table that was designed for this sort of purpose (external service IDs that we can cross reference from multiple providers to find uniqueness) but this is not yet being used - and it has a bug in the structure that needs to get sorted out.

(Add an auto increment id and make it the primary key, as we can have multiple 'iid' (item['id']) for an article and currently this is setup as unique primary key).

For formatting, you can create a quoting filter but basically just convert <>&' to html entities and store it and she'll be right.

Mr. X

9 months ago

The item_id table is fixed in the latest master.

id - auto increment primary key

iid - the item['id'] to associate this with

uid - the uid of that item

sid - whatever unique id this service uses

service - optional but recommended, the name of the service

We use item['extid'] to store facebook post ids, so if you share a post to usenet and facebook it will cause a conflict. The item_id table was meant to provide a generic place to store third party article-ids if we posted to more than one service which insisted on using their own id.

So when you send a post from Friendica, let Friendica create a uri/parent-uri. Whatever message-id is returned by the submission service for that post should go into item_id['sid'].

Then when you're looking at new messages, check for duplicates in both item['uri'] and item_id['sid'], but if it hasn't been seen before and is globally unique you can just store it as item['uri'].

It sounds a bit confusing at first but it isn't really - you'll figure it out.

Michael MD

9 months ago

I already have the item_id table there
(been checking git every now and them and manually doing database updates)

still a little unsure about what settings should be there
Currently it has to be a public forum
The forum page effectively becomes a view of and gateway for the newsgroup.
(so only one newsgroup can be connected to a page)
Friendica users can subscribe to the page just like any forum.

there are these settings for users:
newsgroup
news server (can be disallowed by admin)
news server port (can be disallowed by admin)
news server user/pass (if user news server is allowed by admin)
whether to allow posting

Only public "wall" posts on the page are forwarded.

Michael MD

9 months ago

having trouble with my local test news server swiping posts on local (non-usenet) groups.
Still need to get my head around those inn config files.

Mr. X

9 months ago

I should have a look - there's no reason why we couldn't have a USENET connector which let anybody subscribe to arbitrary newsgroups as they do email and RSS feeds. We shouldn't need a community page to hold an individual newsgroup.

Michael MD

9 months ago

I was thinking about it being also for people who want to use a (non-usenet) newsgroup as part of the federation/app/subscription mix for their public forum as well as for people wanting to subscribe to a usenet group via Friendica.

Subscribing to arbitrary newsgroups could be possible too if there is a hook for detecting that what was entered is a newsgroup
(whether this is allowed on a site probably should be a choice for the site's admin in the context of what news server they are using. If it is not their own news server there may also be restrictions on what they can do)

I think subscribing to a usenet newsgroup or letting people use a newsreader to read their forum are just two of quite a few possible use cases out there.
It might also interest people running existing forums on other platforms (some of which might have existing connectors for nntp/nnrp), bbs sysops, etc
Like connectivity stuff in general there area probably also use-cases I have not yet thought of.

The newsgroup does not have to be a usenet newsgroup,... show more

Mr. X

9 months ago

As a plugin yes - we would probably need a hook to mod/follow that allows a plugin to say "I recognise that address - I'll handle it.". We currently don't have that.

I think that's the only core change needed. We already have hooks to poll and deliver to arbitrary networks.

That would be pretty easy to add - but if that turned out for some reason not to be possible one could always have a list of subscribed newsgroups in their connector settings.

If you've got github, you could have a branch there that others can look at.

Otherwise a tar file or diffs would give us something to work with.

What library are you using for access? I know that IMAP/c-client (which we use for email) will do it, but it's not the best NNTP library. NNTPConnection_class seems to be much better.

Michael MD

Connect

511 Contacts

View Contacts