711 Spooky Mart <[email protected]> writes:

Hello NNTP gurus. I want to host a binary local newsgroup. I want to discourage spammers from trying to push up irrelevant big binaries,
vids, pr0n, etc. The purpose of the binary group is for uploading PDF
and ZIP/XZ/7Z files < 4MB.

I am seeking ideas on different ways to throttle INN connections.

[1] Firstly I want to throttle all client connections to max 64kB/s.

[2] Secondly I want to maintain a timing throttle to all client
connections so that the server does not respond for several seconds
since the last connection from same client / IP, so that there is always
a short delay between the completion of one client command or connect to
the next. I really don't want to differentiate between a client command
and a client connection session--there should be a delay between every command operation during a connect session and a delay between end of a session and the next connection from the same client.

[3] Thirdly, if multiple client connections are detected from the same
IP, I want to throttle bandwidth down to 32kB/s per client and cycle
their access by alternating the delays. This is mainly aimed at
preventing scripted hog connections from Tor exit nodes, although I
don't wish to completely block Tor because anonymity will be one of the
group topics.

I would use iptables (or nftables, which is the new thing) for [1] and [3] these. It's designed to do exactly this kind of work.

I'm not sure off-hand how to do [2] without modifying the source code. I
don't think we have a delay mechanism like that built in. There's support
for doing exponential backoff for posts (see under Posting in the inn.conf manual page), but that's not quite the same thing.

[4] Fourthly, I want to offer the newsgroup to the big world network if anyone wants to sync it, with all the binary mimetypes stripped, so the outgoing feed would only contain body text without the binary content.
If clients want to download the binaries they would need to use my local
INN server.

I'm afraid INN doesn't support this or anything like it. It goes to a lot
of effort to have one and only one article corresponding to a given
message ID and serve exactly the same article through whatever path. It
also has no understanding at all of MIME structure, so doesn't have even
the infrastructure to remove specific parts of a message.

There are hooks to invoke arbitrary Perl or Python code when an article is posted, so you could parse the article there and post a copy (with a
different message ID) to a different group that you then feed to other
servers, but that's as close as I think INN would let you get.

--
Russ Allbery ([email protected]) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hello NNTP gurus. I want to host a binary local newsgroup. I want to
discourage spammers from trying to push up irrelevant big binaries,
vids, pr0n, etc. The purpose of the binary group is for uploading PDF
and ZIP/XZ/7Z files < 4MB.

I am seeking ideas on different ways to throttle INN connections.

[1] Firstly I want to throttle all client connections to max 64kB/s.

[2] Secondly I want to maintain a timing throttle to all client
connections so that the server does not respond for several seconds
since the last connection from same client / IP, so that there is always
a short delay between the completion of one client command or connect to
the next. I really don't want to differentiate between a client command
and a client connection session--there should be a delay between every
command operation during a connect session and a delay between end of a
session and the next connection from the same client.

[3] Thirdly, if multiple client connections are detected from the same
IP, I want to throttle bandwidth down to 32kB/s per client and cycle
their access by alternating the delays. This is mainly aimed at
preventing scripted hog connections from Tor exit nodes, although I
don't wish to completely block Tor because anonymity will be one of the
group topics.

[4] Fourthly, I want to offer the newsgroup to the big world network if
anyone wants to sync it, with all the binary mimetypes stripped, so the outgoing feed would only contain body text without the binary content.
If clients want to download the binaries they would need to use my local
INN server.

I think this would make it reasonably frustrating for spammers and
binary hogs, and not be too onerous for legitimate participants.

Please propose your thoughts on how to proceed with these strange
requirements.

--
──┏━━━━┓──┏━━┓───┏━━┓── ┌────────────────────────┐ ┌────────┐
──┗━━┓─┃──┗┓─┃───┗┓─┃── │ Spooky Mart [chan] 711 │ │ always │
─────┃─┃──┏┛─┗┓──┏┛─┗┓─ │ https://bitmessage.org │ │ open │
─────┗━┛──┗━━━┛──┗━━━┛─ └────────────────────────┘ └────────┘

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

711 Spooky Mart <[email protected]> wrote:

Hello NNTP gurus. I want to host a binary local newsgroup. I want to discourage spammers from trying to push up irrelevant big binaries,
vids, pr0n, etc. The purpose of the binary group is for uploading PDF
and ZIP/XZ/7Z files < 4MB.

Give up now, you don't know what you are doing and obivously don't know INN
and how the nntp protocol works.

You don't UPLOAD to an INN server, people use an NNTP client to publish articles. These articles come from a newsreader.

I kind of find it hard to beleive whatever you are trying to accomplish has
an audience. Who are these pdf's coming from? Is this something common now?

Are they going to just know about this service or are you providing a link
to the server? Do they know they have to have a newsreader?

Binary articles are not normally part of the scene. You can't just UPLOAD a binary file to a server without converting it to plain text. Do a search for uuencode/uudecode or txt2bin/bin2txt. Some newsreaders can do this on the
fly, small ones anyway. Larger ones need to be broken into several parts.

Are your users aware of this?

I think you are better off looking at setting up an ftp server with the anonymous stuff turned on. Bandwidth control would probably be easier using
a Pi with some kind of firewall software on it.

[1] Firstly I want to throttle all client connections to max 64kB/s.

Why? Using your moms internet from your room?

Please propose your thoughts on how to proceed with these strange requirements.

Like I said, give up now and do more research.

-bruce
[email protected]

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

This is a multi-part message in MIME format.
On 10/24/21 9:39 AM, [email protected] wrote:

711 Spooky Mart <[email protected]> wrote:

Hello NNTP gurus. I want to host a binary local newsgroup. I want to
discourage spammers from trying to push up irrelevant big binaries,
vids, pr0n, etc. The purpose of the binary group is for uploading PDF
and ZIP/XZ/7Z files < 4MB.

Give up now, you don't know what you are doing and obivously don't know INN and how the nntp protocol works.

Oh, really? I found a friend like Job had. I will not denounce my own
integrity in this matter.

You don't UPLOAD to an INN server, people use an NNTP client to publish articles. These articles come from a newsreader.

Yes, Captain Obvious! How do you think I posted the opening to this thread?

The NNTP client formats the attachment binaries in MIME format with a
mime boundary marker, usually as base64, which comes after the ASCII or
UTF-8 headers and text body of the NNTP formatted article. Then the
client uploads the formatted message with the POST command and a final
line with a period and linefeed. The reader downloads the message, grabs
the MIME data, and converts attachments back into binary files. I do
this from the command line without a news reader using openssl s_client requests. Here's an example.

$ openssl s_client -ign_eof -connect news.aioe.org:563

I can read articles this way, and cat text files up with header data to
post new articles, which is how this response to you is posted. You can
examine the headers to see how I formatted it and added an attachment.

This is what I meant by 'uploading' a PDF file. People 'upload' split
binaries to paid Usenet providers, and 'download' them with NZB Get or
similar clients. I've known about this for over two decades. I just
never bothered to run my own news server, since I never saw the need of
it until now, for this documentation project. I already know how to set
up INN. I just don't know how to configure it for throttling and
stripping of MIME data for a outgoing feed.

I can attach PostScript since it allows pure text output, but fonts and
MathJax cannot be embedded to ensure proper document rendering on other machines.

I am very comfortable with manually querying NNTP servers with openssl s_client. The knowledge I am lacking here is INN configuration knowledge
to do the things outlined in the enumerated points.

I kind of find it hard to beleive whatever you are trying to accomplish has an audience. Who are these pdf's coming from? Is this something common now?

YGBSM. Every graphical newsreader I know of has a 'Attachment' button
with a paperclip in the composer.

Are they going to just know about this service or are you providing a link
to the server? Do they know they have to have a newsreader?

Of course they are going to know about the service. It's a workgroup
setup for a documentation project. PDF is necessary because of LATEX
maths, and math fonts must be embedded for proper document formatting. I
want the text parts of the feed available to the big world since others
might benefit from it or contribute useful information in the comment
threads, even if they are not drafting the documents.

Binary articles are not normally part of the scene. You can't just UPLOAD a binary file to a server without converting it to plain text. Do a search for uuencode/uudecode or txt2bin/bin2txt. Some newsreaders can do this on the fly, small ones anyway. Larger ones need to be broken into several parts.

I already know this. Most NNTP servers appear to reject base64 / binary,
and don't even carry binary groups. I have tested a few servers before
posting this query thread, and they all rejected the test messages
because they are configured to reject binaries. The servers scan the base64 encoding for binary content, or something to that effect.

Are your users aware of this?

I think you are better off looking at setting up an ftp server with the anonymous stuff turned on. Bandwidth control would probably be easier using
a Pi with some kind of firewall software on it.

No, I am better off with a threaded newsgroup since building the
documentation will be a collaborative effort and switching over to FTP
for every draft exchange is too much friction. Usenet threading is the
simplest and most productive way to proceed. I already know my own work
domain very well, and I am just trying to set up a tool chain to
eliminate friction, and make the work and the process publicly
available. Isn't this the original intent behind distributed big world networks?

[1] Firstly I want to throttle all client connections to max 64kB/s.

Why? Using your moms internet from your room?

I moved out of my mother's house on my 18th birthday over 3 decades ago.
I moved 2000 miles away from my mother. Today I live about 1200 miles
from mom. Her house didn't have a basement, either, so don't bother
going there.

Just because I didn't come in here with a typical nerd attitude of
superior rationale wearing my genius attitude on my shirt sleeve, it
does not follow that I am some dumb kid in his mom's basement. My
questions are cogent, well-worded questions and show the spirit of
ingenuity and of trying to solve a problem to achieve a good workflow
for a project for which I am devoting unpaid time, to benefit other
people's research, when I could be out right now making extra money.

If my objectives can't be achieved via the INN software, that is one
thing. But it is not a cause to insult and belittle me.

Please propose your thoughts on how to proceed with these strange
requirements.

Like I said, give up now and do more research.

No, I am not giving up. My questions here _are_ research. Research
consists of finding answers to questions until the answers satisfy
requirements or solve a problem, or generate new, interesting problems
to research.

-- ──┏━━━━┓──┏━━┓───┏━━┓── ┌─────────�
�”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â” ┌────────┐
──┗━━┓─┃──┗┓─┃───┗┓─┃── │ Spooky Mart [chan] 711 │ │ always │
─────┃─┃──┏┛─┗┓──┏┛─┗┓─ │ https://bitmessage.org │ │ open │
─────┗━┛──┗━━━┛──┗━━━┛─ └────────â�
�€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”˜ └────────┘

_ _ _____ ________ __ _ _ _ ____ _ _______ ____
| | | |/ _ \ \ / / _ \ \ / / | | | | / \ / ___| |/ / ____| _ \
| |_| | | | \ \ /\ / /| | | \ V / | |_| | / _ \| | | ' /| _| | |_) |
| _ | |_| |\ V V / | |_| || | _ | _ |/ ___ \ |___| . \| |___| _ <
|_| |_|\___/ \_/\_/ |____/ |_| ( ) |_| |_/_/ \_\____|_|\_\_____|_| \_\
|/

⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣀⣀⣀⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣠⣶⠟⠛⠛⠛⠛⠛⣛⣻⣿⣿⣿⣿⣿⣟⣛⣛⣛⠛⠒⠲⠶⠦⣤⣤⣤⣀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣼⠏⠁⠀⠀⢀⣤⠶⣛⣩⣥⠤⠤⠤⠤⢤⣤⣤⣭⣭⣉⣉⣛⣛⣻⣭⣥⠬�⠛⢶⣄⡀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⢠⣾⠃⠀⠀⣠⡶⢋⡵⢛⡩⠵⠒⠒⠒⠒⠢⡀⠀⠀⠀⠀⠀⢀⣠⠤⠤⠤⢤⣄⠀⠀⠀⠉⠻⣆⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⢀⣿⠃⠀⠀⠘⢁⡴⢋⣴⢿⠒⠈⠉⣏⠉⠐⠒⡾⣄⠀⠀⠀⠀⠀⡠⠀⠀⢀⣀⣈⣙⣆⡀⠀⠀⢹⡆⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⣠⣾⠃⠀⠀⠀⠀⠀⢀⠟⣁⠀⠁⢀⣤⣦⣤⡀⠘⠀⢈⣷⡄⠀⠀⠀⣇⠖⠉⠙⠅⠀⠀⠉⠉⠑⢦⡈⣷⡀⠀⠀⠀⠀
⠀⠀⠀⠀⢠⣾⢿⣧⠤⠤⠤⠄⠀⠖⣿⠀⠃⠀⠀⣿⣿⣿⣿⡗⠀⠐⠁⢸⡇⠀⣀⣰⠉⠠⠀⠀⣰⣶⣷⣶⠀⠀⠀⢱⡈⢻⣦⠀⠀⠀
⠀⠀⠀⣠⡿⣱⠋⢀⣴⠶⠚⠻⢶⣤⡘⢧⣄⠆⠂⠀⡉⠉⣉⣀⣀⠉⣠⡟⠁⠀⠉⢻⣆⠀⠀⠀⠘⠛⠟⠛⠀⠀⢈⡿�⢢⢹⡇⠀⠀
⠀⠀⢠⣿⠁⡇⢠⣿⠁⠀⢰⣦⡀⠉⠉⠀⠈⠙⠲⠾⠾⠶⠶⠶⠚⠋⠉⠀⠀⠀⠀⢸⣯⡑⠢⢤⣀⣂⣀⣨⠤⠒⠛⠃⠘⡆⡇⡧⠀⠀
⠀⠀⢸⣿⠀⡇⢸⡇⢠⣴⣾⠋⠛⢷⣦⣀⠀⠀⠀⠠⠤⠤⠴⢠⠶⠒⠀⠀⠀⠀⠀⠀⠉⢿⣦⡀⠀⠀⠀⠀⢸⣷⠀⠀⡼⢡⢣⡇⠀⠀
⠀⠀⠀⢿⡇⣧⠘⠿⠀⠀⠸⣧⡀⠀⠈⢻⡿⢶⣦⣄⡀⠀⠀⠸⣆⠐⠟⠻⠷⠀⠀⠀⢀⣾⠛⠃⠑⠤⠀⢀⣼⣿⡇⢀⠤⢂⣾⠃⠀⠀
⠀⠀⠀⠈⢻⣌⠑⠦⠀⠀⠀⢿⣿⣷⣤⣸⣷⡀⠀⠈⠙⠻⢿⣶⣤⣄⣀⡀⠀⠀⠙⠿⠟⠁⠀⠀⢀⣠⡴⣿⠉⣿⣿⠀⠀⣼⠁⠀⠀⠀
⠀⠀⠀⠀⠀⠙⣷⡀⠀⠀⠀⢸⣿⣿⣿⣿⣿⣿⣶⣤⣀⣀⣼⠁⠀⠈⠉⠙⣿⠛⠛⠻⢿⠿⠛⠛⢻⡇⠀⢸⡀⣹⣿⠀⠀⡏⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠈⢿⡀⠀⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣶⣤⣤⣄⣀⣿⣄⣀⣀⣸⣄⣀⣠⣴⣿⣶⣿⣿⣿⣿⡇⠀⡇⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠈⢷⡄⠀⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡇⠀⡇⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠈⢿⣦⠀⠘⣿⠛⢿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡇⠀⣷⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢷⣄⠘⢷⡀⠘⡟⠿⢿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡇⠀⣿⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠹⣧⡀⠻⣾⡃⠀⠀⠈⠙⢿⡿⢿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡇⠀⣿⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⢿⣄⠈⠻⣦⡀⠀⠀⡼⠀⠀⠈⠙⠻⣿⠿⠿⠿⢿⣿⣿⣿⣿⣿⣿⣿⢿⡿⣹⠇⠀⣿⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠹⣷⣄⠈⠛⠷⣼⣇⡀⠀⠀⠀⠀⣿⠀⠀⠀⢸⡇⠀⠀⡿⠀⢸⠇⣘⣧⠟⠀⢀⡿⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠛⢷⣄⡀⠀⠙⠻⠷⠶⣶⣾⣿⣤⣀⣠⣿⣄⣀⣴⠷⠶⠿⠿⠟⠋⠀⢀⣾⠃⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠛⠿⣶⣤⣤⣀⣀⡀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣀⣀⣀⣀⣤⡤⠞⠁⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠉⠙⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠋⠉⠀⠀⠀⠀⠀⠀⠀⠀

__ __ ___ _ _ __ _ ____ ___ ____ ____
| \/ |_ _| | | |/ / / \ | _ \_ _/ ___| / __ \
| |\/| || || | | ' / / _ \ | |_) | | | _ / / _` |
| | | || || |___| . \ / ___ \ | __/| | |_| | | | (_| |
|_| |_|___|_____|_|\_\ /_/ \_\ |_| |___\____| \ \__,_|
\____/
_____ _ _ ____ ____ ___ ___ _ ____ __ __ __ _ ____ _____ |___ / / | / ___|| _ \ / _ \ / _ \| |/ /\ \ / / | \/ | / \ | _ \_ _|
/ /| | | \___ \| |_) | | | | | | | ' / \ V / | |\/| | / _ \ | |_) || |
/ / | | | ___) | __/| |_| | |_| | . \ | | | | | |/ ___ \| _ < | |
/_/ |_|_| |____/|_| \___/ \___/|_|\_\ |_| |_| |_/_/ \_\_| \_\|_|


+--------------------------------------------------------------+
| |
| [chan] 711 |
| https://bitmessage.org |
| __ |
| ____ ________ ,',.`. |
| \`''-.`-._..--...-''' ```--':_ ) ) |
| `-.._` ' -.. ' / |
| ,'`..__..'' -. _ `._ \ |
| ('';` _ ,'' .-' ,' : |
| `-._ `*/ , ' . | |
| _.:._ `-'`-' ; \ ,' ; |
| .':::::'` ,' \,' : ; / |
| `-..__ ,'/ | ,' ,' |
| ``---;'` \ ` ;.____..-'`. ,'\ |
| / / \: : : (\ `\ |
| ,' .' \ : ;' / ) ) |
| /,_,.;::. `. \ / ,',',_(:::. |
| `. `. ,' ;' |
| /,_,'::. `-'`':SSt:. |
| |
+--------------------------------------------------------------+


___ _____ _ ____ ____ ___ _____ _______ _____ _____ _
|_ _|_ _( ) ___| / ___| / _ \ / _ \ \ / / ____| ____|_ _| |
| | | | |/\___ \ \___ \| | | | | | \ \ /\ / /| _| | _| | | | |
| | | | ___) | ___) | |_| | |_| |\ V V / | |___| |___ | | |_|
|___| |_| |____/ |____/ \___/ \___/ \_/\_/ |_____|_____| |_| (_)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 10/24/21 2:37 PM, Aioe wrote:

imho it isn't a good strategy
you have to set some thresholds in the system resources used by each IP,
if one exceeds them you ban it for a certain time
it is not difficult to write a script that reads the logs, builds a
database of the resources used by each client then bans and unbans it

What I'm gathering then, is I got some hacking to do. Maybe I'll post a followup explaining how it all turns out so we can have a cheat sheet.

Thanks all for your advice.

--
──┏━━━━┓──┏━━┓───┏━━┓── ┌────────────────────────┐ ┌────────┐
──┗━━┓─┃──┗┓─┃───┗┓─┃── │ Spooky Mart [chan] 711 │ │ always │
─────┃─┃──┏┛─┗┓──┏┛─┗┓─ │ https://bitmessage.org │ │ open │
─────┗━┛──┗━━━┛──┗━━━┛─ └────────────────────────┘ └────────┘

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Il 23/10/21 21:30, 711 Spooky Mart ha scritto:

Hello NNTP gurus. I want to host a binary local newsgroup. I want to discourage spammers from trying to push up irrelevant big binaries,
vids, pr0n, etc. The purpose of the binary group is for uploading PDF
and ZIP/XZ/7Z files < 4MB.

I am seeking ideas on different ways to throttle INN connections.

[1] Firstly I want to throttle all client connections to max 64kB/s.

if you need a limit for all users, iptables does this
if you need a limit for each user, max_rate does this (see man readers.con)

[2] Secondly I want to maintain a timing throttle to all client
connections so that the server does not respond for several seconds
since the last connection from same client / IP, so that there is always
a short delay between the completion of one client command or connect to
the next.

imho it isn't a good strategy
you have to set some thresholds in the system resources used by each IP,
if one exceeds them you ban it for a certain time
it is not difficult to write a script that reads the logs, builds a
database of the resources used by each client then bans and unbans it

[3] Thirdly, if multiple client connections are detected from the same
IP, I want to throttle bandwidth down to 32kB/s per client and cycle
their access by alternating the delays.

iptables does this

[4] Fourthly, I want to offer the newsgroup to the big world network if anyone wants to sync it, with all the binary mimetypes stripped, so the outgoing feed would only contain body text without the binary content.

you've to patch idd to do this

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Aioe <[email protected]> writes:

Il 23/10/21 21:30, 711 Spooky Mart ha scritto:

Hello NNTP gurus. I want to host a binary local newsgroup. I want to
discourage spammers from trying to push up irrelevant big binaries,
vids, pr0n, etc. The purpose of the binary group is for uploading PDF
and ZIP/XZ/7Z files < 4MB.
I am seeking ideas on different ways to throttle INN connections.
[1] Firstly I want to throttle all client connections to max 64kB/s.

if you need a limit for all users, iptables does this
if you need a limit for each user, max_rate does this (see man readers.con)

Good call on max_rate; I'd forgotten about that.

[4] Fourthly, I want to offer the newsgroup to the big world network if
anyone wants to sync it, with all the binary mimetypes stripped, so the
outgoing feed would only contain body text without the binary content.

you've to patch idd to do this

The easiest place to do so would probably be to patch innxmit (or innfeed,
but for this sort of specialty use, the much simpler innxmit is probably better) to strip the binary attachments out of the outgoing feed.

The hard part is going to be finding a good MIME parser in C. I'm sure
there are tons of them out there in all the different mail programs
written in C, but whether any of them is reusable for your purpose is
another question.

Or, hm, you could probably also do something fancy with a batch feed where
you rewrite the batch in some other program more suitable to doing
parsing, like Python.

--
Russ Allbery ([email protected]) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 10/23/21 12:30 PM, 711 Spooky Mart wrote:

Hello NNTP gurus. I want to host a binary local newsgroup. I want to discourage spammers from trying to push up irrelevant big binaries,
vids, pr0n, etc. The purpose of the binary group is for uploading PDF
and ZIP/XZ/7Z files < 4MB.

I am seeking ideas on different ways to throttle INN connections.

[1] Firstly I want to throttle all client connections to max 64kB/s.

You can use tc to limit outgoing bandwidth per IP. inn has it's
exponential backoff parameters for posting, but it is otherwise
difficult to limit incoming bandwidth.

https://duckduckgo.com/?t=ffsb&q=use+tc+to+limit+bandwidth&ia=web

You would need to tweak the inn source code to put a delay before
commands--I can't imagine why you would want to. To have two different versions of an article, you would need two different spools. You would
need to write your own news forwarder to strip the attachment and send
it to your second spool, but it wouldn't be difficult using Perl Net::NNTP.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sun, 24 Oct 2021 15:09:44 -0500, 711 Spooky Mart <[email protected]> wrote:

On 10/24/21 2:37 PM, Aioe wrote:

imho it isn't a good strategy
you have to set some thresholds in the system resources used by each IP,
if one exceeds them you ban it for a certain time
it is not difficult to write a script that reads the logs, builds a
database of the resources used by each client then bans and unbans it

What I'm gathering then, is I got some hacking to do. Maybe I'll post a followup explaining how it all turns out so we can have a cheat sheet.

Actually, there is fairly popular "fail2ban" program (included in many GNU/Linux distros, if you happen to use that), which does exactly that sort
of job (parses logs and firewalls IPs unless they are in whitelist CIDRs,
and later unbans them etc.), so it could be reused and only a matching rule written for it instead of doing it from scratch.

Perhaps it would be good enough for you use case.

--
Opinions above are GNU-copylefted.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Spooky Mart,

[1] Firstly I want to throttle all client connections to max 64kB/s.

[...]

[4] Fourthly, I want to offer the newsgroup to the big world network if anyone wants to sync it, with all the binary mimetypes stripped, so the outgoing feed would only contain body text without the binary content.
If clients want to download the binaries they would need to use my local
INN server.

What would prevent me from running a program like pullnews or suck at
64kB/s and then provide the big world network the same full articles as
what can be found in your news server? (and feed them to whom is interested)

--
Julien ÉLIE

« – Ils s'arrêtaient tous les jours à 5 heures, pour boire de l'eau
chaude…
– Je prendrai un nuage de lait, je vous prie. » (Astérix)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Spooky Mart,

It's a workgroup
setup for a documentation project. PDF is necessary because of LATEX
maths, and math fonts must be embedded for proper document formatting.

Oh, interesting.
I ran my first INN when I was student, and used the Perl posting hook (mentioned by Russ - https://www.eyrie.org/~eagle/software/inn/docs/hook-perl.html) to
convert plain text articles to HTML on the fly with embedded LaTeX images. Basically, the hook transformed things like $\frac{1}{2}$ to an <img src="mimetex.cgi?..."> call.

Just an idea if you have such a use case in a local newsgroup.
Students could exchange articles with embedded formulae very easily this
way.


Of course if your need is only exchanging existing PDF, it doesn't
answer it.

--
Julien ÉLIE

« – Ils s'arrêtaient tous les jours à 5 heures, pour boire de l'eau
chaude…
– Je prendrai un nuage de lait, je vous prie. » (Astérix)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)