I'm noticing very slow XOVER behavior with some of the larger groups on my server. When I look at what nnrpd is doing it appears to be touching every article on the spool for that group during XOVER. I *thought* the data
provided by the XOVER command came from the overview database. What I see with truss is that it is accessing every article and while I'm watching never accesses the overview database.

Is this expected behavior from XOVER? This is a large group with 1.3 million articles, so I don't expect instant response, but surprised to see the nnrpd process doesn't appear to be using overview data from what I can see with truss:

<snipped millions of similar lines> access("/usr/local/news/spool/articles/rec/arts/tv/1279573",R_OK) = 0 (0x0) access("/usr/local/news/spool/articles/rec/arts/tv/1279574",R_OK) = 0 (0x0) access("/usr/local/news/spool/articles/rec/arts/tv/1279575",R_OK) = 0 (0x0) access("/usr/local/news/spool/articles/rec/arts/tv/1279576",R_OK) = 0 (0x0) access("/usr/local/news/spool/articles/rec/arts/tv/1279577",R_OK) = 0 (0x0) access("/usr/local/news/spool/articles/rec/arts/tv/1279578",R_OK) = 0 (0x0) access("/usr/local/news/spool/articles/rec/arts/tv/1279579",R_OK) = 0 (0x0) access("/usr/local/news/spool/articles/rec/arts/tv/1279580",R_OK) = 0 (0x0) access("/usr/local/news/spool/articles/alt/fan/rush-limbaugh/521666",R_OK) = 0 (0x0)
access("/usr/local/news/spool/articles/alt/politics/usa/127302",R_OK) = 0
(0x0) access("/usr/local/news/spool/articles/alt/fan/rush-limbaugh/521668",R_OK) = 0 (0x0) access("/usr/local/news/spool/articles/alt/fan/rush-limbaugh/521669",R_OK) = 0 (0x0) access("/usr/local/news/spool/articles/alt/fan/rush-limbaugh/521670",R_OK) = 0 (0x0)
access("/usr/local/news/spool/articles/rec/arts/tv/1321380",R_OK) = 0 (0x0) access("/usr/local/news/spool/articles/rec/arts/tv/1321381",R_OK) = 0 (0x0) access("/usr/local/news/spool/articles/alt/fan/rush-limbaugh/521671",R_OK) = 0 (0x0) access("/usr/local/news/spool/articles/alt/fan/rush-limbaugh/521672",R_OK) = 0 (0x0)
access("/usr/local/news/spool/articles/alt/politics/usa/127303",R_OK) = 0
(0x0)
access("/usr/local/news/spool/articles/rec/arts/tv/1321385",R_OK) = 0 (0x0) access("/usr/local/news/spool/articles/alt/survival/84115",R_OK) = 0 (0x0) access("/usr/local/news/spool/articles/rec/arts/tv/1321387",R_OK) = 0 (0x0) write(1,"\^W\^C\^C@\^X_\M-'\M-$>\M^M\M^Y"...,16413) = 16413 (0x401d) write(1,"\^W\^C\^C@\^X_\M-'\M-$>\M^M\M^Y"...,16413) = 16413 (0x401d) write(1,"\^W\^C\^C@\^X_\M-'\M-$>\M^M\M^Y"...,16413) = 16413 (0x401d) write(1,"\^W\^C\^C@\^X_\M-'\M-$>\M^M\M^Y"...,16413) = 16413 (0x401d) write(1,"\^W\^C\^C@\^X_\M-'\M-$>\M^M\M^Y"...,16413) = 16413 (0x401d) write(1,"\^W\^C\^C@\^X_\M-'\M-$>\M^M\M^Y"...,16413) = 16413 (0x401d) write(1,"\^W\^C\^C@\^X_\M-'\M-$>\M^M\M^Y"...,16413) = 16413 (0x401d) write(1,"\^W\^C\^C@\^X_\M-'\M-$>\M^M\M^Y"...,16413) = 16413 (0x401d) write(1,"\^W\^C\^C\^TT_\M-'\M-$>\M^M\M^Y"...,5209) = 5209 (0x1459) sigprocmask(SIG_SETMASK,{ SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIG SEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SI GTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR 1|SIGUSR2 },{ }) = 0 (0x0)
sigaction(SIGALRM,{ 0x80192ba00 SA_RESTART|SA_SIGINFO ss_t },{ SIG_DFL SA_RESTART ss_t }) = 0 (0x0)
sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0)
setitimer(0,{ 0.000000, 1800.000000 },{ 0.000000, 0.000000 }) = 0 (0x0)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

False alarm... Just realized at some point I set nnrpdcheckart to true in inn.conf. Not sure why I thought I needed that or was a good idea. :)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Jesse,

False alarm... Just realized at some point I set nnrpdcheckart to true in inn.conf. Not sure why I thought I needed that or was a good idea.:)

"true" is the default value of nnrpdcheckart, so you did nothing wrong!
In your case, it could indeed be set to "false".
Did it greatly improve the response time of the command?


Also, note that using virtualhost in readers.conf slows down a bit
(X)OVER responses as the Xref header field present in overview data will
be rewritten. Not sure it really takes much time, though!

--
Julien ÉLIE

« C'est comme chercher une aiguille dans du foin en bottes ! »
(Jolitorax)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Jesse,

"true" is the default value of nnrpdcheckart, so you did nothing wrong!
In your case, it could indeed be set to "false".
Did it greatly improve the response time of the command?

Setting it to false *dramatically* improved XOVER performance, hard to quantify but certainly orders of magnitude faster (few seconds for fetching 50,000 records vs. minutes).

Checking the existence of articles (nnrpdcheckart) is indeed really time-consuming for 50,000 records!

I happened to
notice the slow XOVER response because I was subscribing to the largest group in my newsreader to look at something else.

In such cases, the COMPRESS extension is also useful to speed up the transmission of overview data. Unfortunately, I'm not aware of any news
client other than flnews that implemented it :-/

--
Julien ÉLIE

« Piscis primum a capite foetet. »

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Jul 8, 2022 at 10:08:01 AM CDT, "Julien ÉLIE" in <ta9h8h$1eo15$[email protected]> wrote:

Hi Jesse,

False alarm... Just realized at some point I set nnrpdcheckart to true in
inn.conf. Not sure why I thought I needed that or was a good idea.:)

"true" is the default value of nnrpdcheckart, so you did nothing wrong!
In your case, it could indeed be set to "false".
Did it greatly improve the response time of the command?

Ah, my first thought was I turned it on when I was having trouble with expireover, but this makes sense.

Also, note that using virtualhost in readers.conf slows down a bit
(X)OVER responses as the Xref header field present in overview data will
be rewritten. Not sure it really takes much time, though!

Setting it to false *dramatically* improved XOVER performance, hard to
quantify but certainly orders of magnitude faster (few seconds for fetching 50,000 records vs. minutes).

I did read that note about a performance hit using virtualhost, but I can't
say I notice any difference after configuring virtualhost. I happened to
notice the slow XOVER response because I was subscribing to the largest group in my newsreader to look at something else.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Jul 8, 2022 at 10:59:52 AM CDT, "Julien ÉLIE" in <ta9k9o$1eqj6$[email protected]> wrote:

Hi Jesse,

"true" is the default value of nnrpdcheckart, so you did nothing wrong!
In your case, it could indeed be set to "false".
Did it greatly improve the response time of the command?

Setting it to false *dramatically* improved XOVER performance, hard to
quantify but certainly orders of magnitude faster (few seconds for fetching >> 50,000 records vs. minutes).

Checking the existence of articles (nnrpdcheckart) is indeed really time-consuming for 50,000 records!

I happened to
notice the slow XOVER response because I was subscribing to the largest group
in my newsreader to look at something else.

In such cases, the COMPRESS extension is also useful to speed up the transmission of overview data. Unfortunately, I'm not aware of any news client other than flnews that implemented it :-/

I bet there are some newsreaders more focused around downloading binaries that implement it. I sent a question to the Usenapp Support team asking if their client does. Usenapp is relatively new and has zero documentation at the moment.

The client seems well rounded for newsreading and binary downloads and from what I've observed, more conformant to standards than similar predecessors (Unison, for example). I prefer it for reading and composing over Thunderbird.
It has more of the 'old school' newsreader features that have disappeared
from many clients (killfiles, for example).

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)