1. Forum
  2. Usenet
  3. NEWS.SOFTWARE.NNTP

  • INN/inews behavior on \r\n

    From mg@21:1/5 to All on Thu Feb 25 02:14:19 2021
    I'm developing some python scripts to implement an NNTP/WordPress
    gateway, and I ran into unexpected behavior (and a less than helpful
    error message) from inews. INN is from the inn2-2.6.0-2 package through
    apt on linux mint.

    Python's email.message module creates messages that (by default? I
    wasn't able to find a setting for this) terminate lines with \r\n
    (CRLF). Feeding those messages to 'inews -h -O' produces the error
    message (from memory) "No space after colon in <the first line of the
    article body>".

    Digging into the source code (latest version, online), it seems that
    inews is treating the \r character as folding whitespace? Converting
    the \r\n to \n in the message solved the problem, but:

    Is this really the correct behavior? I'm looking at RFCs 5322 and 5536
    here...

    Also, a more specific/helpfull error message would have saved me
    considerable debugging time. INN devs, is this possible?

    Thanks!

    -mg

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From John Levine@21:1/5 to [email protected] on Thu Feb 25 04:54:17 2021
    In article <s1715o$e99$[email protected]>, mg <[email protected]> wrote:
    I'm developing some python scripts to implement an NNTP/WordPress
    gateway, and I ran into unexpected behavior (and a less than helpful
    error message) from inews. INN is from the inn2-2.6.0-2 package through
    apt on linux mint.

    Python's email.message module creates messages that (by default? I
    wasn't able to find a setting for this) terminate lines with \r\n
    (CRLF). Feeding those messages to 'inews -h -O' produces the error
    message (from memory) "No space after colon in <the first line of the
    article body>".

    inews is not nntp just as sendmail is not smtp. Both run inside your Unix system and use the Unix \n line terminator. If you were opening a TCP connection to port 119 and issuing nntp commands, then you would use \r\n
    as RFC 3877 says.

    In python3, email.message normally uses \n line terminators unless you
    tell it to use the email.policy.SMTP or email.policy.SMTPUTF8 policy
    option. So don't do that. If you're still using python2, this would
    be a good time to upgrade.





    --
    Regards,
    John Levine, [email protected], Primary Perpetrator of "The Internet for Dummies",
    Please consider the environment before reading this e-mail. https://jl.ly

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Mike Glover@21:1/5 to John Levine on Fri Feb 26 20:37:08 2021
    John Levine <[email protected]> wrote:
    inews is not nntp just as sendmail is not smtp. Both run inside your Unix system and use the Unix \n line terminator. If you were opening a TCP connection to port 119 and issuing nntp commands, then you would use \r\n
    as RFC 3877 says.

    Point taken.


    In python3, email.message normally uses \n line terminators unless you
    tell it to use the email.policy.SMTP or email.policy.SMTPUTF8 policy
    option. So don't do that. If you're still using python2, this would
    be a good time to upgrade.


    It's python3, and I *was* using email.policy.SMTP. Removing that and my .replace('\r\n', '\n') kludge is working. Thanks.

    But, still, what's the benefit to treating \r as folding whitespace in
    this case?

    -mg

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to Russ Allbery on Fri Feb 26 12:51:44 2021
    Russ Allbery <[email protected]> writes:

    It may make sense to check if the first line of the input ends in \r\n
    and, if so, call wire_to_native on the input buffer before processing
    it. inews isn't really designed to accept arbitrary line endings, but
    it looks easy enough to make it compatible with CRLF line endings.

    Oh, wait, no, this doesn't work because wire_to_native expects
    dot-stuffing and a terminating dot. Bleh. We'd have to actually teach
    inews about \r\n as a line ending.

    --
    Russ Allbery ([email protected]) <https://www.eyrie.org/~eagle/>

    Please post questions rather than mailing me directly.
    <https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to Mike Glover on Fri Feb 26 12:49:44 2021
    Mike Glover <[email protected]> writes:

    But, still, what's the benefit to treating \r as folding whitespace in
    this case?

    It's not. It uses ISWHITE for folding whitespace, which only accepts \t
    and space.

    The problem is that it's not recognizing a line consisting of \r\n as a
    blank line because of this check at the end of StripOffHeaders:

    /* Get start of next header; if it's a blank line, we hit the end. */
    p = NextHeader(p);
    if (*p == '\n')
    break;

    *p is \r here, so it thinks there's another header. I think it then tries
    to parse \r\n as a header line and then errors out because there's no
    colon. It was clearly never tested with CRLF line endings.

    It may make sense to check if the first line of the input ends in \r\n
    and, if so, call wire_to_native on the input buffer before processing it.
    inews isn't really designed to accept arbitrary line endings, but it looks
    easy enough to make it compatible with CRLF line endings.

    --
    Russ Allbery ([email protected]) <https://www.eyrie.org/~eagle/>

    Please post questions rather than mailing me directly.
    <https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From John Levine@21:1/5 to [email protected] on Fri Feb 26 22:47:59 2021
    In article <[email protected]>,
    Russ Allbery <[email protected]> wrote:
    It may make sense to check if the first line of the input ends in \r\n
    and, if so, call wire_to_native on the input buffer before processing it. >inews isn't really designed to accept arbitrary line endings, but it looks >easy enough to make it compatible with CRLF line endings.

    On the other hand, INN is 30 years old and I don't recall this particular question coming up before, so perhaps it doesn't need solving.

    --
    Regards,
    John Levine, [email protected], Primary Perpetrator of "The Internet for Dummies",
    Please consider the environment before reading this e-mail. https://jl.ly

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to John Levine on Fri Feb 26 14:50:18 2021
    John Levine <[email protected]> writes:
    Russ Allbery <[email protected]> wrote:

    It may make sense to check if the first line of the input ends in \r\n
    and, if so, call wire_to_native on the input buffer before processing
    it. inews isn't really designed to accept arbitrary line endings, but
    it looks easy enough to make it compatible with CRLF line endings.

    On the other hand, INN is 30 years old and I don't recall this
    particular question coming up before, so perhaps it doesn't need
    solving.

    The error message being extremely unhelpful is a fair point, though, and probably easier to fix.

    --
    Russ Allbery ([email protected]) <https://www.eyrie.org/~eagle/>

    Please post questions rather than mailing me directly.
    <https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)