1. Forum
  2. Usenet
  3. COMP.LANG.PYTHON

  • Bug in io.TextIOWrapper?

    From Jon Ribbens@21:1/5 to All on Mon Jun 19 14:55:04 2023
    io.TextIOWrapper() wraps a binary stream so you can write text to it.
    It takes an 'encoding' parameter, which it uses to look up the codec
    in the codecs registry, and then it uses the IncrementalEncoder and IncrementalDecoder classes for the appropriate codec.

    The IncrementalEncoder.encode() function is given the object to encode
    of course, and also an optional second parameter which indicates if
    this is the final output.

    The bug is that TextIOWrapper() never sets the second parameter to
    indicate that the output is complete - not even if you call close().

    Example:

    >>> import io
    >>> buffer = io.BytesIO()
    >>> stream = io.TextIOWrapper(buffer, encoding='idna')
    >>> stream.write('abc.example.com')
    15
    >>> stream.flush()
    >>> buffer.getvalue()
    b'abc.example.'

    Obviously using the 'idna' wrapper as an encoding on a stream is a bit unlikely, but nevertheless any other codec which cares about the 'final' parameter will also have this problem.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Inada Naoki@21:1/5 to All on Tue Jun 20 02:15:00 2023
    stream.flush() doesn't mean final output.
    Try stream.close()

    2023年6月20日(火) 1:40 Jon Ribbens via Python-list <[email protected]>:

    io.TextIOWrapper() wraps a binary stream so you can write text to it.
    It takes an 'encoding' parameter, which it uses to look up the codec
    in the codecs registry, and then it uses the IncrementalEncoder and IncrementalDecoder classes for the appropriate codec.

    The IncrementalEncoder.encode() function is given the object to encode
    of course, and also an optional second parameter which indicates if
    this is the final output.

    The bug is that TextIOWrapper() never sets the second parameter to
    indicate that the output is complete - not even if you call close().

    Example:

    >>> import io
    >>> buffer = io.BytesIO()
    >>> stream = io.TextIOWrapper(buffer, encoding='idna')
    >>> stream.write('abc.example.com')
    15
    >>> stream.flush()
    >>> buffer.getvalue()
    b'abc.example.'

    Obviously using the 'idna' wrapper as an encoding on a stream is a bit unlikely, but nevertheless any other codec which cares about the 'final' parameter will also have this problem.
    --
    https://mail.python.org/mailman/listinfo/python-list


    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Peter J. Holzer@21:1/5 to Inada Naoki via Python-list on Mon Jun 19 19:56:10 2023
    On 2023-06-20 02:15:00 +0900, Inada Naoki via Python-list wrote:
    stream.flush() doesn't mean final output.
    Try stream.close()

    After close() the value isn't available any more:

    Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    import io
    buffer = io.BytesIO()
    stream = io.TextIOWrapper(buffer, encoding='idna')
    stream.write('abc.example.com')
    15
    stream.close()
    buffer.getvalue()
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    ValueError: I/O operation on closed file.

    hp

    --
    _ | Peter J. Holzer | Story must make more sense than reality.
    |_|_) | |
    | | | [email protected] | -- Charles Stross, "Creative writing
    __/ | http://www.hjp.at/ | challenge!"

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEETtJbRjyPwVTYGJ5k8g5IURL+KF0FAmSQlrUACgkQ8g5IURL+ KF0+pg//Tl54b/8KZucZAPJRmXccDwDDdviKgGzfhh3hM3BO2VC9jgAoEkkZYI8a 2jz/yAJfGKOIjCcQbzQrtZ5oQWm1ZeS5ZOFXv2zvKZgREv8HlqVcuKcl0Zo/UUV5 LR99P08A7iS/J06irJbLKXQAqoDH8M67BPpv6/LbLzCh0OW39epF5VZ7X+hYxGMO 7h50Lr8RdlhJ+KiKA8j6W6evarZxi3fDpB9gAn8TCLRzZnzA9s1Geb5cB2jz4lsI Fv6FkaXoKAAtRYu281R7gls2LX38a5bd0kCUxmGa1Ocnhac/Yb+htLarNoPaS2xp nmBFNx4cUasCP9zDWut9KHkqJaxLZYQcrnq/Vw3xs1ilT9lyjtqyRS/EKimeAjYz o4ZTS4QI15aVI3ZVqoKbxI543IKc7upVCiG/C8MrHTXq2LT96ZuQ35PClQ1wDOfy K/6vxBSrqiumBNxNx3Hf9YFPybbZe8R0WnGc5/CqJjzogc30VdkBZf+bmK98FBax 0C7LvmT6g8zW884jqciRIdSyt7rpl+z/hCPxstf5PhhCecRt/3Oz5yMJJ81S9rnj /PtNvIERxcR1+wOTsXDs4IEQD3tLCUBT9Y2DjYb9qoAHdNSQ/Z7MYO8gH6FqvuwB REqVMclSozby4b4ZvrgQDKFXCRT9/XoYwZfaDX2
  • From Inada Naoki@21:1/5 to All on Tue Jun 20 03:19:48 2023
    You can use file instead of BytesIO

    2023年6月20日(火) 3:05 Peter J. Holzer via Python-list <[email protected]>:

    On 2023-06-20 02:15:00 +0900, Inada Naoki via Python-list wrote:
    stream.flush() doesn't mean final output.
    Try stream.close()

    After close() the value isn't available any more:

    Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    import io
    buffer = io.BytesIO()
    stream = io.TextIOWrapper(buffer, encoding='idna')
    stream.write('abc.example.com')
    15
    stream.close()
    buffer.getvalue()
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    ValueError: I/O operation on closed file.

    hp

    --
    _ | Peter J. Holzer | Story must make more sense than reality.
    |_|_) | |
    | | | [email protected] | -- Charles Stross, "Creative writing
    __/ | http://www.hjp.at/ | challenge!"
    --
    https://mail.python.org/mailman/listinfo/python-list


    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Inada Naoki@21:1/5 to All on Tue Jun 20 03:46:38 2023
    I checked TextIOWrapper source code and confirmed that it doesn't call encoder.write(text, finish=True) on close.
    Since TextIOWrapper allows random access, it is difficult to call it automatically. So please think it as just limitation rather than bug.
    Please use codec and binary file manually for now.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jon Ribbens@21:1/5 to Inada Naoki on Mon Jun 19 20:15:32 2023
    On 2023-06-19, Inada Naoki <[email protected]> wrote:
    I checked TextIOWrapper source code and confirmed that it doesn't call encoder.write(text, finish=True) on close.
    Since TextIOWrapper allows random access, it is difficult to call it automatically. So please think it as just limitation rather than bug.
    Please use codec and binary file manually for now.

    It could call it on seek() or flush(). It seems like a definite bug to
    me, in that its behaviour appears clearly incorrect - it's just that
    there isn't an entirely obvious "100% correct" behaviour to choose.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)