1. Forum
  2. Usenet
  3. COMP.LANG.TCL

  • Math precision question

    From Luc@21:1/5 to All on Wed Nov 15 13:54:43 2023
    Hi.

    $ set ::tcl_precision 2
    2
    $ expr {1.08688 - 1.08684}
    4e-05
    $ set ::tcl_precision 17
    17
    $ expr {1.08688 - 1.08684}
    4.0000000000040004e-05

    Why don't I just get 0.00004?

    --
    Luc


    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich@21:1/5 to Luc on Wed Nov 15 18:48:57 2023
    Luc <[email protected]d> wrote:
    Hi.

    $ set ::tcl_precision 2

    Note this big warning from the documentation:

    tcl_precision

    This variable controls the number of digits to generate when
    converting floating-point values to strings. It defaults to 0.
    **Applications should not change this value; it is provided for
    compatibility with legacy code.**

    2
    $ expr {1.08688 - 1.08684}
    4e-05
    $ set ::tcl_precision 17
    17
    $ expr {1.08688 - 1.08684}
    4.0000000000040004e-05

    Why don't I just get 0.00004?

    https://0.30000000000000004.com/

    If you want a particular number of decimal digits when converting to a
    string, use [format] and the %f format method.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Luc@21:1/5 to Rich on Wed Nov 15 22:27:03 2023
    On Wed, 15 Nov 2023 18:48:57 -0000 (UTC), Rich wrote:

    https://0.30000000000000004.com/

    If you want a particular number of decimal digits when converting to a >string, use [format] and the %f format method.
    **************************

    Well, I'm glad I asked because I didn't know that site
    and it's really interesting.

    Thank you again.

    --
    Luc


    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Arjen Markus@21:1/5 to Luc on Wed Nov 15 23:53:21 2023
    On Thursday, November 16, 2023 at 2:27:08 AM UTC+1, Luc wrote:
    On Wed, 15 Nov 2023 18:48:57 -0000 (UTC), Rich wrote:

    https://0.30000000000000004.com/

    If you want a particular number of decimal digits when converting to a >string, use [format] and the %f format method.
    **************************

    Well, I'm glad I asked because I didn't know that site
    and it's really interesting.

    Thank you again.

    --
    Luc

    Classic case of floating-point numbers not being decimal numbers. Think of representing 1/3 as a decimal number. That can only be done approximately. The same is true for many fractions that in decimal are finite, when using floating-point numbers, as
    these use 2 as the base instead of 10.

    Regards,

    Arjen

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)