• ::http::geturl with progress callback blocks

    From Alexandru@21:1/5 to All on Mon Jul 24 19:49:52 2023
    I'm using this command below to download a file from own server:

    ::http::geturl $url -binary 1 -progress $callbackproc

    The callback procedure is used to show a progress bar in the UI.

    On some computers to which I don't have access, the download works but the progress bar is not updated until the complete file is downloaded. It looks like the download is blocking the whole time.

    Since the download source is always the same, there must be something else from client to client that causes this behavior. Could it have something to do with the headers exchanged in the http communication?

    Many thanks
    Alexandru

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andreas Leitgeb@21:1/5 to Alexandru on Tue Jul 25 06:58:55 2023
    Alexandru <[email protected]> wrote:
    I'm using this command below to download a file from own server:
    ::http::geturl $url -binary 1 -progress $callbackproc
    The callback procedure is used to show a progress bar in the UI.
    On some computers to which I don't have access, the download works
    but the progress bar is not updated until the complete file is
    downloaded. It looks like the download is blocking the whole time.

    Can you determine whether the callback is actually called?

    Merely from reading it, it could be circumstances that would
    either prevent the callback from being called at all, or just
    prevent the gui from being updated to reflect the progressbar
    updates.

    Maybe, the "readable" events from socket are so fast coming in,
    that the "idletask" for redrawing the progressbar starves...

    Does your callback code call update idletasks? -- not sure if
    it even should... but it might be something to try...

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alexandru@21:1/5 to Andreas Leitgeb on Tue Jul 25 01:10:38 2023
    Andreas Leitgeb schrieb am Dienstag, 25. Juli 2023 um 08:59:00 UTC+2:
    Alexandru <[email protected]> wrote:
    I'm using this command below to download a file from own server:
    ::http::geturl $url -binary 1 -progress $callbackproc
    The callback procedure is used to show a progress bar in the UI.
    On some computers to which I don't have access, the download works
    but the progress bar is not updated until the complete file is
    downloaded. It looks like the download is blocking the whole time.
    Can you determine whether the callback is actually called?

    Merely from reading it, it could be circumstances that would
    either prevent the callback from being called at all, or just
    prevent the gui from being updated to reflect the progressbar
    updates.

    Maybe, the "readable" events from socket are so fast coming in,
    that the "idletask" for redrawing the progressbar starves...

    Does your callback code call update idletasks? -- not sure if
    it even should... but it might be something to try...

    Thanks Andreas. The callback should be called, since I can see how the progress bar is initialized and how it jups to 100% at the end of the download.
    The callback also calls "update" without the "idletasks" argument, because I had the experience, that "update idletasks" does not work.
    The code runs on different machines of different customers. In 90% of the cases there is no issue. For me it looks like the 10% have some network configurations that are incompatible with my code.
    Again: The code on the client PC starts the download from a server computer that is also fixed (always the same).
    I suppose the issue is either something missing on the server or on the client. Until now those are only suppositions.
    I hope somebody made similar experiences and knows the solution.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alexandru@21:1/5 to Alexandru on Tue Jul 25 02:01:37 2023
    Alexandru schrieb am Dienstag, 25. Juli 2023 um 10:10:40 UTC+2:
    Andreas Leitgeb schrieb am Dienstag, 25. Juli 2023 um 08:59:00 UTC+2:
    Alexandru <[email protected]> wrote:
    I'm using this command below to download a file from own server:
    ::http::geturl $url -binary 1 -progress $callbackproc
    The callback procedure is used to show a progress bar in the UI.
    On some computers to which I don't have access, the download works
    but the progress bar is not updated until the complete file is downloaded. It looks like the download is blocking the whole time.
    Can you determine whether the callback is actually called?

    Merely from reading it, it could be circumstances that would
    either prevent the callback from being called at all, or just
    prevent the gui from being updated to reflect the progressbar
    updates.

    Maybe, the "readable" events from socket are so fast coming in,
    that the "idletask" for redrawing the progressbar starves...

    Does your callback code call update idletasks? -- not sure if
    it even should... but it might be something to try...
    Thanks Andreas. The callback should be called, since I can see how the progress bar is initialized and how it jups to 100% at the end of the download.
    The callback also calls "update" without the "idletasks" argument, because I had the experience, that "update idletasks" does not work.
    The code runs on different machines of different customers. In 90% of the cases there is no issue. For me it looks like the 10% have some network configurations that are incompatible with my code.
    Again: The code on the client PC starts the download from a server computer that is also fixed (always the same).
    I suppose the issue is either something missing on the server or on the client.
    Until now those are only suppositions.
    I hope somebody made similar experiences and knows the solution.

    After reading some general things about transfer encoding, I added

    -headers [list Accept-Encoding "gzip, deflate"]

    to the http request and hopefully this will solve the issue.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich@21:1/5 to Alexandru on Tue Jul 25 13:00:50 2023
    Alexandru <[email protected]> wrote:
    Thanks Andreas. The callback should be called, since I can see how
    the progress bar is initialized and how it jups to 100% at the end of
    the download. The callback also calls "update" without the
    "idletasks" argument, because I had the experience, that "update
    idletasks" does not work. The code runs on different machines of
    different customers. In 90% of the cases there is no issue. For me
    it looks like the 10% have some network configurations that are
    incompatible with my code. Again: The code on the client PC starts
    the download from a server computer that is also fixed (always the
    same). I suppose the issue is either something missing on the server
    or on the client. Until now those are only suppositions. I hope
    somebody made similar experiences and knows the solution.

    You appear to have omitted the possibility that those 10% have a
    network link of sufficient speed that the download occurs fast enough
    that "initialize --- 100% done" is actually accurate.

    How large is the file being downloaded, and what is the network
    performace for those that see "direct to 100%" during the download?

    And, while you imply so, for those that see direct to 100%, do they
    have the full file when the download ends?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alexandru@21:1/5 to Rich on Tue Jul 25 06:49:55 2023
    Rich schrieb am Dienstag, 25. Juli 2023 um 15:43:45 UTC+2:
    Alexandru <[email protected]> wrote:
    Rich schrieb am Dienstag, 25. Juli 2023 um 15:00:56 UTC+2:
    Alexandru <[email protected]> wrote:
    Thanks Andreas. The callback should be called, since I can see how
    the progress bar is initialized and how it jups to 100% at the end of
    the download. The callback also calls "update" without the
    "idletasks" argument, because I had the experience, that "update
    idletasks" does not work. The code runs on different machines of
    different customers. In 90% of the cases there is no issue. For me
    it looks like the 10% have some network configurations that are
    incompatible with my code. Again: The code on the client PC starts
    the download from a server computer that is also fixed (always the
    same). I suppose the issue is either something missing on the server
    or on the client. Until now those are only suppositions. I hope
    somebody made similar experiences and knows the solution.
    You appear to have omitted the possibility that those 10% have a
    network link of sufficient speed that the download occurs fast enough
    that "initialize --- 100% done" is actually accurate.

    How large is the file being downloaded, and what is the network
    performace for those that see "direct to 100%" during the download?

    And, while you imply so, for those that see direct to 100%, do they
    have the full file when the download ends?

    The file 27MB large and the networks speed is rather slow.
    I know that, because it takes a minute or so to finish the download.
    But, is that for you, or for the 10%? I.e., are the 10% behind some
    form of caching proxy that has your 27MB file cached, and they are on
    10G or better ethernet to that caching proxy, such that for them a 27MB download does indeed finish nearly instantly?
    Yes the file is fully downloaded when the progress jumps to 100%
    because the file is an executable and it runs correctly after that.
    Ok, that rules out 100% being indicated without the file actually being
    fully downloaded.

    This is no caching issue. The file new and the eventual proxy never saw that file.
    Besides: The download takes as long (order of magnitude) as on computers where the progress is visible.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andreas Leitgeb@21:1/5 to Alexandru on Tue Jul 25 13:41:56 2023
    Alexandru <[email protected]> wrote:
    Andreas Leitgeb schrieb am Dienstag, 25. Juli 2023 um 08:59:00 UTC+2:
    Does your callback code call update idletasks? -- not sure if
    it even should... but it might be something to try...
    Thanks Andreas. The callback should be called, since I can see how the progress bar is initialized and how it jups to 100% at the end of the download.

    You misunderstood me a bit... I see that it is run at least twice,
    but for the steps inbetween... I don't know, if they don't even fire,
    or the effects just aren't updated to screen.

    The callback also calls "update" without the "idletasks" argument,
    because I had the experience, that "update idletasks" does not work.

    The danger with "update" is, that it also allows for the next
    fileevent to slip in, and thereby completely spoiling the event
    loop.

    I'd remove the plain "update" (or change it to "update idletasks")
    and then re-evaluate what's going wrong (if anything actually is), and
    then get help on the problems that came back - they likely have better solutions than "update".

    The code runs on different machines of different customers. In 90% of the cases there is no issue. For me it looks like the 10% have some network configurations that are incompatible with my code.

    I think, "incompatible" might mean, that the network is fast enough
    so that the next fileevent from the http-socket hits the "update"
    within your callback.

    You can "augment" your local test-copy of your program by slowing down
    the callback code with some "after 250" ... I'm pretty sure that with
    that you'll start to see those customers' symptoms.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alexandru@21:1/5 to Rich on Tue Jul 25 06:23:33 2023
    Rich schrieb am Dienstag, 25. Juli 2023 um 15:00:56 UTC+2:
    Alexandru <[email protected]> wrote:
    Thanks Andreas. The callback should be called, since I can see how
    the progress bar is initialized and how it jups to 100% at the end of
    the download. The callback also calls "update" without the
    "idletasks" argument, because I had the experience, that "update
    idletasks" does not work. The code runs on different machines of
    different customers. In 90% of the cases there is no issue. For me
    it looks like the 10% have some network configurations that are incompatible with my code. Again: The code on the client PC starts
    the download from a server computer that is also fixed (always the
    same). I suppose the issue is either something missing on the server
    or on the client. Until now those are only suppositions. I hope
    somebody made similar experiences and knows the solution.
    You appear to have omitted the possibility that those 10% have a
    network link of sufficient speed that the download occurs fast enough
    that "initialize --- 100% done" is actually accurate.

    How large is the file being downloaded, and what is the network
    performace for those that see "direct to 100%" during the download?

    And, while you imply so, for those that see direct to 100%, do they
    have the full file when the download ends?

    The file 27MB large and the networks speed is rather slow.
    I know that, because it takes a minute or so to finish the download.
    Yes the file is fully downloaded when the progress jumps to 100% because the file is an executable and it runs correctly after that.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich@21:1/5 to Alexandru on Tue Jul 25 13:43:41 2023
    Alexandru <[email protected]> wrote:
    Rich schrieb am Dienstag, 25. Juli 2023 um 15:00:56 UTC+2:
    Alexandru <[email protected]> wrote:
    Thanks Andreas. The callback should be called, since I can see how
    the progress bar is initialized and how it jups to 100% at the end of
    the download. The callback also calls "update" without the
    "idletasks" argument, because I had the experience, that "update
    idletasks" does not work. The code runs on different machines of
    different customers. In 90% of the cases there is no issue. For me
    it looks like the 10% have some network configurations that are
    incompatible with my code. Again: The code on the client PC starts
    the download from a server computer that is also fixed (always the
    same). I suppose the issue is either something missing on the server
    or on the client. Until now those are only suppositions. I hope
    somebody made similar experiences and knows the solution.
    You appear to have omitted the possibility that those 10% have a
    network link of sufficient speed that the download occurs fast enough
    that "initialize --- 100% done" is actually accurate.

    How large is the file being downloaded, and what is the network
    performace for those that see "direct to 100%" during the download?

    And, while you imply so, for those that see direct to 100%, do they
    have the full file when the download ends?

    The file 27MB large and the networks speed is rather slow.
    I know that, because it takes a minute or so to finish the download.

    But, is that for you, or for the 10%? I.e., are the 10% behind some
    form of caching proxy that has your 27MB file cached, and they are on
    10G or better ethernet to that caching proxy, such that for them a 27MB download does indeed finish nearly instantly?

    Yes the file is fully downloaded when the progress jumps to 100%
    because the file is an executable and it runs correctly after that.

    Ok, that rules out 100% being indicated without the file actually being
    fully downloaded.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ralf Fassel@21:1/5 to All on Tue Jul 25 18:28:02 2023
    * Andreas Leitgeb <[email protected]>
    | I'd remove the plain "update" (or change it to "update idletasks")
    | and then re-evaluate what's going wrong (if anything actually is), and
    | then get help on the problems that came back - they likely have better
    | solutions than "update".

    I'd also suggest to remove the update, and add stderr output in the
    callback with [clock milliseconds] timestamps, so you see when and how
    often the callback is invoked. If these fire rapidly, the redisplay
    might not have a chance to happen (comparable to what you get when
    setting a "fileevent readable" on a regular file).

    The difficulty for Alexandru might be to find a customer willing in
    helping to debug this :-/

    R'

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From et99@21:1/5 to Alexandru on Sun Jul 30 11:21:21 2023
    On 7/25/2023 6:23 AM, Alexandru wrote:
    Rich schrieb am Dienstag, 25. Juli 2023 um 15:00:56 UTC+2:
    Alexandru <[email protected]> wrote:
    Thanks Andreas. The callback should be called, since I can see how
    the progress bar is initialized and how it jups to 100% at the end of
    the download. The callback also calls "update" without the
    "idletasks" argument, because I had the experience, that "update
    idletasks" does not work. The code runs on different machines of
    different customers. In 90% of the cases there is no issue. For me
    it looks like the 10% have some network configurations that are
    incompatible with my code. Again: The code on the client PC starts
    the download from a server computer that is also fixed (always the
    same). I suppose the issue is either something missing on the server
    or on the client. Until now those are only suppositions. I hope
    somebody made similar experiences and knows the solution.
    You appear to have omitted the possibility that those 10% have a
    network link of sufficient speed that the download occurs fast enough
    that "initialize --- 100% done" is actually accurate.

    How large is the file being downloaded, and what is the network
    performace for those that see "direct to 100%" during the download?

    And, while you imply so, for those that see direct to 100%, do they
    have the full file when the download ends?

    The file 27MB large and the networks speed is rather slow.
    I know that, because it takes a minute or so to finish the download.
    Yes the file is fully downloaded when the progress jumps to 100% because the file is an executable and it runs correctly after that.

    Hi Alexandru, did you ever solve this?

    -progress callback returns the token and 2 values, the
    total expected size, and the current size.

    The expected size is returned via the content-Length
    meta-data but: "The expected total size may be
    unknown, in which case zero is passed to the callback. "

    I presume you use the current size but could it be
    that the expected size is being returned as 0? Do you
    check for that value, and if so, what do you do with a
    0? Might you be just returning from the callback and
    not updating the gui?

    Perhaps the http code knows the expected size only
    when the transfer is finished, and so just fills in
    both values using the final current size? I'm assuming
    you do get a callback with both values the same,
    indicating 100%

    Are you using ttk::progressbar for the progress? It
    has an indeterminate mode which you could use along
    with an entry or label widget with the current size of
    the transfer, if indeed you are getting 0 for the
    expected size.

    Also, there's an option -blocksize size which affects
    when the callback is invoked, I wonder what its
    default value might be.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alexandru@21:1/5 to Alexandru on Sun Jul 30 12:54:18 2023
    Alexandru schrieb am Sonntag, 30. Juli 2023 um 21:40:17 UTC+2:
    et99 schrieb am Sonntag, 30. Juli 2023 um 20:21:27 UTC+2:
    On 7/25/2023 6:23 AM, Alexandru wrote:
    Rich schrieb am Dienstag, 25. Juli 2023 um 15:00:56 UTC+2:
    Alexandru <[email protected]> wrote:
    Thanks Andreas. The callback should be called, since I can see how
    the progress bar is initialized and how it jups to 100% at the end of >>> the download. The callback also calls "update" without the
    "idletasks" argument, because I had the experience, that "update
    idletasks" does not work. The code runs on different machines of
    different customers. In 90% of the cases there is no issue. For me
    it looks like the 10% have some network configurations that are
    incompatible with my code. Again: The code on the client PC starts
    the download from a server computer that is also fixed (always the
    same). I suppose the issue is either something missing on the server >>> or on the client. Until now those are only suppositions. I hope
    somebody made similar experiences and knows the solution.
    You appear to have omitted the possibility that those 10% have a
    network link of sufficient speed that the download occurs fast enough >> that "initialize --- 100% done" is actually accurate.

    How large is the file being downloaded, and what is the network
    performace for those that see "direct to 100%" during the download?

    And, while you imply so, for those that see direct to 100%, do they
    have the full file when the download ends?

    The file 27MB large and the networks speed is rather slow.
    I know that, because it takes a minute or so to finish the download.
    Yes the file is fully downloaded when the progress jumps to 100% because the file is an executable and it runs correctly after that.
    Hi Alexandru, did you ever solve this?

    -progress callback returns the token and 2 values, the
    total expected size, and the current size.

    The expected size is returned via the content-Length
    meta-data but: "The expected total size may be
    unknown, in which case zero is passed to the callback. "

    I presume you use the current size but could it be
    that the expected size is being returned as 0? Do you
    check for that value, and if so, what do you do with a
    0? Might you be just returning from the callback and
    not updating the gui?

    Perhaps the http code knows the expected size only
    when the transfer is finished, and so just fills in
    both values using the final current size? I'm assuming
    you do get a callback with both values the same,
    indicating 100%

    Are you using ttk::progressbar for the progress? It
    has an indeterminate mode which you could use along
    with an entry or label widget with the current size of
    the transfer, if indeed you are getting 0 for the
    expected size.

    Also, there's an option -blocksize size which affects
    when the callback is invoked, I wonder what its
    default value might be.
    It's not solved yet. I added a new option to the header (-headers [list Accept-Encoding "gzip, deflate"] ) beased on some help from general http threads but since I cannot reproduce the issue, I need to wait for feedback from users.
    Your tipp with -blocksize is also interesting, that is also worth a try.
    Ans yes, I also suppose that for some client-server combinations, the total size is not known, so the callback is called one 2 times, with 0% and 100%. The question is rather, when and why it can happen that the file size is not tranfered? I suppose
    that most of the transfers have some default values where the total file size is transfered through the http headers. And that some others block this option. Maybe the -blocksize option can fix this, I'll try to see what exactly the source code does with
    this this option.

    -blocksize defaults to 8192 so it's small enough to show a smooth download progress for a 27MB large files.
    So if cannot be the solution to user specify this value.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alexandru@21:1/5 to All on Sun Jul 30 12:40:14 2023
    et99 schrieb am Sonntag, 30. Juli 2023 um 20:21:27 UTC+2:
    On 7/25/2023 6:23 AM, Alexandru wrote:
    Rich schrieb am Dienstag, 25. Juli 2023 um 15:00:56 UTC+2:
    Alexandru <[email protected]> wrote:
    Thanks Andreas. The callback should be called, since I can see how
    the progress bar is initialized and how it jups to 100% at the end of >>> the download. The callback also calls "update" without the
    "idletasks" argument, because I had the experience, that "update
    idletasks" does not work. The code runs on different machines of
    different customers. In 90% of the cases there is no issue. For me
    it looks like the 10% have some network configurations that are
    incompatible with my code. Again: The code on the client PC starts
    the download from a server computer that is also fixed (always the
    same). I suppose the issue is either something missing on the server
    or on the client. Until now those are only suppositions. I hope
    somebody made similar experiences and knows the solution.
    You appear to have omitted the possibility that those 10% have a
    network link of sufficient speed that the download occurs fast enough
    that "initialize --- 100% done" is actually accurate.

    How large is the file being downloaded, and what is the network
    performace for those that see "direct to 100%" during the download?

    And, while you imply so, for those that see direct to 100%, do they
    have the full file when the download ends?

    The file 27MB large and the networks speed is rather slow.
    I know that, because it takes a minute or so to finish the download.
    Yes the file is fully downloaded when the progress jumps to 100% because the file is an executable and it runs correctly after that.
    Hi Alexandru, did you ever solve this?

    -progress callback returns the token and 2 values, the
    total expected size, and the current size.

    The expected size is returned via the content-Length
    meta-data but: "The expected total size may be
    unknown, in which case zero is passed to the callback. "

    I presume you use the current size but could it be
    that the expected size is being returned as 0? Do you
    check for that value, and if so, what do you do with a
    0? Might you be just returning from the callback and
    not updating the gui?

    Perhaps the http code knows the expected size only
    when the transfer is finished, and so just fills in
    both values using the final current size? I'm assuming
    you do get a callback with both values the same,
    indicating 100%

    Are you using ttk::progressbar for the progress? It
    has an indeterminate mode which you could use along
    with an entry or label widget with the current size of
    the transfer, if indeed you are getting 0 for the
    expected size.

    Also, there's an option -blocksize size which affects
    when the callback is invoked, I wonder what its
    default value might be.
    It's not solved yet. I added a new option to the header (-headers [list Accept-Encoding "gzip, deflate"] ) beased on some help from general http threads but since I cannot reproduce the issue, I need to wait for feedback from users.
    Your tipp with -blocksize is also interesting, that is also worth a try.
    Ans yes, I also suppose that for some client-server combinations, the total size is not known, so the callback is called one 2 times, with 0% and 100%. The question is rather, when and why it can happen that the file size is not tranfered? I suppose that
    most of the transfers have some default values where the total file size is transfered through the http headers. And that some others block this option. Maybe the -blocksize option can fix this, I'll try to see what exactly the source code does with this
    this option.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From et99@21:1/5 to Alexandru on Sun Jul 30 13:10:56 2023
    On 7/30/2023 12:54 PM, Alexandru wrote:
    Alexandru schrieb am Sonntag, 30. Juli 2023 um 21:40:17 UTC+2:
    et99 schrieb am Sonntag, 30. Juli 2023 um 20:21:27 UTC+2:
    On 7/25/2023 6:23 AM, Alexandru wrote:
    Rich schrieb am Dienstag, 25. Juli 2023 um 15:00:56 UTC+2:
    Alexandru <[email protected]> wrote:
    Thanks Andreas. The callback should be called, since I can see how >>>>>> the progress bar is initialized and how it jups to 100% at the end of >>>>>> the download. The callback also calls "update" without the
    "idletasks" argument, because I had the experience, that "update
    idletasks" does not work. The code runs on different machines of
    different customers. In 90% of the cases there is no issue. For me >>>>>> it looks like the 10% have some network configurations that are
    incompatible with my code. Again: The code on the client PC starts >>>>>> the download from a server computer that is also fixed (always the >>>>>> same). I suppose the issue is either something missing on the server >>>>>> or on the client. Until now those are only suppositions. I hope
    somebody made similar experiences and knows the solution.
    You appear to have omitted the possibility that those 10% have a
    network link of sufficient speed that the download occurs fast enough >>>>> that "initialize --- 100% done" is actually accurate.

    How large is the file being downloaded, and what is the network
    performace for those that see "direct to 100%" during the download?

    And, while you imply so, for those that see direct to 100%, do they
    have the full file when the download ends?

    The file 27MB large and the networks speed is rather slow.
    I know that, because it takes a minute or so to finish the download.
    Yes the file is fully downloaded when the progress jumps to 100% because the file is an executable and it runs correctly after that.
    Hi Alexandru, did you ever solve this?

    -progress callback returns the token and 2 values, the
    total expected size, and the current size.

    The expected size is returned via the content-Length
    meta-data but: "The expected total size may be
    unknown, in which case zero is passed to the callback. "

    I presume you use the current size but could it be
    that the expected size is being returned as 0? Do you
    check for that value, and if so, what do you do with a
    0? Might you be just returning from the callback and
    not updating the gui?

    Perhaps the http code knows the expected size only
    when the transfer is finished, and so just fills in
    both values using the final current size? I'm assuming
    you do get a callback with both values the same,
    indicating 100%

    Are you using ttk::progressbar for the progress? It
    has an indeterminate mode which you could use along
    with an entry or label widget with the current size of
    the transfer, if indeed you are getting 0 for the
    expected size.

    Also, there's an option -blocksize size which affects
    when the callback is invoked, I wonder what its
    default value might be.
    It's not solved yet. I added a new option to the header (-headers [list Accept-Encoding "gzip, deflate"] ) beased on some help from general http threads but since I cannot reproduce the issue, I need to wait for feedback from users.
    Your tipp with -blocksize is also interesting, that is also worth a try.
    Ans yes, I also suppose that for some client-server combinations, the total size is not known, so the callback is called one 2 times, with 0% and 100%. The question is rather, when and why it can happen that the file size is not tranfered? I suppose
    that most of the transfers have some default values where the total file size is transfered through the http headers. And that some others block this option. Maybe the -blocksize option can fix this, I'll try to see what exactly the source code does with
    this this option.

    -blocksize defaults to 8192 so it's small enough to show a smooth download progress for a 27MB large files.
    So if cannot be the solution to user specify this value.

    So, have you been able to determine if you're getting
    0 for expected file size on the systems that don't
    update the progress bar?

    What about platform, windows, unix, both? If both,
    does the problem occur on only one?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alexandru@21:1/5 to All on Sun Jul 30 14:20:42 2023
    et99 schrieb am Sonntag, 30. Juli 2023 um 22:11:01 UTC+2:
    On 7/30/2023 12:54 PM, Alexandru wrote:
    Alexandru schrieb am Sonntag, 30. Juli 2023 um 21:40:17 UTC+2:
    et99 schrieb am Sonntag, 30. Juli 2023 um 20:21:27 UTC+2:
    On 7/25/2023 6:23 AM, Alexandru wrote:
    Rich schrieb am Dienstag, 25. Juli 2023 um 15:00:56 UTC+2:
    Alexandru <[email protected]> wrote:
    Thanks Andreas. The callback should be called, since I can see how >>>>>> the progress bar is initialized and how it jups to 100% at the end of >>>>>> the download. The callback also calls "update" without the
    "idletasks" argument, because I had the experience, that "update >>>>>> idletasks" does not work. The code runs on different machines of >>>>>> different customers. In 90% of the cases there is no issue. For me >>>>>> it looks like the 10% have some network configurations that are >>>>>> incompatible with my code. Again: The code on the client PC starts >>>>>> the download from a server computer that is also fixed (always the >>>>>> same). I suppose the issue is either something missing on the server >>>>>> or on the client. Until now those are only suppositions. I hope >>>>>> somebody made similar experiences and knows the solution.
    You appear to have omitted the possibility that those 10% have a
    network link of sufficient speed that the download occurs fast enough >>>>> that "initialize --- 100% done" is actually accurate.

    How large is the file being downloaded, and what is the network
    performace for those that see "direct to 100%" during the download? >>>>>
    And, while you imply so, for those that see direct to 100%, do they >>>>> have the full file when the download ends?

    The file 27MB large and the networks speed is rather slow.
    I know that, because it takes a minute or so to finish the download. >>>> Yes the file is fully downloaded when the progress jumps to 100% because the file is an executable and it runs correctly after that.
    Hi Alexandru, did you ever solve this?

    -progress callback returns the token and 2 values, the
    total expected size, and the current size.

    The expected size is returned via the content-Length
    meta-data but: "The expected total size may be
    unknown, in which case zero is passed to the callback. "

    I presume you use the current size but could it be
    that the expected size is being returned as 0? Do you
    check for that value, and if so, what do you do with a
    0? Might you be just returning from the callback and
    not updating the gui?

    Perhaps the http code knows the expected size only
    when the transfer is finished, and so just fills in
    both values using the final current size? I'm assuming
    you do get a callback with both values the same,
    indicating 100%

    Are you using ttk::progressbar for the progress? It
    has an indeterminate mode which you could use along
    with an entry or label widget with the current size of
    the transfer, if indeed you are getting 0 for the
    expected size.

    Also, there's an option -blocksize size which affects
    when the callback is invoked, I wonder what its
    default value might be.
    It's not solved yet. I added a new option to the header (-headers [list Accept-Encoding "gzip, deflate"] ) beased on some help from general http threads but since I cannot reproduce the issue, I need to wait for feedback from users.
    Your tipp with -blocksize is also interesting, that is also worth a try. >> Ans yes, I also suppose that for some client-server combinations, the total size is not known, so the callback is called one 2 times, with 0% and 100%. The question is rather, when and why it can happen that the file size is not tranfered? I suppose
    that most of the transfers have some default values where the total file size is transfered through the http headers. And that some others block this option. Maybe the -blocksize option can fix this, I'll try to see what exactly the source code does with
    this this option.

    -blocksize defaults to 8192 so it's small enough to show a smooth download progress for a 27MB large files.
    So if cannot be the solution to user specify this value.
    So, have you been able to determine if you're getting
    0 for expected file size on the systems that don't
    update the progress bar?
    No, not yet. But this is highly prpbably, I'll check though. Need a cooperating customer first.

    What about platform, windows, unix, both? If both,
    does the problem occur on only one?
    I'm living in a Windows buble, no unix there.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich@21:1/5 to Alexandru on Sun Jul 30 22:02:10 2023
    Alexandru <[email protected]> wrote:
    Your tipp with -blocksize is also interesting, that is also worth a try.

    That setting is used with [read] to determine how many bytes are read
    from the socket at a time.

    Ans yes, I also suppose that for some client-server combinations, the
    total size is not known, so the callback is called one 2 times, with
    0% and 100%. The question is rather, when and why it can happen that
    the file size is not tranfered? I suppose that most of the transfers
    have some default values where the total file size is transfered
    through the http headers.

    HTTP includes a header that provides for transferring the full size in
    bytes of the file -- but its use is not mandatory.

    And that some others block this option.

    The headers are set by the serving webserver, so it is what determines
    whether the header showing the full file size is transferred.

    You should add some debug logging to your code that logs each callback
    from http::geturl and what the parameters are that get passed, and as
    well log the http headers from the transfer to see what headers are
    being transferred. Without knowing those two items, trying to fix this
    will be like trying to shoot a bullseye while blindfolded.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alexandru@21:1/5 to Rich on Sun Jul 30 15:10:50 2023
    Rich schrieb am Montag, 31. Juli 2023 um 00:02:15 UTC+2:
    Alexandru <[email protected]> wrote:
    Your tipp with -blocksize is also interesting, that is also worth a try.
    That setting is used with [read] to determine how many bytes are read
    from the socket at a time.
    Ans yes, I also suppose that for some client-server combinations, the
    total size is not known, so the callback is called one 2 times, with
    0% and 100%. The question is rather, when and why it can happen that
    the file size is not tranfered? I suppose that most of the transfers
    have some default values where the total file size is transfered
    through the http headers.
    HTTP includes a header that provides for transferring the full size in
    bytes of the file -- but its use is not mandatory.
    And that some others block this option.
    The headers are set by the serving webserver, so it is what determines whether the header showing the full file size is transferred.

    Exactly and the web server is a constant in all this. It's my server and the file is downloaded from my server.
    So it must be something with the client, right?


    You should add some debug logging to your code that logs each callback
    from http::geturl and what the parameters are that get passed, and as
    well log the http headers from the transfer to see what headers are
    being transferred. Without knowing those two items, trying to fix this
    will be like trying to shoot a bullseye while blindfolded.

    Yes I will.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From et99@21:1/5 to Alexandru on Sun Jul 30 17:01:36 2023
    On 7/30/2023 2:20 PM, Alexandru wrote:
    et99 schrieb am Sonntag, 30. Juli 2023 um 22:11:01 UTC+2:
    On 7/30/2023 12:54 PM, Alexandru wrote:
    Alexandru schrieb am Sonntag, 30. Juli 2023 um 21:40:17 UTC+2:
    et99 schrieb am Sonntag, 30. Juli 2023 um 20:21:27 UTC+2:
    On 7/25/2023 6:23 AM, Alexandru wrote:
    Rich schrieb am Dienstag, 25. Juli 2023 um 15:00:56 UTC+2:
    Alexandru <[email protected]> wrote:
    Thanks Andreas. The callback should be called, since I can see how >>>>>>>> the progress bar is initialized and how it jups to 100% at the end of >>>>>>>> the download. The callback also calls "update" without the
    "idletasks" argument, because I had the experience, that "update >>>>>>>> idletasks" does not work. The code runs on different machines of >>>>>>>> different customers. In 90% of the cases there is no issue. For me >>>>>>>> it looks like the 10% have some network configurations that are >>>>>>>> incompatible with my code. Again: The code on the client PC starts >>>>>>>> the download from a server computer that is also fixed (always the >>>>>>>> same). I suppose the issue is either something missing on the server >>>>>>>> or on the client. Until now those are only suppositions. I hope >>>>>>>> somebody made similar experiences and knows the solution.
    You appear to have omitted the possibility that those 10% have a >>>>>>> network link of sufficient speed that the download occurs fast enough >>>>>>> that "initialize --- 100% done" is actually accurate.

    How large is the file being downloaded, and what is the network
    performace for those that see "direct to 100%" during the download? >>>>>>>
    And, while you imply so, for those that see direct to 100%, do they >>>>>>> have the full file when the download ends?

    The file 27MB large and the networks speed is rather slow.
    I know that, because it takes a minute or so to finish the download. >>>>>> Yes the file is fully downloaded when the progress jumps to 100% because the file is an executable and it runs correctly after that.
    Hi Alexandru, did you ever solve this?

    -progress callback returns the token and 2 values, the
    total expected size, and the current size.

    The expected size is returned via the content-Length
    meta-data but: "The expected total size may be
    unknown, in which case zero is passed to the callback. "

    I presume you use the current size but could it be
    that the expected size is being returned as 0? Do you
    check for that value, and if so, what do you do with a
    0? Might you be just returning from the callback and
    not updating the gui?

    Perhaps the http code knows the expected size only
    when the transfer is finished, and so just fills in
    both values using the final current size? I'm assuming
    you do get a callback with both values the same,
    indicating 100%

    Are you using ttk::progressbar for the progress? It
    has an indeterminate mode which you could use along
    with an entry or label widget with the current size of
    the transfer, if indeed you are getting 0 for the
    expected size.

    Also, there's an option -blocksize size which affects
    when the callback is invoked, I wonder what its
    default value might be.
    It's not solved yet. I added a new option to the header (-headers [list Accept-Encoding "gzip, deflate"] ) beased on some help from general http threads but since I cannot reproduce the issue, I need to wait for feedback from users.
    Your tipp with -blocksize is also interesting, that is also worth a try. >>>> Ans yes, I also suppose that for some client-server combinations, the total size is not known, so the callback is called one 2 times, with 0% and 100%. The question is rather, when and why it can happen that the file size is not tranfered? I suppose
    that most of the transfers have some default values where the total file size is transfered through the http headers. And that some others block this option. Maybe the -blocksize option can fix this, I'll try to see what exactly the source code does with
    this this option.

    -blocksize defaults to 8192 so it's small enough to show a smooth download progress for a 27MB large files.
    So if cannot be the solution to user specify this value.
    So, have you been able to determine if you're getting
    0 for expected file size on the systems that don't
    update the progress bar?
    No, not yet. But this is highly prpbably, I'll check though. Need a cooperating customer first.

    What about platform, windows, unix, both? If both,
    does the problem occur on only one?
    I'm living in a Windows buble, no unix there.

    You need to find a customer who's really annoyed by this :)

    Windows, hmmm, and you're downloading an executable file. I've
    always wondered how it knows that and creates a file with the
    blocked status, which I have to unblock in properties. I wonder
    if that could come into play here.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From saitology9@21:1/5 to Alexandru on Sun Jul 30 20:07:57 2023
    On 7/30/2023 6:10 PM, Alexandru wrote:

    Exactly and the web server is a constant in all this. It's my server and the file is downloaded from my server.
    So it must be something with the client, right?



    If the server is yours, try adding "Content-Length" header to the file downloads and see how that goes.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From et99@21:1/5 to All on Sun Jul 30 17:58:22 2023
    On 7/30/2023 5:07 PM, saitology9 wrote:
    On 7/30/2023 6:10 PM, Alexandru wrote:

    Exactly and the web server is a constant in all this. It's my server and the file is downloaded from my server.
    So it must be something with the client, right?



    If the server is yours, try adding "Content-Length" header to the file downloads and see how that goes.

    Since it's both his server and client, he could also have the client
    log the data, say in a global array, with timestamps, and at 100%
    send a different url back with the logging data. This could be a
    debug version to send to a client who has the problem and wants to
    help.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich@21:1/5 to [email protected] on Mon Jul 31 01:08:06 2023
    et99 <[email protected]> wrote:
    Windows, hmmm, and you're downloading an executable file. I've
    always wondered how it knows that and creates a file with the blocked
    status, which I have to unblock in properties. I wonder if that
    could come into play here.

    That would seem to only come into play after the download is over,
    rather than during.

    And the how the file is marked, windows inserts a second stream into
    the file with a particular bit of data inside that essentially says
    "came from the internet". Then, later, if and when you go to execute
    it, if windows finds the second stream present, it pops up the "we
    don't know enough about this file, are you sure" dialog.

    Clearing the second stream out is equal to using the properties to say
    "I vouch for the file".

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From et99@21:1/5 to Rich on Sun Jul 30 19:14:55 2023
    On 7/30/2023 6:08 PM, Rich wrote:
    et99 <[email protected]> wrote:
    Windows, hmmm, and you're downloading an executable file. I've
    always wondered how it knows that and creates a file with the blocked
    status, which I have to unblock in properties. I wonder if that
    could come into play here.

    That would seem to only come into play after the download is over,
    rather than during.

    And the how the file is marked, windows inserts a second stream into
    the file with a particular bit of data inside that essentially says
    "came from the internet". Then, later, if and when you go to execute
    it, if windows finds the second stream present, it pops up the "we
    don't know enough about this file, are you sure" dialog.

    Clearing the second stream out is equal to using the properties to say
    "I vouch for the file".

    Interesting. A second stream, so I guess the application has
    to participate in this. Windows wouldn't know on its own would
    it?

    That's what I wondered about, did windows see .exe in the
    file name and do that w/o the app having to be involved.

    I guess i only see this when I download with a browser, so
    firefox could be the one adding this. Opening a file in tcl
    then shouldn't do it, even with the .exe extension.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich@21:1/5 to [email protected] on Mon Jul 31 02:49:46 2023
    et99 <[email protected]> wrote:
    On 7/30/2023 6:08 PM, Rich wrote:
    et99 <[email protected]> wrote:
    Windows, hmmm, and you're downloading an executable file. I've
    always wondered how it knows that and creates a file with the blocked
    status, which I have to unblock in properties. I wonder if that
    could come into play here.

    That would seem to only come into play after the download is over,
    rather than during.

    And the how the file is marked, windows inserts a second stream into
    the file with a particular bit of data inside that essentially says
    "came from the internet". Then, later, if and when you go to execute
    it, if windows finds the second stream present, it pops up the "we
    don't know enough about this file, are you sure" dialog.

    Clearing the second stream out is equal to using the properties to say
    "I vouch for the file".

    Interesting. A second stream, so I guess the application has
    to participate in this. Windows wouldn't know on its own would
    it?

    Since almost zero windows applications do anything with anything other
    than the primary file stream, most apps are completely unaware.

    Windows (as in the OS) itself is what is adding on the second stream,
    and noticing its presence to pop up the "do you really want to run
    this" dialog.

    That's what I wondered about, did windows see .exe in the
    file name and do that w/o the app having to be involved.

    Windows adds the second streams itself. I don't know if they attach
    them to all downloaded files, or just exe's. My guess is they attach
    them to all downloads -- but the ones attached to files not named .exe
    never trigger a "do you want to run this" dialog because they never get launched as an exe.

    I guess i only see this when I download with a browser, so
    firefox could be the one adding this. Opening a file in tcl
    then shouldn't do it, even with the .exe extension.

    Since it seems to happen for any kind of download I suspect windows
    does it. But as the only time I concern myself with it is on $work
    computer, I've never delved deeply into the who does what where and
    when part of the labeling, only the "how to clear out the labeling"
    part.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From et99@21:1/5 to Rich on Sun Jul 30 23:31:16 2023
    On 7/30/2023 7:49 PM, Rich wrote:
    et99 <[email protected]> wrote:
    On 7/30/2023 6:08 PM, Rich wrote:
    et99 <[email protected]> wrote:
    Windows, hmmm, and you're downloading an executable file. I've
    always wondered how it knows that and creates a file with the blocked
    status, which I have to unblock in properties. I wonder if that
    could come into play here.

    That would seem to only come into play after the download is over,
    rather than during.

    And the how the file is marked, windows inserts a second stream into
    the file with a particular bit of data inside that essentially says
    "came from the internet". Then, later, if and when you go to execute
    it, if windows finds the second stream present, it pops up the "we
    don't know enough about this file, are you sure" dialog.

    Clearing the second stream out is equal to using the properties to say
    "I vouch for the file".

    Interesting. A second stream, so I guess the application has
    to participate in this. Windows wouldn't know on its own would
    it?

    Since almost zero windows applications do anything with anything other
    than the primary file stream, most apps are completely unaware.

    Windows (as in the OS) itself is what is adding on the second stream,
    and noticing its presence to pop up the "do you really want to run
    this" dialog.

    That's what I wondered about, did windows see .exe in the
    file name and do that w/o the app having to be involved.

    Windows adds the second streams itself. I don't know if they attach
    them to all downloaded files, or just exe's. My guess is they attach
    them to all downloads -- but the ones attached to files not named .exe
    never trigger a "do you want to run this" dialog because they never get launched as an exe.

    I guess i only see this when I download with a browser, so
    firefox could be the one adding this. Opening a file in tcl
    then shouldn't do it, even with the .exe extension.

    Since it seems to happen for any kind of download I suspect windows
    does it. But as the only time I concern myself with it is on $work
    computer, I've never delved deeply into the who does what where and
    when part of the labeling, only the "how to clear out the labeling"
    part.


    If windows os does it on its own, then how would it also know the
    file came from the internet. It would have to be able to put the
    read from the internet and the file writes together. I wouldn't
    put that completely past a defender program. But still improbable.

    You're right. A program trace is the only way. We need to eliminate
    the impossible and see what remains.

    Interesting problem though.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ralf Fassel@21:1/5 to All on Mon Jul 31 11:09:52 2023
    * Rich <[email protected]d>
    | et99 <[email protected]> wrote:
    | Windows (as in the OS) itself is what is adding on the second stream,
    | and noticing its presence to pop up the "do you really want to run
    | this" dialog.

    | > That's what I wondered about, did windows see .exe in the
    | > file name and do that w/o the app having to be involved.

    | Windows adds the second streams itself. I don't know if they attach
    | them to all downloaded files, or just exe's. My guess is they attach
    | them to all downloads -- but the ones attached to files not named .exe
    | never trigger a "do you want to run this" dialog because they never get
    | launched as an exe.

    These might have some more info on this:

    https://www.itprotoday.com/mobile-management-and-security/how-does-windows-know-if-file-downloaded-internet

    https://support.microsoft.com/en-gb/topic/information-about-the-attachment-manager-in-microsoft-windows-c48a4dcd-8de5-2af5-ee9b-cd795ae42738

    A quick glance in my Windows Downloads folder shows the Zone.Identifier files/streams for several file types, not only .exe.

    HTH
    R'

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From et99@21:1/5 to Ralf Fassel on Mon Jul 31 17:05:58 2023
    On 7/31/2023 2:09 AM, Ralf Fassel wrote:
    * Rich <[email protected]d>
    | et99 <[email protected]> wrote:
    | Windows (as in the OS) itself is what is adding on the second stream,
    | and noticing its presence to pop up the "do you really want to run
    | this" dialog.

    | > That's what I wondered about, did windows see .exe in the
    | > file name and do that w/o the app having to be involved.

    | Windows adds the second streams itself. I don't know if they attach
    | them to all downloaded files, or just exe's. My guess is they attach
    | them to all downloads -- but the ones attached to files not named .exe
    | never trigger a "do you want to run this" dialog because they never get
    | launched as an exe.

    These might have some more info on this:

    https://www.itprotoday.com/mobile-management-and-security/how-does-windows-know-if-file-downloaded-internet

    https://support.microsoft.com/en-gb/topic/information-about-the-attachment-manager-in-microsoft-windows-c48a4dcd-8de5-2af5-ee9b-cd795ae42738

    A quick glance in my Windows Downloads folder shows the Zone.Identifier files/streams for several file types, not only .exe.

    HTH
    R'

    I think the key sentence in the second link is:

    "When you save files to your hard disk from a program that *uses* the Attachment Manager, the Web content zone information for the file is also saved with the file."

    So, my thoughts about writing to a file with the .exe extension in a tcl program is just a red herring.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)