1. Forum
  2. Usenet
  3. COMP.LANG.TCL

  • New (to me) geturl problem with meta()

    From Ted Nolan @21:1/5 to All on Fri Aug 18 18:29:03 2023
    I have just started seeing an ineresting problem with program I have been
    using to talk to a REST API for the past year or more. In particular
    sometime very recently the indicies in the "meta()" array populated
    after a geturl request have started, sometimes, being populated in
    lower case. It's not consistent, so I suspect my requests are being round-robined across different servers, but to give a particular
    example, I had been doing something like:

    if {[string match -nocase {application/json*} $meta(Content-Type)]} {
    whatever
    }

    and now I am finding that sometimes "meta(Content-Type)" does not exist, but "meta(content-type)" does. Googling around I'm seeing that apparently
    servers are allowed to do this, but the meta() array seems almost useless
    if this is going to happen. I can fix the program of course, I guess
    looking at the man page, I should probably just check "state(type)",
    instead. Should the "meta()" subscripts be lowercased in future
    versions, or should there be a warning on the man page not to use them
    other than for debugging?
    --
    columbiaclosings.com
    What's not in Columbia anymore..

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andreas Leitgeb@21:1/5 to [email protected] on Fri Aug 18 21:14:46 2023
    Ted Nolan <tednolan> <[email protected]> wrote:
    I have just started seeing an ineresting problem with program I have been using to talk to a REST API for the past year or more. In particular sometime very recently the indicies in the "meta()" array populated
    after a geturl request have started, sometimes, being populated in
    lower case. It's not consistent, so I suspect my requests are being round-robined across different servers, but to give a particular
    example, I had been doing something like:
    if {[string match -nocase {application/json*} $meta(Content-Type)]} {
    whatever
    }
    and now I am finding that sometimes "meta(Content-Type)" does not exist, but "meta(content-type)" does. Googling around I'm seeing that apparently servers are allowed to do this, but the meta() array seems almost useless
    if this is going to happen.

    some random thoughts...

    1.) Is Content-Type the only such field, or could this thing happen
    for any other http-header?

    2.) What's happening with headers that are allowed to occur multiple
    times? would these also be allowed to appear in different casings?

    If Content-Type is the only one, and all-lowercase "content-type" the
    only variant of it, then the workaround would be straightforward:
    if {![info exists meta(Content-Type)]} { set meta(Content-Type) $meta(content-type) }


    Otherwise, you might need to loop over the keys of meta, and just
    set a lowercase version of each key with original value in a new
    array.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ted Nolan @21:1/5 to [email protected] on Fri Aug 18 23:33:26 2023
    In article <[email protected]>,
    Andreas Leitgeb <[email protected]> wrote:
    Ted Nolan <tednolan> <[email protected]> wrote:
    I have just started seeing an ineresting problem with program I have been
    using to talk to a REST API for the past year or more. In particular
    sometime very recently the indicies in the "meta()" array populated
    after a geturl request have started, sometimes, being populated in
    lower case. It's not consistent, so I suspect my requests are being
    round-robined across different servers, but to give a particular
    example, I had been doing something like:
    if {[string match -nocase {application/json*} $meta(Content-Type)]} { >> whatever
    }
    and now I am finding that sometimes "meta(Content-Type)" does not exist, but >> "meta(content-type)" does. Googling around I'm seeing that apparently
    servers are allowed to do this, but the meta() array seems almost useless
    if this is going to happen.

    some random thoughts...

    1.) Is Content-Type the only such field, or could this thing happen
    for any other http-header?

    2.) What's happening with headers that are allowed to occur multiple
    times? would these also be allowed to appear in different casings?

    If Content-Type is the only one, and all-lowercase "content-type" the
    only variant of it, then the workaround would be straightforward:
    if {![info exists meta(Content-Type)]} { set meta(Content-Type)
    $meta(content-type) }


    Otherwise, you might need to loop over the keys of meta, and just
    set a lowercase version of each key with original value in a new
    array.


    Well, "Content-Type" is the only one I care about, but yeah, it can happen
    to them all. I have resolved my issue, I think, by using "state(type)",
    so I don't need to do any more groveling around in "meta()".

    However, looking again at the man page, the behavior seems like a bug:

    https://www.tcl.tk/man/tcl8.6/TclCmd/http.html

    specifically lists "Content-Type", "Content-Length", & "Location" with
    those casings, so arguably the package should map whatever comes in
    for those to match the page.

    Oh well.
    --
    columbiaclosings.com
    What's not in Columbia anymore..

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Schelte@21:1/5 to All on Sat Aug 19 10:01:31 2023
    On 18/08/2023 20:29, Ted Nolan <tednolan> wrote:
    I have just started seeing an ineresting problem with program I have been using to talk to a REST API for the past year or more. In particular sometime very recently the indicies in the "meta()" array populated
    after a geturl request have started, sometimes, being populated in
    lower case. It's not consistent, so I suspect my requests are being round-robined across different servers, but to give a particular
    example, I had been doing something like:

    if {[string match -nocase {application/json*} $meta(Content-Type)]} {
    whatever
    }

    and now I am finding that sometimes "meta(Content-Type)" does not exist, but "meta(content-type)" does. Googling around I'm seeing that apparently servers are allowed to do this, but the meta() array seems almost useless
    if this is going to happen. I can fix the program of course, I guess
    looking at the man page, I should probably just check "state(type)",
    instead. Should the "meta()" subscripts be lowercased in future
    versions, or should there be a warning on the man page not to use them
    other than for debugging?

    This is just one of several issues with the http package that prompted
    me to develop the www package (https://chiselapp.com/user/schelte/repository/www)


    Schelte.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From saitology9@21:1/5 to Schelte on Sun Aug 20 14:21:59 2023
    On 8/19/2023 4:01 AM, Schelte wrote:
    On 18/08/2023 20:29, Ted Nolan <tednolan> wrote:

    This is just one of several issues with the http package that prompted
    me to develop the www package (https://chiselapp.com/user/schelte/repository/www)



    This looks like a comprehensive and a well thought-out package. I read
    a little bit of the documentation and may I kindly make a comment for
    your consideration:

    It says that it uses an in-memory sqlite database for managing cookies.
    That sounds good but at the same time, this choice may interfere with
    user scripts that use sqlite for their own purposes. Can you load
    multiple different versions of sqlite simultaneously? Is it thread-safe?
    I am not sure.

    In any case, this would easily result in your package and user scripts
    to fall out of sync with respect to sqlite versions they need.

    IMO, sqlite use here is not really needed. I think for the purposes you
    need it for, a plain dictionary might do the job here, which would make
    the package more attractive.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Gerald Lester@21:1/5 to All on Sun Aug 20 14:54:08 2023
    On 8/20/23 13:21, saitology9 wrote:
    On 8/19/2023 4:01 AM, Schelte wrote:
    On 18/08/2023 20:29, Ted Nolan <tednolan> wrote:

    This is just one of several issues with the http package that prompted
    me to develop the www package
    (https://chiselapp.com/user/schelte/repository/www)



    This looks like a comprehensive and a well thought-out package.  I read
    a little bit of the documentation and may I kindly make a comment for
    your consideration:

    It says that it uses an in-memory sqlite database for managing cookies.
    That sounds good but at the same time, this choice may interfere with
    user scripts that use sqlite for their own purposes.

    It should not.

    Can you load
    multiple different versions of sqlite simultaneously?

    Why would you want to do that?

    Is it thread-safe?

    Yes it is.

    I am not sure.

    In any case, this would easily result in your package and user scripts
    to fall out of sync with respect to sqlite versions they need.

    IMO, sqlite use here is not really needed. I think for the purposes you
    need it for, a plain dictionary might do the job here, which would make
    the package more attractive.

    I agree, sqlite seems to be overkill here.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Schelte@21:1/5 to All on Sun Aug 20 23:13:34 2023
    On 20/08/2023 20:21, saitology9 wrote:
    On 8/19/2023 4:01 AM, Schelte wrote:
    On 18/08/2023 20:29, Ted Nolan <tednolan> wrote:

    This is just one of several issues with the http package that prompted
    me to develop the www package
    (https://chiselapp.com/user/schelte/repository/www)



    This looks like a comprehensive and a well thought-out package.  I read
    a little bit of the documentation and may I kindly make a comment for
    your consideration:

    It says that it uses an in-memory sqlite database for managing cookies.

    The www package uses an in-memory sqlite database, unless you configure
    a file to preserve cookies across application restarts.

    That sounds good but at the same time, this choice may interfere with
    user scripts that use sqlite for their own purposes. Can you load
    multiple different versions of sqlite simultaneously? Is it thread-safe?
    I am not sure.

    In any case, this would easily result in your package and user scripts
    to fall out of sync with respect to sqlite versions they need.

    The www package is happy to use the same version of sqlite3 that may be required by other parts of the application. It doesn't use any advanced features that require the latest sqlite3 version.

    IMO, sqlite use here is not really needed. I think for the purposes you
    need it for, a plain dictionary might do the job here, which would make
    the package more attractive.

    Using plain dicts would have required a lot of extra code to do things
    that can easily be done with sqlite, like searching based on different
    items and storing the information on disk.

    For example: Assuming a dict would be keyed on origin, then path, and
    finally cookie name, the whole dict would have to be examined to find
    expired cookies. Or can you suggest a directory structure that would
    make that easier?

    For storing cookies between application runs, it is of course possible
    to simply dump the dict as a string to file. But then checks and error
    handling would have to be added when reading it back.

    Since sqlite makes all of these tasks easier and it comes bundled with
    Tcl anyway, I chose to make use of it. You haven't yet persuaded me to
    reverse that decision.


    Schelte.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From saitology9@21:1/5 to Schelte on Sun Aug 20 19:53:43 2023
    On 8/20/2023 5:13 PM, Schelte wrote:

    The www package is happy to use the same version of sqlite3 that may be required by other parts of the application. It doesn't use any advanced features that require the latest sqlite3 version.

    I suppose that helps. Once something is deployed, it is quite hard to
    force frequent updates on users (and especially companies). It is
    harder even to say, hey, one of the packages we use has a new version
    with a bug fix, and we have to update the whole thing.


    Using plain dicts would have required a lot of extra code to do things
    that can easily be done with sqlite, like searching based on different
    items and storing the information on disk.

    Just processing the cookies is a big enough help already. I mean, the
    package could fetch the cookies, give it to the user when asked, and
    pass it along, with whatever else the user provides, on to the next
    request. I's say no need to worry about the long-term persistency. It
    becomes user responsibility. They can save it to a file and read it back
    easily on their own. I suspect many may not even bother.


    For example: Assuming a dict would be keyed on origin, then path, and
    finally cookie name, the whole dict would have to be examined to find
    expired cookies. Or can you suggest a directory structure that would
    make that easier?

    I don't know about the internals of the package or what the ideal use
    case is, so I am not sure how big of a problem this is. I'd assume that
    expired cookies would only be relevant when a resource from the same
    domain is requested again. Whenever I have used the http::geturl, it was
    just one request per domain. Also, the number of cookies per site is
    generally in single digits, so even a sequential search should be no
    problem.

    As for the general data structure, it may be better to use a simple list
    stored with the requested object, or even an array, indexed by "$origin,
    $path, $cookie-name". Expiration could be an integer containing seconds
    for time left.



    For storing cookies between application runs, it is of course possible
    to simply dump the dict as a string to file. But then checks and error handling would have to be added when reading it back.

    I am not sure it is a good idea, by default, to maintain state between
    repeated runs of the user program. This makes it sound like a
    full-fledged browser like FF or Chrome.



    Since sqlite makes all of these tasks easier and it comes bundled with
    Tcl anyway, I chose to make use of it. You haven't yet persuaded me to reverse that decision.

    I am sorry - that was not my intention. I thought I'd bring up a point
    that you may or may not have considered. Of course, it is your package
    after all.

    However, if you want to be persuaded :-), here is one: I believe your
    package is pure Tcl code. Sqlite dependency makes it a binary extension. Personally I try to avoid binary packages (notwithstanding the fact that
    it is sqlite :-) ).

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ted Nolan @21:1/5 to [email protected] on Mon Aug 21 02:28:50 2023
    In article <nnd$578bb030$57f092eb@2f63b97640b55fc0>,
    Schelte <[email protected]> wrote:
    On 18/08/2023 20:29, Ted Nolan <tednolan> wrote:
    I have just started seeing an ineresting problem with program I have been
    using to talk to a REST API for the past year or more. In particular
    sometime very recently the indicies in the "meta()" array populated
    after a geturl request have started, sometimes, being populated in
    lower case. It's not consistent, so I suspect my requests are being
    round-robined across different servers, but to give a particular
    example, I had been doing something like:

    if {[string match -nocase {application/json*} $meta(Content-Type)]} { >> whatever
    }

    and now I am finding that sometimes "meta(Content-Type)" does not exist, but >> "meta(content-type)" does. Googling around I'm seeing that apparently
    servers are allowed to do this, but the meta() array seems almost useless
    if this is going to happen. I can fix the program of course, I guess
    looking at the man page, I should probably just check "state(type)",
    instead. Should the "meta()" subscripts be lowercased in future
    versions, or should there be a warning on the man page not to use them
    other than for debugging?

    This is just one of several issues with the http package that prompted
    me to develop the www package >(https://chiselapp.com/user/schelte/repository/www)


    Schelte.


    Thanks! I'll save that link. I'm not sure if I'll be able to use it going forward; I'm already getting some pushback about using packages not in the distro package managers, and I've (I think) solved my issue, but it looks interesting!
    --
    columbiaclosings.com
    What's not in Columbia anymore..

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Keith Nash@21:1/5 to All on Mon Aug 21 20:30:36 2023
    Hi Ted,

    I hadn't noticed this incorrect information in the man page for Tcl 8.6.
    Also the 8.6 man page unhelpfully suggests using

    array set meta $state(meta)

    but this operation will lose some headers if there are multiple headers
    with the same name, and the case-sensitivity issue makes the array much
    less useful than the original list.

    The case-sensitivity issue is fixed in Tcl 8.7, and I believe the
    documentation in 8.7 is accurate. When upgrading from 8.6 to 8.7 users
    may need to alter their scripts to use the new commands and/or adjust
    how they use the "meta" internal state. Tcl 8.6 and 8.7 handle these
    issues differently.

    In http 2.9.x (Tcl 8.6) the header name is stored as a list element name
    in meta, which has the form "header value header value ...", and is
    exactly what was sent by the server. As you point out, the meaning of
    the header name is case-independent and the server can send any case it
    likes. This is inconvenient when writing scripts because every attempt
    to retrieve a header must do a case-insensitive search. As you point
    out the man page gives incorrect information.

    In http 2.10.x (Tcl 8.7) there is a change in both the API and the
    internals of http. The header names stored in meta are lower case. New commands are provided for retrieving header values:
    ::http::responseHeaders token ?headerName?
    ::http::responseHeaderValue token headerName
    The value to be supplied by the script as "headerName" is
    case-insensitive: the command converts it to lower case and then does a
    direct comparison with meta.

    It is possible to use http 2.10 with Tcl 8.6. As well as the module
    file http-*.tm and the man page http.n, it may be helpful to copy the tcl8.7/cookiejar0.2/ directory and the man pages cookiejar.n, idna.n
    from Tcl 8.7.

    Keith.

    On 18/08/2023 19:29, Ted Nolan <tednolan> wrote:
    I have just started seeing an ineresting problem with program I have been using to talk to a REST API for the past year or more. In particular sometime very recently the indicies in the "meta()" array populated
    after a geturl request have started, sometimes, being populated in
    lower case. It's not consistent, so I suspect my requests are being round-robined across different servers, but to give a particular
    example, I had been doing something like:

    if {[string match -nocase {application/json*} $meta(Content-Type)]} {
    whatever
    }

    and now I am finding that sometimes "meta(Content-Type)" does not exist, but "meta(content-type)" does. Googling around I'm seeing that apparently servers are allowed to do this, but the meta() array seems almost useless
    if this is going to happen. I can fix the program of course, I guess
    looking at the man page, I should probably just check "state(type)",
    instead. Should the "meta()" subscripts be lowercased in future
    versions, or should there be a warning on the man page not to use them
    other than for debugging?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ted Nolan @21:1/5 to [email protected] on Tue Aug 22 03:17:07 2023
    In article <[email protected]>,
    Keith Nash <[email protected]> wrote:

    Thanks for the info Keith!

    Hi Ted,

    I hadn't noticed this incorrect information in the man page for Tcl 8.6.
    Also the 8.6 man page unhelpfully suggests using

    array set meta $state(meta)


    Which is what I was doing & why :-)

    The case-sensitivity issue is fixed in Tcl 8.7, and I believe the >documentation in 8.7 is accurate. When upgrading from 8.6 to 8.7 users
    may need to alter their scripts to use the new commands and/or adjust
    how they use the "meta" internal state. Tcl 8.6 and 8.7 handle these
    issues differently.


    The changes you describe sound nice, and I look forward to seeing Tcl 8.7
    in our linux distro package managers! Unfortunately that may take a
    while as our mix of distros is quite diverse.

    Regards,
    Ted Nolan
    --
    columbiaclosings.com
    What's not in Columbia anymore..

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich@21:1/5 to [email protected] on Tue Aug 22 13:04:44 2023
    Ted Nolan <tednolan> <[email protected]> wrote:
    The changes you describe sound nice, and I look forward to seeing Tcl
    8.7 in our linux distro package managers! Unfortunately that may
    take a while as our mix of distros is quite diverse.

    Compiling Tcl/Tk on Linux is usually trivial, so you could compile it
    for yourself if you wanted it sooner than the packagers would provide.

    The most irritating complication comes about not by Tcl/Tk, but by
    distros that /hide/ the development packages behind a separately
    installed -devel package, resulting in the need to install several
    extra packages to get basic compiling to work in the first place.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Harald Oehlmann@21:1/5 to All on Tue Aug 22 16:59:01 2023
    Am 22.08.2023 um 05:17 schrieb Ted Nolan <tednolan>:
    In article <[email protected]>,
    Keith Nash <[email protected]> wrote:

    Thanks for the info Keith!

    Hi Ted,

    I hadn't noticed this incorrect information in the man page for Tcl 8.6.
    Also the 8.6 man page unhelpfully suggests using

    array set meta $state(meta)


    Which is what I was doing & why :-)

    The case-sensitivity issue is fixed in Tcl 8.7, and I believe the
    documentation in 8.7 is accurate. When upgrading from 8.6 to 8.7 users
    may need to alter their scripts to use the new commands and/or adjust
    how they use the "meta" internal state. Tcl 8.6 and 8.7 handle these
    issues differently.


    The changes you describe sound nice, and I look forward to seeing Tcl 8.7
    in our linux distro package managers! Unfortunately that may take a
    while as our mix of distros is quite diverse.

    Regards,
    Ted Nolan

    The http package from TCL is a script only package.
    By just grabbing the http and cookyjar packages from 8.7 and moving them
    into a (sub-)folder of your auto_path would make them automatically work
    in an 8.6 installation.

    So, no need to compile.
    Take care,
    Harald

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Schelte@21:1/5 to All on Wed Aug 23 20:08:13 2023
    On 21/08/2023 01:53, saitology9 wrote:
    I am not sure it is a good idea, by default, to maintain state between repeated runs of the user program.

    I agree. That's why it doesn't do it by default. But it provides the possibility, if you want it.

    Elsewhere in the thread, someone mentioned that 8.7 adds a cookiejar
    package for handling cookies for the http package. And guess what? It
    uses an sqlite database to store the cookies.


    Schelte.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alan Grunwald@21:1/5 to Schelte on Wed Aug 23 20:15:30 2023
    On 23/08/2023 19:08, Schelte wrote:
    On 21/08/2023 01:53, saitology9 wrote:
    I am not sure it is a good idea, by default, to maintain state between
    repeated runs of the user program.

    I agree. That's why it doesn't do it by default. But it provides the possibility, if you want it.

    Elsewhere in the thread, someone mentioned that 8.7 adds a cookiejar
    package for handling cookies for the http package. And guess what? It
    uses an sqlite database to store the cookies.


    Schelte.

    May I offer an observation and a suggestion that I fear is so obvious
    that it's already been considered and discarded?

    I assume that the packages being discussed open a memory-only sqlite3
    database by opening :memory:. This seems to me to risk interfering with
    an application that also wants an in-memory sqlite database.

    Would it not be better to use a disk-based database with published name
    and location that an application writer can choose to interfere with or, preferably, to keep well clear of?

    I appreciate that there will be some performance impact by making the
    database on-disk rather than in-memory but a) that will surely be
    minimal and b) it can be reduced by doing all the package database
    access within a database transaction that is never committed, making the database essentially in-memory anyway.

    Please be gentle as you explain why this is a rotten suggestion.

    Alan

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich@21:1/5 to Alan Grunwald on Wed Aug 23 21:35:16 2023
    Alan Grunwald <[email protected]> wrote:
    On 23/08/2023 19:08, Schelte wrote:
    On 21/08/2023 01:53, saitology9 wrote:
    I am not sure it is a good idea, by default, to maintain state between
    repeated runs of the user program.

    I agree. That's why it doesn't do it by default. But it provides the
    possibility, if you want it.

    Elsewhere in the thread, someone mentioned that 8.7 adds a cookiejar
    package for handling cookies for the http package. And guess what? It
    uses an sqlite database to store the cookies.


    Schelte.

    May I offer an observation and a suggestion that I fear is so obvious
    that it's already been considered and discarded?

    I assume that the packages being discussed open a memory-only sqlite3 database by opening :memory:. This seems to me to risk interfering
    with an application that also wants an in-memory sqlite database.
    [snip]
    Please be gentle as you explain why this is a rotten suggestion.

    It appears that each handle opened upon the :memory: pseudo file
    creates separate databases.

    Testing:

    $ rlwrap tclsh
    % package require sqlite3
    3.37.2
    % sqlite3 db1 :memory:
    % db1 eval {create table db1 (id integer);}
    % db1 eval {select * from db1;}
    % sqlite3 db2 :memory:
    % db2 eval {select * from db1;}
    no such table: db1
    % db2 eval {create table db2 (id integer);}
    % db1 eval {select * from db2;}
    no such table: db2
    %

    Handle db1 does not see the db2 table in db2, nor does handle db2 see
    the db1 table in db1.

    So opening :memory: would seem to be safe (to the extent that someone
    does not explicitly write code to go fish the 'cookie jar' db handle
    out of the loaded module).

    Now, whether this is /documented/ behavior or just a happy side effect
    I have not researched.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Schelte@21:1/5 to Rich on Wed Aug 23 23:58:48 2023
    On 23/08/2023 23:35, Rich wrote:
    So opening :memory: would seem to be safe (to the extent that someone
    does not explicitly write code to go fish the 'cookie jar' db handle out
    of the loaded module). Now, whether this is /documented/ behavior or
    just a happy side effect I have not researched.

    It is documented behavior. https://www.sqlite.org/inmemorydb.html says:
    "Every :memory: database is distinct from every other. So, opening two
    database connections each with the filename ":memory:" will create two independent in-memory databases."


    Schelte.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From saitology9@21:1/5 to Schelte on Wed Aug 23 21:29:06 2023
    On 8/23/2023 2:08 PM, Schelte wrote:

    Elsewhere in the thread, someone mentioned that 8.7 adds a cookiejar
    package for handling cookies for the http package. And guess what? It
    uses an sqlite database to store the cookies.


    So, it too is a binary package then. No problem there. But it doesn't
    change how I feel about binary extensions.

    It may be that we are beating the same poor dead horse here :-)

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From saitology9@21:1/5 to Alan Grunwald on Wed Aug 23 21:38:18 2023
    On 8/23/2023 3:15 PM, Alan Grunwald wrote:
    May I offer an observation and a suggestion that I fear is so obvious
    that it's already been considered and discarded?

    I assume that the packages being discussed open a memory-only sqlite3 database by opening :memory:. This seems to me to risk interfering with
    an application that also wants an in-memory sqlite database.


    You make a great point above. It seems to a non-issue but this is just
    another risk.


    Would it not be better to use a disk-based database with published name
    and location that an application writer can choose to interfere with or, preferably, to keep well clear of?


    Well, I believe the package (as well as sqlite) has options to make an in-memory database persist to a file. So at the end, there is no
    distinction, I suppose.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich@21:1/5 to [email protected] on Thu Aug 24 03:34:46 2023
    saitology9 <[email protected]> wrote:
    On 8/23/2023 3:15 PM, Alan Grunwald wrote:
    May I offer an observation and a suggestion that I fear is so obvious
    that it's already been considered and discarded?

    I assume that the packages being discussed open a memory-only sqlite3
    database by opening :memory:. This seems to me to risk interfering with
    an application that also wants an in-memory sqlite database.


    You make a great point above. It seems to a non-issue but this is just another risk.

    It turns out it is not. In message
    <nnd$36affa55$0b276155@764ea9c06e9f0d0b> is a quote from the Sqlite
    docs where it is documented that each DB handle open on the :memory:
    pseudo file results in a fully separate database.

    So there is no chance of accidental interference.

    The only interference possibility is deliberate interference where an
    author deliberately introspected the http module to extract the db
    handle name in order to access the cookie database.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From saitology9@21:1/5 to Rich on Thu Aug 24 00:01:58 2023
    On 8/23/2023 11:34 PM, Rich wrote:

    It turns out it is not. In message
    <nnd$36affa55$0b276155@764ea9c06e9f0d0b> is a quote from the Sqlite
    docs where it is documented that each DB handle open on the :memory:
    pseudo file results in a fully separate database.

    So there is no chance of accidental interference.

    The only interference possibility is deliberate interference where an
    author deliberately introspected the http module to extract the db
    handle name in order to access the cookie database.


    I am not so sure. Have you considered user script transactions (e.g.,
    looping over a bunch of url's) vs. the package transactions (i.e.,
    saving thee cookies)?

    From sqlite docs:

    "SQLite supports multiple simultaneous read transactions coming from
    **separate database connections**, possibly in separate threads or
    processes, **but only one simultaneous write transaction**."

    Perhaps you could write a small program to test it. Would be
    interesting to find out.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From saitology9@21:1/5 to Rich on Thu Aug 24 00:11:50 2023
    On 8/23/2023 11:31 PM, Rich wrote:

    Given that sqlite ships as a built in part of Tcl (and has starting
    with 8.6), calling a Tcl script only package that uses the built in sqlite
    a "binary extension" is a stretch.


    Well, sqlite is not loaded by default. To use it, you need to load it.
    So it fits the package or extension criterion. And it is binary. QED :-)

    In any case, it is totally OK for www to use sqlite. I don't know
    anything about the www package. I liked the brief paragraph that I read
    on its web page and I made a mental note to consider it in the future. I believe I have expressed this fact more than once. I really don't have
    anything else to say about it.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich@21:1/5 to [email protected] on Thu Aug 24 03:31:55 2023
    saitology9 <[email protected]> wrote:
    On 8/23/2023 2:08 PM, Schelte wrote:

    Elsewhere in the thread, someone mentioned that 8.7 adds a cookiejar
    package for handling cookies for the http package. And guess what?
    It uses an sqlite database to store the cookies.

    So, it too is a binary package then. No problem there. But it
    doesn't change how I feel about binary extensions.

    Given that sqlite ships as a built in part of Tcl (and has starting
    with 8.6), calling a Tcl script only package that uses the built in sqlite
    a "binary extension" is a stretch.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Schelte@21:1/5 to All on Thu Aug 24 10:58:03 2023
    On 24/08/2023 06:01, saitology9 wrote:
    I am not so sure. Have you considered user script transactions (e.g.,
    looping over a bunch of url's) vs. the package transactions (i.e.,
    saving thee cookies)?

    From sqlite docs:

    "SQLite supports multiple simultaneous read transactions coming from **separate database connections**, possibly in separate threads or
    processes, **but only one simultaneous write transaction**."

    This refers to multiple transactions on a single database. It is
    irrelevant here, because cookies are stored in their own database.

    It could only possibly become an issue if a script developer specifies a database file for permanent cookie storage that they also use for other purposes. If anyone decides to do that, it is up to them to make sure
    this design won't cause problems.


    Schelte.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich@21:1/5 to [email protected] on Thu Aug 24 13:36:16 2023
    saitology9 <[email protected]> wrote:
    On 8/23/2023 11:31 PM, Rich wrote:

    Given that sqlite ships as a built in part of Tcl (and has starting
    with 8.6), calling a Tcl script only package that uses the built in sqlite >> a "binary extension" is a stretch.


    Well, sqlite is not loaded by default. To use it, you need to load it.
    So it fits the package or extension criterion. And it is binary. QED :-)

    Yes, the sqlite extension is a binary extension.

    But the fact that sqlite itself is a binary extension does not make the
    www package itself a binary extension. Which is what it seemed your
    earlier comments were implying, that www was a "binary extension"
    because it used the built in sqlite extension.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich@21:1/5 to [email protected] on Thu Aug 24 13:33:09 2023
    saitology9 <[email protected]> wrote:
    On 8/23/2023 11:34 PM, Rich wrote:

    It turns out it is not. In message
    <nnd$36affa55$0b276155@764ea9c06e9f0d0b> is a quote from the Sqlite
    docs where it is documented that each DB handle open on the :memory:
    pseudo file results in a fully separate database.

    So there is no chance of accidental interference.

    The only interference possibility is deliberate interference where an
    author deliberately introspected the http module to extract the db
    handle name in order to access the cookie database.


    I am not so sure. Have you considered user script transactions (e.g.,
    looping over a bunch of url's) vs. the package transactions (i.e.,
    saving thee cookies)?

    From sqlite docs:

    "SQLite supports multiple simultaneous read transactions coming from **separate database connections**, possibly in separate threads or
    processes, **but only one simultaneous write transaction**."

    Perhaps you could write a small program to test it. Would be
    interesting to find out.

    The quote about separation is:

    It is documented behavior. https://www.sqlite.org/inmemorydb.html
    says: "Every :memory: database is distinct from every other. So,
    opening two database connections each with the filename ":memory:"
    will create two independent in-memory databases."

    The way 'distinct' is used in the docs indicates that two different
    openings of :memory: are identical to having done:

    sqlite3 db1 file1
    sqlite3 db2 file2

    Which also works perfectly fine.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alan Grunwald@21:1/5 to Schelte on Thu Aug 24 21:14:43 2023
    On 23/08/2023 22:58, Schelte wrote:
    On 23/08/2023 23:35, Rich wrote:
    So opening :memory: would seem to be safe (to the extent that someone
    does not explicitly write code to go fish the 'cookie jar' db handle
    out of the loaded module). Now, whether this is /documented/ behavior
    or just a happy side effect I have not researched.

    It is documented behavior. https://www.sqlite.org/inmemorydb.html says: "Every :memory: database is distinct from every other. So, opening two database connections each with the filename ":memory:" will create two independent in-memory databases."


    Schelte.

    Thank you. I hadn't realised that about SQLite. I am now even more
    impressed by it.

    PS Thank you everyone for your gentleness.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)