1. Forum
  2. Usenet
  3. COMP.OS.OS2.PROGRAMMER.MI

  • DosSubAllocMem/DosSubFreeMem

    From Iron Spring Software@21:1/5 to All on Mon Apr 4 14:56:12 2022
    Everyone probably knows this, but FWIW here's an interesting problem
    where I ran out of heap on OS/2 for an application that worked for Linux.

    I think the heap is being checkerboarded with a mix of large and and
    small allocations. I pulled out some large allocations (one or more
    chunks of 32K) and used DosAllocMem and DosFreeMem for these, and
    everything worked. I assume this is because the OS/2 heap is
    preallocated while the Linux heap just grows.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Steven Levine@21:1/5 to [email protected] on Tue Apr 5 15:08:07 2022
    On Mon, 4 Apr 2022 21:56:12 UTC, Iron Spring Software
    <[email protected]> wrote:

    Hi Peter,

    small allocations. I pulled out some large allocations (one or more
    chunks of 32K) and used DosAllocMem and DosFreeMem for these, and
    everything worked. I assume this is because the OS/2 heap is
    preallocated while the Linux heap just grows.

    Not really. It's related to how address space is handled by OS/2.
    You don't mention if the runtime for your application supports
    allocating from upper memory. The lack of this capability can cause
    an app to run out of heap space pretty quickly if the has large-ish
    memory needs.

    The kLIBC and OpenWatcom runtimes can allocate heap space from upper
    memory. The VAC runtimes cannot.

    One thing many developers don't understand is that DosAllocMem
    allocates address space in 64K multiples. This is why well written
    runtimes allocate heap chunks in 64K multiples. This avoids wasting
    address space.

    There are a few ancient runtimes that have a fixed heap size. The
    Modula/2 runtime comes to mind. However, this is an exception. Most
    runtimes will allow the heap to expand until the process runs out of
    address space.

    Steven

    --
    ---------------------------------------------------------------------
    Steven Levine <[email protected]>
    DIY/ArcaOS/Warp etc. www.scoug.com www.arcanoae.com www.warpcave.com ---------------------------------------------------------------------

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Marcel Mueller@21:1/5 to All on Sun Apr 17 13:43:02 2022
    Am 04.04.22 um 23:56 schrieb Iron Spring Software:
    Everyone probably knows this, but FWIW here's an interesting problem
    where I ran out of heap on OS/2 for an application that worked for Linux.

    An application using DosSubAllocMem likely /never/ works on Linux. ;-)

    I think the heap is being checkerboarded with a mix of large and and
    small allocations. I pulled out some large allocations (one or more
    chunks of 32K) and used DosAllocMem and DosFreeMem for these, and
    everything worked. I assume this is because the OS/2 heap is
    preallocated while the Linux heap just grows.

    No, you simply fragmeted the address space. There are some patterns that
    force this. In this case the memory requirement may raise with O(n²).

    Normally, I recommend to prefer the C library allocators over using the
    kernel funktions. They are not intended to be used heavily.
    The glibc allocator performs quite well with millions of allocations and
    frees.

    If you really want to write your own allocator, this can be a hard work.
    And it is always a trade off between speed and fill factor.

    AFAIR DosSub... is simply a linked list with not further optimizations.
    If you want to deal with this functions anyway you could use different
    pools for different object sizes - small, medium, large. And for very
    large allocations you could allocate directly from the kernel. However,
    keep in mind that the latter might cause the TLB of your CPU to overflow
    which can cause a significant performance impact. So I do not recommend
    it unless you have really large chunks of e.g. 1MB.


    Marcel

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Peter Flass@21:1/5 to Marcel Mueller on Sun Apr 17 07:04:17 2022
    Marcel Mueller <[email protected]> wrote:
    Am 04.04.22 um 23:56 schrieb Iron Spring Software:
    Everyone probably knows this, but FWIW here's an interesting problem
    where I ran out of heap on OS/2 for an application that worked for Linux.

    An application using DosSubAllocMem likely /never/ works on Linux. ;-)

    Iron Spring PL/I started out as an OS/2 application. It ‘s pure PL/I, and doesn’t use an C code, so I had to port an allocator for the Linux version.
    I could probably move it to OS/2 if there are no other options.


    I think the heap is being checkerboarded with a mix of large and and
    small allocations. I pulled out some large allocations (one or more
    chunks of 32K) and used DosAllocMem and DosFreeMem for these, and
    everything worked. I assume this is because the OS/2 heap is
    preallocated while the Linux heap just grows.

    No, you simply fragmeted the address space. There are some patterns that force this. In this case the memory requirement may raise with O(n²).

    Same thing in different words. This goes back to the beginning of the
    compiler many years ago. I had no idea how much memory would be required
    for the parse table of a large program, so QandD was a a linked list of 32K blocks. There are better ways to handle this. The compiler works fine, but
    I reused the parser logic for the preprocessor, so there’s a different pattern of memory use that breaks the system.


    Normally, I recommend to prefer the C library allocators over using the kernel funktions. They are not intended to be used heavily.
    The glibc allocator performs quite well with millions of allocations and frees.


    See above. I don’t want any dependencies on the C library.

    If you really want to write your own allocator, this can be a hard work.
    And it is always a trade off between speed and fill factor.

    AFAIR DosSub... is simply a linked list with not further optimizations.
    If you want to deal with this functions anyway you could use different
    pools for different object sizes - small, medium, large. And for very
    large allocations you could allocate directly from the kernel. However,
    keep in mind that the latter might cause the TLB of your CPU to overflow which can cause a significant performance impact. So I do not recommend
    it unless you have really large chunks of e.g. 1MB.

    Thanks. I tried looking in the debug handbooks, etc., but couldn’t find any detail on DosSub.. I miss having source to look at. I guess I never
    stressed it before. As I said, it was QandD that always just worked, but
    it’s very inefficient and could probably be improved with minimal effort.



    Marcel




    --
    Pete

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Steven Levine@21:1/5 to All on Sun Apr 17 12:18:12 2022
    On Sun, 17 Apr 2022 14:04:17 UTC, Peter Flass <[email protected]>
    wrote:

    Hi Peter,


    Thanks. I tried looking in the debug handbooks, etc., but couldnt find any detail on DosSub.. I miss having source to look at. I guess I never
    stressed it before. As I said, it was QandD that always just worked, but
    its very inefficient and could probably be improved with minimal effort.

    Switch to allocating 64K blocks with DosAllocMem and you will run out
    of memory much less frequently. See my previous post.

    Steven



    --
    ---------------------------------------------------------------------
    Steven Levine <[email protected]>
    DIY/ArcaOS/Warp etc. www.scoug.com www.arcanoae.com www.warpcave.com ---------------------------------------------------------------------

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)