• Debian initramfs/initrd, was Re: stack smashing detected

    From Finn Thain@21:1/5 to Stan Johnson on Sun Feb 5 23:50:02 2023
    On Sun, 5 Feb 2023, Stan Johnson wrote:


    To save time, I recommend using QEMU and an up-to-date Debian/m68k SID virtual machine to produce the vmlinux and initrd files needed for use
    with Penguin on your slower machines.


    AFAIK, the initrd must be created on the system that is using the initrd
    (or an identical system, at least that was the response I received a
    while back when I was unsuccessfully trying to use the initrd that ships
    on the Debian install CD).

    If that was true, how could Debian ship a single initrd that works on
    Atari, Amiga, Mac etc.? They could not.

    If you boot a random Debian/m68k rootfs with a random Debian/m68k vmlinux/initrd combination, and if you need a kernal module at some point,
    then your random rootfs must contain the modules that match your random
    kernel binary (probably not going to work).

    If you never need to load a module, perhaps because the important ones all
    got loaded from the initrd, then all you need is a valid vmlinux/initrd combination and the rootfs is not relevant.

    When you need to generate a valid Debian/m68k vmlinux/initrd combination
    that is also current, then you'll need a Debian/m68k system that is
    current. The quickest way to get it is an emulator.

    When you need a small initrd, because of RAM limitations, you'll need to customize your initrd for your hardware using /etc/initramfs-tools (as mentioned).

    If you succeed, you'll get an initrd that is missing all the modules for
    all the hardware that you don't own, which saves RAM. It doesn't matter
    what system generates that initrd (could be a build farm).

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From John Paul Adrian Glaubitz@21:1/5 to Finn Thain on Mon Feb 6 00:20:01 2023
    On Mon, 2023-02-06 at 09:29 +1100, Finn Thain wrote:
    On Sun, 5 Feb 2023, Stan Johnson wrote:


    To save time, I recommend using QEMU and an up-to-date Debian/m68k SID virtual machine to produce the vmlinux and initrd files needed for use with Penguin on your slower machines.


    AFAIK, the initrd must be created on the system that is using the initrd (or an identical system, at least that was the response I received a
    while back when I was unsuccessfully trying to use the initrd that ships on the Debian install CD).

    If that was true, how could Debian ship a single initrd that works on
    Atari, Amiga, Mac etc.? They could not.

    You cannot use the initrd that's on the debian-installer medium because it boots directly into the installer. It's not a regular initrd.

    That's completely independent of the architecture/sub-architecture in use.

    When you need to generate a valid Debian/m68k vmlinux/initrd combination that is also current, then you'll need a Debian/m68k system that is
    current. The quickest way to get it is an emulator.

    Correct.

    Adrian


    --
    .''`. John Paul Adrian Glaubitz
    : :' : Debian Developer
    `. `' Physicist
    `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Geert Uytterhoeven@21:1/5 to [email protected] on Mon Feb 6 09:00:01 2023
    Hi Stan,

    On Mon, Feb 6, 2023 at 4:42 AM Stan Johnson <[email protected]> wrote:
    On an SE/30 with 128 MiB memory, the latest Debian SID kernel (vmlinux-6.1.0-2-m68k), using Debian SID modules, and with initrd-6.1.0-2-m68k built on the SE/30, hangs after the initial
    "ABCFGHIJK" (I tried it twice).

    If you get to "K", you're almost at the end of arch/m68k/kernel/head.S,
    and it is very likely the kernel C-code actually started.
    Do you get any output using "debug earlyprintk" on the kernel command
    line?

    Gr{oetje,eeting}s,

    Geert

    --
    Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

    In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
    -- Linus Torvalds

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Geert Uytterhoeven@21:1/5 to [email protected] on Mon Feb 6 21:40:01 2023
    Hi Stan,

    On Mon, Feb 6, 2023 at 9:31 PM Stan Johnson <[email protected]> wrote:
    On 2/6/23 12:52 AM, Geert Uytterhoeven wrote:
    On Mon, Feb 6, 2023 at 4:42 AM Stan Johnson <[email protected]> wrote:
    On an SE/30 with 128 MiB memory, the latest Debian SID kernel
    (vmlinux-6.1.0-2-m68k), using Debian SID modules, and with
    initrd-6.1.0-2-m68k built on the SE/30, hangs after the initial
    "ABCFGHIJK" (I tried it twice).

    If you get to "K", you're almost at the end of arch/m68k/kernel/head.S,
    and it is very likely the kernel C-code actually started.
    Do you get any output using "debug earlyprintk" on the kernel command
    line?
    ...

    Please see the attached serial console log, which all happened within an
    hour after I set the Penguin loose (I have Penguin-19 configured to use
    32768 K memory). There was nothing more after two additional hours.

    Thanks, so the kernel does start, but hangs later.
    Adding "initcall_debug" to the kernel command line may reveal more..

    Gr{oetje,eeting}s,

    Geert

    --
    Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

    In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
    -- Linus Torvalds

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Finn Thain@21:1/5 to Stan Johnson on Tue Feb 7 04:40:01 2023
    On Mon, 6 Feb 2023, Stan Johnson wrote:

    Thanks, so the kernel does start, but hangs later.
    Adding "initcall_debug" to the kernel command line may reveal more..
    ...

    Please see attached.

    ...

    [ 34.440000] calling key_proc_init+0x0/0x5e @ 1
    [ 34.470000] initcall key_proc_init+0x0/0x5e returned 0 after 1307 usecs
    [ 34.500000] calling asymmetric_key_init+0x0/0x10 @ 1
    [ 34.520000] Key type asymmetric registered
    [ 34.540000] initcall asymmetric_key_init+0x0/0x10 returned 0 after 22481 usecs
    [ 34.570000] calling x509_key_init+0x0/0x16 @ 1
    [ 34.580000] Asymmetric key parser 'x509' registered
    [ 34.600000] initcall x509_key_init+0x0/0x16 returned 0 after 14116 usecs
    [ 34.620000] calling crypto_kdf108_init+0x0/0x13a @ 1


    You could try 'initcall_blacklist=key_proc_init' or initcall_blacklist=x509_key_init' etc.

    This kind of thing has come up before. https://lists.debian.org/debian-68k/2019/06/msg00020.html https://lists.debian.org/debian-68k/2019/06/msg00066.html

    These systems are too slow for needless key generation so a bug report
    may be needed.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Brad Boyer@21:1/5 to Stan Johnson on Tue Feb 7 21:10:01 2023
    On Tue, Feb 07, 2023 at 12:31:17AM -0700, Stan Johnson wrote:
    On 2/6/23 8:25 PM, Finn Thain wrote:

    These systems are too slow for needless key generation so a bug report
    may be needed.


    The Mac IIci (25 MHz) is only about 50% faster that the SE/30 (16 MHz).
    The Debian kernel booted on the IIci, though it took somewhere between
    30 and 60 minutes. If it were just slowness, shouldn't the SE/30 be
    expected to boot in about 60 to 120 minutes (I let it run for 3 hours)?


    Is the optional L2 cache card installed in your IIci? If so, that
    could give a really big performance boost on something like this. I
    suspect the key generation routines and data don't fit in the tiny L1
    cache in a 68030, but would fit easily in the L2 cache. Any older
    models like the SE/30 weren't designed to be able to have an L2 cache.
    This difference alone could probably double or triple the performance
    even without the clock speed change.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Brad Boyer@21:1/5 to Stan Johnson on Tue Feb 7 23:20:01 2023
    On Tue, Feb 07, 2023 at 12:41:52PM -0700, Stan Johnson wrote:
    Yes, I do have the L2 cache card installed in the IIci.

    If you think it would be useful, I don't mind letting the SE/30 run
    overnight to see if it eventually boots.

    I don't know if it would be useful in any practical sense, but I do
    admit to being curious how many hours it will take. Overnight should
    be enough. That would confirm that it's just extremely slow.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Finn Thain@21:1/5 to Stan Johnson on Wed Feb 8 00:40:01 2023
    On Tue, 7 Feb 2023, Stan Johnson wrote:

    If you think it would be useful, I don't mind letting the SE/30 run
    overnight to see if it eventually boots.


    Preventing pointless key generation would be beneficial for all Macs,
    Amigas, Ataris, emulators etc. and measuring the performance of one model
    of Mac versus that of another model seems a bit irrelevant to me.

    Moreover, you've shown that your kernel builds produce stack smashing
    errors whereas Debian's build does not. To resolve the problem with your builds, why not begin by eliminating some of the differences between your
    build and Debian's?

    I suggest you should adopt the current Debian SID build environment and toolchain use it to build mainline Linux (stock v6.1) using QEMU.

    If you use your .config and if you still get stack smashing errors then
    you can use the script I wrote to bisect the differences between your
    .config and Debian's .config.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From John Klos@21:1/5 to All on Wed Feb 8 20:30:02 2023
    If anyone knows of a 68030 emulator (maybe Basilisk?) that can boot
    Linux, then I might be able to use that for faster testing.

    I've played around with NetBSD on FS-UAE. I'd use it more, except for the
    fact that the emulation of the Commodore 2065 ethernet card gives very
    flakey networking.

    The emulation was configured with an m68030 & m68882, and I've heard it
    can run Linux, too.

    This difference alone could probably double or triple the performance
    even without the clock speed change.

    Even though the external cache can do burst transfers to the CPU's cache,
    this definitely wouldn't double or triple performance - most of the time
    the CPU isn't waiting for data from memory. It'd be faster, but not significantly.

    John

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Eero Tamminen@21:1/5 to Stan Johnson on Wed Feb 8 20:40:01 2023
    Hi,

    On 8.2.2023 19.39, Stan Johnson wrote:
    The stack smashing appears to be intermittent. And it doesn't show up
    while booting the kernel; it only shows up while sysvinit scripts are
    running (I haven't tested using systemd, since that would be too painful
    on any 68030 slower than about 40 MHz). It takes too long to boot slow systems using Debian's kernel to run repeated tests, and QEMU only
    emulates 68040, so it appears to be necessary to test on real hardware.
    It's not my goal to get my config files closer to Debian's, anyway. My
    goal is to have the fewest number of config files for different groups
    of systems.

    If anyone knows of a 68030 emulator (maybe Basilisk?) that can boot
    Linux, then I might be able to use that for faster testing.

    On quick search I was not able to find out whether Basilisk II (or mini
    vMac) emulate both 030 MMU and CPU i/d-cache. If they don't, they may
    not be accurate enough to find issues like this.


    In case the issue is only 030, not Mac specific, both WinUAE (Amiga) and
    Hatari (Atari) emulators support 68030 + MMU + cache emulation, and can
    boot Linux.

    Hatari CPU core is based on one from WinUAE.)


    Apparently WinUAE has not had problems running Linux, but in Hatari,
    enabling 030 _cache_ emulation breaks Linux boot when it reaches user
    space (kernel boot works fine).

    I'm not sure whether latter is related to the issue you are seeing on Mac.

    (There are some differences e.g. in how Amiga and Atari handle CPU
    exceptions and how MMU is used, which may explain differences in
    behavior. I have no idea whether Mac is closer to Amiga or Atari in
    this respect.)


    For details on using m68k Linux with Hatari, see: https://hatari.tuxfamily.org/doc/m68k-linux.txt


    - Eero

    PS. Debugger in Hatari emulator can provide you with backtraces for
    Linux kernel side, and profile where kernel time goes.

    For a full boot, callgraphs will be so large that you'll probably need
    to throw away 99% fo the data though, to make them readable. For
    details, see:
    https://hatari.tuxfamily.org/doc/debugger.html#Profiling

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Finn Thain@21:1/5 to Stan Johnson on Wed Feb 8 23:30:02 2023
    On Wed, 8 Feb 2023, Stan Johnson wrote:

    On 2/7/23 4:20 PM, Finn Thain wrote:

    On Tue, 7 Feb 2023, Stan Johnson wrote:
    ...
    Preventing pointless key generation would be beneficial for all Macs, Amigas, Ataris, emulators etc. and measuring the performance of one model of Mac versus that of another model seems a bit irrelevant to me.


    Sure, but unless Debian unsupported is willing to manage config files
    for the various systems, then it's not likely to happen.

    It's easy to refute that. Just read my message from 2 days ago in this
    very thread where I pointed to a different Debian kernel key generation
    issue that got fixed.


    Moreover, you've shown that your kernel builds produce stack smashing errors whereas Debian's build does not. To resolve the problem with your builds, why not begin by eliminating some of the differences between your build and Debian's?

    The stack smashing appears to be intermittent. And it doesn't show up
    while booting the kernel; it only shows up while sysvinit scripts are
    running (I haven't tested using systemd, since that would be too painful
    on any 68030 slower than about 40 MHz).

    No-one is asking for systemd tests.

    It takes too long to boot slow systems using Debian's kernel to run
    repeated tests ...

    If your m68k machines are too slow, why do you care about stack smashing
    errors at all?


    If the stack smashing is caused by a kernel bug that is hidden by
    Debian's choice of config options, then it would still be useful to
    identify the bug. If there is something missing from my config files
    that is causing the problem, then that would still be a kernel bug in
    its sanity checking of options.

    This is not about sanity checking.

    Anyway, if you follow the steps I gave, we all get to learn something
    about the cause of the stack smashing error -- if that's what you want.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From John Paul Adrian Glaubitz@21:1/5 to Finn Thain on Thu Feb 9 10:30:02 2023
    On Wed, 2023-02-08 at 10:20 +1100, Finn Thain wrote:
    Preventing pointless key generation would be beneficial for all Macs, Amigas, Ataris, emulators etc. and measuring the performance of one model
    of Mac versus that of another model seems a bit irrelevant to me.

    Feel free to suggest a kernel configuration update for the m68k Debian kernel or even a new kernel flavor such as a »-light« one.

    I am planning to create a new »-virt« kernel anyway. Maybe we can make the main
    m68k kernel flavor leaner and only enable all the »slow« stuff on the »-virt«
    flavor.

    Adrian

    --
    .''`. John Paul Adrian Glaubitz
    : :' : Debian Developer
    `. `' Physicist
    `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From John Paul Adrian Glaubitz@21:1/5 to Stan Johnson on Thu Feb 9 10:30:02 2023
    On Wed, 2023-02-08 at 10:39 -0700, Stan Johnson wrote:
    On 2/7/23 4:20 PM, Finn Thain wrote:

    On Tue, 7 Feb 2023, Stan Johnson wrote:
    ...
    Preventing pointless key generation would be beneficial for all Macs, Amigas, Ataris, emulators etc. and measuring the performance of one model of Mac versus that of another model seems a bit irrelevant to me.


    Sure, but unless Debian unsupported

    What do you mean by »Debian unsupported«? You mean »Debian Ports«?

    is willing to manage config files for the various systems, then it's
    not likely to happen. I currently use separate config files for the
    following Macs, to build kernels with no initrd, no modules, and only
    minimal network and video support:

    1) 68030 8 MiB, no network (PB-170)
    2) 68030 >8 MiB (SE/30, IIci, IIfx, Centris LC III, etc.)
    3) 68040 (Centris 650, PB 550c, etc.)

    Debian supports different kernel configuration per architecture, the concept
    is called »flavors«. We could certainly add a »-lean« or »-light« flavor for smaller systems.

    If the stack smashing is caused by a kernel bug that is hidden by
    Debian's choice of config options, then it would still be useful to
    identify the bug. If there is something missing from my config files
    that is causing the problem, then that would still be a kernel bug in
    its sanity checking of options. Your script will be helpful if it
    becomes necessary to identify specific offending options.

    FWIW, I haven't seen this issue on my Amiga although I haven't run a dist-upgrade for a while now. I guess, I am going to do that in the
    near future.

    Adrian

    --
    .''`. John Paul Adrian Glaubitz
    : :' : Debian Developer
    `. `' Physicist
    `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)