On Sun, 6 Apr 2025 13:02:29 +0300
=?UTF-8?Q?Martin=2D=C3=89ric_Racine?= <[email protected]>
wrote:
On Sun, 6 Apr 2025 10:00:45 +0300
=?UTF-8?Q?Martin=2D=C3=89ric_Racine?= <[email protected]>
wrote:
On Sat, 05 Apr 2025 22:56:46 +0300 =?utf-8?q?Martin-=C3=89ric_Racine?= <[email protected]> wrote:
Package: upgrade-reports
Severity: important
X-Debbugs-Cc: [email protected]
After upgrading an amd64 host from Bookworm to Trixie (2025-04-05 EEST), on reboot, the host gets a kernel panic with:
initramfs unpacking failed invalid magic at start of compressed archive
I just checked whether I can rescue the host using the Trixie mini.iso
on a USB stick:
Debian GNU/Linux 13 (trixie) amd64 - netboot mini.iso 20241227
I get the same kernel panic as soon as GRUB loads the kernel.
If I try with the Bookworm mini.iso, the kernel boots just fine and
gets me to the rescue menu.
Debian GNU/Linux 12 (bookworm) amd64 - netboot mini.iso 20230607+deb12u9
However, using the Bookworm rescue mode is a PITA, because it doesn't know how to mount brtfs sub-volumes (not even with Debian's default @rootfs) so even just chroot'ing myself in using the above recipe requires a LOT of command typing inside the rescue mode's shell.
As a further test, I've tried booting the following Trixie USB stick
in BIOS mode via the rescue menu:
Debian GNU/Linux trixie-DI-alpha1 "Trixie" - Official Alpha amd64
NETINST with firmware 20241230-11:26
This succesfully gets me to the rescue process and allows me to chroot
into the host after typing the long commandline series above.
Basically, a/b testing shows that Trixie kernels per-se work fine, but something fishy is going on with the grub-efi-amd64 in Trixie. The
same host was originally installed with a Bookworm mini.iso on USB and booted fine via EFI, so the problem really does seem to be the grub-efi-amd64 in Trixie.
Given the above, reassigned to bin:grub-efi-amd64.
On Sun, 6 Apr 2025 19:36:03 +0300
=?UTF-8?Q?Martin=2D=C3=89ric_Racine?= <[email protected]>
wrote:
On Sun, 6 Apr 2025 13:02:29 +0300
=?UTF-8?Q?Martin=2D=C3=89ric_Racine?= <[email protected]>
wrote:
On Sun, 6 Apr 2025 10:00:45 +0300
=?UTF-8?Q?Martin=2D=C3=89ric_Racine?= <[email protected]>
wrote:
On Sat, 05 Apr 2025 22:56:46 +0300 =?utf-8?q?Martin-=C3=89ric_Racine?= <[email protected]> wrote:
Package: upgrade-reports
Severity: important
X-Debbugs-Cc: [email protected]
After upgrading an amd64 host from Bookworm to Trixie (2025-04-05 EEST), on reboot, the host gets a kernel panic with:
initramfs unpacking failed invalid magic at start of compressed archive
I just checked whether I can rescue the host using the Trixie mini.iso on a USB stick:
Debian GNU/Linux 13 (trixie) amd64 - netboot mini.iso 20241227
I get the same kernel panic as soon as GRUB loads the kernel.
If I try with the Bookworm mini.iso, the kernel boots just fine and gets me to the rescue menu.
Debian GNU/Linux 12 (bookworm) amd64 - netboot mini.iso 20230607+deb12u9
However, using the Bookworm rescue mode is a PITA, because it doesn't know how to mount brtfs sub-volumes (not even with Debian's default @rootfs) so even just chroot'ing myself in using the above recipe requires a LOT of command typing inside the rescue mode's shell.
As a further test, I've tried booting the following Trixie USB stick
in BIOS mode via the rescue menu:
Debian GNU/Linux trixie-DI-alpha1 "Trixie" - Official Alpha amd64
NETINST with firmware 20241230-11:26
This succesfully gets me to the rescue process and allows me to chroot into the host after typing the long commandline series above.
Basically, a/b testing shows that Trixie kernels per-se work fine, but something fishy is going on with the grub-efi-amd64 in Trixie. The
same host was originally installed with a Bookworm mini.iso on USB and booted fine via EFI, so the problem really does seem to be the grub-efi-amd64 in Trixie.
Given the above, reassigned to bin:grub-efi-amd64.
And, after checking, bumping severity to Critical, since this bug
prevents common hardware (ASUS P8H61 motherboard via UEFI) from
booting.
On Sun, 13 Apr 2025 10:23:17 +0300
=?UTF-8?Q?Martin=2D=C3=89ric_Racine?= <[email protected]>
wrote:
On Mon, 7 Apr 2025 08:28:01 +0300
=?UTF-8?Q?Martin=2D=C3=89ric_Racine?= <[email protected]>
wrote:
On Sun, 6 Apr 2025 19:36:03 +0300
=?UTF-8?Q?Martin=2D=C3=89ric_Racine?= <[email protected]>
wrote:
On Sun, 6 Apr 2025 13:02:29 +0300
=?UTF-8?Q?Martin=2D=C3=89ric_Racine?= <[email protected]> wrote:
On Sun, 6 Apr 2025 10:00:45 +0300 =?UTF-8?Q?Martin=2D=C3=89ric_Racine?= <[email protected]> wrote:
On Sat, 05 Apr 2025 22:56:46 +0300 =?utf-8?q?Martin-=C3=89ric_Racine?=
<[email protected]> wrote:
Package: upgrade-reports
Severity: important
X-Debbugs-Cc: [email protected]
After upgrading an amd64 host from Bookworm to Trixie (2025-04-05 EEST), on reboot, the host gets a kernel panic with:
initramfs unpacking failed invalid magic at start of compressed archive
I just checked whether I can rescue the host using the Trixie mini.iso
on a USB stick:
Debian GNU/Linux 13 (trixie) amd64 - netboot mini.iso 20241227
I get the same kernel panic as soon as GRUB loads the kernel.
If I try with the Bookworm mini.iso, the kernel boots just fine and gets me to the rescue menu.
Debian GNU/Linux 12 (bookworm) amd64 - netboot mini.iso 20230607+deb12u9
However, using the Bookworm rescue mode is a PITA, because it doesn't
know how to mount brtfs sub-volumes (not even with Debian's default @rootfs) so even just chroot'ing myself in using the above recipe requires a LOT of command typing inside the rescue mode's shell.
As a further test, I've tried booting the following Trixie USB stick in BIOS mode via the rescue menu:
Debian GNU/Linux trixie-DI-alpha1 "Trixie" - Official Alpha amd64 NETINST with firmware 20241230-11:26
This succesfully gets me to the rescue process and allows me to chroot
into the host after typing the long commandline series above.
Basically, a/b testing shows that Trixie kernels per-se work fine, but
something fishy is going on with the grub-efi-amd64 in Trixie. The same host was originally installed with a Bookworm mini.iso on USB and
booted fine via EFI, so the problem really does seem to be the grub-efi-amd64 in Trixie.
Given the above, reassigned to bin:grub-efi-amd64.
And, after checking, bumping severity to Critical, since this bug
On 16/04/2025 at 13:38, Martin-Éric Racine wrote:
As the attached screenshot shows, 'fuseblk' seemingly doesn't know
anything about the 'subvol' option for btrfs, which is what causes the failure to mount root. As to whether this implies a GRUB or Linux
module, I wouldn't know.
The failure has nothing to do with fuse.
I guess the kernel tries to mount the root filesystem with fuseblk as a
last resort *after* the initramfs unpacking failure because it is the
only built-in driver, all other block device/filesystem drivers being
modules in the initramfs.
ke 16.4.2025 klo 15.41 Pascal Hambourg ([email protected]) kirjoitti:
On 16/04/2025 at 13:38, Martin-Éric Racine wrote:
As the attached screenshot shows, 'fuseblk' seemingly doesn't know
anything about the 'subvol' option for btrfs, which is what causes the
failure to mount root. As to whether this implies a GRUB or Linux
module, I wouldn't know.
The failure has nothing to do with fuse.
I guess the kernel tries to mount the root filesystem with fuseblk as a
last resort *after* the initramfs unpacking failure because it is the
only built-in driver, all other block device/filesystem drivers being
modules in the initramfs.
Shouldn't a built-in autofs handle that part?
Anyhow, that line about 'subvol' being an unrecognized option for
fuseblk is significant.
If you start grub shell and load the kernel and initramfs by hand, does
it show an error when loading the initramfs ?
No error.
On 16/04/2025 at 15:51, Martin-Éric Racine wrote:
ke 16.4.2025 klo 15.41 Pascal Hambourg ([email protected]) kirjoitti:
On 16/04/2025 at 13:38, Martin-Éric Racine wrote:
As the attached screenshot shows, 'fuseblk' seemingly doesn't know
anything about the 'subvol' option for btrfs, which is what causes the >>> failure to mount root. As to whether this implies a GRUB or Linux
module, I wouldn't know.
The failure has nothing to do with fuse.
I guess the kernel tries to mount the root filesystem with fuseblk as a
last resort *after* the initramfs unpacking failure because it is the
only built-in driver, all other block device/filesystem drivers being
modules in the initramfs.
Shouldn't a built-in autofs handle that part?
I am not sure what you mean by "autofs". AFAIK autofs is an on-demand
mount feature, so irrelevant for the root filesystem. The kernel image (vmlinuz) does not have built-in block/filesystem drivers and needs to
load the related modules from the initramfs in order to be able to mount
the root filesystem.
Anyhow, that line about 'subvol' being an unrecognized option for
fuseblk is significant.
I don't think so. fuseblk is the wrong filesystem type and the "subvol" option is intended for btrfs so it is not surprising that it is not recognized.
If you start grub shell and load the kernel and initramfs by hand, does
it show an error when loading the initramfs ?
No error.
There may be some silent data corruption when grub reads the initramfs.
Can you test with the initramfs on another filesystem type (fat, ext4) ?
Can you test with grub for BIOS/legacy boot ?
On 17/04/2025 at 06:01, Martin-Éric Racine wrote:
ke 16.4.2025 klo 20.40 Pascal Hambourg ([email protected]) kirjoitti:
There may be some silent data corruption when grub reads the initramfs.
Can you test with the initramfs on another filesystem type (fat, ext4) ? >> Can you test with grub for BIOS/legacy boot ?
See above. GRUB-EFI is clearly the culprit. Booting the Trixie d-i in rescue mode works in BIOS mode, but not in EFI mode.
d-i boots in BIOS mode with isolinux, not grub.
On Thu, 17 Apr 2025 10:13:15 +0300
=?UTF-8?Q?Martin=2D=C3=89ric_Racine?= <[email protected]>
wrote:
to 17.4.2025 klo 10.03 Pascal Hambourg ([email protected]) kirjoitti:
On 17/04/2025 at 06:01, Martin-Éric Racine wrote:
ke 16.4.2025 klo 20.40 Pascal Hambourg ([email protected]) kirjoitti:
There may be some silent data corruption when grub reads the initramfs.
Can you test with the initramfs on another filesystem type (fat, ext4) ?
Can you test with grub for BIOS/legacy boot ?
See above. GRUB-EFI is clearly the culprit. Booting the Trixie d-i in rescue mode works in BIOS mode, but not in EFI mode.
d-i boots in BIOS mode with isolinux, not grub.
This still points to GRUB as the culprit. AFAIK the Linux kernel
loaded in both modes is the same. The failure to load initrd appears
when booting the Trixie d-i via UEFI but not via BIOS. Both modes work
fine when booting using the Bookworm d-i. UEFI no longer does with the Trixie d-i.
As a further test, I booted into the host using the Bookworm rescue
mode, chrooted myself in, then purposely downgraded the host's grub*
to the Bookworm versions and APT-pinned them to 1000. The host boots
fine now.
$ dpkg -l | grep grub | awk '{print $2,$3}'
grub-common 2.06-13+deb12u1
grub-efi-amd64 2.06-13+deb12u1
grub-efi-amd64-bin 2.06-13+deb12u1
grub-efi-amd64-signed 1+2.06+13+deb12u1
grub-ipxe 1.21.1+git20250317.42a29d56+dfsg-2
grub2-common 2.06-13+deb12u1
$ uname -a
Linux p8h62 6.12.21-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.21-1 (2025-03-30) x86_64 GNU/Linux
$ ls /sys/firmware/efi/efivars/
(a very full directory indeed - I won't copy-paste it here)
Again, this clearly point to Trixie's GRUB-EFI as the culprit.
Meanwhile, my testing host (i386, GRUB-PC) boots fine:
$ dpkg -l | grep grub | awk '{print $2,$3}'
grub-common 2.12-7
grub-ipxe 1.21.1+git20250317.42a29d56+dfsg-2
grub-pc 2.12-7
grub-pc-bin 2.12-7
grub2-common 2.12-7
Further googling shows that users of Ubuntu derivatives started
experiencing similar issues on hosts with a btrfs rootfs about a year
ago. Given how Ubuntu developers happen to be the key contributors to
GRUB at Debian, I'd really hope to see a more proactive reaction from
them to this bug report.
The Debian changelog suggests what caused this:
Someone enabled compiling GRUB with FUSE3 to enhance QEMU support.
The problem with FUSE3 is that it only has very basic support for
btrfs, lacking among other things support for subvolumes. This is a
serious issue considering how widely spread btrfs has become and how debian-installer uses subvolumes by default whenever someone selects
brtfs as the filesystem type. Rendering hosts unbootable is a big
no-no. I would really hope the release team to step in on this one.
Martin-Éric
Hey Felix,
pe 18.4.2025 klo 19.39 Felix Zielcke ([email protected]) kirjoitti:
Am Freitag, dem 18.04.2025 um 19:10 +0300 schrieb Martin-Éric
Racine:
The Debian changelog suggests what caused this:
Someone enabled compiling GRUB with FUSE3 to enhance QEMU
support.
The problem with FUSE3 is that it only has very basic support for
btrfs, lacking among other things support for subvolumes. This is
a
serious issue considering how widely spread btrfs has become and
how
debian-installer uses subvolumes by default whenever someone
selects
brtfs as the filesystem type. Rendering hosts unbootable is a big
no-no. I would really hope the release team to step in on this
one.
FUSE support in GRUB 2 is only used for the grub-mount binary. And
not
at all in the actual booting code loaded from EFI/BIOS.
You could though test the fs drivers of it with grub-mount:
grub-mount /dev/sda2 /mnt (replace sda2 with your / partition)
sha1sum /mnt/@rootfs/boot/initrd.img-6.12.20-amd64
sha1sum /@rootfs/boot/initrd.img-6.12.20-amd64
both commands should output the same hash sum.
And yes, this is safe with a mounted fs because the GRUB fs code is
100% complete read-only.
Exactly how does the above address the issue I reported?
Martin-Éric
Am Freitag, dem 18.04.2025 um 21:17 +0300 schrieb Martin-Éric Racine:
Hey Felix,
pe 18.4.2025 klo 19.39 Felix Zielcke ([email protected]) kirjoitti:
Am Freitag, dem 18.04.2025 um 19:10 +0300 schrieb Martin-Éric
Racine:
The Debian changelog suggests what caused this:
Someone enabled compiling GRUB with FUSE3 to enhance QEMU
support.
The problem with FUSE3 is that it only has very basic support for btrfs, lacking among other things support for subvolumes. This is
a
serious issue considering how widely spread btrfs has become and
how
debian-installer uses subvolumes by default whenever someone
selects
brtfs as the filesystem type. Rendering hosts unbootable is a big no-no. I would really hope the release team to step in on this
one.
FUSE support in GRUB 2 is only used for the grub-mount binary. And
not
at all in the actual booting code loaded from EFI/BIOS.
You could though test the fs drivers of it with grub-mount:
grub-mount /dev/sda2 /mnt (replace sda2 with your / partition)
sha1sum /mnt/@rootfs/boot/initrd.img-6.12.20-amd64
sha1sum /@rootfs/boot/initrd.img-6.12.20-amd64
both commands should output the same hash sum.
And yes, this is safe with a mounted fs because the GRUB fs code is
100% complete read-only.
Exactly how does the above address the issue I reported?
Well, then we at least know that the bug is not in GRUB's btrfs
filesystem code
So your initramfs should be correctly read by it
Will the release team publish Trixie fully knowing that btrtfs hosts
will no longer be bootable via UEFI?
On Sun, Apr 13, 2025 at 10:23:17AM +0300, Martin-Éric Racine wrote:
Is there any way I can help the maintainers pinpoint the source of the problem?
This bug needs two things:
1) A full reproducer step list, starting from "I have nothing but an empty amd64 VM".
2) A bisect of GRUB between 2.06 and 2.12.
la 3.5.2025 klo 14.42 Chris Hofstaedtler ([email protected]) kirjoitti:
Control: tags -1 + moreinfo unreproducible
On Fri, May 02, 2025 at 08:14:37PM +0300, Martin-Éric Racine wrote:
pe 2.5.2025 klo 19.44 Chris Hofstaedtler ([email protected]) kirjoitti:
On Sun, Apr 13, 2025 at 10:23:17AM +0300, Martin-Éric Racine wrote:
Is there any way I can help the maintainers pinpoint the source of the problem?
This bug needs two things:
1) A full reproducer step list, starting from "I have nothing but an empty
amd64 VM".
There isn't much to reproduce:
1) Start with a fully functional host running Bookworm with GRUB-EFI
on a brtfs filesystem created using d-i/Bookworm's default @rootfs
subvolume name.
2) Change APT sources from Bookworm to Trixie.
3) Dist-upgrade.
4) Reboot.
5) Find the above kernel panic.
6) Using Bookworm d-i's rescue mode via EFI, APT pin and downgrade
grub* to Bookworm. All other packages remain at Trixie versions.
7) Reboot.
8) The host boots normally.
I've followed these steps today, and cannot reproduce the problem.
The upgraded VM boots successfully.
I beleive you. Again, googling this exact error mostly pulls up
reports of this failing on ASUS motherboards of various models.
la 19.4.2025 klo 12.43 Pascal Hambourg ([email protected]) kirjoitti:
On 19/04/2025 at 10:19, Martin-Éric Racine wrote:
Will the release team publish Trixie fully knowing that btrtfs hosts
will no longer be bootable via UEFI?
Not all btrfs hosts are affected. A fresh UEFI trixie btrfs virtual
machine boots fine on bookworm QEMU+OVMF. IIUC you wrote that trixie installer is affected too on your host, but it does not use btrfs to
boot so non-btrfs systems may be affected.
Point taken. Trixe d-i doesn't boot in UEFI on this host, which would
suggest that the filesystem type used for the root partition may well
be a red herring.
Point taken. Trixe d-i doesn't boot in UEFI on this host, which would
suggest that the filesystem type used for the root partition may well
be a red herring.
Trixie DI RC1 does boot via UEFI so there's hope. However, I have no
idea of what change makes RC1 succeed where December's Alpha didn't.
If anyone knows, please do tell.
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 715 |
| Nodes: | 16 (4 / 12) |
| Uptime: | 29:21:48 |
| Calls: | 12,108 |
| Calls today: | 8 |
| Files: | 15,006 |
| Messages: | 6,518,244 |