Hi Daniel,
On Mon, Apr 28, 2025 at 05:20:31PM +0200, Daniel Baumann wrote:
On 4/28/25 14:58, Helmut Grohne wrote:
I am sorry to tell you that the brittle /usr-move mitigations broke
again.
no worries, I'm honestly so lucky that you're able and willing to help, I'll gladly wait for your patch.
Well, it was me who got you into this misery as I claimed that
duplicating diversions would allow us to move all the files from / to
/usr. How foolish of me.
Evidently, I am unable to fully grasp the complexity involved. In the
hope of not screwing up again, I started writing more extensive tests
than last time.
Testing
~~~~~~~
You find a testcase.sh and a Makefile.test attached. These are based on
my earlier work in #1059534. Compared to back then, I expanded the test
cases significantly. The original method spelled out cases as a way of installing or removing packages before or after upgrading to trixie.
I've now taken this down to the dpkg level driving unpacks and
configurations directly in order to exercise more control. In
particular, I am also testing the wicked case of temporarily removing
zutils as part of the upgrade and in doing so, I can reproduce the
reported problem. Let me quote the test matrix from Makefile.test as
it's important that this matrix is exhaustive.
unpgzip-confgzip-unpzutils-confzutils-rmzutils
unpgzip-confgzip-unpzutils-rmzutils
unpgzip-unpzutils-confgzip-confzutils-rmzutils
unpgzip-unpzutils-confgzip-rmzutils
unpgzip-unpzutils-confzutils-confgzip-rmzutils
unpgzip-unpzutils-confzutils-rmzutils-confgzip
unpgzip-unpzutils-rmzutils-confgzip
unpzutils-unpgzip-confgzip-confzutils-rmzutils
unpzutils-unpgzip-confzutils-confgzip-rmzutils
unpzutils-unpgzip-rmzutils-confgzip
unpzutils-confzutils-unpgzip-confgzip-rmzutils
unpzutils-confzutils-rmzutils
unpzutils-rmzutils-unpgzip-confgzip
withzutils-deinstzutils-unpgzip-rmzutils-confgzip
withzutils-deinstzutils-unpgzip-rmzutils-unpzutils-confgzip-confzutils-rmzutils
withzutils-deinstzutils-unpgzip-rmzutils-unpzutils-confzutils-confgzip-rmzutils
withzutils-unpzutils-unpgzip-confgzip-confzutils-rmzutils
withzutils-unpzutils-unpgzip-confzutils-confgzip-rmzutils
withzutils-unpzutils-unpgzip-rmzutils-confgzip
withzutils-unpzutils-confzutils-unpgzip-confgzip-rmzutils
We always start with a bookworm installation. As gzip is essential,
that's always included. zutils is installed if the case starts with "withzutils". Then the sequence of events is executed. "unp*" means
unpacking the relevant package (from trixie). Likewise "conf*" and "rm*"
refer to configuring and removing the package. The failure case involves scheduling a package for removal and that's what "deinst*" does. If you
can think of any valid interaction that's not listed here, please tell.
For instance, we see no withzutils-unpgzip-*, because trixie's gzip
Conflicts with bookworm's zutils and therefore we have to upgrade
zutils (unpzutils), remove zutils (skip withzutils) or schedule
deinstallation of zutils (deinstzutils) before unpacking trixie's gzip.
Fixing
~~~~~~
The actual changes are significant. For one thing, I argued that zutils
should Breaks gzip to ensure ordering of postinst. Not sure how I got
there as there is no postinst. Meanwhile, that Breaks combined with
gzip's Conflicts causes apt to invoke that wicked code path of
temporarily uninstalling zutils that presently is broken. However, the
preinst argues that wrongly renaming is ok, because gzip will be
upgraded and fix that. Once dropping that Breaks, we may no longer
wrongly rename.
That leads us to preinst changes. How we have to rename depends on
whether gzip installs its tools in /bin or /usr/bin as those diversions
have differing targets beyond aliasing. One option would be to
repeatedly dpkg-query -S those files, but each such invocation takes
0.2s even on fast CPUs and a hot cache. Instead, I propose examining
the gzip version to compute the intended location.
Then, an initial installation may be an upgrade performed by temporary
removal. In other words, gzip may have diverted its tools. We need to
actively undo that diversion.
As mentioned earlier, we should not be using --rename, so I'm changing
that to --no-rename and as a result I have to do the renaming myself.
The original location simply is the result of dpkg-divert --truename as
the removal of gzip's diversion also uses --no-rename. The destination
location depends on whether gzip installed aliased or canonical (i.e.
the gzip version).
While looking at the code, I noticed that the upgrade path was
implemented with --no-rename as well, but not doing any rename while it
really should. I'm adding that missing rename, but I also figured why
that didn't cause any problems. Due to gzip's Conflicts that branch
really should be dead code.
Last but not least, I'm being more strict on certain conditions and have preinst abort when it is faced with unexpected situations.
Retrospective
~~~~~~~~~~~~~
How did we get into this last failure? The problem existed in the
initial move as the test suite did not cover the relevant case. It was
driven by apt and at that time apt did not exercise the temporary
removal. The addition of Breaks via #1092737 caused apt to choose the
temporary removal path and that's why it is now practically broken.
I hope to find someone who can review this work before we apply it.
Helmut
#!/bin/shdie() { echo "error: $*" 1>&2 exit 1}diverted="zcat zcmp zdiff zegrep zfgrep zgrep"examine_diversions() { diversions_found= for suffix in gzip gzip.usr-is-merged usr-is-merged; do if test -e "$1/usr/bin/$2.$suffix" -o -h "$1/usr/bin/
$2.$suffix"; then if test "$suffix" = usr-is-merged && test -e "$1/usr/bin/$2" -o -h "$1/usr/bin/$2"; then # gzip's self-diversion should cause the files to be absent suffix=bad-usr-is-merged fi diversions_found="${diversions_found:+$
diversions_found }$suffix" fi done}if test -n "$MMDEBSTRAP_HOOK"; then if dpkg-query --root "$1" -f '${db:Status-Status}\n' -W zutils | grep -q '^\(installed\|unpacked\)$'; then for cmd in $diverted; do examine_diversions "$1" "$cmd" case "
$diversions_found" in gzip|gzip.usr-is-merged) ;; *) ls -l "$1/usr/bin/z"* PAGER="cat" dpkg --root "$1" -l gzip zutils die "diversion check failed for $cmd: $diversions_found" ;; esac done else for cmd in $diverted;
do examine_diversions "$1" "$cmd" case "$diversions_found" in ""|usr-is-merged) ;; *) ls -l "$1/usr/bin/z"* PAGER="cat" dpkg --root "$1" -l gzip zutils die "non-diversion check failed for $cmd" ;; esac done fi
exit 0fibasedist=bookwormnextdist=sidmirror=
http://deb.debian.org/debianactions="$1-"set -- \ --variant=apt \ "$basedist" \ /dev/null \ "$mirror" \ --customize-hook="upload zutils_1.14-3.1_amd64.deb /zutils.deb" \ --customize-hook="sed -i -
e 's/$basedist/$nextdist/' \"\$1/etc/apt/sources.list\"" \ --chrooted-customize-hook="apt-get update && apt-get -y install libc6 && apt-get download gzip zutils" \ --customize-hook="$0"while test -n "$actions"; do next="${actions%%-*}" actions="${
actions#*-}" case "$next" in withzutils) set -- "$@" --include=zutils ;; unpgzip) set -- "$@" --chrooted-customize-hook="dpkg --unpack --auto-deconfigure /gzip_*.deb" --customize-hook="$0" ;; unpzutils) set -- "$@" --chrooted-
customize-hook="dpkg --unpack --auto-deconfigure /zutils.deb" --customize-hook="$0" ;; conf*) set -- "$@" --chrooted-customize-hook="dpkg --configure ${next#conf}" --customize-hook="$0" ;; rmzutils) set -- "$@" --chrooted-customize-hook="
dpkg --remove zutils" --customize-hook="$0" ;; deinstzutils) set -- "$@" --chrooted-customize-hook="echo zutils deinstall | dpkg --set-selections" ;; *) echo "unknown action $next" exit 1 ;; esacdoneecho "mmdebstrap $*"exec auto-
apt-proxy mmdebstrap "$@"
TESTS= \
unpgzip-confgzip-unpzutils-confzutils-rmzutils \
unpgzip-confgzip-unpzutils-rmzutils \
unpgzip-unpzutils-confgzip-confzutils-rmzutils \
unpgzip-unpzutils-confgzip-rmzutils \
unpgzip-unpzutils-confzutils-confgzip-rmzutils \
unpgzip-unpzutils-confzutils-rmzutils-confgzip \
unpgzip-unpzutils-rmzutils-confgzip \
unpzutils-unpgzip-confgzip-confzutils-rmzutils \
unpzutils-unpgzip-confzutils-confgzip-rmzutils \
unpzutils-unpgzip-rmzutils-confgzip \
unpzutils-confzutils-unpgzip-confgzip-rmzutils \
unpzutils-confzutils-rmzutils \
unpzutils-rmzutils-unpgzip-confgzip \
withzutils-deinstzutils-unpgzip-rmzutils-confgzip \
withzutils-deinstzutils-unpgzip-rmzutils-unpzutils-confgzip-confzutils-rmzutils \
withzutils-deinstzutils-unpgzip-rmzutils-unpzutils-confzutils-confgzip-rmzutils \
withzutils-unpzutils-unpgzip-confgzip-confzutils-rmzutils \
withzutils-unpzutils-unpgzip-confzutils-confgzip-rmzutils \
withzutils-unpzutils-unpgzip-rmzutils-confgzip \
withzutils-unpzutils-confzutils-unpgzip-confgzip-rmzutils \
all: $(foreach t,$(TESTS),testout/$(t))
testout/%:
./testcase.sh "$*" >"$@" 2>&1; echo $$? >> "$@"
diff --git a/debian/control b/debian/control
index 0020d21..31a2a1a 100644
--- a/debian/control
+++ b/debian/control
@@ -18,11 +18,6 @@ Architecture: any
Depends:
${misc:Depends},
${shlibs:Depends},
-Breaks:
-# We must ensure that gzip is upgraded before zutils.postinst runs. As it is -# essential, Breaks is sufficient here and the janitor may propose dropping
-# this relation eventually.
- gzip (<< 1.12-1.1~),
Suggests:
bzip2,
lzip,
diff --git a/debian/zutils.preinst b/debian/zutils.preinst
index 0b242e6..3512868 100755
--- a/debian/zutils.preinst
+++ b/debian/zutils.preinst
@@ -4,14 +4,45 @@ set -e
# DEP17 M18: Duplicate diversion in aliased location /bin.
+die() {
+ printf '%s, cannot proceed\n' "$*" 1>&2
+ exit 1
+}
+
case "${1}" in
install)
+ # Situations considered:
+ # * bookworm or earlier gzip is fully installed.
+ # -> No diversions, /bin/${FILE}
+ # * trixie or later gzip is unpacked.
+ # -> No diversions, /usr/bin/${FILE}
+ # * trixie or later gzip has been unpacked while bookworm's
+ # zutils was inst