Hi,

I wanted to bring fakeroot bugs #1023286 and #1030638 to the attention of a wider audience because even though I filed these bugs, I do not see myself finding the time to track down and correct their consequence for all the source package builds that happened since their introduction with the upload of glibc 2.34.

Essentially, fakeroot uses LD_PRELOAD to "fake" being able to change file ownership as if the process were run as the root user. If a new version of glibc gets uploaded that uses different functions to do the same thing, this mechanism breaks down and fakeroot needs to be adjusted to support these new functions as well. Most of the breakage (except for mipsel) was done as part of #1023286 but I discovered another problem and reported as #1030638 and there is no fix for it yet.

The problem affects 32 bit architectures as glibc chooses different functions to perform stat() functionality on those architectures, so the problem effects armel, armhf, i386 and mipsel. Those architectures also not among the most used once which explains why not much fuzz was created about this yet (at least not as far as I've heard).

By now the fakeroot package carries tests that check for these problems and you can see how fakeroot FTBFS on the architectures affected by this problem:

https://buildd.debian.org/status/package.php?p=fakeroot

For the affected architectures, package builds that change ownership of the files shipped in *.deb packages will remain at root:root instead of being set to the user and group that should own them. I found two examples for this problem: mutt and bsdutils

$ dpkg-deb --fsys-tarfile mutt_2.2.9-1_amd64.deb | tar --numeric-owner-tv | grep -v 0/0
-rwxr-sr-x 0/8 14584 2022-11-13 18:01 ./usr/bin/mutt_dotlock
$ dpkg-deb --fsys-tarfile mutt_2.2.9-1_i386.deb | tar --numeric-owner -tv | grep /usr/bin/mutt_dotlock
-rwxr-sr-x 0/0 13804 2022-11-13 18:01 ./usr/bin/mutt_dotlock

$ dpkg-deb --fsys-tarfile bsdutils_1%3a2.38.1-4_amd64.deb | tar --numeric-owner -tv | grep -v 0/0
-rwxr-sr-x 0/5 39224 2022-11-25 16:19 ./usr/bin/wall
$ dpkg-deb --fsys-tarfile bsdutils_1%3a2.38.1-4_i386.deb| tar --numeric-owner -tv | grep /usr/bin/wall
-rwxr-sr-x 0/0 38436 2022-11-25 16:19 ./usr/bin/wall

/usr/bin/mutt_dotlock should be owned by group "mail" and /usr/bin/wall by group "tty" but this is only the case for the amd64 version of those two packages. On i386, they are owned by the root group.

So fakeroot should get fixed and the affected source packages should be rebuilt. I lack the time to organize either, so here I am writing you this mail.

Maybe somebody else finds this interesting enough to invest their time into this. :)

Thanks!

cheers, josch

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

El 9/2/23 a las 15:37, Johannes Schauer Marin Rodrigues escribió:

I wanted to bring fakeroot bugs #1023286 and #1030638 to the attention of a wider audience because even though I filed these bugs,

Thanks for bringing this up!

Can you confirm if all this is correct?

- Only packages uploaded after 2022-08-07 (when glibc 2.34 hit unstable)
are potentially affected.

- Packages with "Rules-Requires-Root: no" are never affected.

- No intervention from individual maintainers is required for fixing this,
as we already have a binNMU mechanism which we already use for transitions.

- A minor observation: Only packages which use dh_fixperms with -X (or --exclude)
are apparently affected. Those which instead do chmod/chown after dh_fixperms are apparently not affected, at least after the first fakeroot bug (2022-11) was fixed.

Thanks.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

Quoting Santiago Vila (2023-02-09 17:32:08)

El 9/2/23 a las 15:37, Johannes Schauer Marin Rodrigues escribió:

I wanted to bring fakeroot bugs #1023286 and #1030638 to the attention of a wider audience because even though I filed these bugs,

Thanks for bringing this up!

Can you confirm if all this is correct?

- Only packages uploaded after 2022-08-07 (when glibc 2.34 hit unstable) are potentially affected.

Correct. 2022-08-07 is a lower bound.

Or in a bit more detail: only packages rebuilt after toolchain packages have been rebuilt with glibc 2.34 can be affected. For example all problems I observed so far came from coreutils (cp, chown, chgrp) which was only rebuilt with glibc 2.34 when 9.1-1 was uploaded in October 2022. But of course there could be other software that modifies ownership information during builds that was rebuilt with glibc 2.34 earlier than that and that I just do not know about. So 2022-08-07 is a lower bound but not necessarily the largest lower bound.

- Packages with "Rules-Requires-Root: no" are never affected.

Only packages built inside fakeroot are affected. As far as I understand it, fakeroot is not used by dpkg-buildpackage if Rules-Requires-Root is set to "no".

- No intervention from individual maintainers is required for fixing this, as we already have a binNMU mechanism which we already use for transitions.

Once fakeroot is fixed, binNMUs can be used to fix packages, yes. Without the fakeroot fix in place, individual maintainers could do things to fix their packages on the affected architectures but I do not think doing so is a good idea.

- A minor observation: Only packages which use dh_fixperms with -X (or --exclude) are apparently affected. Those which instead do chmod/chown after dh_fixperms are apparently not affected, at least after the first fakeroot bug (2022-11) was fixed.

I do not understand what makes you think that only packages using dh_fixperms -X are affected? I think what makes the two packages that I found fail to have correct permissions is that they both use dh_install which in turn uses 'cp -a' which is broken under fakeroot on our 32bit architectures right now. I patched mutt and added an 'ls -lha' in execute_before_dh_builddeb to show the problem:

$ ls -lha debian/tmp/usr/bin/mutt_dotlock debian/mutt/usr/bin/mutt_dotlock -rwxr-sr-x 1 root root 9.6K Feb 9 22:06 debian/mutt/usr/bin/mutt_dotlock -rwxr-sr-x 1 root mail 28K Feb 9 22:05 debian/tmp/usr/bin/mutt_dotlock

So the chgrp call in Makefile.am worked correctly and set the group owner to "mail" but after dh_install moved mutt_dotlock from debian/tmp/ to debian/mutt/ using 'cp -a' (if I'm reading the code correctly) the group ownership information is lost.

Thanks!

cheers, josch

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Fri, 10 Feb 2023 at 03:18:16 +0100, Johannes Schauer Marin Rodrigues wrote:

Quoting Santiago Vila (2023-02-09 17:32:08)

- No intervention from individual maintainers is required for fixing this, as
we already have a binNMU mechanism which we already use for transitions.

Once fakeroot is fixed, binNMUs can be used to fix packages, yes. Without the fakeroot fix in place, individual maintainers could do things to fix their packages on the affected architectures but I do not think doing so is a good idea.

There is one thing that maintainers can do to fix their packages on the affected architectures that I think *might* be a good idea: if their
package builds correctly with Rules-Require-Root: no, they could add that field, resulting in fakeroot not being used.

The safe way to do this (helped by all the work that has gone into making
our builds more reproducible-by-default) is:

- build the package as-is, and copy the resulting binaries as a reference
- add Rules-Require-Root: no
- build the package again in the same environment
- compare the two sets of binaries with diffoscope
- if there are no differences other than the .buildinfo, then
Rules-Require-Root: no is OK to use
- if there are differences, investigate them, and do not upload if unsure

If we weren't so close to soft freeze, I would be saying maintainers
of affected packages should at least try this, because it permanently
avoids fakeroot-related issues for that package; but for the minority
of packages where it's wrong to use Rules-Require-Root, that could lead
to regressions, which makes me less confident about that advice.

smcv

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

El 10/2/23 a las 3:18, Johannes Schauer Marin Rodrigues escribió:

I do not understand what makes you think that only packages using dh_fixperms -X are affected? I think what makes the two packages that I found fail to have
correct permissions is that they both use dh_install which in turn uses 'cp -a'
which is broken under fakeroot on our 32bit architectures right now. I patched
mutt and added an 'ls -lha' in execute_before_dh_builddeb to show the problem:

$ ls -lha debian/tmp/usr/bin/mutt_dotlock debian/mutt/usr/bin/mutt_dotlock -rwxr-sr-x 1 root root 9.6K Feb 9 22:06 debian/mutt/usr/bin/mutt_dotlock -rwxr-sr-x 1 root mail 28K Feb 9 22:05 debian/tmp/usr/bin/mutt_dotlock

So the chgrp call in Makefile.am worked correctly and set the group owner to "mail" but after dh_install moved mutt_dotlock from debian/tmp/ to debian/mutt/
using 'cp -a' (if I'm reading the code correctly) the group ownership information is lost.

I mean: There are mainly two ways to ship files not root:root inside a package.

One way is to accept the special permissions/ownerships resulting from dh_install
and then avoid resetting them in dh_fixperms using -X or --exclude.
This is from mutt's debian/rules:

override_dh_fixperms:
dh_fixperms --exclude usr/bin/mutt_dotlock

The other way is not to care about the permissions from dh_install and set them after dh_fixperms no matter what. This is from procmail's debian/rules:

override_dh_fixperms:
dh_fixperms
chgrp mail debian/procmail/usr/bin/procmail
chgrp mail debian/procmail/usr/bin/lockfile
chmod 6755 debian/procmail/usr/bin/procmail
chmod 2755 debian/procmail/usr/bin/lockfile

The minor observation is that packages which set permissions in the second
way which were uploaded after the first fakeroot bug was fixed seem
not to be affected:

dpkg -c procmail_3.24-1_i386.deb | grep -v root/root
-rwxr-sr-x root/mail 26292 2023-01-05 22:35 ./usr/bin/lockfile
-rwsr-sr-x root/mail 112668 2023-01-05 22:35 ./usr/bin/procmail

Thanks.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Quoting Santiago Vila (2023-02-10 11:12:09)

El 10/2/23 a las 3:18, Johannes Schauer Marin Rodrigues escribió:

I do not understand what makes you think that only packages using dh_fixperms
-X are affected? I think what makes the two packages that I found fail to have
correct permissions is that they both use dh_install which in turn uses 'cp -a'
which is broken under fakeroot on our 32bit architectures right now. I patched
mutt and added an 'ls -lha' in execute_before_dh_builddeb to show the problem:

$ ls -lha debian/tmp/usr/bin/mutt_dotlock debian/mutt/usr/bin/mutt_dotlock -rwxr-sr-x 1 root root 9.6K Feb 9 22:06 debian/mutt/usr/bin/mutt_dotlock -rwxr-sr-x 1 root mail 28K Feb 9 22:05 debian/tmp/usr/bin/mutt_dotlock

So the chgrp call in Makefile.am worked correctly and set the group owner to
"mail" but after dh_install moved mutt_dotlock from debian/tmp/ to debian/mutt/
using 'cp -a' (if I'm reading the code correctly) the group ownership information is lost.

I mean: There are mainly two ways to ship files not root:root inside a package.

One way is to accept the special permissions/ownerships resulting from dh_install
and then avoid resetting them in dh_fixperms using -X or --exclude.
This is from mutt's debian/rules:

override_dh_fixperms:
dh_fixperms --exclude usr/bin/mutt_dotlock

The other way is not to care about the permissions from dh_install and set them
after dh_fixperms no matter what. This is from procmail's debian/rules:

override_dh_fixperms:
dh_fixperms
chgrp mail debian/procmail/usr/bin/procmail
chgrp mail debian/procmail/usr/bin/lockfile
chmod 6755 debian/procmail/usr/bin/procmail
chmod 2755 debian/procmail/usr/bin/lockfile

The minor observation is that packages which set permissions in the second way which were uploaded after the first fakeroot bug was fixed seem
not to be affected:

dpkg -c procmail_3.24-1_i386.deb | grep -v root/root
-rwxr-sr-x root/mail 26292 2023-01-05 22:35 ./usr/bin/lockfile
-rwsr-sr-x root/mail 112668 2023-01-05 22:35 ./usr/bin/procmail

Firstly, when talking about "after the first fakeroot bug was fixed", remember that while it was fixed for i386 it was not fixed for mipsel. So all timestamps we use in this discussion depend on the architecture we are talking about.

Secondly, this is still not about dh_fixperms so much as it is about dh_install being called *before* the chgrp and chmod calls. You could add these calls in other dh overrides as long as those are executed after dh_install has run. For example you could place them into execute_before_dh_builddeb and achieve the same effect. But yes, since the fundamental difference in these two styles of how dh_fixperms is overridden makes the group owneership either depend on 'cp -a' (which is broken) or on another manual chgrp (which works) you could call it like that, I guess.

I didn't want to go into this in my last mail for similar reasons as mentioned by smcv in their reply but since you are now explicitly asking about it: it's true that yes, while not required, individual maintainers *could* do something about this problem even without fakeroot being fixed: add a manual chgrp call into their d/rules. As long as that chgrp call happens *after* dh_install (either in execute_after_dh_fixperms or in execute_before_dh_builddeb, for example) and *on* the files installed by dh_install, it will correct the missing ownership information due to 'cp -a' usage in dh_install.

This will of course just paper over the issue without fixing the underlying cause. It's what I called "not a good idea" in my last mail.

Thanks!

cheers, josch

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

El 10/2/23 a las 3:18, Johannes Schauer Marin Rodrigues escribió:

So the chgrp call in Makefile.am worked correctly and set the group owner to "mail" but after dh_install moved mutt_dotlock from debian/tmp/ to debian/mutt/
using 'cp -a' (if I'm reading the code correctly) the group ownership information is lost.

Ok, I agree. To clarify even more: My minor observation was not to
show how individual maintainers could fix this in their own packages
(as it's much better that this is done via NMUs in a centralized way)
but more something like "don't panic, the number of affected packages
could be smaller than we think".

Thanks a lot.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

"Johannes" == Johannes Schauer Marin Rodrigues <[email protected]> writes:

Johannes> This will of course just paper over the issue without
Johannes> fixing the underlying cause. It's what I called "not a
Johannes> good idea" in my last mail.

I like the explicit chgrp style because it explicitly documents all the permission variations between dh_fixperms's defaults in debian/rules
rather than split between debian/rules and the upstream build system.
If you as a maintainer think like I do and believe that's actually an
improve to your package,
"papering over" fakeroot bugs but happening to improve things in other
ways
seems like a fine thing to do.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Simon McVittie:

On Fri, 10 Feb 2023 at 03:18:16 +0100, Johannes Schauer Marin Rodrigues wrote:

Quoting Santiago Vila (2023-02-09 17:32:08)

- No intervention from individual maintainers is required for fixing this, as
we already have a binNMU mechanism which we already use for transitions.

Once fakeroot is fixed, binNMUs can be used to fix packages, yes. Without the
fakeroot fix in place, individual maintainers could do things to fix their >> packages on the affected architectures but I do not think doing so is a good >> idea.

There is one thing that maintainers can do to fix their packages on the affected architectures that I think *might* be a good idea: if their
package builds correctly with Rules-Require-Root: no, they could add that field, resulting in fakeroot not being used.

[...]

smcv

Packages that need static non-root ownership cannot do that at the
moment using debhelper / dpkg. These are in turn the most likely
packages to exhibit this problem that triggered this discussion.

For everything else, you can pretty much always migrate to "Rules-Requires-Root: no". It is "just" a question of:

1) Stop the accidental root usage in d/rules. E.g., remove -o root
-g root passed to install and left over chown calls.
2) Convince the upstream build system to stop using root during
installation in the rare cases they do that.

Example from sudo: https://salsa.debian.org/sudo-team/sudo/-/merge_requests/13/diffs?commit_id=fa2a3a3ce37eb356b79ce31838e8b415a7dc31d2

It is not very difficult to do. However, it does take human time and
effort, which is a scares resource.


But the moment you see a non "root/root" line in the data.tar listing,
it is checkmate and game-over. I think we may be able to provide better
debian package tooling for the next release that can solve the static
ownership problem, but not the human time/effort part.

Thanks,
~Niels

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)