• Cross libextutils-pkgconfig-perl (was: Bug#1078010: libnet-z3950-zoom-p

    From gregor herrmann@21:1/5 to Helmut Grohne on Thu Aug 29 19:20:01 2024
    On Wed, 07 Aug 2024 18:29:50 +0200, Helmut Grohne wrote:

    Hi Gregor,

    Hi Helmut,

    thanks for taking the time to write this long and detailed email!

    On Tue, Aug 06, 2024 at 06:27:19PM +0200, gregor herrmann wrote:
    I played around a bit:
    - crosscompiled the package as is from amd64->i386 (with pbuilder's
    --host-arch) and it failed
    - created a patch similar to [0] using ExtUtils::PkgConfig [1]
    I believe our goal should be to somehow make that patch of yours work, because it is a natural way to write it and I expect other upstreams to
    do it that way. To me, this looks like the better solution even though
    it does not work right now. If you use that, you don't fix the FTCBFS,
    but you push the problem elsewhere and it no longer is
    libnet-z3950-zoom-perl that needs changes to fix it. ;)

    I already "fixed" libnet-z3950-zoom-perl by applying your patch but
    that shouldn't prevent us from taking a look at the underlying issue
    (hence To and Subject adjusted).

    Hm, does libextutils-pkgconfig-perl (arch:all) need "Multi-Arch: foreign"?
    That is the right way of looking at it.

    Thanks :)

    My answer here is maybe.

    Hm :)

    It depends on how you define it. We want
    ExtUtils::PkgConfig->cflags to yield flags for the "host" architecture,
    but what "host" means from a libextutils-pkgconfig-perl view is not
    entirely clear. It could be that we communicate the architecture to libextutils-pkgconfig-perl via its dependency and then it certainly
    would not be architecture-independent as we'd expect a libextutils-pkgconfig-perl:amd64 to behave differently from a libextutils-pkgconfig-perl:i386 one. The other way is that you somehow
    pass the architecture at runtime via some variable (explicit or
    implicit, argument or global or environment). In that way, the
    architecture is part of your input and we expect it to yield the same
    flags if given the same architecture and then the answer would be yes.

    The latter sounds more attractive to me.

    The other way to look at it is looking at its dependencies. It depends
    on pkg-config, which is an architecture-dependent package (because each pkg-config:$arch provides the $triplet-pkg-config entry point). If libextutils-pkgconfig-perl uses this entry point, it must be its package architecture that defines what "host" is as it forwards this
    architecture on its dependency. Otherwise, it can only call into
    pkg-config via /usr/bin/pkg-config, but then it gets the results for the build architecture, which isn't what we wanted.
    So without further modifications, we cannot just mark it Multi-Arch:
    foreign (and that also is the reason why I didn't file a bug asking for
    that yet).

    Ack.
    So if ExtUtils::PkgConfig calls /usr/bin/$prefix-pkgconf (with
    $prefix derived from the host architecture), we should be good with "Multi-Arch: foreign"?

    Any time we want to use libextutils-pkgconfig-perl in a package build,
    we need to ensure that our installation set contains a pkg-config for
    the host architecture. No matter how we do it, as long as it remains Architecture: all, it technically has no way of requesting the host architecture's pkg-config via its own dependencies, so the only way to
    keep it Architecture: all is to state that any package that declares a
    build dependency on libextutils-pkgconfig-perl must also declare a build dependency on pkg-config.

    Hm, probably doesn't scale and is prone to errors.

    The other way is converting it to
    Architecture: any + Multi-Arch: same (this conversion is often dubbed
    "the multiarch interpreter workaround for the multiarch interpreter problem"). Once doing this a regular build dependency (which is
    implicitly constrained to the host architecture) gets its architecture constraint passed through to the pkg-config dependency and stuff might
    work.

    Also not very elegant, and also error-prone, as you write:

    I note that this also poses a very unusual use of the architecture qualification of the libextutils-pkgconfig-perl module. For most other
    perl modules, the expectation of an architecture-qualified dependency is
    that you can load the module (and any extension modules that it
    transitively depends on) into a perl interpreter of said architecture.
    With libextutils-pkgconfig-perl, there are no extension modules (outside
    core perl) involved, so this does not matter. Rather the above
    interprets the architecture qualification as a means of requesting
    package configuration for the qualified architecture. Such an
    inconsistency is prone to biting us later, so maybe the route of pushing
    the need to depend on pkg-config down to the consumer is the better
    choice in the end.


    Note that all of this only looks at the packaging metadata only. I
    haven't yet implied anything about the implementation of libextutils-pkgconfig-perl that also likely needs to change to support
    such use.

    Ack.

    The bottom line roughly is "it's difficult". I hope we can agree with
    that.

    Totally.

    If I put ExtUtils::PkgConfig manually into ./ExtUtils/PkgConfig.pm
    and run the Makefile.PL call from above manually with an additional
    -I. I get:
    […]
    This likely is due to libextutils-pkgconfig-perl not prepending the
    toolchain prefix to pkg-config and thus using the build architecture
    (likely amd64) one.

    *nod*

    That's the other piece of the puzzle, we need to
    enhance ExtUtils::PkgConfig to somehow pick up the toolchain prefix from
    the perl Config.pm and we pass -I/usr/lib/i386-linux-gnu/perl/cross-config-5.38.2 to supply a config matching the host architecture. To make matters worse, the toolchain
    prefix isn't readily available in there. I suppose the best we could do
    is examine cc and strip "-gcc", but that's a hack at best. Possibly perl needs to record this value or it needs to be passed in some other way.

    That's a question for Niko and Dom, if we can get this variable into
    %Config.

    The bottom line roughly is "it's difficult". I hope we can agree with
    that.

    Yes, but it doesn't look that different to my naïve eyes:
    - All that ExtUtils::PkgConfig does is calling `pkg-config'
    - If we need to change it to call `/usr/bin/$prefix-pkg-config',
    we need the $prefix.
    - If we can't get it from %Config directly, we could abuse
    $Config{cc}, as you write; not elegant but well.
    - I wonder, and maybe this is too trivial to be true, if we couldn't
    just use `dpkg-architecture -qDEB_HOST_GNU_TYPE'?

    And in the end ExtUtils::PkgConfig just calls `pkg-config`, so any
    prefix would need to be found and added there. Alright, sorry for the detour :)
    Thank you for the detour. I think you're spot-on. If you want to further
    this adventure, I'm happy to continue.

    I think I'm happy to continue as well.

    Making ExtUtils::PkgConfig just
    work would fix a pile of packages and make for fewer patches and thus
    less maintenance cost. I'm not looking for quick solutions with
    long-term maintenance costs, but for good solutions with low permanent
    costs.

    Ack.


    Cheers,
    gregor

    --
    .''`. https://info.comodo.priv.at -- Debian Developer https://www.debian.org
    : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D 85FA BB3A 6801 8649 AA06
    `. `' Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
    `-

    -----BEGIN PGP SIGNATURE-----

    iQKTBAEBCgB9FiEE0eExbpOnYKgQTYX6uzpoAYZJqgYFAmbQpxhfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEQx RTEzMTZFOTNBNzYwQTgxMDREODVGQUJCM0E2ODAxODY0OUFBMDYACgkQuzpoAYZJ qgbdkw//d0ULUpfrRFjbAbZsbcABI4vAKMcQCqNErCfxPEwzTmjIdxXKZOOwOppa 4mNC2hp1k0EAxleFTTHDsW0IwIcwU4HNspuCIUINbX5c1bPkXGViiOJCx1a4hIMx n0GZ2ZPOhhAI34p5Bjh6YsnK2rx8fGHxzvoC3s6QZLM8fu0f2U3KtAGRwL+de2IQ dhWePyvGP1qyW9SBYRsgsFkW0m9dLuS+Lca8ifNL7DgHaDBOYBTUJyXplVQyWD10 hgr3zyauhOruPl6lpr5sY2H0wK4psRz95/GXa9PWnE0DDsVmsC/uPrwKC2tPha1x a8wtTuh9u2m3z7W6/zF0lq9ArmPGEst+5j5CkODuXW7QyWpf3+cLgsYMIboQxpoQ gxZMB7iQ7vAfI5IG0cxR0eLrGi6Cu67kgnEW1Z7ICfDaaUe23h9i5+/JNGwaRRa1 LZmeUsuw8cvDR7dSWs4JWj40lGTX/T7KorKn++K9H8mOZQZHCasdEDoWYmWNdqVV
    oY34l+yc
  • From Helmut Grohne@21:1/5 to gregor herrmann on Thu Aug 29 22:10:01 2024
    Hi Gregor,

    On Thu, Aug 29, 2024 at 06:51:35PM +0200, gregor herrmann wrote:
    thanks for taking the time to write this long and detailed email!

    Thanks for having taken the time to read such a long email.

    Ack.
    So if ExtUtils::PkgConfig calls /usr/bin/$prefix-pkgconf (with
    $prefix derived from the host architecture), we should be good with "Multi-Arch: foreign"?

    The implication of Multi-Arch: foreign would be that users have to
    additionally depend on pkg-config. Your reply to this proposition was:

    Hm, probably doesn't scale and is prone to errors.



    That's the other piece of the puzzle, we need to
    enhance ExtUtils::PkgConfig to somehow pick up the toolchain prefix from the perl Config.pm and we pass -I/usr/lib/i386-linux-gnu/perl/cross-config-5.38.2 to supply a config matching the host architecture. To make matters worse, the toolchain
    prefix isn't readily available in there. I suppose the best we could do
    is examine cc and strip "-gcc", but that's a hack at best. Possibly perl needs to record this value or it needs to be passed in some other way.

    That's a question for Niko and Dom, if we can get this variable into
    %Config.

    Indeed, getting the GNU triplet into the cross Config.pm is something technically solvable with reasonable effort. A very common way to name
    it is $CROSS_COMPILE and when doing so it would have a trailing dash.
    For instance CROSS_COMPILE=aarch64-linux-gnu- and using it would work
    like ${CROSS_COMPILE}pkg-config such that it also works when the
    variable is empty or unset in the native case.

    Yes, but it doesn't look that different to my na�ve eyes:
    - All that ExtUtils::PkgConfig does is calling `pkg-config'
    - If we need to change it to call `/usr/bin/$prefix-pkg-config',
    we need the $prefix.
    - If we can't get it from %Config directly, we could abuse
    $Config{cc}, as you write; not elegant but well.
    - I wonder, and maybe this is too trivial to be true, if we couldn't
    just use `dpkg-architecture -qDEB_HOST_GNU_TYPE'?

    Ideally, we look at this from two sides simultaneously. Thus far, our
    focus has been how to achieve this from a Debian packaging point of
    view. The other angle is from an upstream point of view supporting cross compilation in multiple Linux distributions (such as Yocto,
    OpenEmbedded, PtxDist, Alpine, ...). Would you want one mechanism to communicate the architecture for each distribution or would you want one mechanism that works for all of them in the same way? Also consider that
    people may use Debian for cross compiling their own software without
    using dpkg-buildpackage. Long story short, upstream code should not call
    into dpkg-architecture nor inspect DEB_HOST_* environment variables.


    I think I'm happy to continue as well.

    Yeah. I just do not happen to see the solution that obviously has no
    downsides.

    I also see a spectrum of modifications to the module implementation that
    would change the equation.
    A. Reimplement pkg-config entirely in Perl avoiding the need to depend
    on pkg-config.
    B. Reimplement the search path configuration (i.e. compute a suitable
    PKG_CONFIG_LIBDIR depending on the architecture) and pass this
    environment variable to pkg-config (not triplet-pkg-config). Beware
    though that the variable contains paths containing
    DEB_HOST_MULTIARCH and that may differ from DEB_HOST_GNU_TYPE.
    C. Stop working with pkg-config and require pkgconf. Then you may
    instead call pkgconf --personality=$DEB_HOST_GNU_TYPE ... rather
    than using the triplet-prefixed form.

    All of these approaches have in common that we avoid requiring pkg-config:$DEB_HOST_ARCH and can work with pkgconf-bin, which happens
    to be M-A:foreign and thus allows our module to also be M-A:foreign.

    Do you think any of A/B/C would be upstreamable? Note that all of them
    would benefit other cross compiling Linux distributions even though most
    of them don't cross Perl extensions to the extend Debian does.

    Helmut

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From gregor herrmann@21:1/5 to Helmut Grohne on Sun Jan 12 01:10:01 2025
    On Thu, 29 Aug 2024 21:55:47 +0200, Helmut Grohne wrote:

    Hi Gregor,

    Hi Helmut!

    On Thu, Aug 29, 2024 at 06:51:35PM +0200, gregor herrmann wrote:
    thanks for taking the time to write this long and detailed email!
    Thanks for having taken the time to read such a long email.

    Thanks again for writing up all this information, and sorry for my
    very late reply.

    There's just one point I want to add today:

    I also see a spectrum of modifications to the module implementation that would change the equation.
    A. Reimplement pkg-config entirely in Perl avoiding the need to depend
    on pkg-config.

    This already exists: https://metacpan.org/dist/PkgConfig
    and is even packaged as libpkgconfig-perl, which contains
    /usr/bin/ppkg-config.

    All of these approaches have in common that we avoid requiring pkg-config:$DEB_HOST_ARCH and can work with pkgconf-bin, which happens
    to be M-A:foreign and thus allows our module to also be M-A:foreign.

    /usr/bin/ppkg-config is even pure perl, parsing .pc files itself. No pkg-config, no pkgconf-bin, no pkgconf, only dependency perl:any.

    Replacing pkg-config with ppkg-config in libextutils-pkgconfig-perl
    would be trivial.


    Cheers,
    gregor

    --
    .''`. https://info.comodo.priv.at -- Debian Developer https://www.debian.org
    : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D 85FA BB3A 6801 8649 AA06
    `. `' Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
    `-

    -----BEGIN PGP SIGNATURE-----

    iQKTBAEBCgB9FiEE0eExbpOnYKgQTYX6uzpoAYZJqgYFAmeDBf1fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEQx RTEzMTZFOTNBNzYwQTgxMDREODVGQUJCM0E2ODAxODY0OUFBMDYACgkQuzpoAYZJ qgYxCg//WZSWOaCXEnAzdee5zaEGD8cOqfgNX52dhkj44AuLd2H7Znqa7DjMWGoq JHefpqkCrY4DvWXsVS3iCewwx3Txgtgd4711DQpIH0pzqwbfSLxgq217Pu04Jj5R +p36dJ6/j1tLJZLqsIdNqZw7BVJxgfqBChp2P5tRdTeA+jD+DbiBdQj5XVE0CNrN bfWkZ5DBBbF42ZfvVWhrN5wbGU+LdbXNP3p/PTw5dqeCzEW9+QxhhmDYELkcwOwd ljBcVmxBx2QjwJl4iGXsHCs+WgHu6WDXLR5hnSOjMdID8Nsvryye8bKNB/GMta+Z qN/wR4ULMGkthssDCERmfffkGVUB6LFDXjh9CQVXhWwrz8Ltgd0eUVWobG/B8EBv +4WoRQ5gPh2ZNteXqrpzcRVnut0UiReIFCxJGSF9z5QOa+NZ1ztKsmjEp3Fv/BTG 69ME9fm8Yf22SKUFIQAXb9gTCb0YtSvI7X/5D3rk/CptQcr2zE+zdts0lRHeLCpQ K/2jRMPC/h8vtDtH4Mxydn5N8gtRcAKGLDbO4ppjte8ncduFOEjE/CrVdPzdJonR 8/zAltueJqvUd564EVzBb4KFnr0XPBtiPo2Du8ip2jQFqdCEVg3sKmYW5ogW0+S1 M8os5awfTLkkfcxCHjDL5k2TgJBsowDNcXJTwTxgQvrJsKh+yas=
    =xCr0
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)