[continued from previous message]

to be auto-detectable from just the current toolchain being used.

So you are saying to configure a build environment for, say, x86-64-v3 you would configure gcc with --with-arch64=x86-64-v3 and then dpkg-architecture would parse the output of gcc -Q --help=target to set DEB_HOST_ARCH_VARIANT appropriately? (modulo mistakes in details) Or do you mean something else entirely?

That would be one solution yes, which could give automatic bijective
mappings, although ideally with a machine-readable way to get at it,
which I'm not sure we have currently. For example code in dpkg-dev
already runs «$CC -dumpmachine» to infer the host architecture to use
during builds.

While using a triplet variation could be a way to do that, that would
require such triplet support for each variant/ISA, which tends to be
very painful to introduce if it's not there already, so I'd not
consider this specific way a viable option.

Some of the above problems could perhaps be avoided if we introduced
a concept of architecture aliases/ISAs (similar to what rpm has), which would side-step the pool sharing issue, the binary package renaming,
etc. One big issue with this is that it requires for dpkg to have an exhaustive table of all such aliases, and if there's ever a new alias added, old dpkg versions need to be updated or they will not understand what they match with. So this does not seem ideal either. So I guess this is a variation over your proposal, but perhaps this could still be used
in specific contexts, say only at build-time (but not for dependency relationships), for repo management (say binary-arm64v9/Packages.xz),
or binary package names where the field would specify the actual name
for the filename, say:

Architecture: arm64
ArchitectureIsa: arm64v9

or maybe better:

Architecture: arm64
ArchitectureIsa: v9

resulting in dpkg-deb generating:

binpkg_1.0-1_arm64v9.deb

but targeting arm64.

I'm not sure but I think you have talked yourself into suggesting something very similar to my proposal here?

Ah sorry, yeah, didn't mean to present it as a new idea, I was mostly
trying to walk over the issues, and refine upon your initial idea,
with those constraints applied. :)

On Fri, 2023-09-01 at 08:43:55 +1200, Michael Hudson-Doyle wrote:

Is there a better way of doing this?

I think starting from 5, the rest are probably just details to hammer
out, but not insurmountable things.

Great. The things I see as a bit vague at a base level currently are:

* Should the ISA influence the toolchain via toolchain defaults or dpkg-buildflags?
* How is the default ISA for a buildd chroot selected?

So the clear downsides of either modifying the default toolchain or
having to provide an additional one is that this seems pretty heavy
weight. Also because people might want to build optimized variants
locally w/o having to mess with their already existing toolchains.
(I'm not sure whether something going along the lines of <https://git.hadrons.org/cgit/debian/fakecross.git> could be an
option, although as mentioned above, if that would imply new triplets,
then probably not.)

So the easiest way might indeed be by controlling this via an envvar,
which dpkg-buildpackage could also setup internally via a new option,
say --arch-isa=amd64v3 or similar to make this slightly more
discoverable. Which would be easy to use from the buildds too I guess.

There is also the question of whether partial coverage of an ISA is handled by the package publisher or client side in apt but that's at least one
level higher.

Yeah, that would be of no concern to dpkg, I think.

Thanks,
Guillem

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

Yeah, that works, and follows the current DPKG_*_ARCH_ABI lead for
example.

which dpkg-buildpackage could also setup internally via a new option,
say --arch-isa=amd64v3 or similar

--host-arch-isa would be more coherent I think.

Ah absolutely! For some reason had --arch in mind as a valid option
(I only see it now in dpkg-scanpackages :D, or maybe I was thinking
about --host-isa :).

I guess one could add support for --target-host-arch-isa to build a
toolchain that defaults to a particular ISA. But well.

Yes, the ISA support in dpkg should be extensive enough (so that if
this needs to be supported in the toolchain, then it is possible).

So to summarise, here are the generic changes that I think need to be made
to src:dpkg to support variant ISAs as a thing:

* add get_host_arch_isa() to Dpkg::Arch

Yes (perhaps as mentioned below also just get_host_isa()).

* dpkg-gencontrol records DEB_HOST_ARCH_ISA into DEBIAN/control as ArchitectureIsa

Probably better Architecture-Isa, otherwise the current automatic
case folding would make it end up as Architectureisa.

* dpkg-architecture emits DEB_HOST_ARCH_ISA and grows --host-arch-isa flag

Also DEB_BUILD_ARCH_ISA and DEB_TARGET_ARCH_ISA, and also grows a --target-arch-isa (but I'm thinking whether the shorter --host-isa would
be nicer, for example the --match-bits are not spelled --match-arch-bits,
which would seem also a bit redundant).

* dpkg-buildpackage passes --host-arch-isa to dpkg-architecture

Yes, but only when not the baseline.

* dpkg-genchanges should record the ISA in the changes file somehow I
guess?

Yes, also dpkg-genbuildinfo. This could be done either from the
envvars, or perhaps through the debian/files attributes support. But
given that this is supposedly build global (I think it would be rather
weird to end up with a .changes including say _amd64.deb and
_amd64v3.deb file references from the same build), then probably using
the envvar might be the better way.

* dpkg-deb records the ISA in the file name

Yes.

Have I missed anything?

Nothing else comes to mind right now (except what I might have already
added).

(Hmm does anything need to reject unknown values
found in DEB_HOST_ARCH_ISA / --host-arch-isa? Probably...)

I guess it indeed makes sense to define what ISAs are supported, and
either error out or warn and ignore such values. So there might be a
need to add something like a new data/isatable.

Conceptually slightly separately, it might make sense to add a build "feature" to Dpkg::Vendor::Debian to allow setting -march (and -mtune?)

Then when we want to add support to an ISA, we add a little thing to set_build_features (in either Vendor::Debian or Vendor::Ubuntu or wherever) that maps get_host_arch_isa() to values for the march-influencing feature.

Hmm, right, how to hook this. I'm not sure the current interface is good
enough to describe this via build flags features, because such new feature
area would expose arch-specific features. I have been thinking through
the build flags and will try to send a proposal/RFC to revamp parts of
it during the weekend. But I think the ISA stuff is better just handled
(at leas for now) directly by injecting whatever flags when the requested
ISA is different to the baseline.

Thanks,
Guillem

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

Guillem

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

ArchitectureIsa name. While *this specific use case* is a variant of
the architecture instruction set; you could just as well build other
variants such as "compiled with -O3", "compiled with frame pointers",
"compiled with -O0", or other shenanigans (I haven't thought about
others outside compiler flags)

Hence I prefer Architecture-Variant, Subarchitecture, or anything
like that rather than have to invent another field or abuse this
one the next time we want to build a special variant of an architecture
with special optimizations for a special customer or whatnot.

This has stalled a bit, but we really gotta get going on this, we
should really have this working in APT by end of year or something.
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

one the next time we want to build a special variant of an architecture
with special optimizations for a special customer or whatnot.

yes, it sounds like a bit too specific.

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

&gt; &gt; &gt; added, old dpkg versions need to be updated or they will not understand<br>
&gt; &gt; &gt; what they match with. So this does not seem ideal either. So I guess this<br>
&gt; &gt; &gt; is a variation over your proposal, but perhaps this could still be used<br>
&gt; &gt; &gt; in specific contexts, say only at build-time (but not for dependency<br>
&gt; &gt; &gt; relationships), for repo management (say binary-arm64v9/Packages.xz),<br>
&gt; &gt; &gt; or binary package names where the field would specify the actual name<br>
&gt; &gt; &gt; for the filename, say:<br>
&gt; &gt; &gt; <br>
&gt; &gt; &gt;    Architecture: arm64<br>
&gt; &gt; &gt;    ArchitectureIsa: arm64v9<br>
&gt; &gt; &gt; <br>
&gt; &gt; &gt; or maybe better:<br>
&gt; &gt; &gt; <br>
&gt; &gt; &gt;    Architecture: arm64<br>
&gt; &gt; &gt;    ArchitectureIsa: v9<br>
&gt; &gt; &gt; <br>
&gt; &gt; &gt; resulting in dpkg-deb generating:<br>
&gt; &gt; &gt; <br>
&gt; &gt; &gt;    binpkg_1.0-1_arm64v9.deb<br>
&gt; &gt; &gt; <br>
&gt; &gt; &gt; but targeting arm64. I also think I prefer naming this explicitly as ISA<br>
&gt; &gt; &gt; variants, if you will, than just architecture variants as that gives<br>
&gt; &gt; &gt; way too much room (which perhaps we want, but then that has other<br>
&gt; &gt; &gt; implications over compatibility), and for the field perhaps just Isa is<br>
&gt; &gt; &gt; better, to avoid the implicit repetition of<br>
&gt; &gt; &gt; ArchitectureInstructionSetArchitecture :), but that makes it less easy<br>
&gt; &gt; &gt; to associate both as related.<br>
&gt; &gt; <br>
&gt; &gt; I have thought more about this and I&#39;m not particularly fond of the<br>
&gt; &gt; ArchitectureIsa name. While *this specific use case* is a variant of<br>
&gt; &gt; the architecture instruction set; you could just as well build other<br>
&gt; &gt; variants such as &quot;compiled with -O3&quot;, &quot;compiled with frame pointers&quot;,<br>
&gt; &gt; &quot;compiled with -O0&quot;, or other shenanigans (I haven&#39;t thought about<br>
&gt; &gt; others outside compiler flags)<br>

These looks really out of scope for the initially proposed purpose,<br>
as they have nothing to do with the architecture, they are rebuilds<br>
with different compilation flags. Creating a new pseudo-architecture<br>
for those things seems just wrong and out of place. I understand that<br>
might seem convenient, but it does not make sense to me, conceptually<br>
and from a design PoV.<br></blockquote><div><br></div><div>I think I agree although I would probably phrase it a bit the other way around. The facility we are talking about adding here technically speaking /could/ apply to any compiler settings that do
not affect ABI, but /should/ it?</div><div><br></div><div>The motivation behind different -march flags seems sound. There are some other things we could technically do that just seem like a bad idea (like a variant built without hardening flags). I don&#
39;t think having a variant with -O3 makes sense -- I think it is the role of the distro to evaluate which of -O2 and -O3 is better and not just punt that sort of thing to the user. So I think a tighter target scope is best and if the naming of fields
can point in that direction it&#39;s probably a good thing? As much as ArchitectureISA doesn&#39;t seem particularly aesthetic to me.</div><div><br></div><div>OTOH, something like this might be useful for experimentation, to be able to spin up a variant
as part of finding out if -O3 is better. We have other ways to do this for Ubuntu but they are annoying and clunky.</div><div><br></div><div>Cheers,</div><div>mwh</div><div> </div></div></div>

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)