XPost: linux.debian.maint.dpkg
Hi!
On Thu, 2024-12-05 at 09:23:24 +0100, Helmut Grohne wrote:
On Wed, Dec 04, 2024 at 02:03:29PM +0100, Guillem Jover wrote:
On Thu, 2024-11-28 at 10:54:37 +0100, Helmut Grohne wrote:
For one thing, I propose extending debhelper to provide --min-ram-per-parallel-core as that seems to be the most common way to
do it. I've proposed https://salsa.debian.org/debian/debhelper/-/merge_requests/128
to this end.
To me this looks too high in the stack (and too Linux-specific :).
I don't think being Linux-specific is necessarily bad here and note that
the /proc interface is also supported by Hurd (I actually checked on a
porter box). The problem we are solving here is a practical one and the solution we pick now probably is no longer relevant in twenty years.
That's about the time frame I am expect Linux to be the preferred kernel
used by Debian (could be longer, but unlikely shorter).
See below for the portability part.
I think adding this in dpkg-buildpackage itself would make most sense
to me, where it is already deciding what amount of parallelism to use
when specifying «auto» for example.
Given that this would be and outside-in interface, I think this would
imply declaring these parameters say as debian/control fields for example, or some other file to be parsed from the source tree.
I find that outside-in vs inside-out distinction quite useful, but I
actually prefer an inside-out approach. You detail that picking a
sensible ram-per-core value is environment-specific. Others gave
examples of how build-systems address this in ways of specifying linker groups with reduced parallelism and you go into detail of how the
compression parallelism is limited based on system ram already. Given
all of these, I no longer am convinced that reducing the package-global parallelism is the desired solution. Rather, each individual step may
benefit from its own limiting and that's what is already happening in
the archive. It is that inside-out approach that we see in debian/rules
in some packages. What I now find missing is better tooling to support
this inside-out approach.
Not all outside-in interfaces are made equal, as I hinted on that
other mail, some are (let's call them) permeable, where the build
driver performs some defaults setup or data gathering that it does
not necessarily uses itself, and which can be easily overridden by
the inner packaging files.
I don't have a strong opinion on this case though, my initial reaction
was that because dpkg-buildpackage is already trying to provide a good
default for the number of parallel jobs to use, it seemed like a good
global place to potentially improve that number to influence all users,
if say the only thing needed is a declarative hint from the packing
itself. This being a permeable interface also means the inner processes
could still ignore or tune that value further or whatever (except for
the --jobs-force option which is harder to revert from inside).
But I guess it depends on whether we can have a better general heuristic
in the outer parallel job number computation. Or whether for the cases
that we need to tune, any such general heuristic would serve no actual
purpose and all/most of them would need to be overridden anyway.
My main concerns would be:
* Portability.
I am not concerned. The parallelism limit is a mechanism to increase efficiency of builder deployments and not much more. The portable
solution is to stuff in more RAM or supply a lower parallel value
outside-in. A 90% solutions is more than good enough here.
Right, I agree with the above, because this should be considered an opportunistic quality of life improvement, where the user can always
manually override it, if the tool does not get it right. My concern
was about the above debhelper MR failing hard on several conditions
where it should just simply disable the improved clamping. See for
example the parallel auto handling in dpkg-buildpackage (after the «run_hook('preinit')»), or lib/dpkg/compress.c:filter_xz_get_memlimit()
and lib/dpkg/meminfo.c:meminfo_get_available_from_file()) for the
dpkg-deb one, where these should gracefully fallback to less accurate
methods if they cannot gather the needed information.
(Now that you mention it, I should probably enable Hurd for the
/proc/meminfo codepath. :)
* Whether this is a local property of the package (so that the
maintainer has the needed information to decide on a value, or
whether this depends on the builder's setup, or perhaps both).
All of what I wrote in this thread thus far assumed that this was a
local property. That definitely is an oversimplification of the matter
as an upgraded clang, gcc, ghc or rustc has historically yielded
increased RAM consumption. The packages affected tend to be sensitive to changes in these packages in other ways, so they generally know quite
closely what version of dependencies will be in use and can tailor their guesses. So while this is a non-local property in principle, my
expectation is that treating it as if it was local is good enough for a
90% solution.
My thinking here was also about the general case too, say a system
that has many cores relative to its available memory, where each core
would get what we'd consider not enough memory per core (assuming for
example a baseline for what dpkg-deb might require, plus build helpers
and their interpreters, and what a compiler with say an empty C, C++
or similar file might need, etc).
* We might need a way to percolate these parameters to children of
the build/test system (as Paul has mentioned), where some times
you cannot specify this directly in the parent. Setting some
standardize environment variables would seem sufficient I think,
but while all this seems kind of optional, this goes a bit into
reliance on dpkg-buildpackage being the only supported build
entry point. :)
To me, this reads as an argument for using an inside-out approach.
Given all of the other replies (on-list and off-list), my vision of how
I'd like to see this approached has changed. I see more and more value
in leaving this in close control of the package maintainer (i.e.
inside-out) to the point where different parts of the build may use
different limits.
I think this would be fine too. The above point was more an option to
implement an hybrid permeable outside-in, where dpkg-buildpackage
could then try to gather say the amount of available memory, number of
cores, and any other potentially relevant additional data, then perhaps
try to use a declarative hint from the packaging to tune either the
default parallel jobs value or provide a new variable, and then let
the inner processes decide based on those variables, which would avoid
having to duplicate much of the data gathering and potential
portability issues.
This could also imply alternatively or in addition, providing a tool
or adding some querying logic in an existing tools (in the dpkg toolset)
to gather that information which the packaging could use, or…
How about instead we try to extend coreutils' nproc? How about adding
more options to it?
--assume-units=N
--max-units=N
--min-ram-per-unit=Z
…as you mention, another build-essential package's tools, so that again
we do not need to duplicate much of the logic. :)
Then, we could continue to use buildopts.mk and other mechanism to
extract the passed parallel value from DEB_BUILD_OPTIONS as before and
run it through an nproc invocation for passing it down to a build system
in the specific ways that the build system requires. More options could
be added to nproc as-needed.
So, yeah that would also work.
Thanks,
Guillem
--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)