On 12 Oct 2022, at 05:46, Michał Górny <[email protected]> wrote:

On Tue, 2022-10-11 at 23:54 +0100, Sam James wrote:

Note that this will only take effect if compressing
modules with xz or zstd (COMPRESS_MODULE_XZ is set or
COMPRESS_MODULE_ZSTD is set respectively)

... or if compiling the kernel with xz or zstd (COMPRESS_KERNEL_XZ
or COMPRESS_KERNEL_ZSTD is set respectiely).

Signed-off-by: Sam James <[email protected]>
---
eclass/kernel-build.eclass | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/eclass/kernel-build.eclass b/eclass/kernel-build.eclass
index 98a385bf72c0..936258e9e8af 100644
--- a/eclass/kernel-build.eclass
+++ b/eclass/kernel-build.eclass
@@ -29,7 +29,7 @@ esac

PYTHON_COMPAT=( python3_{8..11} )

-inherit python-any-r1 savedconfig toolchain-funcs kernel-install
+inherit multiprocessing python-any-r1 savedconfig toolchain-funcs kernel-install

BDEPEND="
${PYTHON_DEPS}
@@ -90,6 +90,14 @@ kernel-build_src_configure() {
ARCH=$(tc-arch-kernel)
)

+ if type -P xz ; then
+ export XZ_OPT="-T$(makeopts_jobs)"
+ fi
+
+ if type -P zstd ; then
+ export ZSTD_NBTHREADS="$(makeopts_jobs)"
+ fi
+
restore_config .config
[[ -f .config ]] || die "Ebuild error: please copy default config into .config"

I'm think that we need an override for these values. Just because
someone is using -j12 to compile source files in parallel (which has no
real drawback), it doesn't mean that that person wants to use 12-thread parallel xz/zstd that will actually reduce compression ratio for minimal
gain (I mean, most of these files won't even use parallel compression
because they're too small).

Not to mention that someone using -j32 distcc setup necessarily wants 32-threaded local compression.

I'll make it use XZ_OPT if present in environment to avoid clobbering, ditto zstd.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, 2022-10-11 at 23:54 +0100, Sam James wrote:

Note that this will only take effect if compressing
modules with xz or zstd (COMPRESS_MODULE_XZ is set or
COMPRESS_MODULE_ZSTD is set respectively)

... or if compiling the kernel with xz or zstd (COMPRESS_KERNEL_XZ
or COMPRESS_KERNEL_ZSTD is set respectiely).

Signed-off-by: Sam James <[email protected]>
---
eclass/kernel-build.eclass | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/eclass/kernel-build.eclass b/eclass/kernel-build.eclass
index 98a385bf72c0..936258e9e8af 100644
--- a/eclass/kernel-build.eclass
+++ b/eclass/kernel-build.eclass
@@ -29,7 +29,7 @@ esac

PYTHON_COMPAT=( python3_{8..11} )

-inherit python-any-r1 savedconfig toolchain-funcs kernel-install
+inherit multiprocessing python-any-r1 savedconfig toolchain-funcs kernel-install

BDEPEND="
${PYTHON_DEPS}
@@ -90,6 +90,14 @@ kernel-build_src_configure() {
ARCH=$(tc-arch-kernel)
)

+ if type -P xz ; then
+ export XZ_OPT="-T$(makeopts_jobs)"
+ fi
+
+ if type -P zstd ; then
+ export ZSTD_NBTHREADS="$(makeopts_jobs)"
+ fi
+
restore_config .config
[[ -f .config ]] || die "Ebuild error: please copy default config into .config"

I'm think that we need an override for these values. Just because
someone is using -j12 to compile source files in parallel (which has no
real drawback), it doesn't mean that that person wants to use 12-thread parallel xz/zstd that will actually reduce compression ratio for minimal
gain (I mean, most of these files won't even use parallel compression
because they're too small).

Not to mention that someone using -j32 distcc setup necessarily wants 32-threaded local compression.

--
Best regards,
Michał Górny

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, 12 Oct 2022, Michał Górny wrote:

(I mean, most of these files won't even use parallel compression
because they're too small).

xz(1) says about the block size: "The default size is three times the
LZMA2 dictionary size or 1 MiB, whichever is more." The default (-6)
dictionary size is 8 MiB.

So IIUC only files larger than 24 MiB would be compressed in parallel?

Ulrich

--=-=-Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQFDBAEBCAAtFiEEtDnZ1O9xIP68rzDbUYgzUIhBXi4FAmNGfUgPHHVsbUBnZW50 b28ub3JnAAoJEFGIM1CIQV4uTjcIAKzam5grB51Dlwwn9oGF+ygEhTVvlMywTajE +XfHNUpSO5Zob/kO9cz4KieaTjjiq+iwSig/iFiONZfuIzadygKp9ltJGnh0TruB 49+x1GRc1qe+MnKD2ptg+NS7Yv3lnnM/WtGMHr/5NV2x7wZHkFvE+fou9e9Qi6Ry GdhJSyY+/XyMaAzaWgdIzQvprWjqK3DnEG8J8zwye57HxihKp5bPPwdoHTM+bhLK acMljNchHpufftOUqAX5sdv8abLFYc4asKfI2IAVt1EOWJVm3MvKVietNXB9Ds64 nQoGOzv0YrTMCHroqUaydfK+C3+jXrufeHZtTVx49Kt7j6CryaE=AWej
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Michał Górny <[email protected]> writes:

On Tue, 2022-10-11 at 23:54 +0100, Sam James wrote:

Note that this will only take effect if compressing
modules with xz or zstd (COMPRESS_MODULE_XZ is set or
COMPRESS_MODULE_ZSTD is set respectively)

... or if compiling the kernel with xz or zstd (COMPRESS_KERNEL_XZ
or COMPRESS_KERNEL_ZSTD is set respectiely).

Signed-off-by: Sam James <[email protected]>
---
eclass/kernel-build.eclass | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/eclass/kernel-build.eclass b/eclass/kernel-build.eclass
index 98a385bf72c0..936258e9e8af 100644
--- a/eclass/kernel-build.eclass
+++ b/eclass/kernel-build.eclass
@@ -29,7 +29,7 @@ esac

PYTHON_COMPAT=( python3_{8..11} )

-inherit python-any-r1 savedconfig toolchain-funcs kernel-install
+inherit multiprocessing python-any-r1 savedconfig toolchain-funcs kernel-install

BDEPEND="
${PYTHON_DEPS}
@@ -90,6 +90,14 @@ kernel-build_src_configure() {
ARCH=$(tc-arch-kernel)
)

+ if type -P xz ; then
+ export XZ_OPT="-T$(makeopts_jobs)"
+ fi
+
+ if type -P zstd ; then
+ export ZSTD_NBTHREADS="$(makeopts_jobs)"
+ fi
+
restore_config .config
[[ -f .config ]] || die "Ebuild error: please copy default config into .config"

I'm think that we need an override for these values. Just because
someone is using -j12 to compile source files in parallel (which has no
real drawback), it doesn't mean that that person wants to use 12-thread parallel xz/zstd that will actually reduce compression ratio for minimal
gain (I mean, most of these files won't even use parallel compression
because they're too small).

Note that >= xz-5.3.3_alpha defaults to the threaded compressor
anyway, just with -j1 and the difference is tiny.

--=-=-Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iOUEARYKAI0WIQQlpruI3Zt2TGtVQcJzhAn1IN+RkAUCY0buyV8UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0MjVB NkJCODhERDlCNzY0QzZCNTU0MUMyNzM4NDA5RjUyMERGOTE5MA8cc2FtQGdlbnRv by5vcmcACgkQc4QJ9SDfkZDkwAD6AuDtYraDom93Qq35W+mEuu9o2qoP6EiXstdf Ta/LUC4A/2y1N5R+gnop8Jf0BRSYaGRBMSBZh1WgURTdYzLYT7EI
=8umK
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)