1. Forum
  2. Usenet
  3. LINUX.DEBIAN.DEVEL

  • =?utf-8?B?4oCmL2RvYyDigKYvbG9nOiAuZ3og?= =?utf-8?B?4oaS?= .zst

    From Adam Borowski@21:1/5 to All on Fri Aug 19 17:30:01 2022
    Meow!
    Because of the trimming changelogs discussion, I just wondered whether
    it'd be beneficial to switch compression from .gz to .zst (gzip to zstd).

    Numbers I got from the desktop I sit at:

    * du of a copy of /usr/share/doc
    366692
    * rm ! -name "*.gz"
    194184
    * decompress
    656852
    * repack as zstd
    163340

    31MB saving, decent but not world shattering. Let's do so...?


    But then, I looked at /var/log

    * du
    257612
    * rm ! -name "*.gz"
    53028
    * decompress
    471892
    * repack as zstd
    16360

    Holy apt! Now this is a find! This box is a desktop thus doesn't produce
    a lot of logs (just sbuild spam), but there's quite a few of my boxen where logs get very bulky.

    Thus: let's switch logrotate from gzip to zstd? As changing the default
    would upset scripts, let's do it at the same time as enabling dateext...


    Meow!
    --
    ⢀⣴⠾⠻⢶⣦⠀
    ⣾⠁⢠⠒⠀⣿⡁
    ⢿⡄⠘⠷⠚⠋⠀ You're alive. But that's just a phase. ⠈⠳⣄⠀⠀⠀⠀

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?utf-8?Q?Hakan_Bay=C4=B1nd=C4=B1r?@21:1/5 to All on Fri Aug 19 17:50:01 2022
    Hi Adam,

    I’d object that, because after we rotate the logs, we use a lot of z commands, namely zcat, zgrep, zless. Which allows us process many gigabytes of gzip files without extracting them first.

    We have a big cluster at office and a central logging system. That system handles close to a thousand machines at the same time, and we neither have disk space, nor processing power problems.

    Sorry for not agreeing with your idea, but operations needs this kind of interoperability and composability.

    I think we shouldn’t fix it, because it’s not broken from my point of view,

    Best regards,

    Hakan

    On 19 Aug 2022, at 18:22, Adam Borowski <[email protected]> wrote:

    Meow!
    Because of the trimming changelogs discussion, I just wondered whether
    it'd be beneficial to switch compression from .gz to .zst (gzip to zstd).

    Numbers I got from the desktop I sit at:

    * du of a copy of /usr/share/doc
    366692
    * rm ! -name "*.gz"
    194184
    * decompress
    656852
    * repack as zstd
    163340

    31MB saving, decent but not world shattering. Let's do so...?


    But then, I looked at /var/log

    * du
    257612
    * rm ! -name "*.gz"
    53028
    * decompress
    471892
    * repack as zstd
    16360

    Holy apt! Now this is a find! This box is a desktop thus doesn't produce
    a lot of logs (just sbuild spam), but there's quite a few of my boxen where logs get very bulky.

    Thus: let's switch logrotate from gzip to zstd? As changing the default would upset scripts, let's do it at the same time as enabling dateext...


    Meow!
    --
    ⢀⣴⠾⠻⢶⣦⠀
    ⣾⠁⢠⠒⠀⣿⡁
    ⢿⡄⠘⠷⠚⠋⠀ You're alive. But that's just a phase. ⠈⠳⣄⠀⠀⠀⠀

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Johannes Schauer Marin Rodrigues@21:1/5 to All on Fri Aug 19 22:20:01 2022
    Quoting Hakan Bayındır (2022-08-19 17:47:42)
    I’d object that, because after we rotate the logs, we use a lot of z commands, namely zcat, zgrep, zless. Which allows us process many gigabytes of gzip files without extracting them first.

    We have a big cluster at office and a central logging system. That system handles close to a thousand machines at the same time, and we neither have disk space, nor processing power problems.

    Sorry for not agreeing with your idea, but operations needs this kind of interoperability and composability.

    $ apt-file search /bin/zcat
    gzip: /bin/zcat
    zutils: /bin/zcat

    It sounds like you are using zcat from gzip which indeed can only handle gzip compression. Why don't you use zcat from zutils?

    $ apt-file show zutils
    zutils: /bin/zcat
    zutils: /bin/zcmp
    zutils: /bin/zdiff
    zutils: /bin/zegrep
    zutils: /bin/zfgrep
    zutils: /bin/zgrep
    zutils: /bin/ztest
    zutils: /bin/zupdate
    zutils: /etc/zutilsrc

    $ apt-cache show zutils
    [...]
    Description-en: utilities for dealing with compressed files transparently
    Zutils is a collection of utilities for dealing with any combination of
    compressed and non-compressed files transparently. Currently the supported
    compressors are gzip, bzip2, lzip, xz, and zstd.

    Problem solved?
    --==============E31819341320200941=MIME-Version: 1.0
    Content-Transfer-Encoding: 7bit
    Content-Description: signature
    Content-Type: application/pgp-signature; name="signature.asc"; charset="us-ascii"

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEElFhU6KL81LF4wVq58sulx4+9g+EFAmL/7/cACgkQ8sulx4+9 g+Fafw/9E7+ujNmUHmPBzoUBZbF/4N9mApKC4F/ya+hKdbMVy34DzfEgGduO9NXe qZHQjnRgYbncDYU3ZREgubqtRwqR0V+Om4/sTqBdWaDuWdRoc0FFbg9F0iRhOIBq MBNpmYAG03lFA/524mKuIFaveDGwU6UAjwCAcSzxvhC2JCXHFUBP9kZHqBEYdzZ8 qsw9YvXhBFVl3YwUvP8BD5AGvd11E8QzmKOy01cTiHcgwwJs+tiUo7VpVKenupmK iMkyyapCBAHDYAnEUB4+0B8bMYXo7F5MQ+7hD0kLODNyp3LWsAi+anCSrofg5xtb p2fappTjKs9ybg8QtMXpiv55m9ypXLji6OZxQKrbXFcT8zxN/eG4oFnpiPPT7IXt JaOxbxDpMCUZnhmc1jjgUnwS0RbXQ9J9o/+hyz3U9WaqUtHQVnyn6btZN3IRoNC7 ayBY+LU+j55ZLKDL5+4+VuMOQDb3C4ZqBR3q9kMFGRq5iOpqd5QsYC9y5whEv15G ad0JtE/1ElJkXftXjuuec7LHHe3IaMz4m6Nk86vfgmvrgsfqie5f3uYKoSoeXvpw 03opndl4jUu91CmazsI4MwUpc9149R3ndMHrTKfTsBH83e0izKK2srXbYQethXi7 d+gsUaLz/0P5WQvo2aZJGmrZNeDfK2worEFYy+lrbh+W312TRag=
    =edQQ
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Joerg Jaspert@21:1/5 to All on Fri Aug 19 22:20:01 2022
    On 16595 March 1977, Hakan Bayındır wrote:

    I’d object that, because after we rotate the logs, we use a lot of z commands, namely zcat, zgrep, zless. Which allows us process many
    gigabytes of gzip files without extracting them first.

    So you use zstdcat, zstdgrep, etc.pp, done.

    --
    bye, Joerg

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)