• [gentoo-dev] www-client/chromium stabilisation and arch testing

    From Matt Jolly@21:1/5 to All on Wed Jan 29 00:10:01 2025
    This is a multi-part message in MIME format.
    Hi All,

    Take 2; emailing the affected arch aliases directly only got
    one response which included a suggestion to take this straight to -dev.

    As you're probably aware, www-client/chromium is a frequently-updated
    package with a not-insignificant build. Chromium has an enormous attack
    surface and is used to parse often-untrusted data from the web; this
    results in frequent CVEs due to the back-and-forth between black hats
    and upstream developers, and upstream's (quite generous) bug bounty
    program.

    Fortunately for us, upstream has a fixed 4-week release cycle[1]:
    Every 4 weeks the 'beta' channel is promoted to 'stable', the
    'dev' channel is promoted to 'beta'; each channel is 'refreshed'
    weekly. The 'stable' refreshes typically contain security fixes and
    urgent regression fixes (at the discretion of the upstream release
    team).

    Approximately 48 weeks of the year we package an update for the 'stable' channel for ::gentoo. The current stabilisation process often delays the delivery of these security updates to users running stable keywords by
    hours to days depending on arch-tester availability.

    Upstream provide official binaries for (and even encourage side-by-side installs of) each channel on Windows and Mac. Upstream also support
    amd64 and arm64 arches on Linux and have a robust development
    process with significant investment in CI/CD infrastructure.

    In ::gentoo we currently carry all 3 channels, with 'dev' being
    unkeyworded, 'beta' keyworded for 'testing', and 'stable' going through
    the standard arch stabilisation process for amd64 and arm64 (with
    'ppc64' 'testing' keywords when patches are available).

    Between the (Gentoo) Chromium project building and testing the software
    and various arch testers tying up resources that could otherwise be used
    to look at other packages there's a reasonable case to be made that
    overall we're spending 10+ hours on a process that may be superfluous.

    With all of this in mind, I'd like to see about changing the way that we
    handle stabilisation of www-client/chromium for amd64 and arm64. I see
    two options that significantly reduce packaging overhead and, more
    importantly, enable us to get security fixes into the hands of our users
    more quickly (hours rather than days after an upstream release) and
    free up valuable arch tester time.

    I see two options for handling this better:

    1. File a stablereq on promotion to 'stable' channel for each major
       version and commit any updates directly with stable keywords.

    Pros: Provides a checkpoint to ensure dependent packages (dev-build/gn, media-video/ffmpeg-chromium) are keyworded, allows for a brief review
    period.

    Cons: Still introduces some delay in delivering updates, might be
    unnecessary given upstream's robust testing and the nature of the
    'stable' channel which only receives backported security fixes.

    2. Commit the 'stable' channel with stable keywords unless there's a
       specific reason to do otherwise.

    Pros: Fastest possible delivery of security updates, minimises packaging overhead.

    Cons: Relies more heavily on upstream's stability, less opportunity for Gentoo-specific review before release. We would need a clear policy for handling potential issues and rollbacks, possibly retaining more package versions in-tree and demoting a release to 'testing' as required.

    My Recommendation: I lean towards option 2 as it prioritises getting
    security fixes to users as quickly as possible. The 'stable' channel's
    design and Google's extensive CI/CD infrastructure give a good level
    of comfort for this approach. For managing dependencies like
    dev-build/gn and media-video/ffmpeg-chromium, we can rely on pkgcheck
    and our own CI (hi croaker) to ensure that keywords are kept in sync.
    To mitigate the reduced opportunity for Gentoo-specific review, we will
    retain the previous 2 stable 'refreshes' in the tree, allowing for swift demotion to 'testing' should any critical issues arise. We will also
    establish a clear policy for demoting a release based on user reports.

    As a stretch goal it would be possible to action the keyword updates for dependencies like dev-build/gn and media-video/ffmpeg-chromium using
    some of the existing automation that is used to monitor for new upstream releases, though I'm wary of automating too much of this and would need
    to look at Gentoo policy.

    I'm interested in your thoughts on this, or if continuing with the
    status-quo is otherwise desirable from an arch project perspective. Specifically, I'd like to know if you have any reservations about
    adopting option 2.

    If we want to evaluate the success of this change, we can track the time between upstream releases and the availability of stable keywords in
    Gentoo, as well as monitoring for any increase in bug reports related to Chromium's stable channel. We can also leverage our existing security
    bugs to track the number of security vulnerabilities addressed more
    promptly under the new approach. I'm also interested in any other
    thoughts on evaluating its success if we do go ahead, or whether we
    bother with any of this at all and just watch for a sudden flood of bugs
    logged against the stable channel.

    Thanks for your time.

    Cheers,

    Matt

    1: https://chromium.googlesource.com/chromium/src/+/HEAD/docs/process/release_cycle.md

    <!DOCTYPE html>
    <html>
    <head>

    <meta http-equiv="content-type" content="text/html; charset=UTF-8">
    </head>
    <body>
    <p>
    </p>
    <div class="moz-text-flowed"
    style="font-family: -moz-fixed; font-size: 12px;" lang="x-unicode">Hi
    All,
    <br>
    </div>
    <div class="moz-text-flowed"
    style="font-family: -moz-fixed; font-size: 12px;" lang="x-unicode"><br>
    </div>
    <div class="moz-text-flowed"
    style="font-family: -moz-fixed; font-size: 12px;" lang="x-unicode">Take
    2; emailing the affected arch aliases directly only got</div>
    <div class="moz-text-flowed"
    style="font-family: -moz-fixed; font-size: 12px;" lang="x-unicode">one
    response which included a suggestion to take this straight to
    -dev.<br>
    </div>
    <div class="moz-text-flowed"
    style="font-family: -moz-fixed; font-size: 12px;" lang="x-unicode"><br>
    </div>
    <div class="moz-text-flowed"
    style="font-family: -moz-fixed; font-size: 12px;" lang="x-unicode">As
    you're probably aware, www-client/chromium is a frequently-updated
    <br>
    package with a not-insignificant build. Chromium has an enormous
    attack
    <br>
    surface and is used to parse often-untrusted data from the web;
    this
    <br>
    results in frequent CVEs due to the back-and-forth between black
    hats
    <br>
    and upstream developers, and upstream's (quite generous) bug
    bounty
    <br>
    program.
    <br>
    <br>
    Fortunately for us, upstream has a fixed 4-week release cycle[1]:
    <br>
    Every 4 weeks the 'beta' channel is promoted to 'stable', the
    <br>
    'dev' channel is promoted to 'beta'; each channel is 'refreshed'
    <br>
    weekly. The 'stable' refreshes typically contain security fixes
    and
    <br>
    urgent regression fixes (at the discretion of the upstream release
    <br>
    team).
    <br>
    <br>
    Approximately 48 weeks of the year we package an update for the
    'stable'
    <br>
    channel for ::gentoo. The current stabilisation process often
    delays the
    <br>
    delivery of these security updates to users running stable
    keywords by
    <br>
    hours to days depending on arch-tester availability.
    <br>
    <br>
    Upstream provide official binaries for (and even encourage
    side-by-side
    <br>
    installs of) each channel on Windows and Mac. Upstream also
    support
    <br>
    amd64 and arm64 arches on Linux and have a robust development
    <br>
    process with significant investment in CI/CD infrastructure.
    <br>
    <br>
    In ::gentoo we currently carry all 3 channels, with 'dev' being
    <br>
    unkeyworded, 'beta' keyworded for 'testing', and 'stable' going
    through
    <br>
    the standard arch stabilisation process for amd64 and arm64 (with
    <br>
    'ppc64' 'testing' keywords when patches are available).
    <br>
    <br>
    Between the (Gentoo) Chromium project building and testing the
    software
    <br>
    and various arch testers tying up resources that could otherwise
    be used
    <br>
    to look at other packages there's a reasonable case to be made
    that
    <br>
    overall we're spending 10+ hours on a process that may be
    superfluous.
    <br>
    <br>
    With all of this in mind, I'd like to see about changing the way
    that we
    <br>
    handle stabilisation of www-client/chromium for amd64 and arm64. I
    see
    <br>
    two options that significantly reduce packaging overhead and, more
    <br>
    importantly, enable us to get security fixes into the hands of our
    users
    <br>
    more quickly (hours rather than days after an upstream release)
    and
    <br>
    free up valuable arch tester time.
    <br>
    <br>
    I see two options for handling this better:
    <br>
    <br>
    1. File a stablereq on promotion to 'stable' channel for each
    major
    <br>
       version and commit any updates directly with stable keywords.
    <br>
    <br>
    Pros: Provides a checkpoint to ensure dependent packages
    (dev-build/gn,
    <br>
    media-video/ffmpeg-chromium) are keyworded, allows for a brief
    review
    <br>
    period.
    <br>
    <br>
    Cons: Still introduces some delay in delivering updates, might be
    <br>
    unnecessary given upstream's robust testing and the nature of the
    <br>
    'stable' channel which only receives backported security fixes.
    <br>
    <br>
    2. Commit the 'stable' channel with stable keywords unless there's
    a
    <br>
       specific reason to do otherwise.
    <br>
    <br>
    Pros: Fastest possible delivery of security updates, minimises
    packaging
    <br>
    overhead.
    <br>
    <br>
    Cons: Relies more heavily on upstream's stability, less
    opportunity for
    <br>
    Gentoo-specific review before release. We would need a clear
    policy for
    <br>
    handling potential issues and rollbacks, possibly retaining more
    package
    <br>
    versions in-tree and demoting a release to 'testing' as required.
    <br>
    <br>
    My Recommendation: I lean towards option 2 as it prioritises
    getting
    <br>
    security fixes to users as quickly as possible. The 'stable'
    channel's
    <br>
    design and Google's extensive CI/CD infrastructure give a good
    level
    <br>
    of comfort for this approach. For managing dependencies like
    <br>
    dev-build/gn and media-video/ffmpeg-chromium, we can rely on
    pkgcheck
    <br>
    and our own CI (hi croaker) to ensure that keywords are kept in
    sync.
    <br>
    To mitigate the reduced opportunity for Gentoo-specific review, we
    will
    <br>
    retain the previous 2 stable 'refreshes' in the tree, allowing for
    swift
    <br>
    demotion to 'testing' should any critical issues arise. We will
    also
    <br>
    establish a clear policy for demoting a release based on user
    reports.
    <br>
    <br>
    As a stretch goal it would be possible to action the keyword
    updates for
    <br>
    dependencies like dev-build/gn and media-video/ffmpeg-chromium
    using
    <br>
    some of the existing automation that is used to monitor for new
    upstream
    <br>
    releases, though I'm wary of automating too much of this and would
    need
    <br>
    to look at Gentoo policy.
    <br>
    <br>
    I'm interested in your thoughts on this, or if continuing with the
    <br>
    status-quo is otherwise desirable from an arch project
    perspective.
    <br>
    Specifically, I'd like to know if you have any reservations about
    <br>
    adopting option 2.
    <br>
    <br>
    If we want to evaluate the success of this change, we can track
    the time
    <br>
    between upstream releases and the availability of stable keywords
    in
    <br>
    Gentoo, as well as monitoring for any increase in bug reports
    related to
    <br>
    Chromium's stable channel. We can also leverage our existing
    security
    <br>
    bugs to track the number of security vulnerabilities addressed
    more
    <br>
    promptly under the new approach. I'm also interested in any other
    <br>
    thoughts on evaluating its success if we do go ahead, or whether
    we
    <br>
    bother with any of this at all and just watch for a sudden flood
    of bugs
    <br>
    logged against the stable channel.
    <br>
    <br>
    Thanks for your time.
    <br>
    <br>
    Cheers,
    <br>
    <br>
    Matt
    <br>
    <br>
    1: <a class="moz-txt-link-freetext" href="https://chromium.googlesource.com/chromium/src/+/HEAD/docs/process/release_cycle.md">https://chromium.googlesource.com/chromium/src/+/HEAD/docs/process/release_cycle.md</a>
    <br>
    </div>
    </body>
    </html>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)