1. Forum
  2. Usenet
  3. UK.COMP.OS.LINUX

  • cgroups are not nice

    From #Paul@21:1/5 to All on Tue Dec 5 19:19:07 2023
    Does anyone here understand cgroups?

    I have to put up with running some really quite intensive processing
    on an ubuntu machine (running 20.04 LTS, on the basis of uname -a)
    that I do not admin (or have any sudo privileges). You may assume I
    have zero influence over those who do admin it.

    Just as some context, generally each processing run - managed by a
    bash script - creates as many background jobs (via &, but not nohup)
    as I need to saturate the cpu cores. Some processing is run manually
    from a terminal, some automatically from cron.

    Once upon a time I could use nice (and/or renice) to change priorities,
    so that things I wanted finished sooner would run faster than ones I
    was happy to wait for; but if the fast ones finished, I didn't have to
    waste any cpu time as the others jobs could take up the slack.


    Now we have "cgroups", which no doubt has many advantages, but seems
    imo bizarrely complicated. And although I can still "nice" processes,
    it never makes any difference at all -- not even between a set of jobs
    created by the same shell or processing script, which I would have
    thought of as sensible, and as seems to be implied by some doco as what
    should happen (but ime does not).


    Unfortunately, despite several attempts, I can't make head or tail of
    what to do (if anything can be done) from any of: the cgroups
    documentation, various blog posts, posts on stackoveflow/exchange,
    "askubuntu", etc. Many of the what-to-do" "explanations" seem to
    involve being root and/or sudo; and the most useful looking questions
    I have found are without answers.

    Can anyone help?

    #Paul

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Richard Kettlewell@21:1/5 to [email protected]. on Tue Dec 5 22:14:33 2023
    [email protected] (#Paul) writes:
    Does anyone here understand cgroups?

    I have to put up with running some really quite intensive processing
    on an ubuntu machine (running 20.04 LTS, on the basis of uname -a)
    that I do not admin (or have any sudo privileges). You may assume I
    have zero influence over those who do admin it.

    Just as some context, generally each processing run - managed by a
    bash script - creates as many background jobs (via &, but not nohup)
    as I need to saturate the cpu cores. Some processing is run manually
    from a terminal, some automatically from cron.

    Once upon a time I could use nice (and/or renice) to change priorities,
    so that things I wanted finished sooner would run faster than ones I
    was happy to wait for; but if the fast ones finished, I didn't have to
    waste any cpu time as the others jobs could take up the slack.


    Now we have "cgroups", which no doubt has many advantages, but seems
    imo bizarrely complicated. And although I can still "nice" processes,
    it never makes any difference at all -- not even between a set of jobs created by the same shell or processing script, which I would have
    thought of as sensible, and as seems to be implied by some doco as what should happen (but ime does not).

    I think it would be worth investigating why you can’t get ‘nice’
    working. For example, start by checking what nice value your processes
    get when you do or don’t nice them.

    Unfortunately, despite several attempts, I can't make head or tail of
    what to do (if anything can be done) from any of: the cgroups
    documentation, various blog posts, posts on stackoveflow/exchange, "askubuntu", etc. Many of the what-to-do" "explanations" seem to
    involve being root and/or sudo; and the most useful looking questions
    I have found are without answers.

    AFAIK directly createing a control group normally needs root
    permission. But you can create a user slice with a CPU resource policy
    attached and systemd will create a suitable one for you.

    A very simplistic example:

    richard@sfere:~$ cat .config/systemd/user/test.slice
    [Slice]
    CPUQuota=50%
    richard@sfere:~$ systemctl --user daemon-reload
    richard@sfere:~$ systemd-run --user --slice test.slice sh -c 'yes > /dev/null'
    Running as unit: run-r2fa6a01b9fdb4ce49d208f7f58f8187c.service

    [...]

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    2504272 richard 20 0 5792 864 780 R 50.0 0.0 0:24.58 yes
    805434 news 20 0 122264 96872 52212 S 1.0 0.6 40:13.91 innd

    See ‘man systemd.resource-control’ for further options. (You should probably find an introduction to systemd first if you’re not already comfortable with it.)

    --
    https://www.greenend.org.uk/rjk/

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Grant Taylor@21:1/5 to All on Wed Dec 6 10:33:41 2023
    On 12/5/23 13:19, #Paul wrote:
    Does anyone here understand cgroups?

    I have a vague understanding of them.

    Once upon a time I could use nice (and/or renice) to change priorities,
    so that things I wanted finished sooner would run faster than ones I
    was happy to wait for; but if the fast ones finished, I didn't have to
    waste any cpu time as the others jobs could take up the slack.

    Sounds like you were leveraging (,re,io)nice to adjust the weight /
    priority of your processes as you see fit. I would consider this to be
    a very traditionally Unix thing to do.

    Now we have "cgroups", which no doubt has many advantages, but seems
    imo bizarrely complicated. And although I can still "nice" processes,
    it never makes any difference at all -- not even between a set of jobs created by the same shell or processing script, which I would have
    thought of as sensible, and as seems to be implied by some doco as what should happen (but ime does not).

    My speculation is that more of your processes than you care for are run
    in different (instances of a) cgroup(s). So when you're shuffling
    priority as you had been doing, you're probably doing so against one or
    maybe two other processes inside of a cgroup. Unfortunately this has no relation to much less impact on or between other cgroup(s).

    Can anyone help?

    I would work with the people that do have root / sudo access to the
    system your on and get them to help enable you to do what you want.
    Even if that means that all of your processes end up in one cgroup so
    that you can shuffle things therein as you desire.

    This seems to me like what a system administrator is supposed to do, as
    in help their users achieve their desired goal.

    But, sadly, it sounds like you might have more of a dictator form of
    system administrator.

    I'm sorry, but I don't have an answer for you.



    --
    Grant. . . .

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From #Paul@21:1/5 to All on Fri Dec 15 10:32:07 2023
    Richard Kettlewell <[email protected]d> wrote:
    [...]

    Grant Taylor <[email protected]> wrote:
    [...]

    Thanks both, the extra context helps. I'll try to remember to
    post back here if I manage to get something to work out, but
    atm the box is less heavily loaded atm so it's not such a
    priority.

    #Paul

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)