1. Forum
  2. Usenet
  3. COMP.UNIX.SHELL

  • Run multiple make in parallel with appropriate cores settings for each

    From [email protected]@21:1/5 to All on Mon Apr 25 19:50:08 2022
    I noticed the BuildPackages.sh [1] script is used by GAP to build the packages supported by it. Because there are so many packages, there is another option in the script to support parallel make. The following code snippet is the relevant logic
    corresponding to this option implemented in this script:

    ```
    PARALLEL=no
    [...]
    while [[ "$#" -ge 1 ]]; do
    option="$1" ; shift
    case "$option" in
    [...]
    --parallel) PARALLEL=yes; ;;
    [...]
    esac
    done
    [...]

    if [ "x$PARALLEL" = "xyes" ] && [ "x$STRICT" = "xyes" ]; then
    error "The options --strict and --parallel cannot be used simultaneously"
    fi

    if [ "x$PARALLEL" = "xyes" ]; then
    export MAKEFLAGS="${MAKEFLAGS:--j3}"
    fi;

    [...]

    BUILD_PID=$!
    if [ "x$PARALLEL" = "xyes" ]; then
    # If more than 4 background jobs are running, wait for one to finish (if
    # <wait -n> is available) or for all to finish (if only <wait> is available)
    if [[ $(jobs -r -p | wc -l) -gt 4 ]]; then
    wait -n 2>&1 >/dev/null || wait
    fi
    else
    # wait for this package to finish building
    if ! wait $BUILD_PID && [[ $STRICT = yes ]]
    then
    exit 1
    fi
    fi;
    ```
    As you can see, by default, the parallel make is disabled, and when the `--parallel` option is used, 3 cores will be assigned to each make process.

    I want to enhance this logic and tried the following simple and crude modification [2]:

    Add the following line:

    NCORE=$(sudo dmidecode -t 4 | grep 'Core Enabled:' | awk '{a+=$NF}END{print a}')

    And change the following two lines respectively:

    Change
    export MAKEFLAGS="${MAKEFLAGS:--j3}"

    to

    export MAKEFLAGS="${MAKEFLAGS:--j$NCORE}"


    Change

    if [[ $(jobs -r -p | wc -l) -gt 4 ]]; then

    to

    if [[ $(jobs -r -p | wc -l) -gt $((NCORE +1)) ]]; then

    But the above method has the drawbacks and possibly more feasible alternatives, as commented here [3]:

    ```
    Note that by telling both make and jobs to use that many cores, you are in fact instructing the system to use ~NCORE^2 cores.

    One could possibly rearrange things so that a single make job server is shared across all jobs, then it's be "only" 2*NCORE jobs.
    ```

    But I don't know how to implement this or other more powerful and feasible solutions to solve the problems discussed here. Any hints will be highly appreciated.

    [1] https://github.com/gap-system/gap/blob/master/bin/BuildPackages.sh
    [2] https://github.com/gap-system/gap/pull/4879/commits/f0375b2be81900687e678e2c61a28f6643dffb97
    [3] https://github.com/gap-system/gap/pull/4879#discussion_r857318524

    Regards,
    HZ

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Richard Harnden@21:1/5 to [email protected] on Tue Apr 26 12:13:41 2022
    On 26/04/2022 03:50, [email protected] wrote:
    I noticed the BuildPackages.sh [1] script is used by GAP to build the packages supported by it. Because there are so many packages, there is another option in the script to support parallel make. The following code snippet is the relevant logic
    corresponding to this option implemented in this script:

    ```
    PARALLEL=no
    [...]
    while [[ "$#" -ge 1 ]]; do
    option="$1" ; shift
    case "$option" in
    [...]
    --parallel) PARALLEL=yes; ;;
    [...]
    esac
    done
    [...]

    if [ "x$PARALLEL" = "xyes" ] && [ "x$STRICT" = "xyes" ]; then
    error "The options --strict and --parallel cannot be used simultaneously" fi

    if [ "x$PARALLEL" = "xyes" ]; then
    export MAKEFLAGS="${MAKEFLAGS:--j3}"
    fi;

    [...]

    This [...] desends into a subdir and starts the build in the background.


    BUILD_PID=$!
    if [ "x$PARALLEL" = "xyes" ]; then
    # If more than 4 background jobs are running, wait for one to finish (if
    # <wait -n> is available) or for all to finish (if only <wait> is available)
    if [[ $(jobs -r -p | wc -l) -gt 4 ]]; then
    wait -n 2>&1 >/dev/null || wait
    fi
    else
    # wait for this package to finish building
    if ! wait $BUILD_PID && [[ $STRICT = yes ]]
    then
    exit 1
    fi
    fi;

    So: either wait for each background job in finish if PARALLEL != yes, or
    let 4 jobs run at once.

    ```
    As you can see, by default, the parallel make is disabled, and when the `--parallel` option is used, 3 cores will be assigned to each make process.

    I want to enhance this logic and tried the following simple and crude modification [2]:

    You make everything way too complicated.

    I would say that if you want to start 4 build jobs at a time, then you
    should reduce -j3 ; not try to increace it. If you do anything at all.

    I would just leave it alone. You are trying to add complexity for zero benefit.


    Add the following line:

    NCORE=$(sudo dmidecode -t 4 | grep 'Core Enabled:' | awk '{a+=$NF}END{print a}')

    And change the following two lines respectively:

    Change
    export MAKEFLAGS="${MAKEFLAGS:--j3}"

    to

    export MAKEFLAGS="${MAKEFLAGS:--j$NCORE}"


    Change

    if [[ $(jobs -r -p | wc -l) -gt 4 ]]; then

    to

    if [[ $(jobs -r -p | wc -l) -gt $((NCORE +1)) ]]; then

    But the above method has the drawbacks and possibly more feasible alternatives, as commented here [3]:

    ```
    Note that by telling both make and jobs to use that many cores, you are in fact instructing the system to use ~NCORE^2 cores.

    One could possibly rearrange things so that a single make job server is shared across all jobs, then it's be "only" 2*NCORE jobs.
    ```

    But I don't know how to implement this or other more powerful and feasible solutions to solve the problems discussed here. Any hints will be highly appreciated.

    [1] https://github.com/gap-system/gap/blob/master/bin/BuildPackages.sh
    [2] https://github.com/gap-system/gap/pull/4879/commits/f0375b2be81900687e678e2c61a28f6643dffb97
    [3] https://github.com/gap-system/gap/pull/4879#discussion_r857318524

    Regards,
    HZ

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Kees Nuyt@21:1/5 to [email protected] on Tue Apr 26 14:05:25 2022
    On Tue, 26 Apr 2022 12:13:41 +0100, Richard Harnden
    <[email protected]> wrote:

    You make everything way too complicated.

    I would say that if you want to start 4 build jobs at a time, then you
    should reduce -j3 ; not try to increace it. If you do anything at all.

    I would just leave it alone. You are trying to add complexity for zero benefit.

    +1
    --
    Kees Nuyt

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)