1. Forum
  2. Usenet
  3. COMP.UNIX.SHELL

  • Re: Let cat use multiple cores for parallel processing.

    From [email protected]@21:1/5 to [email protected] on Mon Apr 18 16:29:05 2022
    On Tuesday, April 19, 2022 at 7:27:48 AM UTC+8, [email protected] wrote:
    Today, I noticed that some of the discussions here are entitled

    See here [1] for the original discussion.

    [1] https://unix.stackexchange.com/questions/81705/can-the-cat-command-use-multi-processes

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From [email protected]@21:1/5 to All on Mon Apr 18 16:27:45 2022
    Today, I noticed that some of the discussions here are entitled

    ```
    can the "cat" command use multi-processes"
    ```

    At the bottom of the discussion, I see the following comment and script snippet:


    Update On further thought, this will only prefetch up to the pipe buffer size, 64k. The following modification will pull all of the files into memory; if they can't fit, don't use it:
    ```
    #!/bin/bash

    fd=3
    for file in *
    do
    eval exec $fd\<<(content=$(<$file); echo "$content")

    fd=$((fd + 1))
    done

    out_fd=3
    while [ "$out_fd" -lt "$fd" ]
    do
    cat <&${out_fd}
    out_fd=$((out_fd + 1))
    done
    ```

    But I'm still not sure whether I can let `cat` use multiple cores for parallel processing. Any hints will be highly appreciated.

    Regards,
    HZ

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From David W. Hodgins@21:1/5 to [email protected] on Mon Apr 18 20:24:55 2022
    On Mon, 18 Apr 2022 19:29:05 -0400, [email protected] <[email protected]> wrote:

    On Tuesday, April 19, 2022 at 7:27:48 AM UTC+8, [email protected] wrote:
    Today, I noticed that some of the discussions here are entitled

    See here [1] for the original discussion.

    [1] https://unix.stackexchange.com/questions/81705/can-the-cat-command-use-multi-processes

    The cat command is not designed to merge files and is the wrong tool to use to try and do so.

    The sort command has a merge option, which is suitable for use if the files are each already sorted.

    Regards, Dave Hodgins

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From David W. Hodgins@21:1/5 to [email protected] on Mon Apr 18 20:18:51 2022
    On Mon, 18 Apr 2022 19:27:45 -0400, [email protected] <[email protected]> wrote:
    But I'm still not sure whether I can let `cat` use multiple cores for parallel processing. Any hints will be highly appreciated.

    The cat command is supposed to concatenate one or more files, optionally with some
    modifications such as suppressing repeated empty output lines, and write them to a
    single output. The limited processing available in the cat command ensures it will
    be i/o bound, not cpu bound. The first bottleneck the cat command has to deal with
    is the write speed. Since there is only one output, there is no possible benefit,
    and lots of complications, to try to use more than one core.

    To even think of using multiple cores for the cat command, strongly suggests you
    should be using some other command.

    What are you trying to accomplish? Is this just a test question for some course?

    Ah. Just noticed your headers. You're using google groups, not usenet. After
    a quick search, the thread you're referring to is from 8 years ago, and it's answered in that thread, that there is no benefit.

    Regards, Dave Hodgins.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Aragorn@21:1/5 to All on Tue Apr 19 12:24:09 2022
    On 18.04.2022 at 20:18, David W. Hodgins scribbled:

    Ah. Just noticed your headers. You're using google groups, not
    usenet. After a quick search, the thread you're referring to is from
    8 years ago, and it's answered in that thread, that there is no
    benefit.

    It's Hongyi Zhao, the longstanding resident help vampire. Don't expect
    any clarification from him, ever. He only asks questions — whether
    on-topic or off-topic, he doesn't care — but he'll never answer any
    question asked of him, and he'll never tell you what he's trying to
    accomplish.

    The worst part is that everyone here in comp.unix.shell is always eager
    to reply to him, regardless of what he's asking, just to show everyone
    else how well they know the different POSIX standards, without ever
    questioning why Hongyi can't be bothered reading the manuals or doing
    his own research, not to mention what he's trying to concoct with his experiments.

    And well, ultimately, he's a Google Grouper. That should say it all.
    <shrug>


    --
    With respect,
    = Aragorn =

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)