• wget Uses Multiple Connections??

    From Lawrence D'Oliveiro@21:1/5 to All on Thu Jun 12 00:33:34 2025
    Jack Wallen’s enthusiasm, I think, exceeds his grasp on facts, again <https://www.zdnet.com/article/heres-a-faster-way-to-download-files-on-linux-without-a-web-browser/>.

    Yes, wget is a convenient way to download large items, and lots of
    items: give it a list of URLs, and it will go away and do them all,
    quietly, in the background. It has lots of options for controlling its automatic retry-on-failure feature, too.

    But nowhere in the docs do I find mention that it can take advantage
    of multiple simultaneous connections to speed up a download.

    In the standard Debian repo, I see a separate package called “aria2”,
    that does indeed advertise the ability to use multiple simultaneous
    connections to speed up downloads <https://aria2.github.io/>.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Lawrence D'Oliveiro@21:1/5 to John McCue on Thu Jun 12 02:14:59 2025
    On Thu, 12 Jun 2025 01:47:23 -0000 (UTC), John McCue wrote:

    But I do remember reading somewhere if using wget, if downloads
    are too fast the site will kill the download.

    There is a flag to fake random downloads, "--random-wait", to
    prevent that logic that could exist on the remote site.

    Also --limit-rate, options for waiting before retrying etc. And you can
    use an alternative --user-agent (for sites that block the default one),
    and even specify a “Referer” [sic] header, for those sites that try to prevent deep-linking.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From John McCue@21:1/5 to Lawrence D'Oliveiro on Thu Jun 12 01:47:23 2025
    Lawrence D'Oliveiro <[email protected]d> wrote:
    Jack Wallen?s enthusiasm, I think, exceeds his grasp on facts, again <https://www.zdnet.com/article/heres-a-faster-way-to-download-files-on-linux-without-a-web-browser/>.

    <snip>

    But nowhere in the docs do I find mention that it can take advantage
    of multiple simultaneous connections to speed up a download.

    I did not see that either, maybe they are referring to wget2 ?

    But I do remember reading somewhere if using wget, if downloads
    are too fast the site will kill the download.

    There is a flag to fake random downloads, "--random-wait", to
    prevent that logic that could exist on the remote site.

    In the standard Debian repo, I see a separate package called ?aria2?,
    that does indeed advertise the ability to use multiple simultaneous connections to speed up downloads <https://aria2.github.io/>.

    I never heard of "aria2", but the repo of my system does have
    it:

    aria2-1.36.0nb12 - Multi-threaded, multi-protocol, flexible
    download accelerator:

    Description:

    aria2 is a utility for downloading files. It has completely
    new design concept from its predecessor, Aria, and is written
    from scratch. aria2 has a segmented downloading engine in its
    core. It can download one file from multiple URLs or multiple
    connections from one URL. This results in very high speed
    downloading, much faster than ordinary browsers. This engine
    in was implemented in a single-thread model. The architecture
    is clean and easy to extend. aria2 currently supports HTTP,
    FTP, and BitTorrent. It also supports Metalink version 3.0.
    Please visit the project web site at http://aria2.sourceforge.net

    Homepage:
    https://aria2.github.io/


    --
    [t]csh(1) - "An elegant shell, for a more... civilized age."
    - Paraphrasing Star Wars

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Borax Man@21:1/5 to John McCue on Thu Jun 12 10:56:16 2025
    On 2025-06-12, John McCue <[email protected]> wrote:
    Lawrence D'Oliveiro <[email protected]d> wrote:
    Jack Wallen?s enthusiasm, I think, exceeds his grasp on facts, again
    <https://www.zdnet.com/article/heres-a-faster-way-to-download-files-on-linux-without-a-web-browser/>.

    <snip>

    But nowhere in the docs do I find mention that it can take advantage
    of multiple simultaneous connections to speed up a download.

    I did not see that either, maybe they are referring to wget2 ?

    But I do remember reading somewhere if using wget, if downloads
    are too fast the site will kill the download.

    There is a flag to fake random downloads, "--random-wait", to
    prevent that logic that could exist on the remote site.

    In the standard Debian repo, I see a separate package called ?aria2?,
    that does indeed advertise the ability to use multiple simultaneous
    connections to speed up downloads <https://aria2.github.io/>.

    I never heard of "aria2", but the repo of my system does have
    it:

    aria2-1.36.0nb12 - Multi-threaded, multi-protocol, flexible
    download accelerator:

    Description:

    aria2 is a utility for downloading files. It has completely
    new design concept from its predecessor, Aria, and is written
    from scratch. aria2 has a segmented downloading engine in its
    core. It can download one file from multiple URLs or multiple
    connections from one URL. This results in very high speed
    downloading, much faster than ordinary browsers. This engine
    in was implemented in a single-thread model. The architecture
    is clean and easy to extend. aria2 currently supports HTTP,
    FTP, and BitTorrent. It also supports Metalink version 3.0.
    Please visit the project web site at http://aria2.sourceforge.net

    Homepage:
    https://aria2.github.io/


    I use aria2 as my downloader, more specifically, I have an aria2c daemon
    that runs in the background, and I can add files to download and
    torrents, and they'll just work away in the background. Downloads will continue if you shutdown your computer and restart later.

    Aria2 does support downloading from multiple sources, its one of its
    selling points. Never seen this in wget.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Carlos E.R.@21:1/5 to Borax Man on Thu Jun 12 18:55:58 2025
    On 2025-06-12 12:56, Borax Man wrote:
    On 2025-06-12, John McCue <[email protected]> wrote:
    Lawrence D'Oliveiro <[email protected]d> wrote:

    ...

    I use aria2 as my downloader, more specifically, I have an aria2c daemon
    that runs in the background, and I can add files to download and
    torrents, and they'll just work away in the background. Downloads will continue if you shutdown your computer and restart later.

    Aria2 does support downloading from multiple sources, its one of its
    selling points. Never seen this in wget.

    Another is it supports metalinks.

    --
    Cheers, Carlos.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Computer Nerd Kev@21:1/5 to Lawrence D'Oliveiro on Fri Jun 13 09:03:46 2025
    Lawrence D'Oliveiro <[email protected]d> wrote:
    Jack Wallen's enthusiasm, I think, exceeds his grasp on facts, again <https://www.zdnet.com/article/heres-a-faster-way-to-download-files-on-linux-without-a-web-browser/>.

    One wonders why you keep reading this clickbait stuff then.

    Yes, wget is a convenient way to download large items, and lots of
    items: give it a list of URLs, and it will go away and do them all,
    quietly, in the background. It has lots of options for controlling its automatic retry-on-failure feature, too.

    Far more than that. The recursive mode is extremely useful for
    mirroring sections of websites or downloading files in a
    directory index. Unfortunately most of the time it takes many
    attempts to stop it from accidentally trying to download the
    entire internet, but the web is just not well suited to that
    a lot of the time (alternatives like HTTTrack are more capable
    but less convenient). When you have a plain server-generated
    directory list it works very well.

    The "--spider" option is great for checking the size of download
    links before deciding if/where to download them.

    I also just set up a system using "wget --post-file=" to submit
    logs to a Bash CGI script on a web server which sorts and stores
    them. Should be more reliable than using scp and running into all
    the endless incompatibility pain you get with anything SSH these
    days (Dropbear doesn't even support SFTP which OpenSSH calls 'scp'
    too now).

    But nowhere in the docs do I find mention that it can take advantage
    of multiple simultaneous connections to speed up a download.

    True it doesn't do that. It will reuse the same connection to fetch
    multiple files from the same server sequentially though.

    In the standard Debian repo, I see a separate package called "aria2",
    that does indeed advertise the ability to use multiple simultaneous connections to speed up downloads <https://aria2.github.io/>.

    I've used "lftp" for that in the past, but I don't need it often.

    --
    __ __
    #_ < |\| |< _#

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From candycanearter07@21:1/5 to Lawrence D'Oliveiro on Fri Jun 13 01:10:04 2025
    Lawrence D'Oliveiro <[email protected]d> wrote at 00:33 this Thursday (GMT):
    Jack Wallen’s enthusiasm, I think, exceeds his grasp on facts, again
    <https://www.zdnet.com/article/heres-a-faster-way-to-download-files-on-linux-without-a-web-browser/>.

    Yes, wget is a convenient way to download large items, and lots of
    items: give it a list of URLs, and it will go away and do them all,
    quietly, in the background. It has lots of options for controlling its automatic retry-on-failure feature, too.

    But nowhere in the docs do I find mention that it can take advantage
    of multiple simultaneous connections to speed up a download.

    In the standard Debian repo, I see a separate package called “aria2”, that does indeed advertise the ability to use multiple simultaneous connections to speed up downloads <https://aria2.github.io/>.


    aria2 is very useful, and I have it aliased over wget.
    --
    user <candycane> is generated from /dev/urandom

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Lawrence D'Oliveiro@21:1/5 to Computer Nerd Kev on Fri Jun 13 23:30:43 2025
    On 13 Jun 2025 09:03:46 +1000, Computer Nerd Kev wrote:

    Lawrence D'Oliveiro <[email protected]d> wrote:
    Jack Wallen's enthusiasm, I think, exceeds his grasp on facts, again
    <https://www.zdnet.com/article/heres-a-faster-way-to-download-files-on-linux-without-a-web-browser/>.

    One wonders why you keep reading [him].

    He has more patience for trying different Linux distros than I do.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ant@21:1/5 to Lawrence D'Oliveiro on Sun Jun 15 01:18:31 2025
    Lawrence D'Oliveiro <[email protected]d> wrote:
    Jack Wallen?s enthusiasm, I think, exceeds his grasp on facts, again <https://www.zdnet.com/article/heres-a-faster-way-to-download-files-on-linux-without-a-web-browser/>.

    Yes, wget is a convenient way to download large items, and lots of
    items: give it a list of URLs, and it will go away and do them all,
    quietly, in the background. It has lots of options for controlling its automatic retry-on-failure feature, too.

    But nowhere in the docs do I find mention that it can take advantage
    of multiple simultaneous connections to speed up a download.

    In the standard Debian repo, I see a separate package called ?aria2?,
    that does indeed advertise the ability to use multiple simultaneous connections to speed up downloads <https://aria2.github.io/>.

    Original wget or wget2?
    --
    "Who shall separate us from the love of Christ? Shall trouble or hardship or persecution or famine or nakedness or danger or sword? ... No, in all these things we are more than conquerors through him who loved us." --Romans 8:35-37. Will 2day B loco in
    USA with its protests, festival, parade, DJT's 79th BD, flag day, etc.?
    Note: A fixed width font (Courier, Monospace, etc.) is required to see this signature correctly.
    /\___/\ Ant(Dude) @ http://aqfl.net & http://antfarm.home.dhs.org.
    / /\ /\ \ Please nuke ANT if replying by e-mail.
    | |o o| |
    \ _ /
    ( )

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From candycanearter07@21:1/5 to Ant on Wed Jun 18 05:30:05 2025
    Ant <[email protected]> wrote at 01:18 this Sunday (GMT):
    Lawrence D'Oliveiro <[email protected]d> wrote:
    Jack Wallen?s enthusiasm, I think, exceeds his grasp on facts, again
    <https://www.zdnet.com/article/heres-a-faster-way-to-download-files-on-linux-without-a-web-browser/>.

    Yes, wget is a convenient way to download large items, and lots of
    items: give it a list of URLs, and it will go away and do them all,
    quietly, in the background. It has lots of options for controlling its
    automatic retry-on-failure feature, too.

    But nowhere in the docs do I find mention that it can take advantage
    of multiple simultaneous connections to speed up a download.

    In the standard Debian repo, I see a separate package called ?aria2?,
    that does indeed advertise the ability to use multiple simultaneous
    connections to speed up downloads <https://aria2.github.io/>.

    Original wget or wget2?


    There's a sequel?
    --
    user <candycane> is generated from /dev/urandom

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)