Hello all.

I wrote a shell scraper of a news website and one of the options is to
keep re-accessing the initial webpage (while loop) at regular intervals,
grab all news links and scrape text of the ones which are new. This
option keeps the script running for days straight. When stdout is
redirected to a *file*, it works as expected.

Instead if we pipe output to `less', there is a buffering stand. When
`less' buffering is full at about 8-64KB, the whole while loop in the
script hangs and only continues to run when we scroll down `less'
display buffer. But if it hangs for a few hours, it means the scraping
tool will only resume scraping some hours later, too, failing to scrape
a lot of news from the initial webpage during that time.

I tried piping to `less -B -b4096' and `less -B -b-1' to no avail:

-B or --auto-buffers
By default, when data is read from a pipe, buffers are allocated
automatically as needed. If a large amount of data is read from
the pipe, this can cause a large amount of memory to be allo‐
cated. The -B option disables this automatic allocation of buf‐
fers for pipes, so that only 64 KB (or the amount of space spec‐
ified by the -b option) is used for the pipe.

-bn or --buffers=n
Specifies the amount of buffer space less will use for each
file, in units of kilobytes (1024 bytes). By default 64 KB of
buffer space is used for each file (unless the file is a pipe; see
the -B option). The -b option specifies instead that n kilobytes
of buffer space should be used for each file. If n is -1, buffer
space is unlimited; that is, the entire file can be read into
memory.


I use GNU coreutils and Arch Linux. My understanding is that `less
-Bb-1' should have worked but there is some internal buffering system
that still holds. Any suggestion? I know of `stdbuf', tried a little but
could not make it work. If I was not clear, please let me know.

Thanks,
JSN

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

frogger <[email protected]> writes:

I wrote a shell scraper of a news website and one of the options is to
keep re-accessing the initial webpage (while loop) at regular
intervals, grab all news links and scrape text of the ones which are
new. This option keeps the script running for days straight. When
stdout is redirected to a *file*, it works as expected.

Instead if we pipe output to `less', there is a buffering stand. When
`less' buffering is full at about 8-64KB, the whole while loop in the
script hangs and only continues to run when we scroll down `less'
display buffer. But if it hangs for a few hours, it means the scraping
tool will only resume scraping some hours later, too, failing to
scrape a lot of news from the initial webpage during that time.

I'm not 100% sure what you want, but does:

$ scraper >file & less file

and then using the F command do something like you want?

--
Ben.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 19/03/2022 19:06, Ben Bacarisse wrote:

frogger <[email protected]> writes:

I wrote a shell scraper of a news website and one of the options is to
keep re-accessing the initial webpage (while loop) at regular
intervals, grab all news links and scrape text of the ones which are
new. This option keeps the script running for days straight. When
stdout is redirected to a *file*, it works as expected.

Instead if we pipe output to `less', there is a buffering stand. When
`less' buffering is full at about 8-64KB, the whole while loop in the
script hangs and only continues to run when we scroll down `less'
display buffer. But if it hangs for a few hours, it means the scraping
tool will only resume scraping some hours later, too, failing to
scrape a lot of news from the initial webpage during that time.

I'm not 100% sure what you want, but does:

$ scraper >file & less file

and then using the F command do something like you want?

Hey Ben!

That is *exactly* what I am trying just now. Was not sure it would be a
good solution but it seems straightforward that is, as it was so obvious
to you.

So what I just did and works is:

scraperFunction >file & tail -f file | less

Also added a trap inside the script to kill forks and remove temp file.

Thanks, Ben!
JSN

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)