On 1 Jun 2024 10:11 +0200, from [email protected]:

for years have i been using a self-made backup script [...]

I won't get into that -- I can't even fathom why you'd need coproc
for a backup script. I tend to keep things simple -- they tend to
thank me in failing less often and in more understandable ways.

I agree. There are plenty enough of ready-made backup solutions to
cater to different needs that making one's own shouldn't be needed.

Depending on the format you want your backup in, it's quite possible
that a simple rsync invocation would do. rsnapshot works on top of
rsync and gives you incremental backups with history. That's what I
use (plus some surrounding homegrown scripts, particularly one to
implement a more intelligent old backups purge policy than what
rsnapshot itself offers) and it has worked near perfectly for probably
a decade.

I didn't try your script, but may be there is a "\n" missing down
there?

printf "%s\n" "sleep 3;exit" >&6

^^^

^^

--
Michael Kjörling 🔗 https://michael.kjorling.se “Remember when, on the Internet, nobody cared that you were a dog?”

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sat, Jun 01, 2024 at 08:20:08AM +0000, Michael Kjörling wrote:

On 1 Jun 2024 10:11 +0200, from [email protected]:

for years have i been using a self-made backup script [...]

I won't get into that -- I can't even fathom why you'd need coproc
for a backup script. I tend to keep things simple -- they tend to
thank me in failing less often and in more understandable ways.

I agree. There are plenty enough of ready-made backup solutions to
cater to different needs that making one's own shouldn't be needed.

(Full disclosure: I do mine with a shell script wrapped around
rsync, and using its wonderful "dir-merge" feature to fine tune
what to leave out).

[...]

I didn't try your script, but may be there is a "\n" missing down
there?

printf "%s\n" "sleep 3;exit" >&6

^^^

^^

Good catch, thanks Michael -- to DdB: please, disregard my hunch. I
didn't look closely enough.

Cheers
--
t

-----BEGIN PGP SIGNATURE-----

iF0EABECAB0WIQRp53liolZD6iXhAoIFyCz1etHaRgUCZlrdnQAKCRAFyCz1etHa Rky1AJ9+23NBawsFhDGqlJVl/zfiff/WkgCfWrR/BjSwQZifNMFVeukgE5KnGoY=
=EFXP
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sat, Jun 01, 2024 at 09:20:59AM +0200, DdB wrote:

Hello,

for years have i been using a self-made backup script [...]

I won't get into that -- I can't even fathom why you'd need coproc
for a backup script. I tend to keep things simple -- they tend to
thank me in failing less often and in more understandable ways.

I didn't try your script, but may be there is a "\n" missing down
there?

printf "%s\n" "sleep 3;exit" >&6

^^^

That would be my first hunch.

Cheers
--
t

-----BEGIN PGP SIGNATURE-----

iF0EABECAB0WIQRp53liolZD6iXhAoIFyCz1etHaRgUCZlrXpAAKCRAFyCz1etHa RlyKAJ99dR2vMumsn/mJJ2qy2bqNSlsnLwCeL28h5Cw7ETwri7AmEB07rhVRUOo=
=ullW
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sat, Jun 01, 2024 at 10:53:45AM +0200, DdB wrote:

Am 01.06.2024 um 09:20 schrieb DdB:

Hello,

I get it: you wouldnt trust my scripts.

That wasn't the point. I'm just not in the situation to
debug it at the moment.

Thats fine with me. But my
experience is quite different: Software, i prefer using is such, that i
keep control.

Definitely -- that's why I concoct my backup scripts myself.
I was rather commenting at the concrete construction (that
is: concurrent processes communicating over two way pipes),
which seems somewhat fragile to me. If you haven't async
primitives, as in the shell, you might bump into deadlocks
sooner rather than later :-)

Cheers
--
t

-----BEGIN PGP SIGNATURE-----

iF0EABECAB0WIQRp53liolZD6iXhAoIFyCz1etHaRgUCZlrmlQAKCRAFyCz1etHa RsHnAJ0dl1FLq4pZRZpD+tk3n5HebfIHVwCfVftjW0k0erdjuSGdZ1ag1Lqam8c=
=lF77
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sat, Jun 01, 2024 at 09:20:59AM +0200, DdB wrote:

#!/bin/bash -e

coproc { bash; }
exec 5<&${COPROC[0]} 6>&${COPROC[1]}
fd=5

echo "ls" >&6
while IFS= read -ru $fd line
do
printf '%s\n' "$line"
done

printf "%s\n" "sleep 3;exit" >&6
while IFS= read -ru $fd line
do
printf '%s\n' "$line"
done

exec 5<&- 6>&-

wait
echo waited, done

i get the output from ls, but then the thing is hanging indefinitely, apparently not reaching the exit line. :(

Your first while loop never terminates. "while read ..." continues
running until read returns a nonzero exit status, either due to an
error or EOF. Your coproc never returns EOF, so the "while read"
loop just keeps waiting for the next line of output from ls.

If you're going to communicate with a long-running process that can
return multiple lines of output per line of input, then you have
three choices:

1) Arrange for some way to communicate how many lines, or bytes,
of output are going to be given.

2) Send a terminator line (or byte sequence) of some kind that
indicates "end of current data set".

3) Give up and assume the end of the data set after a certain amount
of time has elapsed with no new output arriving. (This is usually
not the best choice.)

These same design issues occur in any kind of network communication, too. Imagine an IMAP client or something, which holds open a network connection
to its IMAP server. The client asks for the body of an email message,
but needs to keep the connection open afterward so that it can ask for
more things later. The server has to be able to send the message back
without closing the connection at the end. Therefore, the IMAP protocol
needs some way to "encapsulate" each server response, so the client
knows when it has received the full message.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 6/1/24 00:20, DdB wrote:

Hello,

for years have i been using a self-made backup script, that did mount a
drive via USB, performed all kinds of plausibility checks, before
actually backing up incrementally. Finally verifying success and logging
the activities while kicking the ISB drive out.

Since a few months, i do have a real backup server instead, connecting
to it via ssh i was able to have 2 terminals open and back up manually.
Last time, i introduced a mistake by accident and since, i am trying to automate the whole thing once again, but that is difficult, as the load
on the net is huge, mbuffer is useful in that regard. So i was intending
to have just one script for all the operations using coproc to
coordinate the 2 servers.

But weird things are going on, i cant reliably communicate between host
and backup server, at least not automatically.

Searching the web, i found: https://github.com/reconquest/coproc.bash/blob/master/REFERENCE.md
But i was unable to get this to work, which seems to indicate, that i am misunderstanding something.
The only success i had was to "talk" to a chess engine in a coprocess,
which did go well. But neither bash nor ssh are cooperating, i may have timing issues with the pipes or some other side effects.

How can i describe? For example if i start this:

#!/bin/bash -e

coproc { bash; }
exec 5<&${COPROC[0]} 6>&${COPROC[1]}
fd=5

echo "ls" >&6
while IFS= read -ru $fd line
do
printf '%s\n' "$line"
done

printf "%s\n" "sleep 3;exit" >&6
while IFS= read -ru $fd line
do
printf '%s\n' "$line"
done

exec 5<&- 6>&-

wait
echo waited, done

i get the output from ls, but then the thing is hanging indefinitely, apparently not reaching the exit line. :(

Anyone who can share his experience to advance my experimenting?
DdB

https://en.wikipedia.org/wiki/XY_problem


Please define the root problem you are trying to solve.


David

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sat Jun 1, 2024 at 8:20 AM BST, DdB wrote:

for years have i been using a self-made backup script, that did mount a
drive via USB, performed all kinds of plausibility checks, before
actually backing up incrementally. Finally verifying success and logging
the activities while kicking the ISB drive out.

I'd keep using this for now, if I were you, and work on
implementing/fixing a replacement at the same time, and only stop doing
the simple solution when the newer solution has reached the same level
of reliability.

--
Please do not CC me for listmail.

👱🏻 Jonathan Dowland
✎ [email protected]
🔗 https://jmtd.net

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 6/2/24 21:35, DdB wrote:

Am 02.06.2024 um 02:41 schrieb DdB:

Will share my findings, once i made more progress...

Here is what i've got before utilizing it:

datakanja@PBuster-NFox:/mnt/tmp$ cat test
#!/bin/bash -e
# testing usefulness of coprocess to control host and backup machine from a single script.
# beware: do not use subprocesses or pipes, as that will confuse the pipes setup by coproc!
# At this point, this interface may not be very flexible
# but trying to follow best practices for using coproc in bash scripts
# todo (deferred): how to handle stderr inside coproc?
# todo (deferred): what, if coproc dies unexpectedly?

# setting up the coprocess:
stdout_to_ssh_stdin=5 # arbitrary choice outside the range of used file desciptors
stdin_from_ssh_stdout=6

coproc SSH { bash; } # for testing purposes, i refrain from really involving ssh just yet and replace it with bash:

# save filedescriptors by duplicating them:
eval "exec ${stdin_from_ssh_stdout}<&${SSH[0]} ${stdout_to_ssh_stdin}>&${SSH[1]}"
echo The PID of the coproc is: $SSH_PID # possibly useful for inspection

unique_eof_delimirer="<EOF>"
line=""
# collect the output available and print it locally (synchonous):
function print-immediate-output () {
while IFS= read -r -u "${stdin_from_ssh_stdout}" line
do
if [[ "${line:0-5:5}" == "$unique_eof_delimirer" ]] # currently, the length is fixed
then
line="${line%<EOF>}"
if [[ ! -z $line ]]
then
printf '%s\n' "$line"
fi
break
fi
printf '%s\n' "$line"
done
}

# send a single command via ssh and print output locally
function send-single-ssh-command () {
printf '%s\n' "$@" >&"${stdout_to_ssh_stdin}"
printf '%s\n' "echo '"$unique_eof_delimirer"'" >&"${stdout_to_ssh_stdin}"
print-immediate-output
}


send-single-ssh-command "find . -maxdepth 1 -name [a-z]\*" # more or less a standard command, that succeeds
send-single-ssh-command "ls nothin" # more or less a standard command, that fails

# tearing down the coprocess:
printf "%s\n" "exit" >&"${stdout_to_ssh_stdin}" # not interested in any more output (probably none)
wait
# Descriptors must be closed to prevent leaking.
eval "exec ${stdin_from_ssh_stdout}<&- ${stdout_to_ssh_stdin}>-"

echo "waited for the coproc to end gracefully, done"

datakanja@PBuster-NFox:/mnt/tmp$ ./test
The PID of the coproc is: 28154
./test
./out
ls: Zugriff auf 'nothin' nicht möglich: Datei oder Verzeichnis nicht gefunden
waited for the coproc to end gracefully, done
datakanja@PBuster-NFox:/mnt/tmp$

"test" is both a program and a shell builtin. I suggest that you pick
another, non-keyword, name for your script.


I suggest adding the Bash option "-u' (nounset).


Your file descriptor duplication, redirection, etc., seems overly
complex. Would not it be easier to use the coproc handles directly?

2024-06-03 08:49:41 dpchrist@laalaa ~/sandbox/bash
$ nl coproc-demo
1 #!/usr/bin/env bash
2 # $Id: coproc-demo,v 1.3 2024/06/03 15:49:36 dpchrist Exp $
3 set -e
4 set -u
5 coproc COPROC { bash ; }
6 echo 'echo "hello, world!"' >&"${COPROC[1]}"
7 read -r reply <&"${COPROC[0]}"
8 echo $reply
9 echo "exit" >&"${COPROC[1]}"
10 wait $COPROC_PID

2024-06-03 08:49:44 dpchrist@laalaa ~/sandbox/bash
$ bash -x coproc-demo
+ set -e
+ set -u
+ echo 'echo "hello, world!"'
+ bash
+ read -r reply
+ echo hello, 'world!'
hello, world!
+ echo exit
+ wait 4229


David

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)