1. Forum
  2. FidoNet
  3. SYNC PROGRAMMING

  • docs/zmodem_comparison.md

    From Rob Swindell (on Debian Linux)@1:103/705 to Git commit to main/sbbs/master on Thu Jul 23 23:03:46 2026
    https://gitlab.synchro.net/main/sbbs/-/commit/7471d1b776cb2d7ea274a41e
    Added Files:
    docs/zmodem_comparison.md
    Log Message:
    docs: add ZMODEM implementation comparison (sexyz vs lrzsz vs rzsz)

    A performance, scalability, and robustness analysis of Synchronet's ZMODEM (sexyz + the shared zmodem.c, which SyncTERM also uses) benchmarked against lrzsz 0.12.21rc and Chuck Forsberg's final rzsz 3.73 (modern branch).

    Headline findings, all empirically measured:

    - sexyz's sender is ~25x slower than lrzsz's lsz on a fast link (~8 vs
    ~204 MB/s), from two stacked overheads isolated by linking the real
    zmodem.o behind two send paths: zmodem.c's per-byte callback + escape +
    CRC design caps any sender at ~26 MB/s (shared, so SyncTERM inherits it),
    and sexyz.c's ring-buffer + output-thread drops that a further ~3x to
    ~8 MB/s (sexyz-only; a futex storm + ~84-byte writes). SyncTERM avoids
    the second layer, so its upload runs ~26 MB/s.
    - A windowed (-w) send hangs the instant the file offset crosses 2 GiB:
    zmodem.c narrows transmit-window/ACK positions to signed int32_t. Shared
    with SyncTERM. The data path (fseeko/int64) is otherwise 64-bit clean;
    4 GiB is a hard protocol ceiling (32-bit wire position field).
    - sexyz's receiver is fine past 2 GB; Forsberg's sender runs ~92 MB/s but
    its receiver won't complete headlessly (serial-tty assumptions).

    Includes a conditions matrix (block size, CRC, latency, bandwidth
    asymmetry, injected errors) and per-finding attribution to sexyz.c vs the shared zmodem.c vs SyncTERM's term.c. Relates to the sexyz-throughput and 2-GiB-window GitLab issues.

    Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
    --- SBBSecho 3.37-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell (on Debian Linux)@1:103/705 to Git commit to main/sbbs/master on Thu Jul 23 23:03:46 2026
    https://gitlab.synchro.net/main/sbbs/-/commit/f29244a40663ce7f99d3e4e6
    Modified Files:
    docs/zmodem_comparison.md
    Log Message:
    docs: note the sexyz SINGLE_THREADED build is not a throughput fix

    Document that sexyz.c:84 still carries a source-level
    `#define SINGLE_THREADED FALSE` (no makefile toggle), but its
    single-threaded send_byte (sexyz.c:682) writes one byte per write()
    syscall via sendbuf() — trading the two-thread futex storm for a write-syscall storm, generally worse. Add a three-way send-architecture
    table (threaded / single-threaded / buffered) and a recommendation caveat:
    the fix is output buffering (flush per subpacket, as SyncTERM's term.c
    does), not thread count.

    Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
    --- SBBSecho 3.37-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell (on Debian Linux)@1:103/705 to Git commit to main/sbbs/master on Thu Jul 23 23:03:46 2026
    https://gitlab.synchro.net/main/sbbs/-/commit/5c0e802235bfc797afbdadb1
    Modified Files:
    docs/zmodem_comparison.md
    Log Message:
    docs/zmodem: correct throughput magnitudes to steady-state; version-tag

    The initial throughput figures were measured on 32 MB transfers whose
    short duration was dominated by per-transfer startup, which deflated and distorted the rates. Re-measured at 256 MB steady-state:

    - sexyz sender was ~18x (not ~25x) slower than lsz: ~11 vs ~204 MB/s.
    - The two layers are ~7.5x sexyz.c (ring per-byte, 85->11, now fixed by
    the send-batching change -> ~66 MB/s) and ~2.4x zmodem.c (per-byte send
    design, 204->85, shared with SyncTERM), not the ~3x/~8x quoted before.
    - The SyncTERM-model (ztx_buf) sender runs ~85 MB/s, not ~26; sexyz
    receiver ~130; Forsberg sender ~107.

    The mechanism findings (futex storm, ~84-byte writes, ring-per-byte) were unaffected -- only the magnitudes changed.

    Also: tag every number to a component version (sexyz.c revision 3.3->3.4, zmodem.c rev 2.2->2.3) since Synchronet's side is actively changing, and
    mark the recommendations with status (Fix A and the 2 GB fix done, Fix B
    and the adaptive-block-length model pending).

    Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
    --- SBBSecho 3.37-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell (on Debian Linux)@1:103/705 to Git commit to main/sbbs/master on Thu Jul 23 23:03:46 2026
    https://gitlab.synchro.net/main/sbbs/-/commit/9c73c787d0f036e0eff35085
    Modified Files:
    docs/zmodem_comparison.md
    Log Message:
    docs/zmodem: record that the send-batching fixes were reverted

    Multi-run error-recovery testing showed both batching prototypes regress
    error recovery under heavy injected corruption: the shipped per-byte
    sender passes 3/3 (~50 s), the Fix A prototype (batch spans into the ring, sexyz.c) timed out 0/3, and the Fix B prototype (send_buf span callback, zmodem.c) hard-failed 1/3. Root cause: batching a whole subpacket before
    one blocking flush starves the back-channel during retransmit storms (the sender is stuck in the flush and can't service ZRPOS), stalling the
    transfer; the original per-byte-to-ring keeps the pipeline draining and
    stays responsive.

    Both prototypes were reverted. The only shipped code change remains the
    2 GB signed-position fix in zmodem.c (rev 2.3, GitLab #1196); sexyz.c is unchanged (stays 3.3). The ~66/~88 MB/s figures are retained in the doc, relabeled as reverted prototypes, because the root-cause analysis (ring per-byte -> futex storm / 84-byte writes) is still valid and the eventual
    fix must service the back-channel while sending.

    Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
    --- SBBSecho 3.37-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell (on Debian Linux)@1:103/705 to Git commit to main/sbbs/master on Sun Aug 30 17:37:54 2026
    https://gitlab.synchro.net/main/sbbs/-/commit/c9fb4003646dcbbaf56bb784
    Modified Files:
    docs/zmodem_comparison.md
    Log Message:
    docs/zmodem: re-measure zmtx/zmrx at 2.05, which fixes both reported defects

    Deuce released 2.05 the day after this document reported two defects
    against 2.04, and fixed both, plus implemented the ESC8 encoding the DSZ investigation had recovered. Re-measure everything the document claims
    about zmtx/zmrx.

    ESCCTL is fixed (82be5ad). CR is folded into the control class --
    action = TX_ESCAPE_CONTROL | TX_ESCAPE_CR -- with the '@' conditional
    kept as a separate test, which is the same shape as our own fix six days earlier. zmtx 2.05 now transfers byte-identically to lrz -e, to sexyz
    -e and to its own zmrx -e, putting 5,245,010 bytes on the wire where
    sexyz puts 5,245,034 and lsz 5,245,032 for the same file. The
    32,400-byte shortfall, which was the unescaped CR count, is gone.

    The retry budget is fixed (6e32e97). The gate at 3e-6 goes from 0 of 5
    to 3 of 5 and the rate sweep now completes where 2.04 failed from 5e-7
    up. Still the weakest of the three -- sexyz manages 10 of 10 and lrzsz
    7 of 10 -- but no longer structurally unable to finish.

    And ESC8 now interoperates (399e3b6). The commit adopts Forsberg's
    encoding rather than inventing one; its comment names DSZ.EXE's
    ZMODEM-90 seven-bit encoder and the code emits SO followed by the byte
    with bit 7 cleared, which is what the known-plaintext capture recovered. Verified against the 1997 binary both ways on a 16 KiB file: zmtx 2.05
    to DSZ rz -E and DSZ sz to zmrx -b both arrive identical. So a de facto normative ESC8 now exists, with two implementations agreeing on it,
    where days ago there were two encodings and no agreement.

    The features cost the sender something: 0.38 CPU-seconds per 256 MiB at
    2.04, 0.48 at the 2.05 tag, 0.44 after dfa97f0 "Recover standard
    transfer performance", which is measured here because that commit lands
    after the tag. Still the cheapest sender measured, but the margin over
    lsz narrowed from 2.5x to 2.2x.

    The 2.02 and 2.04 measurements are kept throughout rather than
    overwritten, since they are how the encoding was established and why the options are the way they are.
    --- SBBSecho 3.37-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell (on Debian Linux)@1:103/705 to Git commit to main/sbbs/master on Sun Aug 30 17:56:37 2026
    https://gitlab.synchro.net/main/sbbs/-/commit/60b70e526213285b61ae11f8
    Modified Files:
    docs/zmodem_comparison.md
    Log Message:
    docs/zmodem: zmtx/zmrx supports CRC-16, it just cannot be asked for

    The feature table listed CRC-16 as "--" for zmtx/zmrx, which reads as
    not supported. That is wrong in a way that matters to anyone pairing it
    with a CRC-16-only peer.

    want_fcs_32 is hard-coded true in zmdm.c and neither program offers a
    CRC flag, so it cannot be asked to prefer 16-bit -- but the machinery is present in both the header and data paths and the selection is
    can_fcs_32 && want_fcs_32, so it falls back whenever the peer omits
    CANFC32. Distinguish "supported but not selectable" from "absent",
    since sexyz and lsz can both force CRC-16 and zmtx/zmrx can only fall
    into it.

    Measured rather than read: zmtx to sexyz -o puts 4,311,016 bytes on the
    wire against 4,312,065 with CRC-32, and sexyz -o to zmrx 4,310,744
    against 4,311,816. A 4 MiB file is 512 subpackets at 8 KiB and two CRC
    bytes saved each is about 1,024, which is what those deltas are. Both
    verified byte-identical, so the fallback is real in both directions.

    Also corrects a stale count in the prose above the table.
    --- SBBSecho 3.37-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell (on Debian Linux)@1:103/705 to Git commit to main/sbbs/master on Sun Aug 30 23:50:56 2026
    https://gitlab.synchro.net/main/sbbs/-/commit/0822029d905accebd45d3265
    Modified Files:
    docs/zmodem_comparison.md
    Log Message:
    docs/zmodem: measure what MobyTurbo is worth

    Rob asked what sexyz would gain by supporting MobyTurbo. zmtx/zmrx 2.05 implements it, so it can be measured rather than estimated.

    It is worth about 2.3% on compressed or otherwise incompressible data and essentially nothing on text. ZMODEM must escape seven byte values --
    ZDLE, 0x10, 0x11, 0x13, 0x90, 0x91 and 0x93 -- at one extra wire byte
    each, which is 7/256 = 2.73% of uniformly distributed data; MobyTurbo
    drops that escaping on links known to be transparent and recovers exactly
    it. Measured on 1 MiB: random data goes from +2.84% overhead to +0.49%,
    while text goes from +0.09% to +0.09%. The split is the escape rate
    itself -- those seven values occupy 2.746% of the random file and 0.000%
    of the text one, none of them occurring in ordinary ASCII.

    Three bounds on what that is worth, all recorded: it is a wire saving, so
    it converts to throughput only where bandwidth binds rather than in this document's CPU-bound regime; it needs a transparent link, the same
    precondition ESC8 exists to work around, inverted; and it is negotiated
    by the receiver, so DSZ sz -m to sexyz puts exactly the same bytes on the
    wire as without it.

    Forsberg measured about the same himself. DSZ.DOC's benchmark table has Pro-YAM at 231 cps standard against 237 with MobyTurbo on a 2400 bps
    link, +2.6%, within half a point of the figure here thirty-six years
    later.

    Also adds feature-table rows for MobyTurbo and Pack-7, both of which
    zmtx/zmrx 2.05 now implements and neither sexyz nor lrzsz does.
    --- SBBSecho 3.37-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)