debian-user:

I had the need to zero-fill a Seagate Constellation ES.2 3 TB SATA
magnetic hard disk drive using an Intel DQ67SW motherboard, Intel Core
i7-2600S processor, 8 GB dual-channel memory, StarTech PEXSAT32 adapter,
and Debian:

2024-09-08 08:05:02 root@bullseye-bios
~/hardware/seagate/ST33000650NS/REDACTED
# cat /etc/debian_version ; uname -a
11.10
Linux bullseye-bios 5.10.0-32-amd64 #1 SMP Debian 5.10.223-1
(2024-08-10) x86_64 GNU/Linux


When done, I ran some benchmarks to compare sequential write performance
for various synchronization methods and block sizes:

1. Using dd(1) with option oflag=sync:

2024-09-08 09:19:27 dpchrist@laalaa ~/hardware/seagate/ST33000650NS/REDACTED
$ tail -n 214 log.txt | perl -pe 's/^\s+//' | egrep -A 3 'oflag=sync' |
egrep 'time|GiB'
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=1G count=1
oflag=sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.3181 s, 147 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=256M count=4
oflag=sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.39608 s, 145 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=64M count=16
oflag=sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.48021 s, 144 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=16M count=64
oflag=sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.92566 s, 135 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=4M count=256
oflag=sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 8.97546 s, 120 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=1M count=1K
oflag=sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.3814 s, 69.8 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=256K count=4K oflag=sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 40.8931 s, 26.3 MB/s

2. Using dd(1) with sync(1):

2024-09-08 09:19:38 dpchrist@laalaa ~/hardware/seagate/ST33000650NS/REDACTED
$ tail -n 214 log.txt | perl -pe 's/^\s+//' | egrep -A 3 '&& sync' |
egrep 'time|GiB'
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=1G count=1 && sync 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.3975 s, 145 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=256M count=4 && sync 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.21795 s, 149 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=64M count=16 && sync 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.21845 s, 149 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=16M count=64 && sync 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.20049 s, 149 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=4M count=256 && sync 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.11058 s, 151 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=1M count=1K && sync 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.12372 s, 151 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=256K count=4K &&
sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.10426 s, 151 MB/s

3. Using dd(1) without explicit synchronization:

2024-09-08 09:20:45 dpchrist@laalaa ~/hardware/seagate/ST33000650NS/REDACTED
$ tail -n 214 log.txt | perl -pe 's/^\s+//' | egrep -A 3 time | egrep 'time|GiB' | egrep -v -A 1 '(MB\/s|sync)$' | egrep -v '^--'
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=1G count=1
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.30077 s, 147 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=256M count=4
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.27323 s, 148 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=64M count=16
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.21564 s, 149 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=16M count=64
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.17308 s, 150 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_Z294MZFA bs=4M count=256
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.20907 s, 149 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=1M count=1K
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.11303 s, 151 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=256K count=4K 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.13726 s, 150 MB/s


Additional observations:

1. The Bash shell 'time' built-in command and the per-second/
per-transfer performance values displayed by dd(1) option
'status=progress' can show the interaction between dd(1) option
'oflag=sync', kernel I/O buffering, dd(1) write completion, and sync(1).

2. dd(1) does not exit until the final block has been written.


Conclusions:

1. dd(1) option 'oflag=sync' causes block size to affect write
performance -- increasing the block size increases the performance. The
effect is non-linear, with a knee at around 30 IOPS. For best
performance, block size should be tuned to produce 30 IOPS or less.

2. sync(1) is unnecessary.

3. Use dd(1) as-is, without 'oflag=sync' or sync(1).


David

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

David Christensen <[email protected]> writes:

2. sync(1) is unnecessary.

Does it even do anything wrt device files? sync(2) says "sync() causes
all pending modifications to filesystem metadata and cached file data to
be written to the underlying filesystems."

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon 09 Sep 2024 at 11:52:19 (+0300), Anssi Saari wrote:

David Christensen <[email protected]> writes:

2. sync(1) is unnecessary.

Does it even do anything wrt device files? sync(2) says "sync() causes
all pending modifications to filesystem metadata and cached file data to
be written to the underlying filesystems."

I think you also need to read the other manpage, fsync(2).

Cheers,
David.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

David Wright <[email protected]> writes:

On Mon 09 Sep 2024 at 11:52:19 (+0300), Anssi Saari wrote:

David Christensen <[email protected]> writes:

2. sync(1) is unnecessary.

Does it even do anything wrt device files? sync(2) says "sync() causes
all pending modifications to filesystem metadata and cached file data to
be written to the underlying filesystems."

I think you also need to read the other manpage, fsync(2).

Why do you think that? Which part of the fsync manpage explicitly covers fsync's effect on device files? Share share, it's fair.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

David Wright <[email protected]> writes:

On Tue 10 Sep 2024 at 11:56:25 (+0300), Anssi Saari wrote:

Why do you think that? Which part of the fsync manpage explicitly covers
fsync's effect on device files? Share share, it's fair.

“fsync() transfers ("flushes") all modified in-core data of (i.e.,
modified buffer cache pages for) the file referred to by the file
descriptor fd to the disk device (or other permanent storage device)
so that all changed information can be retrieved even after the
system crashed or was rebooted. This includes writing through or
flushing a disk cache if present. The call blocks until the device
reports that the transfer has completed. It also flushes metadata
information associated with the file (see stat(2)).”

You just parroted a man page I had already read. Why did you think
that'd be helpful? I asked the questions because the man page didn't
answer my questions. As you are apparemntly unable I found for example https://unix.stackexchange.com/questions/473854/block-device-cache-v-s-a-filesystem
and so arrived to the conclusion that the final close(2) call on a block
device already flushes all buffers before returning. So the answer to
the question "is running sync needed after dd to block device" is
no. Someone else posted that too on this list recently, in another
thread.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Fri, Sep 13, 2024 at 03:00:14PM +0300, Anssi Saari wrote:

David Wright <[email protected]> writes:

On Tue 10 Sep 2024 at 11:56:25 (+0300), Anssi Saari wrote:

Why do you think that? Which part of the fsync manpage explicitly covers >> fsync's effect on device files? Share share, it's fair.

“fsync() transfers ("flushes") all modified in-core data of (i.e.,
modified buffer cache pages for) the file referred to by the file
descriptor fd to the disk device (or other permanent storage device)
so that all changed information can be retrieved even after the
system crashed or was rebooted. This includes writing through or
flushing a disk cache if present. The call blocks until the device
reports that the transfer has completed. It also flushes metadata
information associated with the file (see stat(2)).”

You just parroted a man page I had already read. Why did you think
that'd be helpful? I asked the questions because the man page didn't
answer my questions. As you are apparemntly unable [...]

Please, Anssi. You can do better than that.

[...] I found for example https://unix.stackexchange.com/questions/473854/block-device-cache-v-s-a-filesystem
and so arrived to the conclusion that the final close(2) call on a block device already flushes all buffers before returning. So the answer to
the question "is running sync needed after dd to block device" is
no. Someone else posted that too on this list recently, in another
thread.

So I did an (admittedly very rough) experiment, and this seems to confirm the above:

tomas@caliban:~$ time sudo dd if=/dev/zero of=/dev/sdb bs=1M ; time sync
dd: error writing '/dev/sdb': No space left on device
15268+0 records in
15267+0 records out
16008609792 bytes (16 GB, 15 GiB) copied, 3614.25 s, 4.4 MB/s

real 60m14.260s
user 0m0.005s
sys 0m0.007s

real 0m0.077s
user 0m0.002s
sys 0m0.000s

I interpret that this way: after 16G of dd (the stick is 16G) there isn't
a significant amount of buffers lying around waiting to be flushed to the
stick (it's a slow USB, as suggested by the 60m the dd took, so if there
were a lot of buffers, we'd see more than .077 sec for the sync).

The box has plenty of RAM (16G) for what it's doing (14G free), so there
is no particular memory pressure which would favour quick flushing of
buffers.

So yes, it seems when the dd is done, it's done.

Cheers
--
t

-----BEGIN PGP SIGNATURE-----

iF0EABECAB0WIQRp53liolZD6iXhAoIFyCz1etHaRgUCZuSFjAAKCRAFyCz1etHa RvXEAJ4k2h+v467f0m/w4A2EE7S30PWhaACfQ+zE/1aYWP3UbPjniI6CcxPtfY8=
=/soe
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Fri, Sep 13, 2024, at 5:00 AM, Anssi Saari wrote:
...

So the answer to
the question "is running sync needed after dd to block device" is
no. Someone else posted that too on this list recently, in another
thread.

On the other hand, it may not be necessary, but it doesn't do any harm. If it is really unnecessary, it will probably cost only a fraction of a second to do. And if it is actually necessary, you should do it, no matter how long it takes.

Rick

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 9/13/24 12:06, Jeffrey Walton wrote:

To add a datapoint...

My daily driver workstation is really fast with lots of RAM. It has
3.4 GHz cpu and 64 GB of RAM. I also set swappiness to a low value
to avoid spilling out of RAM.

I use a lot of SBC's/dev boards for testing. They usually use a
SDcard. The SDcard is really slow. The card can only provide 10 MB/s
or 30 MB/s write speeds. Some cards I use are so cheap they are even
slower.

If I dd an image from the workstation to the SDcard, it happens in
under a second. dd exits, and closes its file descriptors. Something
is obviously wrong since the image is 1 or 2 GB, and the SDcard write
speed is 10 MB/s or 30 MB/s.

The file system cache is still holding the writes. If I remove the
SDcard and try to use it, the image is corrupt. When I say "remove",
I mean pop the card out of the card reader since the write has
supposedly finished.

What I found is, I have to manually call sync to ensure the image is
written from cache to the SDcard. When I call `dd if=... of=/dev/sdd
&& sync`, the command takes 30 seconds or so to complete. The time is
spent in sync, not dd.

Based on my experience with lots of RAM and slow media, you have to
call sync to get the cache manager to write back to the disk.

That makes me think there is a bug in the secure digital card firmware,
the Linux secure digital card device driver, and/or the Linux file
system driver (?).


Have you seen a dd(1) invocation work correctly on a SD card without any synchronization options and/or trailing sync(1)?


On 9/13/24 13:57, Rick Thomas wrote:

On the other hand, [running sync after dd] may not be necessary, but
it doesn't do any harm. If it is really unnecessary, it will probably
cost only a fraction of a second to do. And if it is actually
necessary, you should do it, no matter how long it takes.

+1


David

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sun, Sep 08, 2024 at 10:59:58AM -0700, David Christensen wrote:

2. Using dd(1) with sync(1):
3. Using dd(1) without explicit synchronization:

These two tests are identical, as dd has exited (and all the performance information has been printed) before sync runs. The flag you want[1] is conv=fsync which runs sync before exiting, thus making the stats reflect
the time taken to flush to disk.

Mike Stone

[1] insofar as you want to use dd at all; as I've said before, it's
almost never the best tool for the task unless you're trying to reblock
data from a mainframe 3490 tape to a unix 9 track or somesuch. I haven't
needed to do that in almost 25 years. In general cp will provide the
best possible performance using familiar parameters and without the need
to try to understand how to apply flags based on a 1960s-era mainframe directive to a modern unix system. I suppose if you enjoy impressing
people with obscure command syntax (and if you know people who can be
impressed by same) dd is its own reward.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Fri, Sep 13, 2024 at 08:33:55PM +0200, [email protected] wrote:

tomas@caliban:~$ time sudo dd if=/dev/zero of=/dev/sdb bs=1M ; time sync
dd: error writing '/dev/sdb': No space left on device
15268+0 records in
15267+0 records out
16008609792 bytes (16 GB, 15 GiB) copied, 3614.25 s, 4.4 MB/s

real 60m14.260s
user 0m0.005s
sys 0m0.007s

real 0m0.077s
user 0m0.002s
sys 0m0.000s

I interpret that this way: after 16G of dd (the stick is 16G) there isn't
a significant amount of buffers lying around waiting to be flushed to the >stick (it's a slow USB, as suggested by the 60m the dd took, so if there
were a lot of buffers, we'd see more than .077 sec for the sync).

It's an invalid test because you hit the end of the device; there's
nothing left to sync.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sun, Sep 15, 2024 at 10:47:46PM -0400, Michael Stone wrote:

On Fri, Sep 13, 2024 at 08:33:55PM +0200, [email protected] wrote:

[rough dd test]

It's an invalid test because you hit the end of the device; there's nothing left to sync.

This doesn't make sense to me. The question was whether dd writes "raw" to
the device or "buffered" (meaning that its output ends up in OS buffers to
be flushed later). Since the whole thing (dd + OS) can't look into the
future, stuff will end in buffers to be flushed if the second is the case, regardless on whether dd "hits the end of the device" or not.

Those buffers would have to be flushed, thus showing up in sync's time.
They do not, therefore I conclude that we have the first case.

What am I missing?

Cheers
--
t

-----BEGIN PGP SIGNATURE-----

iF0EABECAB0WIQRp53liolZD6iXhAoIFyCz1etHaRgUCZue0UgAKCRAFyCz1etHa RuIPAJ9nNIfiK6Gu742kT+DRTYDuBmYaxQCffiPdwoHMmSUU1/4Usbygqw5kJ0E=
=gAYw
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Jeffrey Walton <[email protected]> writes:

The file system cache is still holding the writes. If I remove the
SDcard and try to use it, the image is corrupt. When I say "remove", I
mean pop the card out of the card reader since the write has
supposedly finished.

All right. So in fact, block devices either have write back write
caching or they don't and apparently the device driver decides
that. hdparm may be able to control caching but I don't know it works
for USB devices.

It'd be interesting to know if you have caching related kernel messages
about your SD cards when plugged in? I did have from my USB SD reader.

I remember I have a USB-SSD case which demonstrated this situation,
super fast writes to a block device. I may dig that up for some testing.

In my little experiments with media on hand I found only uncached writes
this weekend. Test devices were one old 1 GB SD card, one newish 64 GB
SDXC card, one old 4 GB USB stick, two SD card readers, one USB and one PCI-connected built in to a laptop.

Interestingly with the old memory stick there's a kernel message which
says outright write caching is write through which seems to indicate synchronous writes.

[3675744.043264] sd 6:0:0:0: [sdc] No Caching mode page found
[3675744.048781] sd 6:0:0:0: [sdc] Assuming drive cache: write through [3675744.113885] sdc: sdc1
[3675744.139820] sd 6:0:0:0: [sdc] Attached SCSI removable disk

Writes to that stick were about 6 MB/s as reported by dd.

With my USB SD card reader there was a similar message:
[17740.817424] sd 5:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA

Laptop's built in PCI-connected SD card reader didn't say anything about
cache.

Write speeds reported from dd were pretty much as expected, from a few
MB/s to a few dozen with the SDXC card.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Sep 16, 2024 at 12:00:53PM +0300, Anssi Saari wrote:

Jeffrey Walton <[email protected]> writes:

The file system cache is still holding the writes. If I remove the
SDcard and try to use it, the image is corrupt. When I say "remove", I
mean pop the card out of the card reader since the write has
supposedly finished.

All right. So in fact, block devices either have write back write
caching or they don't and apparently the device driver decides
that. hdparm may be able to control caching but I don't know it works
for USB devices.

You are talking device cache. Jeffrey is talking file system cache.
Two different (and independent) layers.

Actually, dd doesn't "do" file system cache (unless you are dd'ing
to a file, of course).

Cheers
--
t

-----BEGIN PGP SIGNATURE-----

iF0EABECAB0WIQRp53liolZD6iXhAoIFyCz1etHaRgUCZuf3MgAKCRAFyCz1etHa RrMBAJ9T7LfxL7Sl9as/RtAtx4eW5qUdOwCfbt+hsqgWvOgr6M+Vvv+R7/u6phI=
=20Y/
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

<[email protected]> writes:

On Mon, Sep 16, 2024 at 12:00:53PM +0300, Anssi Saari wrote:

Jeffrey Walton <[email protected]> writes:

The file system cache is still holding the writes. If I remove the
SDcard and try to use it, the image is corrupt. When I say "remove", I
mean pop the card out of the card reader since the write has
supposedly finished.

All right. So in fact, block devices either have write back write
caching or they don't and apparently the device driver decides
that. hdparm may be able to control caching but I don't know it works
for USB devices.

You are talking device cache. Jeffrey is talking file system cache.
Two different (and independent) layers.

I think Jeffrey's post was pretty clear in that it was not about writing
to a file system. Notwithstanding the words "file system cache" in that
quote above which I think that was just a mistake.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Michael Stone <[email protected]> writes:

[1] insofar as you want to use dd at all ...

dd does conveniently provide progress and stats whereas cp does not.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Sep 16, 2024 at 01:40:52PM +0300, Anssi Saari wrote:

<[email protected]> writes:

[...]

You are talking device cache. Jeffrey is talking file system cache.
Two different (and independent) layers.

I think Jeffrey's post was pretty clear in that it was not about writing
to a file system. Notwithstanding the words "file system cache" in that
quote above which I think that was just a mistake.

This way around it'd make sense. The quoted kernel messages:

[3675744.043264] sd 6:0:0:0: [sdc] No Caching mode page found

and so on talk device cache, not FS cache. To avoid confusion, it is
important to keep them apart.

(Still, if you yank the device out while its caches are dirty, you might
get corruption, depending on how those react to power loss).

Cheers
--
t

-----BEGIN PGP SIGNATURE-----

iF0EABECAB0WIQRp53liolZD6iXhAoIFyCz1etHaRgUCZugNzQAKCRAFyCz1etHa RqJHAJ4gmh1+TaxXKmntYcmxE4p3U9ScagCdEVcDrwq97YiYTqywKlWiPuv4Em4=
=EMmQ
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Sep 16, 2024 at 01:12:35PM +0300, Anssi Saari wrote:

Michael Stone <[email protected]> writes:

[1] insofar as you want to use dd at all ...

dd does conveniently provide progress and stats whereas cp does not.

stats which are misleading without the proper arguments, so it's a bit
of a wash. This is actually a hard problem because what people expect as
the desired output varies considerably depending on the task at hand.
There are a number of tools which will do variations of this, without
the baggage of dd. (E.g., pv)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Sep 16, 2024 at 11:15:50AM +0200, [email protected] wrote:

Actually, dd doesn't "do" file system cache (unless you are dd'ing
to a file, of course).

Incorrect; linux block device accesses cache unless specifically asked
not to. This is very easy to demonstrate: run something like "dd if=/dev/nvme0n1 bs=1M count=1000" twice, and you'll see that the second
run is significantly faster than the first. (Correct for a block device
you actually have, and this assumes that you have more than 1G of
unused RAM available for cache.)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Fri 13 Sep 2024 at 15:00:14 (+0300), Anssi Saari wrote:

David Wright <[email protected]> writes:

On Tue 10 Sep 2024 at 11:56:25 (+0300), Anssi Saari wrote:

Why do you think that? Which part of the fsync manpage explicitly covers >> fsync's effect on device files? Share share, it's fair.

“fsync() transfers ("flushes") all modified in-core data of (i.e.,
modified buffer cache pages for) the file referred to by the file
descriptor fd to the disk device (or other permanent storage device)
so that all changed information can be retrieved even after the
system crashed or was rebooted. This includes writing through or
flushing a disk cache if present. The call blocks until the device
reports that the transfer has completed. It also flushes metadata
information associated with the file (see stat(2)).”

You just parroted a man page I had already read. Why did you think
that'd be helpful?

Um, because you asked me too, above.

I asked the questions because the man page didn't
answer my questions. As you are apparemntly unable I found for example https://unix.stackexchange.com/questions/473854/block-device-cache-v-s-a-filesystem

I see that I looked at that page back in May 2021 (the page has been
updated since then), when the list had a previous conversation on dd,
partly about blocksize, but in which we were told that we were only
meant to be using dd with tapes or when converting, say, EBCDIC to
ASCII, or whatever.

I've yet to find another tool that can give you progress when already
in progess.

I had landed on a different page from the above this time around,
https://stackoverflow.com/questions/58276098/fsync-with-raw-device-fd
but I couldn't determine a definitive answer there. There still seems
to be some disagreement in this thread.

and so arrived to the conclusion that the final close(2) call on a block device already flushes all buffers before returning. So the answer to
the question "is running sync needed after dd to block device" is
no. Someone else posted that too on this list recently, in another
thread.

Well, if you knew that already, you could have just posted your view
rather than "Share share, it's fair".

Of course, we all know that the time it takes to add ;sync
to a command line is hardly going to delay you more than a
suitably sized choice of blocksize, so much of this discussion
is moot in practical terms.

Cheers,
David.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)