Solved a minor annoyance I had with arch today

[u/arbaralla]

Occasionally I need to copy big files to relatively slow USB sticks.

(Usually ISO images to my pendrive with Ventoy to install an OS somewhere or run a rescue mission. BTW if you are still using stuff like Rufus or dd for that purpose, check out Ventoy, it's awesome)

And for a long time, a pet peeve of mine was that the big file was copied "instantly", but then unmounting that pendrive took forever with no progress bar. I'd much rather have a real progress bar that shows how fast the stick actually writes.

After some research, I learned that Linux happily lets gigabytes of dirty data pile up in RAM before it starts throttling the program doing the writing. So the file manager thinks the copy is done when really it’s just sitting in memory, and unmount has to flush all of it. So I made a udev rule that sets a 16 MB limit on dirty cache for USB disks:

# /etc/udev/rules.d/90-usb-dirty-limit.rules
ACTION=="add", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_BUS}=="usb", RUN+="/bin/sh -c 'echo 1 > /sys/class/bdi/%M:%m/strict_limit; echo 16777216 > /sys/class/bdi/%M:%m/max_bytes'"

16 MB of dirty cache should be plenty for most pendrives (and it's the limit of data the OS will "pretend" to have instantly copied). If you regularly use a fast USB SSD you might want a bigger number there like 128 MB, since a tiny buffer can cost some throughput on really fast drives.

[u/bitwaba]

Yeah, this is a really old annoyance in Linux.

After the gui shows your data is copied, you need to run sync in a terminal and wait for it to finish.

I have no idea why, and I have no idea why every major distro hasn't come up with some kind of elegant way to handle it.

[u/C0rn3j]

It would be better to fix whatever file manager you're using, to my knowledge the required interfaces to make them show real progress have existed for years at this point.

[u/PacmanAteMyRAM → u/C0rn3j]

It's not the fault of anything userspace. A lot of file copying will hit the kernel cache before being flushed to disk. Even dd does this by default if you have enough ram to fill with what you wrote to a removable storage device. The software has to intentionally do synchronous writes to avoid this, which is visibly much slower.. but the true speed. For dd I also add conv=sync oflag=sync to make this happen as well. I don't want a flashed image to take 2 seconds and then sync for 5 minutes.. I want it to finish when it actually finishes.

I picked up a bad habit from my days working with embedded hardware and raspberry pi's of running the sync command after most file operations I did to really make sure that something was written and wouldn't result in a corrupted transfer.

I never need it these days, but once in a year or so.. the sync command hangs... because it was actually syncing something that hadn't finished being written.

[u/C0rn3j → u/PacmanAteMyRAM]

It's not the fault of anything userspace.

It is, to my knowledge, you can track until the file is actually written.

[u/yevelnad]

Have the same problem. I always use the resources app(the new gnome task manager) to view the disk write of the drive. It is a better indicator than the file managers. 😅

[u/archover]

Alternatively, use fast flash drives.

The ones I only use are 400MB/sec to 1100MB/sec, and act very similarly to internal drives for all use cases I care about, OUT OF THE BOX. Look on amazon for SSK and Vansuny. These in long service and reliable.

Good day.