r/linux • u/adminvasheypomoiki • 18h ago
Kernel Spent three years blaming rsync and SSH. It was a DMA cache bug in Qualcomm's Wi-Fi driver corrupting 64 bytes at a time
https://blog.dfb.sh/blog/eap770-root-for-two-zeros/70
u/jthill 15h ago
Gotta say, was not expecting to read
Shout-out to TP-Link though: the system is pretty hardened. I found zero vectors in their own binaries, they use whitelists everywhere and load only the modules they need. So no free
; echo 0 > /sys/sfe/ppe_rfs_featurecheese for me. So the way in had to be the kernel.
30
27
u/scheimong 15h ago
My god what an odyssey. Have the persistence (or stubbornness) to will through all that bullshit is something else man.
14
u/adminvasheypomoiki 15h ago
It wasn't that bad. Just a couple of hours from time to time :) And I have mesh at home, so only 1 ap was misbehaving
14
u/edward_snowedin 14h ago
I enjoyed your article; it is so refreshing to see a pre-LLM tech blogging style again.
13
u/AminoOxi 17h ago
Qualcomm!
Damnit 🎃
3
u/spyingwind 9h ago
Qualcomm and RealTek are the two that I try to avoid. Those to have given me more problems than any other.
6
7
62
u/SPEZ_IS_A_JABRONI 18h ago
blaming rsync is crazy
96
u/adminvasheypomoiki 18h ago
Read an article :) I mean rsync was failing with gibberish errors about Mac checks, direct ssh was working, tcp checksums clear..
23
u/Brillegeit 16h ago
Crazy is a choice of words when talking about rsync considering the last few months.
3
u/TheG0AT0fAllTime 8h ago
Uh oh.. what did they do
•
u/Brillegeit 24m ago
Suddenly large amount of Claude generated code coming in before summer including a regression that broke backups:
https://news.ycombinator.com/item?id=48342705
Then a lot of conspiracy theories and comments all over the place about the sanity of the developer, nothing backed up by anything but "trust me, bro" of course.
7
u/nou_spiro 12h ago
I had similiar issue. I was transfering files with tar c | nc ---> nc | tar x over a network. And sometimes the files got corrupted. But only when transfer was crossing ethernet and wifi boundary on AP. Ethernet to ethernet or wifi to wifi was fine. When compared the files content it was about 1000 bytes.
6
16
u/ipsirc 18h ago
Later I happened to start using Tailscale for traffic inside my home network, and the transfers stopped needing retries. WireGuard drops a packet if its authentication fails, so the TCP connection inside the tunnel sees a lost packet and retransmits it. Since the transfers worked, I stopped digging.
This is always the way. I've never run rsync without wg/ovpn.
86
u/autogyrophilia 17h ago
That's like saying you don't need to cure your gonorrhoea because you always use a condom
10
16
0
u/Big-Report3344 8h ago
yeah but if the condom works you kinda stop caring about the gonorrhea until you take it off
35
1
u/BlokeInTheMountains 15h ago
If you have root, can you just patch the driver and get your 8% back?
6
u/adminvasheypomoiki 15h ago
Or brick the ap. It's ro squshfs. And I don't have sources. So I need to build firmware somehow and flush it. Doable, but I don't have a tty to usb thingy
1
u/throwaway490215 11h ago
So the real lesson is ; just have wireguard on everything. It would seem it would have saved you 9% of speed as well right?
I also noticed that for some reason my wifi TCP is extremely slow and choppy, probably because it uses the router's TCP stack's throughput downgrade on dropped packets.
WireGuard (with a little raspberry pi as the hub) gives me much better results.
-24
u/mrtruthiness 16h ago
Horrible article in my opinion.
TL;DR He was having a problem with his Access Point. He blamed lots of other things for dubious reasons. In the end he's probably been mislead by an AI (Opus 5.5) hallucination into thinking this final blame is correct. Who knows.
21
u/adminvasheypomoiki 16h ago
Sure. Root on the AP is my dream and working downloads are a hallucination :)
The A/B test is at the bottom of the post: IT WAS IMPOSIBLE TO DONWLOAD THE FILE before the fix. And if you actually click "Driver details," it links to the exact lines in edma_tx.c and edma_rx.c where fast_recycled skips cache invalidation. You don't have to trust me or the AI, just go read the code
228
u/adminvasheypomoiki 18h ago
Author here. Most of the post is about rooting the AP, the kernel part is in the "DMA cache coherency bug" section: the driver skips cache invalidation on recycled rx buffers, so a dirty line from the previous packet gets written back over the new one after DMA, and the offload engine then recomputes the checksum over the garbage.
I think I've seen the same pattern in a few other drivers, but I only read the code, didn't test. And I still don't know where to send a patch for qca-nss-dp, the github repo is a mirror. Ideas welcome. And I don't know where to discuss stuff with cache coherency in doma to. I guess on some mail list