This reminds me of a (completely different, but felt equally) bizzare problem with ghostty (the terminal emulator) on Linux and Steam, specifically Baldur's Gate 3.
At the start of the game after an update, BG3 compiles the (Vulkan) shaders. To avoid hammering the computer, it checks the load, including I/O pressure, and then throttles itself. It turns out that Ghostty uses io_uring, and even if it's not doing anything this somehow counts as pressure. This was made worse by the fact that I usually have a dozen or so Ghostty windows.
End result, BG3 compiles shaders using 1 core, throttled, instead of all available 20 cores. Instead of a minute or so, it'd take several hours.
(For closure: I configured ghostty to use epoll instead. Probably not optimal but works around the problem.)
Interesting to see the "io_uring vs io pressure measurement" issue come up here; it's turned up on qemu-devel recently too: https://lore.kernel.org/qemu-devel/4EF13FE7-E6F3-42E2-83F9-6...
If I am reading the thread correctly, the io_uring maintainer's position is that applications trying to use the metric that "poll via io_uring" handles differently from "poll via epoll or select" are fundamentally misguided and measuring something that doesn't make sense, so they don't want to slow down io_uring to placate those apps.
From https://lore.kernel.org/qemu-devel/962d44d3-ca7b-4cc8-b0f7-7...
> iowait is fundamentally incompatible with sane accounting in the first place, that is the problem. It really tells you nothing, and was deprecated since the days of uniprocessor systems. The problem here is not how iowait is done in the kernel, it's that people are using this metric in the first place. Which is why you just want to NOT do that, and do sane accounting on your disk drives, if you care about that kind of thing.
> Which is why I refuse to cater to any kind of iowait accounting metrics, it's just adding overhead to support a metric that is largely nonsensical in the first place. It doesn't answer any questions that an application/tier owner would want to know. Those questions might be things like "am I I/O bound?" or "is the device I'm using being slow?". I wish people would realize that, rather than continue to place value in iowait as a metric for ANYTHING, and stop trying to work around it.
It sounds more like their position is "iowait is a useless metric" and that polling with io_uring completely breaks iowait metrics but fixing it would add a minor slowdown so they refuse to do so because they don't think it's a valuable metric to begin with.
More context: https://lwn.net/Articles/989272/
Is there a good replacement for iowait?
An app can use io_uring_set_iowait to change that behavior. Since a terminalâs waiting on I/O isnât hugging the disk but just waiting on mouse and keyboard input, that seems like the honest thing to do.
In certain circumstances shader pre-compilation may only use one core, however this can be overridden by the user, example to use 8 cores:
~/.steam/steam/steam_dev.cfg
unShaderBackgroundProcessingThreads 8
From https://wiki.archlinux.org/title/SteamAmazing. How did you figure this out?!
I asked Claude to help me debug and research it.
It looked at fossilize code (open source component used in the process), found a couple of threads online that suggested io_uring/iowait and told me first to try closing the ghostty windows and then to try to switch to epoll. This did the trick.
I have a very vague understanding of what's going on (actually better now that I've look at the links/resources from the replies in this thread!), and would probably have been able to figure it by googling in my own. But, my first hunch was that it's a problem with steam, proton, or the game, so I'd probably waste a lot of time in that direction first.
Did you report it upstream? Proton, dxvk, Larian or Ghostly? It would be neat if we fixed it for everyone
Better description: "My USB network switch was cutting my download speed".
The fact that it's a monitor is a little bit irrelevant.
Same old adapter worked perfectly if the monitor was taken out of the path. Kind of an interaction thing with both, really.
The twist is it would happen with any additional device in the signal path. The mere presence of a hub causes the USB control message to take slightly longer to reach the adapterâabout the same amount of time it takes to receive 5 Ethernet frames at 2Gbps.
The fix is to cut the overhead by sending more than 4 frames per interrupt. Unfortunately that required buying a new adapter with a different chip.
There are plenty of USB hubs with built-in NICs behind the hub, that can achieve full ethernet bandwidth. I even have a hub with a built-in RTL8156 and it achieves 2.4 gbps. And a separate RTL8156 plugged into a random hub also hits that.
So it certainly seems possible to have an additional device in the signal path.
The fix is actually to plug it directly into the host, and not use a hub. Very common thing to need to do with cheap or older USB Hardware. Paying more money for something new when the old one works fine is pointless and wasteful.
> 2026 Studio Display XDR, with everything plugged into it so switching laptops is one cable
I think I'd have accepted having two cables from the get-go. And if I observed this problem, the first diagnostic I would do is to plug the Ethernet directly into the laptop.
Fewer technical pieces in a puzzle sometimes trumps ease of use.
With a $3300 monitor? Not a chance. When you are dropping that kind of money on your setup and it performs worse than a $35 docking station, having to mess around with that kind of ugly workaround is unacceptable.
"Ease of use" is the whole point of monitors with integrated docking stations, or even of USB-C in general. Going back to a worse UX than a 2005 Dell laptop dock? Hell no, not going to happen: something is broken here and it will end up getting replaced.
The first rule of troubleshooting is to try to cut down the search space as much as possible with each question. My first reaction would have also been to plug directly into the computer to see if there's a bottleneck or failure at the monitor.
That's what I did in the blogpost.
I was under the impression that in the Apple ecosystem, you pay for better integration. Appleâs XDR display should really be able to handle this better. Or have the drivers negotiate with the chipset better.
I mean, the guyâs using a USB Ethernet adapter with one of the cheapest NICs on the market that can do 2.5GbE, and itâs the behaviour of its firmware that caused this problem. It would have performed identically behind any hub. How is this appleâs fault?
How cheap are we talking?
I've never had a problem getting a steady 2.5 Gbits between a pair of USB NICs purchased for $20 several years ago on Amazon.
"Better" is subjective. I think monitor is a better title, because it gives misdirection, so we the readers learn the same time OP did in the story. Unreliable narrator and all that.
I understand that not everybody likes this, because it can feel like a trick, or patronizing, or lying. But in my experience, it works very well for people.
I didnât like it because it was a very generic and common technical occurrence that I would not be interested in reading about normally.
Now I feel as though my time was wasted by a dishonest author trying to get me to believe that their story was interesting when it was not.
In the grand scheme of things, itâs not a very big deal, but I feel quite annoyed since I was baited into expecting a fun read.
Yeah I feel you. If you start wondering how a real monitor could do this, only to discover a common other component, it's a disappointment.
Since lots of comments misunderstood the article, the extra latency from going through the monitor meant his laptop couldn't acknowledge the packets quickly enough. That led a buffer on the ethernet adapter to fill and that led it to drop packets.
A different adapter which required less frequent acknowledgement fixed the problem for him.
It is quite a nice post which gets to the bottom of incredibly opaque behaviour of USB hubs.
Speaking of opaque behavior, I installed a 7-port PCIe USB card for a coworker last week that I noticed consists of a four-port controller with a 4-port hub connected to one of the ports, with no identifiable markings or documentation suggesting differences between any of the ports.
I also once had an unpowered USB 3.0 switch that somehow managed to supply power to my workstation's +5V standby rail after disconnecting it from the wall, presumably from the PC on the other side of the switch, since the only other connected devices were a mouse and a keyboard (not that powered devices' B connectors should be supplying power to unpowered hosts' A connectors in the first place).
Thatâs really still the adapter though because daisy chaining usb-c exists and is kind of a given so wouldnât that latency exist in any other configuration such as a dock? Do you know anyone who plugs Ethernet into their laptop? I donât.
The fix is that it could send more packets per ack not that it needed to happen less. If the buffer was bigger but still capped at 4 packets per ack, the full buffer drain speed would remain at ~2.
Probably a stupid question, but why is USB behaviour so opaque?
There's not much monitoring built into usb hubs. It's actually far less when you think about all the monitoring a network switch has.
Meta:
> Somehow I am one of those few very lucky people to have the opportunity of owning a 2.5 gig symmetrical line for my internet.
I'm not going to complain against more bandwidth (as a GenXer I stared my online journey at (IIRC) 9600bps (maybe lower): much celebration with V.32/V.90), but I'm curious to know how many residential use-cases can take advantage of that much bandwidth. Linux ISOs?
How many services allow that much speed on a single stream / (TCP) session? Maybe you could have five at 500 Mbps?
Certainly downloading security updates or games may be quicker, but I would think that above 500Mbps (and certainly 1 Gbps) there would be major diminishing returns. Is anyone regularly getting close to that utilization in their daily work / hobbies (and not just a momentary spike on patch Tuesday)?
Torrents, and things like downloading games from Steam feel really good if you have a very strong connection. A lot of games can be downloaded in the time it takes to go grab a drink from the fridge. It adds some nice quality of life, there isn't as much need to keep a large downloaded set - just download when you need it, and it'll be there quickly.
I think going from something like 100 Mb download to 1 Gb is slightly overrated. You hit heavy diminishing returns. Symmetric Upload, for me, is what really changes your quality of life a lot. Uploading a youtube video suddenly isn't an event, it takes a few seconds. Want to upload an image in any chat? Done before you consciously see it uploading. Heavy commits are instant, even with a bunch of resources. It changed my behavior. I started uploading more stuff to remote servers because the cost in time got so much lower.
Yup as someone on a 1000/20 link (canât get anything better in my little village in BE), decent upload is far more useful than more download after a point (far before 1gbps).
Depends on how many people you have in your household and what they do with the connection. Mine has two people who are both gamers and who use streaming media. 1000/50 makes a huge difference vs 500/50 when both are downloading.
The major CDNs routinely deliver full throughput.
Itâs great when experimenting with LLMs because I can quickly try new models and download from the internet almost as fast as if coming from my NAS.
Game updates are also a great use case. I donât game often so when my friends want to play I often have to update the game. Some patches are 60GB or more. When they come down at full line speed thatâs valuable time Iâm getting back to spend with my friends in game.
Also hyperscalers. On several occasions I've used the Netflix content freely available on S3[1] as an ad-hoc download speed test.
I think itâs less about being able to max it out and more about having background events (like backups) or other users on the network not impact your own usage.
But I run servers at home and regularly download enormous video files so I can think of plenty of ways to max out a 2.5 Gbps connection (alas my own is nowhere near as fast).
Indeed. We have 4Gbps because it's not much more expensive than 1Gbps fiber (plus it required our ISP to upgrade our fiber connection from AON to XGS-PON, which is more future-proof).
The benefits are two-fold:
1. Background tasks don't impact your own issue (as you say).
2. Many CDNs can saturate 4 Gbps, so if you download something, the waiting time is much shorter than say 100Mbps.
Whether it's worth it all depends on the price difference.
4Gbit is kind of the sweet spot currently. 2.5Gbit equipment became quite cheap (e.g. Ubiquiti Flex Mini 2.5Gbit managed switches are often around 50 Euro), existing Cat 5e cabling still works reliably, and if you have a router with 10Gbit upstream and multiple 2.5Gbit downstream ports, none of the subnets can completely saturate the upstream bandwidth.
Also, these bandwidths rock for doing an initial backup, downloading a full backup, pulling many Docker images, large and frequent NixOS rebuilds (unstable-small, yay). Downloading GTA 6 (not me though, not a gamer).
> (plus it required our ISP to upgrade our fiber connection from AON to XGS-PON, which is more future-proof).
Isn't "AON" basically switched Ethernet?
* https://www.fs.com/blog/aon-vs-pon-networks-whats-the-differ...
What is meant by XGS-PON being "more future-proof"?
> plus it required our ISP to upgrade our fiber connection from AON to XGS-PON, which is more future-proof
The opposite, actually! See for example Init7 in Switzerland offering 25Gbps fiber to home users, and having done so since 2021 - for exactly the same ~$80/mo as 1Gbps: that would not be possible on new XGS-PON deployments as it requires having direct access to the fiber going to this one customer. Init7 could trivially offer 100Gbps over the exact same fiber using existing off-the-shelf plug-and-play equipment, and many times that for customers willing to spend money on WD equipment.
PON inherently ties your fate to that of your neighbors: whatever they are getting, you are getting as well. You can just about have multiple generations of PON living on the same fiber, but that's it. The sensible thing to do would be to give each building dedicated fiber to a central PoP and put all the PON splitters in there, but as that would allow for legislation mandating access to the PoP to competitors wanting to offer superior services via direct fiber access, that is often made deliberately impossible in new PON installations by putting the splitters into tiny curbside cabinets where there is physically no space for anyone but the incumbent - which of course means the only way "open access" is possible is via a virtual provider scenario where any kind of meaningful differentiation is impossible.
Do not underestimate the possibilities. I have 1.0/0.1Gb cable, and doing remote access is a complete joke. Even streaming a 1080p movie up from my house is extremely challenged.
This also locks me out of hosting my own âcloudâ storage at home where it belongs - I donât want to be out and waiting forever to get a file, so I stick to traditional cloud services for now. Slower than local storage on the LAN of course though.
A fiber isp is digging trenches soon. Cannot wait!
I have about 45 Mbit upstream on my gigabit connection, and I've never had a problem with remote access so long as it's not competing with other traffic on the connection, in which case USB tethering on my iPhone works well enough.
I suppose this also depends the protocol and the implementation. VNC often feels bad even over a point-to-point Ethernet link, while RDP feels fine for basic desktop applications on any sufficiently low-latency connection (within reason; it's not exactly snappy over dialup or basic rate ISDN).
Yeah upload is usually the game changer for home internet. I have symmetric gigabit and I would happily halve my download speed if it meant I could double my upload. Sending large backup .tars to cloud storage, sending data to VPCs, object storage, allowing more concurrent Plex users remotely, etc etc, 2Gbps would make so many things easier.
I mean, yeah, you have a 100 Mbit/s (or 12.5 MB/s) upload rate under ideal conditions. If my math is right - havenât had my coffee yet - thatâs just 4% of the OPs bandwidth.
I recently had this same problem with a multigig copper SFP+ module. These SFP+ modules (usually) advertise 10Gbps to the NIC they're plugged in, but will negotiate lower speeds on the copper side. So when NIC tries to send to SFP+ at higher rate than whatever speed SFP+ has negotiated internally, the SFP+ module will just drop a bunch of packets and incur a significant hit in throughput.
These tend to not support sending pause frames to its SFP+ side either, so the best option I found was to set bandwidth limit on the NICâŠ
Good tip! I was looking into those recently, as I have a switch whose SFP+ slots only support 1 or 10G, but I want the option to connect 1/2.5/5/10 devices.
Sounds like that may be more trouble than it's worth. Hmm. I may still poke at it, but this definitely opened my eyes to a potential trouble spot that hadn't been on my radar.
I have nothing useful to add other than I feel so validated reading this. Ive been struggling with this issue on and off for over a year. With the release of Astra, it was able to finally unstick me to the point of enabling flow control which has been good enough.
I experienced this problem with a macbook and a thunderbolt dock. I had tried multiple different brands of docks and ethernet adapters, and various macbooks. One interesting thing I discovered is if I plugged a non-thunderbolt cable between the dock and the laptop, I would achieve full speed with no packet loss, so interestingly the issue doesn't occur over pure USB but only over thunderbolt.
This has got to be a bug on Apple's side right? Would be nice if they fixed it.
What kind of Thunderbolt cable, and what kind of USB cable? The downside of using the USB-C connector for everything is that it obscures the difference between active and passive Thunderbolt cables.
Oof. That is the kind of thing where Apple will never do anything. They implement standards, but they guarantee absolutely nothing if even so much as one cable involved isnât Apple-branded. Using docks and Ethernet adapters? Good luck! Theyâll just ask if youâve considered buying a $5000 monitor with a $1000 stand to use as a dock instead. (And ironically, thatâs what OP did and still had the problem :D)
> Using docks and Ethernet adapters? Good luck!
All the more irritating is that all this dock and dongle bullshit was popularlized by them.
I had the same problem recently when trying to figure out high bandwidth game streaming from my desktop to my Steam deck in the living room. The beta streaming tech is âpyrowaveâ and it is super low latency and looks INCREDIBLE compared to traditional codecs, but as a trade off it is extremely sensitive to packet loss, and uses an appropriately large amount of bandwidth (7-800mbit at 4k120). Wired network isnât _strictly_ a requirement but it might as well be, and the steam deck doesnât have a physical port - but the steam deck dock does.
Anyway, the exact same situation arises, where some network condition or physical configuration (thereâs a lot of hardware in between these two devices, including MoCA adapters) meant that the network adapter buffer fills up and starts dropping packets. The buffer size is configurable through some terminal command and maxing that out makes the issue disappear immediately.
Unrelated to this particular issue, but I've had subtle monitor behaviors catch me off guard as well. Usually I think of monitors as being unrelated to the underlying program running, but this is not always the case. For example I released a JavaScript programming game on steam that uses chrome (Electron) for sandboxing. This game has a tiny community but already there are a couple players doing speed running [1], and one discovered that you can complete the game many seconds faster if using a 120Hz monitor instead of 60Hz. The reason is that the JS API requestAnimationFrame [2] is tied to monitor's refresh rate (which I didn't realize) and I originally didn't do any sort of logic limiting it to 60fps (and actually I was using PixiJS tickers[3] which I didn't realize were using requestAnimationFrame under the hood).
[1] https://www.speedrun.com/Pragma_Twice
[2] https://developer.mozilla.org/en-US/docs/Web/API/Window/requ...
Get a daily email with the the top stories from Hacker News. No spam, unsubscribe at any time.