There are some hilarious tidbits in here
> Starting in the first week of 2024, the FreeBSD boot process suddenly got about 3x slower. I started bisecting commits, and tracked it down to... a commit which increased the root disk size from 5 GB to 6 GB. Why? Well, I reached out to some of my friends at Amazon, and it turned out that the answer was somewhere between "magic" and "you really don't want to know"; but the important part for me was that increasing the root disk size to 8 GB restored performance to earlier levels.
The original object size limit for S3 was 5 GB, as noted in my 2006 blog post:
https://aws.amazon.com/blogs/aws/amazon_s3/
I do not know if this has anything to do with the cliff that you saw.
Pretty sure that's not related. For one thing I don't think EBS snapshots are stored in S3 as 5 GB segments.
Now I really want to know though.
My understanding is that EBS has some heuristics for deciding whether to keep data cached; an AMI which has a cached snapshot as its root disk will boot much faster than an AMI where all the data needs to be pulled from S3.
Some huge customer chunked their data into 5GB pieces so now there's a "if size == 5GB" in the cache code.
What's the smallest size for which those heuristics keep the snapshot cached?
(I'm currently using 1GB snapshots, because my actual disk image is a tiny fraction of that size. But if bumping that to 2GB or 4GB would make it faster, that's a small price to pay.)
Yeah, I am constantly curious about how the sausage that is cloud services like AWS is made. It seems generally slick on the surface, but whatās holding it all together? I imagine it as a tangled ball of tools like Puppet, Chef, etc. and custom glue.
A lot of AWS services are built on other AWS services. Like Lambda, SQS, and other such "core services" are used by others under the hood.
At Amazon scale mostly everything is custom
Less puppet/chef
I wonder how long did it take to bisect such issue. Build image every time and reboot a vm?
I can't remember exactly but it was a few hours. I already knew which week the issue arose (from comparing weekly snapshots) so that gave me a head start.
But yes, I built a lot of AMIs. And launched new EC2 instances for each of them -- it wasn't just a matter of rebooting since the first time an AMI launches there's different behaviour (both from FreeBSD, e.g. growing the root disk, and from EC2, e.g. disk caching).
Thanks for the additional information, a few hours sounds great, I was expecting multiple days to narrow it down, given a lengthy feedback loop.
Sweet! By the way we just added FreeBSD to the download page on ziglang.org (as of today), so FreeBSD users can grab master branch builds automatically built by the CI.
It's also now a first-class supported cross-compilation target, including when linking libc, so you can do stuff like `zig cc -o hello hello.c -target riscv64-freebsd`.
And then of course if you have any C/C++ dependencies, you can fetch and build them with the zig build system, so it should be possible to easily cross-compile even quite complex projects for FreeBSD now.
Hopefully that helps more projects decide to add FreeBSD support and respective testing to their CI!
Zig's cross compilation is awesome, and it's nice to see FreeBSD on the supported target list.
There is also a lot of work on the laptop front, I read that the BSD foundation invested $750k for this implementing: (S0ix Sleep State, etc )
you can find the project laptop here https://github.com/FreeBSDFoundation/proj-laptop
Yep there's a lot of work going on. I was just writing about the work I was doing. ;-)
Lots of respect for cperciva.
Donāt know how he manages all of this + Tarsnap.
It turns out that at a certain point, money can buy time. Do I fix the leaky tap myself, or hire a plumber? After electricians rip up my basement drywall (perfectly reasonably -- I was getting solar panels installed and the electrical panel needed to be upgraded) do I fix it myself or do I hire a professional drywaller?
To be fair, some of the time I spent on this came away from Tarsnap. But less than you might imagine.
> or do I hire a professional drywaller
When it comes to drywall, always hire a professional. Learn from other's mistakes... it's not as easy as you think and it won't turn out well.
Itās really not hard. I added onto a house I owned back in 2009 after returning from Iraq and saving a bunch of money. IRS also gave me a huge refund that year. Something with GWOT vets and taxes with combat pay. Anyhow, converted a covered porch to a room, had permits, raised the floor, added electric, energy efficient Argon gas windows, etc. You give 84 lumber the blue print plans from the architect and they tell you how much material you need. After I got to the stage to add furring strips for the concrete block part and insulation (which is easy too), I was ready to add the dry wall. Just lift it up to the wall then screw it in. Dry wall tape, corner bead, and mud to cover the seams. Orange peel can hide imperfections. Then paint. Pretty straightforward
True. In this case it was drywall in a poorly lit basement, so I wasn't all that concerned about it turning out perfectly -- but it absolutely did turn out much better than it would have if I tried to do it myself.
I know how to do drywall as well as a pro - it just takes me 6x as long. It is easy to do, but you can't do what the pros do without a lot of practice. By planning on 6x longer you can slow down, do thiner coats and such (pros do 3 coats of mud, I do 6 for the same thickness). Which falls into the do or hire.
Dry wallers are amazing. As a diyer, itās one of the few things I can never seem to get right. Iām happy to put holes in it, but seeing professionals patch it is another level.
At a certain point money can even buy money.
The other day I had the opportunity to get a 10% discount on a fridge if I could pay the whole thing in one payment. If I didn't have the money I wouldn't get the discount, so in a way being poor means everything is more expensive.
"The reason that the rich were so rich, Vimes reasoned, was because they managed to spend less money. Take boots, for example. ... A really good pair of leather boots cost fifty dollars. But an affordable pair of boots, which were sort of OK for a season or two and then leaked like hell when the cardboard gave out, cost about ten dollars. ... But the thing was that good boots lasted for years and years. A man who could afford fifty dollars had a pair of boots that'd still be keeping his feet dry in ten years' time, while a poor man who could only afford cheap boots would have spent a hundred dollars on boots in the same time and would still have wet feet. This was the Captain Samuel Vimes 'Boots' theory of socio-economic unfairness."
Being more poor is always more expensive than being less poor.
All poor people know this in their bones because they face this every day of their lives.
[flagged]
[flagged]
A fourteen year old bug is relevant how?
If you've found a method to write software that is 100% bug-free, we're all ears.
There are bugs and then there are BUGS!
[flagged]
I was rather hoping Amazon would spend and contribute more. But it seems they basically only want to pay for the minimum FreeBSD support.
Amazon isn't even on FreeBSD sponsors [1]. And Google only sponsored $9K last year. Apple isn't there. Edit: And Credit to Microsoft being at least on the list! And forgot to mention Meta / Facebook missing from it as well.
I would have expect them to sponsor FreeBSD and OpenBSD annually by default given they use and continue to benefits the work out of both.
[1] https://freebsdfoundation.org/our-donors/donors/?donationYea...
I'd love to see Amazon contribute more, of course; but the fact they don't show up as donors to the FreeBSD Foundation doesn't mean they're not supporting FreeBSD. The money they paid me didn't flow through the Foundation, for example; I'd guess that Foundation-funded development is maybe 10% of all corporate-funded FreeBSD development. (It's an important 10%, especially because it can be focused on "what does FreeBSD need" rather than "what does company X need" -- but it's still a small minority.)
This comment does not present the full picture.
First, it presents the snapshot of donations within a given year to the Foundation. The history of donations is not represented by definition.
Second, it does not present contributed development. Those are typically summarily available on the release notes of each release [1].
Has Amazon donated more to FreeBSD than Notch the Minecraft guy?
I wonder for what reason Microsoft funds them. Their Hyper-V extensions are not as complete as Linux. There's no Microsoft-supported port of .NET. I can't think of any services that run on *BSD from Microsoft, cloud or otherwise.
There are customers that run FreeBSD on Azure and Microsoft officially supports it: https://learn.microsoft.com/en-us/azure/virtual-machines/lin...
I don't know, but Microsoft has some developers working on Hyper-V for FreeBSD. They've even come to FreeBSD developer summits.
My thought process was that it is such a small amount of money for Advertisement or marketing to have your logo on some prominent Open source project to at least try and redeem / make themselves look good.
I wonder if this is a very, very, very long-term holdover from the Hotmail team having some expertise, and Microsoft wanting to be able to offer something a bit nix-y that wasn't Linux
Amazon does the least for FOSS out of the FAANGs.
I wanted to use FreeBSD for my home gateway/firewall/dns/dhcp server, but unfortunately my 10 GbE NICs didn't seem to have drivers, so I ended up going with Nix instead. I used FreeBSD many years ago as a workstation, and found the experience to be quite memorable. It's nice to see that it's still chugging along.
I use FreeBSD for all my (and my companies) infrastructure. I only use hardware with Intel NICs because they are 100% reliable on FreeBSD. Anything with Realtek seem to crap out under load despite the hard work of the FreeBSD engineers who maintain the drivers (I'm not complaining and I respect their efforts).
It's a small price to pay and it stops me having to install less stable operating systems.
I remember the time around version 7 or 8 when FreeBSD had better drivers than Linux for stuff like Atheros wifi cards.
I favored FreeBSD until around 2021 when computers with different CPU mixed together started to become common. I first bought a RockPro64 with 2 big and 4 little cores and then an Intel Alder lake. As far as I understand FreeBSD scheduler to this day don't know how to properly play with these so it brings the system to the lowest denominator of the slower cores.
Out of curiosity, who are the top users of FreeBSD/EC2?
I have no clue. Seriously, the users who talk to me are maybe 0.1% of the total FreeBSD/EC2 user base.
I would love to know who is using FreeBSD in EC2.
I can tell you some financial services I have worked for do use FreeBSD on EC2 as well as on the metal in data centers to do millions of transactions a month. I like the OS, thanks for your work.
Does netflix only use it on their edge boxes?
Yes, their CDN is FreeBSD but last I heard all of their cloud operations are Linux.
Get a daily email with the the top stories from Hacker News. No spam, unsubscribe at any time.