2024-09-21 · Infra Log

Trailing 2 Weeks Incidents

(Larger boxes are longer, darker boxes are higher-impact; see last week’s bulletin for details about the top row of incidents).

This Week In Infra Engineering

Will did a deep dive into I/O scheduling in our LHR clusters, after Tigris nudged us about performance/reliability issues in their FoundationDB cluster running in that region. Using metrics, system configuration, and statekeeping data, Will isolated Tigris’s workloads to a concentrated cluster of SSDs with a particular make/model, which we now know to have an iffy performance envelope for the kinds of work we do. The bigger problem wasn’t so much the drives as it was the scheduling we did: because Tigris created their series of volumes for this region in rapid succession, they all got scheduled on a small subset of our storage capacity in the region. Worse, a consequence of our scheduling algorithm was that the periodic snapshot backups of these volumes were all scheduled to fire in tandem, concentrating a large amount of I/O activity on a small number of drives (the graph of what was happening looked like a stable EKG). Scheduling improvements are a theme of the infra work that we’re doing right now, but this issue in particular surfaced an unexpected (and straightforward to fix) issue: we needed to be adding jitter to the timing of our backup jobs.

Ben is hip-deep in a fleetwide upgrade of our worker OS distributions. This is a tricky and annoying problem. Most of what runs on a worker physical for us is software we build and ship ourselves, in some cases several times a week. Beyond that, we have an established runbook for upgrading our OS kernels; we don’t run the distro-version-native kernel version anywhere (we have fussy eBPF dependencies, among other things). But the distro itself, which in particular sticks us with a specific version of systemd, is a gigantic pain to upgrade; we have consistent OS kernels across the fleet, but not consistent distro versions. That’s changing, but it’s a complicated process, involving workload migrations and, in some cases, reprovisioning servers, which surfaces fun bugs like “the semi-random identifiers we create for Fly Machines are influenced by the provenance of the worker physical on which it was created, meaning a reprovisioned host can cause Machine ID collisions”. That bug hasn’t happened, because Ben is auditing the stack for problems like that.

Dusty has been the capacity czar for the past couple months. As you can see from this week’s BOS outage, this is an important issue for us. We’ve integrated our scheduler state with system metrics and used that to create new capacity threshold numbers for the fleet, which now informs our provisioning; a lot of the guesswork has been taken out of where we’re shipping and racking new servers. We’ve shifted some capacity (ORD gave some servers to EWR, for instance), and reallocated a backlog of servers to different regions. We now have a runbook for provisioning new capacity in existing regions that uses a lightweight version of our machine migration (for apps without storage) to rebalance as capacity is added.

JP has been working on improving Fly Machine scheduling across the fleet, which was also implicated in an incident this week. We now have stricter placement logic that ensures multiple Fly Machines for the same app created concurrently are distributed across worker physicals. Our Machines API now handles some of the retry logic that our CLI, flyctl, was using before; unlike flyctl, which is open-source and doesn’t have APIs that can specifically place a Fly Machine on a specific worker physical, our API gateways do have visibility into the physicals in a region, and we now take advantage of that.