I'm really kind of surprised they let us do that - like, why didn't they just have you upload the binaries after building on your local machine?

As a side note, I started getting these symptoms (actions staying queued forever or not running at all) at 5pm PDT yesterday, intermittently. And definitely full outage by 8:30pm PDT. So for sure full outage for 7 hours and counting, even if you use other runners, and I strongly believe partial outage for ~15.5+ hrs before that, even if it hasn't been acknowledged by GitHub yet.

I should eat lunch.

As someone who has spent many years working in high load environments, this is not an uncommon pattern.

You design a system and it works great. It can handle failures, load spikes, it is horizontally scalable, things are great. You think you figured it out.

And then load keeps increasing and you suddenly hit a tipping point where everything keeps failing, and you cant keep up. The things that you thought were perfectly horizontally scalable turn out to have a bottleneck you didn’t even think about until you got to a truly massive scale. Your systems suddenly don’t have the excess capacity to handle load spikes or catchup work, so suddenly any failure cascades and recovery is more and more difficult. You can’t solve the problem with additional hardware, and your perfect scalable design actually can’t scale any more.

This doesn’t have to be about GitHub using LLMs in their code to still be related to LLMs. GitHub gets a lot more commits now because of LLMs and probably get a lot more reads because of LLMs as well.

It could be that the extra usage just pushed them past one of those capacity thresholds.

With what to show for it? If GH did 10x in volume/git commits, it's all LLM sloppy-pasta. Where's the 10x productivity? Where are the amazing apps?

These are folks that routinely make it a point to press on system design and scalability during interviews.

Now they suddenly can’t scale or design systems but we should accept that?

I'm not sure why this particular industry is so abysmal at making things even semi-reliable after decades of research, educated workforces, and loads of cash.

The fix is merged, but won't deploy... it's been hours

Thankfully it's a batch job, and isn't interrupting production ATM

I thought they should rename to Anglia Busways and have bus replacement trains instead.

Yes, we call it: Microslop.

Sitting on Github these days is the same as sticking to twitter a decade ago, expect next mecha hitler, I suppose.

[0] https://depot.dev

This is a man who's spent a significant portion of every day for the last 15 years on GitHub.

This is multiple times this month that this has been a problem.

Has GitHub completed it's internal migration to Azure yet? Or is it still ongoing? None of our devs want to switch away from GH, but we will have to at this point.

Probably both. But it still throws a thorn into the theory that LLMs are about to replace software engineers any day now.

You'd think they could LLM-code their way out of this situation easily if LLMs were the software engineer replacement they are being marketed as.

Even my own systems at home are burgeoning under the load of the my more ambitious hobby projects I'm doing for fun. I don't envy "we had to build five more datacenters to keep up with demand" class problems just from a how many people have to sign off on them perspective let alone the technical difficulty of doing so

For like 10 years the only feature development they did was by stealing ideas from GitLab. I wouldn't be shocked if little if none engineering discipline has took place at all during this time if it's this brittle to frequent change.

Guessing it was mostly held together with duct tape and poorly written tests/monitoring systems if any other corporate driven software.

We're really going to find out over the next few years which businesses have good practices or not.

Outages happen, but this many outages so close together, and so many of them so major/long lasting, something is systematically wrong for sure. It's been seriously hamstringing our ability to ship code at my company.

This is no surprise given standard Microsoft operating procedure - https://news.ycombinator.com/item?id=47616242

If the scheduling was self hosted it would be inexcusable but you can always just connect whatever you want to webhooks.

This is annoying and I'm here because it's down. But it would have to be far worse to come close to actually being worth changing.

If I had to guess it's because Github is sitting on top on infrastructure held up by toothpicks and duct tape

It's hard to draw a direct analogy there, but I feel like it echoes the same sentiment.

A soapbox I have is that GHA workflows are scripts that could run on your machine without any of the YAML stuff. Who gives a flying about the DAG or the logs? Which, by the way, if you're willing to walk to the milk store to buy your milk, could be recreated in a much more testable and maintainable way without any of the YAML bs that GHA prescribes....

But DAGs are pretty, and logstreams showing up in a browser application instill trust (for reasons that fly far above the head of yours truly). So people go for that. Pretty DAG, nice logstream; therefore, deliver my milk. All of a sudden.... The CI/CD platform is having its merry way with your SLAs, contract abidements, and hotfix deployments.

What a time to be alive.

Not to rail on the South Park thing, but the blast radius of this issue also reminds me of the episode where the internet dried up.

If this bs with GitHub continues, Parker/Stone will have to make a GitHub episode. How seen would we all feel if that happened?

There's always the escape hatch of running you GHA workflows locally, but unfortunately, despite the existence of packages like `act`, there is no way to fully recreate the GHA runtime locally. Tons of the special YAML syntax just can't (more accurately, "just doesn't") get interpreted by those local actions runners.

We never went this route, but at my old org, I always advocated for considering GHA to be wrapper around a single bash script (or whatever script you want to run), as a means of completely breaking out of the GHA hellscape that is programming in YAML, who's turing-completeness is pretty dubious.

Unless you have things set up this way, you (the client of GitHub) would have to completely redesign your CI on the fly, run it locally, and then figure out how to get the D compliment of the I to work in a way that is auditable. Fat chance for most teams I bet.

Thank god you're dealing with a batch scenario. Silver lining for sure. Still, embrace the anger.

What makes my blood boil is that there's millions of DEVs literally crying at the moment worrying about how GitHub's failure to be responsible will put their jobs in jeopardy.

And fingers crossed for you my friend. We're at 5+ hours at the time of this writing.... You're batch job may still have a chance!!!

Edit: to be clear, I mean the part where they maintain state on their end. I have no idea if they do that with another runner instance, that seems unlikely.

They have a strong motivation (self preservation) to continue to misunderstand the problem. If they did what is best for us, then we could avoid a substantial fraction of all GH subscriptions by using a FOSS tool to hit the Pareto frontier by replicating just enough GH services to watch commits and PRs.

I think people were so excited to move away from jenkins to something 'managed' just because of how much a dinosaur jenkins is and how much a pain in the ass it is to upgrade it... but now we are seeing how managed can bite you in the ass if the manager is incompetent.

Step one - migrate my build workflows to Docker.

My Github actions are now basically: "checkout / set env vars from secrets / docker-compose builder run make".

I used large machine runners to run full the Docker (Podman actually) on Github first to avoid dealing with docker-in-docker complications. This step also provided some very nice robustness advantages, as I can now trigger deployments from my laptop if needed.

Step two:

Migrate to self-hosted runners. I used my former homelab server to set up a build machine. It has 16Tb of fast NVMe SSDs and thanks to Podman container layer caching, my entire lint workflow now takes 30 seconds. Faster than just one "npm install" on Github before.

And Github's self-hosted runners are actually surprisingly easy to set up and use. They are also somewhat more robust.

Step three:

Swap Github for something else.

Seems like the only reliable way to run GHA jobs is to not use their runners. Hope they at least didn’t break self-hosted runners operations

What is happening at GH?

Rate of change trying to keep up with new challengers? Over-reliance on AI? Engineers trying to debug slop?

Their MO was to court an executive and sell second-rate tools to them before the people who had to use them had a chance to say anything. It doesn't matter how much evidence you can provide to the contrary, once the million dollar deal is signed, you are going to be tasked with finding reasons to say that your executive was shrewd for buying this pile of junk and unfulfilled promises, and not an insane idiot sucking away your job satisfaction as fast as they can.

They did a lot of deals based on how their products would have features their competitors already have 'soon' when they haven't even started them, and a long track record of taking 3 major releases to get from something to good, and then breaking everything again by doing a 4th major release that re-imagined everything and made it horrible again.

I'm not going to claim that Apple was or is a panacea. Apple doesn't use vaporware which is big, and their Cycle of Awful is 2 releases instead of 3. You could afford to skip 1 waiting for the next even-numbered version, instead of being 2 versions behind and getting pressed to upgrade.

Self-hosting a service like GitHub that operates at GitHub scale is difficult.

Self-hosting a service like GitHub that operates at the typical small/medium company's scale is trivial.

A single machine (with separate runners for CI) will cover many companies' needs. It being a single machine eliminates a lot of the complexity and failure modes associated with a distributed system and makes backups/restores/maintenance easy.

> Customers using self-hosted runners may see errors or rate limiting when runners register.

Windows 95 was a brilliant hack that allowed 32-bit GUI programs to run on hardware with 2Mb of RAM (Windows had 4Mb of RAM as the official minimum, but you could run it on 2Mb) while preserving compatibility with the majority of DOS software.

Maybe they're hosting in us-east-1 though :)

They're at 93.91% uptime over the past 90 days, according to https://mrshu.github.io/github-statuses/ , and that doesn't even include today's outage yet.

A glorious one nine of reliability.

In large part, the move from AWS to Azure. Azure's just bad.

Windows NT was one of them. Up to Windows XP the products were pretty solid and each had visible improvement against the previous one.

Their language products were/are still solid IMO. Maybe Visual Studio is sluggish, but we can still use an older version if we want. Plus they put a lot of effort optimizing VSCode, too.

Even back in the MS-DOS/16-bit Windows days, when things broke down quite easily, I think they still provide the best bang for individual users and developers. There was no competitors who could provide so much value back then.

GitHub Will Prioritize Migrating to Azure Over Feature Development - https://news.ycombinator.com/item?id=45517173 - October 2025 (63 comments)

I'm sorry but this made me laugh out loud. That isn't true at all, this has been going on for years. This conversation[0] from six years ago has discussion about the outages starting to become much more frequent in December 2019. It has never gotten better in that time, it's just continually degraded.

[0] https://news.ycombinator.com/item?id=22935941

Seems pretty conclusive. Very similar story when they bought skype.

I have an Ops background and I strongly suspect they were given a stupid timeline for the Azure migration.

I've got to believe Microsoft have decent Ops people but the management wanted to move faster than was reasonable and screwed it up. Move one thing at a time and double check it all works and you can do a migration like this.

If I could right now:

1. go sign-up elsewhere 2. Log into GitHub and point Actions to that new host 3. All my actions files immediately worked without question

I'd probably give it a spin and make a wiki page explaining how to swap back and forth. No meetings. No design issues. No scheduling. Just a flip switch on who to pay for computers.

But for people who either don't pay anything at all or phenomenal amount one 9 of up time is all you need.

If you're actually trying to run a business I guess you can call and gitlab and get an Enterprise contract

I don't disagree that it's obvious they've got problems but I'm just saying it's obvious to me the part that falls over (the scheduling of jobs) and why that would impact self hosted runners, which do no scheduling but depend on it to function.

As for 'just a message queue with some database updates and sharding that's easy to reason about'... Here's a job scheduling problem as an example: imagine you schedule a job, and there's no runner available. How do you disambiguate between no runners available because you've reached capacity, runners not being available because they're on a real network with faulty connections, and runners not being available because of a faulty rollout of internal updates?

A simple message queue for job scheduling is fine if you own everything and can deal with the operational overhead of identifying those cases by hand, but Github can't do that.

Disagree. I was admining NT4.0 boxes back then and it was

1. very slow

2. constantly leaking that required weekly scheduled reboots

3. security wise it was nightmare even by that times standard

Sure, but the part that actually schedules where a 'job' gets run is based on a relatively simplistic tag system. Reading the yaml and plopping some job metadata into a queue-like system isn't where I would expect their issues to be, but at their scale I'm sure everything becomes fragile and inscrutable.

> imagine you schedule a job, and there's no runner available. How do you disambiguate between no runners available because you've reached capacity, runners not being available because they're on a real network with faulty connections, and runners not being available because of a faulty rollout of internal updates?

You don't need to. GitHub Actions runners, and most CI runners that I've interacted with appear to have a pull-based model where they ask for work that matches their declared tags/shape (usually platform/runtime/OS/etc.). This probably amounts to a database query, but who knows.

> A simple message queue for job scheduling is fine if you own everything and can deal with the operational overhead of identifying those cases by hand, but Github can't do that.

I highly doubt it's a simple message queue. My issue is git repos and their CI infrastructure have very low coupling to other repos or entities in most circumstances, at least conceptually, so parts of the system (ie. regions, shards, etc.) should be able to function even when others are down (ie. it shouldn't break for everyone). There's clearly centralization and coupling that isn't obvious from an outside perspective, which sorta tells me it's incidental, but that's a guess.

For the last point what was the golden standard back in the mid-late 90s, if we don't include mainframe/minicomputers? Was it Solaris or BSD?

Now that I think about it, I feel sad that I do not have the technical prowess to compare operating systems :/

9% availability would be an uptime of ~33 days a year, I think at that point, we're pushing the semantics of "available" if the service is down the entire year except one month on average.