I'm getting cf-mitigated: challenge on openai API requests.
https://www.cloudflarestatus.com/ https://status.openai.com/
But the day comes that I need to tweak a deploy flow, or update our testing infra and about halfway through the task I take the whole thing down. It's gotten to the point where when there's an outage I'm the first person people ask what I'm doing...and it's pretty dang consistent....
[1]: https://www.reddit.com/r/ProgrammerHumor/comments/1p204nx/ac... [2]: https://news.ycombinator.com/item?id=47230704
Our health check checks against githubstatus.com to verify 'why' there may be a GHA failure and reports it, e.g.
Cannot run: repo clone failed — GitHub is reporting issues (Partial System Outage: 'Incident with Copilot and Actions'). No cached manifests available.
But, if it's not updated, we get more generic responses. Are there better ways that you all employ (other than to not use GHA, you silly haters :-))
There are heavier solutions, but even setting something like this up as a backstop might be useful. If your blog is being hammered by ChatGPT traffic, spare a thought for Github. I can only imagine their traffic has ballooned phenomenally.
1: https://duggan.ie/posts/self-hosting-git-and-builds-without-...
It can be a pain to setup a break-glass, especially if you have a lot of legacy CI cruft to deal with. But it pays off in spades during outages.
I'm biased because we (dagger.io) provide tooling that makes this break-glass setup easier, by decoupling the CI logic from CI infrastructure. But it doesn't matter what tools you use: just make sure you can run a bootstrap CI pipeline from your local machine. You'll thank me later.
Of course, once you have the momentum it doesn't matter nearly as much, at least for a while. If it happens too much though, people will start looking for alternatives.
The key to remember is Momentum is hard to redirect, but with enough force (reasons), it will.
https://mrshu.github.io/github-statuses/
Most individual services have two nines... but not all of them.
Which is really baffling when talking about a service that has at least weekly hicups even when it's not a complete outage.
There's almost 20 outages listed on HN over the past two months: https://news.ycombinator.com/from?site=githubstatus.com so much for “always available”.
That being said, GitHub is Microsoft now, known for that Microsoft 360 uptime.
https://www.windowscentral.com/microsoft/using-ai-is-no-long...
https://thenewstack.io/github-will-prioritize-migrating-to-a...
I mean... It's right in the name! It's up for 360 days a year.
And the frequency they can tolerate is surprisingly high given that we're talking about the 20th or so outage of 2026 for github. (See: https://news.ycombinator.com/from?site=githubstatus.com)
I'm on the lookout for an alternative, this really is not acceptable.
Should have self hosted.
If anyone is using Github professionally and pays for github actions or any github product, respectfully, why?
You can switch to a VPS provider and self host gitea/forejo in less time than you might think and pay a fraction of a fraction than you might pay now.
The point becomes more moot because github is used by developers and devs are so so much more likely to be able to spin up a vps and run forejo and run terminal. I don't quite understand the point.
There are ways to run github actions in forejo as well iirc even on locally hosted which uses https://github.com/nektos/act under the hood.
People, the time where you spent hundreds of thousands of dollars and expected basic service and no service outage issues is over.
What you are gonna get is service outage issues and lock-ins. Also, your open source project is getting trained on by the parent company of the said git provider.
PS: But if you do end up using Gitea/forejo. Please donate to Codeberg/forejo/gitea (Gitea is a company tho whereas Codeberg is non profit). I think that donating 1k$ to Codeberg would be infinitely better than paying 10k$ or 100k$ worth to Github.
I generally recommend that the break glass solution always be pair programmed.
Crazy in 2026, but installable software has some pros still, for both the developer and for the customer. And I would personally love if I could do things that way for more things.
Yeah. You probably do want to make sure you turn your .git/ into a "bare" git repository but that's basically it.
And it's what I do too: an OCI container that gives me access to all my private Git repos (it sets up SSH with U2F so I get to use my Yubikey to push/pull from various machines to those Git repos).
does anyone know where these "detailed root cause analysis" reports are shared? is there maybe an archive?
PRs are a defacto communication and coordination bus between different code review tools, its all a mess.
LLMs make it worse because I'm pushing more code to github than ever before, and it just isn't setup to deal with this type of workload when it is working well.
GitHub was the pinnacle of GitForge a couple of years back, and it seems like they wanted to hit a wall.
Otherwise, you cannot explain how you can enshittify a software that much.
A self-hosted git server is trivial. Making sure everything built on top of that is able to fallback to that is not. Especially when GH has so many integrations out of the box
We built a CI platform using dagger.io on top of GH Actions, and the "break glass" pattern was not an afterthought; it was a requirement (and one of the main reasons we chose dagger as the underlying foundation of the platform in the first place)
Even if I get the idea of an automation before there’s a run book for it.
I did a PoC of Dagger for an integration and delivery workload and loved the local development experience. Being able to define complex pipelines as a series of composable actions in a language which can be type checked was a great experience, and assembling these into unix-style pipelines felt very natural.
I struggled to go beyond this and into an integration environment, though. Dagger's current caching implementation is very much built around there being a single long-lived node and doesn't scale out well, at least without the undocumented experimental OCI caching implementation. Are you able to share any details on how Fastly operates Dagger?
Ironically, this makes Dagger even more relevant in the age of coding agents: the bottleneck increasingly is not the ability to generate code, but to reliably test it end-to-end. So the more we all rely on coding agents to produce code, the more we will need a deterministic testing layer we can trust. That's what Dagger aspires to be.
For reference, a few other HN threads where we discussed this:
There are also monthly availability reports: https://github.blog/tag/github-availability-report/
Have you ever considered that this is the problem? GH never planned for this sort of pointless and unpaid activity before. Now they have a large increase (I've seen figures of 100x) in activity and they can't keep up.
It doesn't help that almost none of the added activity is actually useful; it's just thousands and thousands of clones of some other pointless product.
Also, how would PRs and code review be handled?
Your suggestion really only makes sense for a small single developer hobby project in an interpreted language. Which, if that is what you intended, fair enough. But there really wasn't enough context to ascertain that.
Yes, I agree on your assessment. AI means a higher rate of code changes, so you need more robust and fast CI.
If you're already at the point where you're fielding pull requests, lots of long running tests, etc., you'll probably already know you need more than git over ssh.
Born just in time to talk about this situation on hackernews xD (/jk)
> Too slow: https://github-incidents.pages.dev/
I am not even mad that I am slow honestly, this is really funny lol.