GitHub Is Having Issues

I would prefer we have posts when github is not having issues to cut down on noise.

A directory over SSH can be your git server. If your CI isn't too complex, a post-receive hook looping into Docker can be enough. I wrote up about self hosting git and builds a few weeks ago[1].

There are heavier solutions, but even setting something like this up as a backstop might be useful. If your blog is being hammered by ChatGPT traffic, spare a thought for Github. I can only imagine their traffic has ballooned phenomenally.

1: https://duggan.ie/posts/self-hosting-git-and-builds-without-...

Insert the standard comment about how git doesn't even need a hub. The whole point of it is that it's distributed and doesn't need to be "hosted" anywhere. You can push or pull from any repo on anyone's machine. Shouldn't everyone just treat GitHub as an online backup? Zero reason it being down should block development.

In moments like this, it's useful to have a "break glass" mode in your CI tooling: a way to run a production CI pipeline from scratch, when your production CI infrastructure is down. Otherwise, if your CI downtime coincides with other production downtime, you might find yourself with a "bricked" platform. I've seen it happen and it is not fun.

It can be a pain to setup a break-glass, especially if you have a lot of legacy CI cruft to deal with. But it pays off in spades during outages.

I'm biased because we (dagger.io) provide tooling that makes this break-glass setup easier, by decoupling the CI logic from CI infrastructure. But it doesn't matter what tools you use: just make sure you can run a bootstrap CI pipeline from your local machine. You'll thank me later.

Microslop ruins everything it touches.

Is this related to Cloudflare?

I'm getting cf-mitigated: challenge on openai API requests.

https://www.cloudflarestatus.com/ https://status.openai.com/

What’s interesting about outages like this is how many things depend on GitHub now beyond just git hosting. CI pipelines, package registries, release automation, deployment triggers, webhooks — a lot of infrastructure quietly assumes GitHub is always available. When GitHub degrades, the blast radius is surprisingly large because it breaks entire build and release chains, not just repo browsing.

I swear this is my fault. I can go weeks without doing infra work. Github does fine, I don't see any hiccups, status page is all green.

But the day comes that I need to tweak a deploy flow, or update our testing infra and about halfway through the task I take the whole thing down. It's gotten to the point where when there's an outage I'm the first person people ask what I'm doing...and it's pretty dang consistent....

I would so very much love to see GitHub switch gears from building stuff like Copilot etc and focus on availability

Maybe we should turn these weekly posts into an actionable item we can use to move organizations away from this critical infrastructure that is failing in realtime.

codeberg might be a little slower on git cli, but at least it's not becoming a weekly 'URL returned error: 500' situation...

> This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.

does anyone know where these "detailed root cause analysis" reports are shared? is there maybe an archive?

I really wish Graphite had just gone down the path of better Git hosting and reviewing, instead of trying to charge me $40 a month for an AI reviewer. It would be nice to have a real first class alternative to Github

I've taken to hosting everything critical like this myself on a single system with Docker Compose with regular off premises backups and a restore process that I know works because I test it every 6 months. I can swap from local hosting to a VPS in 30 mins if I need to. It seems like the majority of large services like GitHub have had increasingly annoying downtime while I try to get work done. If you know what you're doing it's a false premise that you'll just have more issues with self hosting. If you don't know what you are doing it's becoming an increasingly good time to learn. I've had 4 years of continuous uptime on my services at this point. I still push to third parties like GitHub as yet another backup and see the occasional 500 and my workflow keeps chugging along. I've gotten old and grumpy and rather just do it myself.

Happening very often lately

I am getting really tired of github. outages happen that's a given. but on so much stuff they don't even care or try. Github is becoming the bottleneck in my agentic coding workflows. unless I make Claude do it intelligently, I hit rate limits checking on CI jobs (5000 api requests in an hour). Depot makes their CI so much better, but it is still tied to github in a couple of annoying places.

PRs are a defacto communication and coordination bus between different code review tools, its all a mess.

LLMs make it worse because I'm pushing more code to github than ever before, and it just isn't setup to deal with this type of workload when it is working well.

How reliable is githubstatus.com? I know that status pages are generally not updated until Leadership and/or PR has a chance to approve the changes; is that the case here?

Our health check checks against githubstatus.com to verify 'why' there may be a GHA failure and reports it, e.g.

Cannot run: repo clone failed — GitHub is reporting issues (Partial System Outage: 'Incident with Copilot and Actions'). No cached manifests available.

But, if it's not updated, we get more generic responses. Are there better ways that you all employ (other than to not use GHA, you silly haters :-))

Microslop is farting too hard on vibecoding

In many companies I worked for, there were a bunch of infrastructure astronauts who made everything very complicated in the name of zero downtime and sold them to management as “downtime would kill pur credibility and our businesses ”, and then you have billion dollar companies everyone relies on (GitHub, Cloudflare) who have repeated downtime yet it doesn't seem to affect their business in any way.

I have a bug bash in an hour and fixes that need to go in beforehand. So of course GitHub is down.

Only on days with a "y"...

How many 9s is GitHub at now? 2?

You know that it's bad when the status page doesn't have the availability stats anymore.

Seems like the xkcd [1] for internet infrastructure that was posted earlier [2] should have github somewhere on it, even if just for how often it breaks. Maybe it falls under "whatever microsoft is doing"

[1]: https://www.reddit.com/r/ProgrammerHumor/comments/1p204nx/ac... [2]: https://news.ycombinator.com/item?id=47230704

I spent hours trying to figure out what was wrong with GHCR, &^$% Github.

I'm on the lookout for an alternative, this really is not acceptable.

Lowendtalk providers who take 7$ per year deals can provide more reliability than Github at this moment and I am not kidding.

If anyone is using Github professionally and pays for github actions or any github product, respectfully, why?

You can switch to a VPS provider and self host gitea/forejo in less time than you might think and pay a fraction of a fraction than you might pay now.

The point becomes more moot because github is used by developers and devs are so so much more likely to be able to spin up a vps and run forejo and run terminal. I don't quite understand the point.

There are ways to run github actions in forejo as well iirc even on locally hosted which uses https://github.com/nektos/act under the hood.

People, the time where you spent hundreds of thousands of dollars and expected basic service and no service outage issues is over.

What you are gonna get is service outage issues and lock-ins. Also, your open source project is getting trained on by the parent company of the said git provider.

PS: But if you do end up using Gitea/forejo. Please donate to Codeberg/forejo/gitea (Gitea is a company tho whereas Codeberg is non profit). I think that donating 1k$ to Codeberg would be infinitely better than paying 10k$ or 100k$ worth to Github.

So Tay.ai and Zoe are still wrecking GitHub infrastructure.

Should have self hosted.

has anyone at MS tried unplugging azure and plugging azure back in yet?

the day ends in y, water is wet. I really hate that github doesn't have any real competition. Yes, I know about gitlab, but it isnt real competition.

GitHub has been shit lately. What the fuck is going on?

The appearance of a thread here is so consistent that HN needs a black-bar style indicator for GH outages that points to it.

Microslop ruins everything it touches.

Is this related to Cloudflare?

I'm getting cf-mitigated: challenge on openai API requests.

https://www.cloudflarestatus.com/ https://status.openai.com/

I swear this is my fault. I can go weeks without doing infra work. Github does fine, I don't see any hiccups, status page is all green.

I would so very much love to see GitHub switch gears from building stuff like Copilot etc and focus on availability

I would prefer we have posts when github is not having issues to cut down on noise.

Yeah, right. I mean, I'm so happy that only one of my clients is using GitHub as their GitForge. Every single other one hosts their own GitForge. And I can't state how much better every single other GitForge is.

GitHub was the pinnacle of GitForge a couple of years back, and it seems like they wanted to hit a wall.

Otherwise, you cannot explain how you can enshittify a software that much.

Only on days with a "y"...

You know that it's bad when the status page doesn't have the availability stats anymore.

[1]: https://www.reddit.com/r/ProgrammerHumor/comments/1p204nx/ac... [2]: https://news.ycombinator.com/item?id=47230704

> a lot of infrastructure quietly assumes GitHub is always available

Which is really baffling when talking about a service that has at least weekly hicups even when it's not a complete outage.

There's almost 20 outages listed on HN over the past two months: https://news.ycombinator.com/from?site=githubstatus.com so much for “always available”.

codeberg might be a little slower on git cli, but at least it's not becoming a weekly 'URL returned error: 500' situation...

These days it feels like people have simply forgotten that you could also just have a bare repository on a VPS and use it over ssh.

I mean, this isn't a 'URL returned error: 500' situation for anything that Codeberg provides considering this is an issue with Copilot and Actions.

I rarely successfully get Codeberg URLs to load. Which is sad because I actually would very much like to recommend it but I find it unreliable as a source.

That being said, GitHub is Microsoft now, known for that Microsoft 360 uptime.

I mean... you understand the scale difference right?

Happening very often lately

How reliable is githubstatus.com? I know that status pages are generally not updated until Leadership and/or PR has a chance to approve the changes; is that the case here?

Our health check checks against githubstatus.com to verify 'why' there may be a GHA failure and reports it, e.g.

Cannot run: repo clone failed — GitHub is reporting issues (Partial System Outage: 'Incident with Copilot and Actions'). No cached manifests available.

But, if it's not updated, we get more generic responses. Are there better ways that you all employ (other than to not use GHA, you silly haters :-))

Right now the page says Copilot and Actions are affected but I can't even push anything to a repo from the CLI.

To be fair - it SUPER does. Being down frequently makes your competition look better.

Of course, once you have the momentum it doesn't matter nearly as much, at least for a while. If it happens too much though, people will start looking for alternatives.

The key to remember is Momentum is hard to redirect, but with enough force (reasons), it will.

The reality is that consumers don't really care about downtime unless it's truly frequent.

How many 9s is GitHub at now? 2?

If you count every service together, it's down to one nine.

https://mrshu.github.io/github-statuses/

Most individual services have two nines... but not all of them.

90 day non-degraded uptime of Github Actions is 98.8% if the official numbers can be believed

Github proudly boasts an industry-leading seven 9s of uptime. 49.999999%

GitHub has been shit lately. What the fuck is going on?

Top-down mandates to use AI as much as possible, and to rip up their infrastructure and move everything to Azure.

https://www.windowscentral.com/microsoft/using-ai-is-no-long...

https://thenewstack.io/github-will-prioritize-migrating-to-a...

Does anything running on Azure have an acceptable uptime?

A directory over SSH can be your git server. If your CI isn't too complex, a post-receive hook looping into Docker can be enough. I wrote up about self hosting git and builds a few weeks ago[1].

1: https://duggan.ie/posts/self-hosting-git-and-builds-without-...

It can be a pain to setup a break-glass, especially if you have a lot of legacy CI cruft to deal with. But it pays off in spades during outages.

I mean... you understand the scale difference right?

To be fair - it SUPER does. Being down frequently makes your competition look better.

Of course, once you have the momentum it doesn't matter nearly as much, at least for a while. If it happens too much though, people will start looking for alternatives.

The key to remember is Momentum is hard to redirect, but with enough force (reasons), it will.

90 day non-degraded uptime of Github Actions is 98.8% if the official numbers can be believed

If you count every service together, it's down to one nine.

https://mrshu.github.io/github-statuses/

Most individual services have two nines... but not all of them.

Github proudly boasts an industry-leading seven 9s of uptime. 49.999999%

Does anything running on Azure have an acceptable uptime?

> The origin of a git repo is more or less just the contents of the .git directory in a remote location. That's it. You don't even need to run a git server if you're happy enough using ssh for transport.

Yeah. You probably do want to make sure you turn your .git/ into a "bare" git repository but that's basically it.

And it's what I do too: an OCI container that gives me access to all my private Git repos (it sets up SSH with U2F so I get to use my Yubikey to push/pull from various machines to those Git repos).

Doesn't post-receive block the push operation and get cancelled when you cancel the push?

The problem is that any kind of automatic code change process like CI, PRs, code review, deployments, etc etc are based on having a central git server. Even security may be based on SSO roles synced to GH allowing access to certain repos.

A self-hosted git server is trivial. Making sure everything built on top of that is able to fallback to that is not. Especially when GH has so many integrations out of the box

At times like this is when I'm so happy I don't work with deploying to a production environment, but rather we release software that (after extensive qualification), customers can install in their environment on their airgapped networks. Using a USB stick to cross the air gap. If we miss a release by a day or thrre, there is enough slack in the process before it goes to the customer that no one will be any the wiser.

Crazy in 2026, but installable software has some pros still, for both the developer and for the customer. And I would personally love if I could do things that way for more things.

This is a must when your systems deal with critical workloads. At Fastly, we process a good chunk of the internet's traffic and can't afford to be "down" while waiting for the CI system to recover in the event of a production outage.

We built a CI platform using dagger.io on top of GH Actions, and the "break glass" pattern was not an afterthought; it was a requirement (and one of the main reasons we chose dagger as the underlying foundation of the platform in the first place)

It’s a hard sell. I always get blank looks when I suggest it, and often have to work off book to get us there.

I generally recommend that the break glass solution always be pair programmed.

100%. We used to design the pipeline a way that is easily reproducible locally, e.g. doesn’t rely on plugins of the CI runtime. Think build.sh shell script, normally invoked by CI runner but just as easy to run locally.

A while back I think I heard you on a podcast describing these pain points. Experienced them myself; sounded like a compelling solution. I remember Dagger docs being all about AI a year or two ago, and frankly it put me off, but that seems to have gone again. Is your focus back to CI?

> a lot of infrastructure quietly assumes GitHub is always available

Which is really baffling when talking about a service that has at least weekly hicups even when it's not a complete outage.

There's almost 20 outages listed on HN over the past two months: https://news.ycombinator.com/from?site=githubstatus.com so much for “always available”.

Part of it is probably historical momentum. GitHub started as “just git hosting,” so a lot of tooling gradually grew around it over the years — Actions, package registries, webhooks, release automation, etc. Once teams start wiring all those pieces together, replacing or decoupling them becomes surprisingly hard, even if everyone knows it’s a single point of failure.

I mean, this isn't a 'URL returned error: 500' situation for anything that Codeberg provides considering this is an issue with Copilot and Actions.

Except actually it was, that was what my git client was reporting trying to run a pull.

These days it feels like people have simply forgotten that you could also just have a bare repository on a VPS and use it over ssh.

Most developers don’t even know git and GitHub are different things…

I rarely successfully get Codeberg URLs to load. Which is sad because I actually would very much like to recommend it but I find it unreliable as a source.

That being said, GitHub is Microsoft now, known for that Microsoft 360 uptime.

> Microsoft 360 uptime

I mean... It's right in the name! It's up for 360 days a year.

Right now the page says Copilot and Actions are affected but I can't even push anything to a repo from the CLI.

Yep getting 500 errors intermittently on fetch and checkout operations in my CI pretty consistently at the moment. Like 1 in 2 attempts

Agreed. I believe that's marked under "Git Operations" and it's all green. Just began being able to push again a minute ago.

The reality is that consumers don't really care about downtime unless it's truly frequent.

Exactly.

And the frequency they can tolerate is surprisingly high given that we're talking about the 20th or so outage of 2026 for github. (See: https://news.ycombinator.com/from?site=githubstatus.com)

Top-down mandates to use AI as much as possible, and to rip up their infrastructure and move everything to Azure.

https://www.windowscentral.com/microsoft/using-ai-is-no-long...

https://thenewstack.io/github-will-prioritize-migrating-to-a...

This is very worrying if their mandate doesn't include quality control.

I figured that it would be something like that. But it's been so frequent that I expect the leadership to act decisively towards a long-term reliability plan. Unfortunately they have near monopoly in this space, so I guess there's not enough incentive to fix the situation.

Most developers don’t even know git and GitHub are different things…

Yep getting 500 errors intermittently on fetch and checkout operations in my CI pretty consistently at the moment. Like 1 in 2 attempts

> Microsoft 360 uptime

I mean... It's right in the name! It's up for 360 days a year.

Exactly.

And the frequency they can tolerate is surprisingly high given that we're talking about the 20th or so outage of 2026 for github. (See: https://news.ycombinator.com/from?site=githubstatus.com)

Agreed. I believe that's marked under "Git Operations" and it's all green. Just began being able to push again a minute ago.

This is very worrying if their mandate doesn't include quality control.

Except actually it was, that was what my git client was reporting trying to run a pull.

I'm going to trust the constant stream of updates from the company itself which shows exactly what went down and came back up rather than a random anecdote.