That one would be like iscaliforniaonfire.com

Actions aren't secondary to most paid GH users; and if they are down it usually means no deployments and no tests, which can often halt work.

Relevant xkcd: https://xkcd.com/2236/

> they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI.

That is an active choice they are making, over and instead of any reliability for the rest of their users.

It is a problem they are embracing, and actively encouraging, for themselves.

GitHub Actions is really, really badly designed. The security model is fundamentally broken, the YAML hell is as bad as any, the log streaming lags like hell, they charge self-hosted runners for using their coordination plane, jobs queuing is really slow, and their software for actually running jobs is cursed and designed in a way that is practically hostile to self-hosting.

GitHub Actions is god-awful. Have you ever used any other CI tools?

I don't typically have sympathy for businesses that fail to deliver a service as advertised.

I can have sympathy for the humans caught in the crossfire but only managing one nine of availability on a commercial service is not acceptable.

I don't know enough about Github or Microsoft to say anything intelligent, but I agree with your sentiment in general. And, one thing is for sure - I have used Github for years, pushing many thousands of commits on personal, private projects. I've burned through as many free action minutes as I can, and I use my own action runners as well. I've done this all for free, so on balance I am grateful for them.

6 years is a good run. For comparison, every 6 years there's a not insignificant chance you die. A restaurant that lasted 6 years would be an outstanding success. 6 years ago the very first LLMs were a novelty toy. 6 years before that was just after HTTPS became popular. 6 years before that was about when the iPhone came out.

I have a lot of sympathy for the people working at GitHub trying to keep it going, but I do not have sympathy for GitHub. That's what they're paid for. I have things to get done, and it has no sympathy for me.

I think somewhere down the line you have an answer to problem itself. Its for a while that companies are not into the business of sympathies, we should probably not get into that side for the topic.

1. Github has enterprise users who paid for the service, their day job requires Github to be available and working

2. Github has generous free tier which is the one which is exploring a lot more with the AI generated code.

It is a complexity in itself but the traffic should have been separated, the free users should not be allowed to bring down Github for enterprise customers (Just to clarify, I am free user myself). And if they do not have capacity it would have been perfectly fine to push back or throttle new users/repositories.

> have a little more sympathy for the billion dollar company

No.

Their leadership went all in on AI and in the last blog post essentially admitted some missing test coverage for a critical path. Time to learn lessons and fix your vibe coded shitslop and stop using "user graph go up" as some kind of excuse.

Hey! isgithubcooked.com is my site; I made this site in February '26, I think

The contribution graph as an outage calendar idea is a commonly recurring one :). I definitely saw it somewhere else as a static asset first before I made this site.

Not sure whether it has been updated since your comment, but the sentence now reads:

    GitHub has had 1125 incidents since March 2016. Over the last 3 months, they've averaged 24 incidents per month

edit: although they also have 1.2 days of downtime (in a day) for their 'worst days' of downtime table, which suggests some auto number crunching is not working as expected.

> I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers.

I don't think this is commendable at all. I give GitHub a lot of money and I'm tired of it being wasted with downtime.

We have a business relationship so no we shouldn’t have any sympathy. They sell a service and they’re failing to provide it.

They did that to themselves, and doubly so.

They decided they needed to capture the whole open source ecosystem by turning open source work into social networking... on a proprietary platform (because open source is great, especially when it's others' software). That was before they joined Microsoft.

And then Microsoft pushed AI everywhere, including on GitHub itself with copilot.

I would have liked if they had left the open source projects alone and didn't create that FOMO for not using them.

I have no sympathy.

Sympathy? It's a Microsoft company that is being ran with a consistency of a startup in early seed rounds. Their downtime is abhorrent and unacceptable as far as enterprise goes. Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime.

As a daily GitHub user in Australia I still haven’t figured out why everyone’s complaining about uptime. :)

what's normal working hours? very US centric comment IMO. Europe, India, China, Latam etc. don't fall into your 9-5 EST normal working hour bucket.

> GitHub has had 1125 incidents since March 2016. Over the last 3 months, they've averaged 24 incidents per month (↓ 5% vs prev 3mo).

Looks like they fixed it already

Ahhh, I think the author mixed up two values here. That value seems to actually be the average over the past 3 months.

    const incidents = e.detail.incidents;
    
    // ...snip...
    
    const now = new Date();
    const threeMonthsAgo = new Date(now);
    threeMonthsAgo.setMonth(threeMonthsAgo.getMonth() - 3);
    
    // ...snip...
    
    var currentFreq = recentIncidents.length / 3; // <- We out here, smoking these guns with our homeboy Claude
    
    // ...snip...
    
    var earliest = null;
    for (var j = 0; j < incidents.length; j++) {
      var dd = new Date(incidents[j].started_at); // <- eventually incidents[j].started_at is "2016-03-01T07:07:37.000Z"
      if (!earliest || dd < earliest) earliest = dd;
    }
    
    // ...snip...
    
    document.getElementById('n-since').textContent = earliest
      ? earliest.toLocaleDateString('en-US', { month: 'long', year: 'numeric' })
      : '?';
    document.getElementById('n-rate').textContent = Math.round(currentFreq * 10) / 10;

#n-since is going to be either march or feburary. It'll change depending on your timezone because JS's Date object always shifts the date around to match the same instant but in the system's timezone.

#n-rate has nothing to do with the #n-since month, it's just the last trailing 3 months. And even then, it's sort of underbaked? It's moving the date back by 3 calendar months not taking into account differing numbers of days, so it'll under-report short months.

I wouldn't trust the stats here.

Edit: whoops, author updated the template while I was writing this! It now says "Over the last 3 months", though that's still calendar months.

> Their incident-free uptime on 9-5 EST, Mon-Fri, is probably like 60%.

And it may be even worse in EDT, which is currently in effect!

Prob used Copilot to write the excel formula...

they could rate limit pushes to free projects.

I agree that they should reserve capacity for enterprises and paying users.

But I think they’re in a tough spot. GitHub has historically been a huge supporter, proponent, and provider for open source projects. Engineers are difficult customers, to say the least, and the community would likely freak tf out of the segmented traffic.

The logical, pragmatic, and justifiable answer doesn’t always align with your market.

It's a shame that the GitHub org that we use at my job that we pay a lot of money for gets affected the same way my personal nonsense does.

I don't know the architecture or any of that, but I feel like there could be (and it's not like they would've really known this until the last year or two with the massive spike) separate infrastructure for paid users/orgs vs free the same way they make the distinction with enterprise.

I get the massive load changes that they are under over the last two years, but why does a bunch of vibe coded slop take down the same resources that my company pays for every single month and has for years? I imagine properly splitting that out would be an absolute headache and not worthwhile for them vs stabilizing the rest of the service, but damn it sucks when I get blocked at work because GH is down.

If you haven't read this, you really have no idea what they are dealing with. https://cursor.com/blog/git-at-any-scale

Their engineers almost certainly are not the ones making the decision to let all the new traffic degrade their service for their existing paid customers.

Yes I tweaked it! The number and copy were mismatched and are no longer!

That worst day is likely an overlapping incidents accounting issue; I tried to account for overlapping incidents in another view but probably failed to port it over there.

Should be fixed soon!

Typically when a company gets acquired, they get thrown a bunch of terrible initiatives like “change your cloud provider” that distract from their mission, executives saying “more AI!” and they definitely don’t hear anyone in management say “prioritize technical debt”.

> And then Microsoft pushed AI everywhere, including on GitHub itself with copilot.

Ding ding ding, we have a winner. I like AI. I work for an AI company. Still, Microsoft aggressively pushed GitHub users toward Copilot. They don't get to do that and complain about increased volume from AI-generated changes.

No Copilot + reasonable operation: the way things were

Copilot + reasonable operation: Well done!

No Copilot + being overwhelmed by AI commits: Sympathy.

Copilot + being overwhelmed by AI commits: "Where did that petard come from that's hoisting us?"

Im fairly certain that SWEs only exist on the US west coast

Sympathy negated by their 2026 Pwnie award: "Lamest Vendor"

https://this.weekinsecurity.com/microsoft-wins-lamest-vendor...

Why? If we have sympathy for Apple then we can have sympathy for Microsoft ...

Do we have a breakdown on how much traffic is free users vs paid users?

> Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime.

I gather that you have intimate and deep knowledge on the teams and the problems they try to solve there.

This is uncharitable, rude, and pretty baseless. Unless you have a lot of direct, personal information about GitHub engineers, they’re dealing with a huge spike in traffic.

Is their uptime acceptable? No. But personal attacks aren’t necessary or constructive.

That seems a bit crass but the underlying sentiment stands. Microsoft has more money than God. 4T valuation. When you say you’re worth that much, no excuses. Figure it out.

and it has been so damn slow for a number of years now, it drives me insane.

Not sure if I would agree to that completely. Free tier is always ripe for abuse so in the beginning for a long time, Github has this requirement that you can only have a public repository if you are a non-paying user. Somewhere down the road when machine learning picked up (everyone was on the band wagon of hoarding the data, datalakes, hadoop, flink were at the top) they relaxed that condition allowing one to have a private repository on the free tier.

They are in a tough spot cause they want to continuously ingest all the data to train their LLMs. Any fork or decentralization at large scale of git is going to impact the training pipeline.

A service like GH will still show a daily usage pattern, often with a peak somewhere around 16:00 UTC when most of the US and Europe are at work.

The phrasing could have been better, but we've talked as a team about cancelling a $100k/yr contract nearly half a dozen times the last year, and the only thing keeping us from doing so is various "compliance" issues, and to a lesser degree the friction from physically moving. That's a tenuous moat, and if I were a GH PM I'd probably want to know that the current instability is somewhere near critical mass.

not constructive, but also not personal. rude but based on observed facts.

One, logically, your statement doesn't follow. They're two different companies that have primarily in different market segment. Two, it's not true in the first place - did we side with apple and its cut of the app store revenue for instance?

You're the only mention of Apple in this thread, and I don't see what they have to do with it.

I’d be surprised if paid traffic was more than 1%.

Our GitlabCE instance has been sitting in the racks for nearly two years with almost 100% uptime running on 10 year old xeons that have long since paid for themselves.

Of course it is not free of all management but for our use case it is working.

There are hiccups with the CI runners from time time but nothing major and we have another machine in another rack that serves as a backup which can be brought up ~< 20 minutes.

I know companies have long since tossed their expertise for hosting their own stuff in favour of SaaS but at some point its hard to beat the up time of a single machine.

So your proposal is for Microsoft to invest their free cash in helping support a bunch of AI coders maintain their pet projects? Sounds like you should be a CEO!

How low would uptime have to get before you'd consider holding the engineers personally responsible for it?

You can pay for GitHub Enterprise Cloud which is essentially their private GitHub deployment stack that they'll host for you. It's theoretically isolated, supposed to be completely separate from these outages.

It's a lot more expensive and has a good bit more limitations to the regular SaaS product.

Wait, I don't sympathize Apple at all... Or any other American corporation.

Please, read between the lines. Github actions has been a mess since 2019 at least. None of the instability is new, before this unprecedented growth (for a service that's supposed to scale horizontally) the excuse du jour was the azure migration, before that it was the high rate of shipping post acquisition.

Core parts of the product, like navigating to individual files in a code review, are broken

If I own a restaurant and buy bread from a supplier — BreadHub.

And 99% of the bread that I get is good but 1% of the loaves, they forgot to add flour. Consistently, for years, they always have loaves missing a key ingredient that I still end up paying for.

I can be pretty sure that BreadHub have a pretty major internal issue, and should probably be questioning their competence, regardless of their “scale”, and without any knowledge of the “problems they’re solving”

For CI there are a number of drop-in commercially-available options. You can make it a staged migration, first trivially migrating the build runners, followed by the more complicated integration test and deployment runners. The CI harness itself can follow, and finally moving to a code hosting and visualization service is last.

The final step is challenging; likely the most difficult part is changing all the code references and imports. Shadowing changes would be straightforward. Training your likely 25-50 engineers to use the new code review UX would likely not take that long.

Considering the wasted engineering velocity during Github outages, it's worthwhile to do even a partial migration. Github's action runners have in my experience, been the most fragile part of the platform. Given the ease of moving build and merge queue runners to alternates, it's a no-brainer.

We have the expertise. We don’t have the interest in fighting our internal “security” teams to keep running VMs.

You are certainly free to attempt to migrate your workloads to your own or leased data center space in your area then buy your own on prem cloud like Dell Private Cloud and deal with all of the maintenance yourself. I don’t think it would amount to $100k/yr and you would be taking on all the risk.

Why are we holding the engineers accountable and not the CEO? The company isn’t one department. Responsibility should bubble up to the top.

Start charging more, cut down on CI, figure it out. You don't NEED to give away so much if it means your service is going down.

> Core parts of the product, like navigating to individual files in a code review, are broken

I think this is a good argument to underline "It's not _just_ the scale". Adding to this, the Github Code Review experience is kind-of broken, the way comments/threads are stacked in the PR overview has not improved, pagination isn't really a thing, and these issues are age old. Hopefully, one day, Github will mature.

I’m not saying users don’t have a right to be pissed or aren’t justified in looking at other options.

I’m saying that GH is operating at a huge scale with (probably) lots of technical debt and a forced migration to new infrastructure.

I would be (and am) highly critical of leadership. I’m not going to make strong assertions about ICs without knowing their context. I’ve worked at a company with a sterling reputation for engineering excellence where brilliant ICs were kneecapped by poor leadership.

I think a lot of us, at one point or another in our careers, have worked with potato leadership that can be short-sighted or political. It isn’t a comment on the engineers.

I don't think this is an apt analogy. The bread makeup is still there, as far as I'm aware, no users have lost any data or are missing "key" ingredients.

An outage is more like a shipping issue with the supplier, if it's owned wholly by them.

Please don't do it. Now somebody will come and will try to improve your bread analogy. We will be discussing bread for eons

Agreed. At some point to have to stop hugging your ops team and start firing them.

I see you haven't experienced USFoods and Sysco.

There isn’t mission critical work being done on GH. You start charging more and the people in charge of finance at companies paying for GH will actually start paying attention. Those finance people don’t care rn because GH is affordable and helps get stuff done. There’s a threshold that exist at every company and once it’s crossed the people who can make money decisions start asking questions

Self-hosted is not the only alternative...

Okay, let's use a car analogy...

They dont have a CEO ever since their last CEO said that human programming "wasn't going anywhere".

Agreed. In fact, keep firing ops until someone gets it done. There's no way your company will build a reputation for not supporting their teams.

No, you hire more of them, you give them what they need to do their jobs, and you actually listen to their recommendations. It's not even complicated. It just takes time and money, neither of which companies want to spend on reliability until their customers scream very loud en masse.