GitHub Is Experiencing Difficulties

At this point maybe they should consider sending out announcements when it is working instead

Given the recent history of outages, I wonder how many are seriously considering moving off GitHub for anything other than code hosting. I mean actions / workflows.

Dealing with an incident on prod is extra difficult when your pipelines are not running.

I knew that when the build actions started failing en masse and my newly submitted PRs did not trigger the build actions at all (besides the mandatory one in my organization), it seemed logical to me to look at HN first..

Especially troublesome in the middle of trying to fix a high score security vulnerability when the release vehicle is Github.

I noticed my CI throwing errors all of a sudden. I sure wish they would become more active in alerting folks or build it into these tools - especially as these occurrences are becoming more frequent. Possibly an API-accessible services status. Then, at least we could build in our own checks when we hit errors and not have to hunt down what all is broken.

using github in the big 26 is self ownage

I had tried downloading release assets of two different OSS that I use, and all requests failed.

It was before it became a news and a trend in X.

Really frustrating experience.

Self-hosting is king.

This seems persistent and endemic, have they said anything about why it's been so bad and what they are going to doing to improve going forward?

As I said 6 years ago. [0] You would be better off self-hosting than using GitHub or GitHub actions. No CEO of GitHub exists and now it is falling over again.

There is no better time to self-host.

[0] https://news.ycombinator.com/item?id=22868406

so... what happened?

Especially troublesome in the middle of trying to fix a high score security vulnerability when the release vehicle is Github.

I had tried downloading release assets of two different OSS that I use, and all requests failed.

It was before it became a news and a trend in X.

Really frustrating experience.

using github in the big 26 is self ownage

Comments moved thither. Thanks!

Dealing with an incident on prod is extra difficult when your pipelines are not running.

Given the recent history of outages, I wonder how many are seriously considering moving off GitHub for anything other than code hosting. I mean actions / workflows.

At this point maybe they should consider sending out announcements when it is working instead

As I said 6 years ago. [0] You would be better off self-hosting than using GitHub or GitHub actions. No CEO of GitHub exists and now it is falling over again.

There is no better time to self-host.

[0] https://news.ycombinator.com/item?id=22868406

We've been working on this at Depot[0]. We built the internals of Depot CI as a general-purpose workflow engine that understands GitHub Actions as an input (with plans to extend) - we've been working to make Actions runners more performant and more reliable for a while, but eventually you hit a ceiling if you don't also control the control plane.

[0] https://depot.dev

To offer a data point (not a tribalist argument): GitHub Actions being down this often is still less expensive than the cost of switching everything somewhere else. Would rather have my team walk away from the PRs and do other tasks than have engineering effort, meetings, design pages, scheduling, etc. for switching over.

This is annoying and I'm here because it's down. But it would have to be far worse to come close to actually being worth changing.

GHA still beats Circle CI or trying to run your own Jenkins - even with all the downtime. I'd move in a heartbeat if there were good alternatives.

Mitchell Hashimoto had a blog post a few months ago talking about him moving off of GitHub: https://mitchellh.com/writing/ghostty-leaving-github

This is a man who's spent a significant portion of every day for the last 15 years on GitHub.

Our deployments are triggered by GitHub Actions, and it is a pain to deploy without it (specifically collecting the credentials and putting them into variables the shell will read).

This is multiple times this month that this has been a problem.

Has GitHub completed it's internal migration to Azure yet? Or is it still ongoing? None of our devs want to switch away from GH, but we will have to at this point.

Forgejo selfhosted for a few months now. Its all good and cost nothing.

What would you guys recommend as an alternative? Both hosted and/or self hosted

I seriously want to. I'm just waiting for the day where I have enough extra energy to simply bite the bullet and do it.

I would conjecture that the long tail of GH users are not dependent on any of the CI stuff in the first place.

how hard could it be these days for a mid to large size eng company to have their own gitlab/hub type solution hosted in aws

I once saw a sign at a railway station in the UK that actually did say "normal service will be operating between the 12th of July and 18th of August" (or something like that). Made me laugh. At the time Anglia Railways had a rail replacement bus literally every weekend.

I thought they should rename to Anglia Busways and have bus replacement trains instead.

Self-hosting is king.

I’ve been self-hosting lore and gitea the past 6 months or so and it’s been a breeze. For anything I really care to share I throw up on Tangled.

Except anything I'd self host would be down much more often. I've hardly ever actually experienced down time issues with github when I needed it.

This seems persistent and endemic, have they said anything about why it's been so bad and what they are going to doing to improve going forward?

Can't they haven't hooked up CoPilot to their Xitter account just yet.

so... what happened?

Sorry folks, guess my GHCP query was too large...

Comments moved thither. Thanks!

[0] https://depot.dev

GHA still beats Circle CI or trying to run your own Jenkins - even with all the downtime. I'd move in a heartbeat if there were good alternatives.

I think I agree if you don't have someone dedicated to running and maintaining Jenkins. If you have someone at your org that knows what they are doing though, I'd still take Jenkins over GHA any day.

gitlab self hosted though? no outage. i remember equivalent functionality at my last job taking this path

This is annoying and I'm here because it's down. But it would have to be far worse to come close to actually being worth changing.

A case study in vendor lock-in.

We've started writing some new workflows against argo-workflows for the last like 9months or so. Especially for use cases where the actions are like self-service type automation.

I think people were so excited to move away from jenkins to something 'managed' just because of how much a dinosaur jenkins is and how much a pain in the ass it is to upgrade it... but now we are seeing how managed can bite you in the ass if the manager is incompetent.

Mitchell Hashimoto had a blog post a few months ago talking about him moving off of GitHub: https://mitchellh.com/writing/ghostty-leaving-github

This is a man who's spent a significant portion of every day for the last 15 years on GitHub.

I thought they should rename to Anglia Busways and have bus replacement trains instead.

I would conjecture that the long tail of GH users are not dependent on any of the CI stuff in the first place.

Our deployments are triggered by GitHub Actions, and it is a pain to deploy without it (specifically collecting the credentials and putting them into variables the shell will read).

This is multiple times this month that this has been a problem.

Has GitHub completed it's internal migration to Azure yet? Or is it still ongoing? None of our devs want to switch away from GH, but we will have to at this point.

Sorry folks, guess my GHCP query was too large...

Can't they haven't hooked up CoPilot to their Xitter account just yet.

I’ve been self-hosting lore and gitea the past 6 months or so and it’s been a breeze. For anything I really care to share I throw up on Tangled.

how hard could it be these days for a mid to large size eng company to have their own gitlab/hub type solution hosted in aws

What would you guys recommend as an alternative? Both hosted and/or self hosted

We have set up a new enterprise and are migrating to GHE. For us this was a no-brainer; we get our own isolated environment, Enterprise Managed Users with Entra OIDC provisioning and Github Copilot in EU data residency. https://eu.githubstatus.com/posts/dashboard looks pretty good to me. We will still keep our github.com enterprise around for public repositories. I’m genuinely curious to know why a company would prefer github.com over GHE?

We self-host gitlab and it’s great. It’s not cheap, but it’s also never gone down.

Forgejo seems decent but I've only scratched the surface.

Here's what I did.

Step one - migrate my build workflows to Docker.

My Github actions are now basically: "checkout / set env vars from secrets / docker-compose builder run make".

I used large machine runners to run full the Docker (Podman actually) on Github first to avoid dealing with docker-in-docker complications. This step also provided some very nice robustness advantages, as I can now trigger deployments from my laptop if needed.

Step two:

Migrate to self-hosted runners. I used my former homelab server to set up a build machine. It has 16Tb of fast NVMe SSDs and thanks to Podman container layer caching, my entire lint workflow now takes 30 seconds. Faster than just one "npm install" on Github before.

And Github's self-hosted runners are actually surprisingly easy to set up and use. They are also somewhat more robust.

Step three:

Swap Github for something else.

WoodpeckerCI for DIY, Drone.io for paid support. Self-hosted, OAuth login, repo-permission-based authorization, container-native design, easy web UI, optional RDBMS, supports many platforms. All the core features needed for scalable CI, deploy as few/many as you want for multiple teams, small enough to easily run on one box for one repo. Nothing else is as simple, easy, powerful, compatible.

Forgejo selfhosted for a few months now. Its all good and cost nothing.

Codeberg ironed out a lot of issues in Forgejo

Hey, there's still plenty if you don't mind the leading 8

Have they aimed for 86 uptime?

I seriously want to. I'm just waiting for the day where I have enough extra energy to simply bite the bullet and do it.

Never mind. It's happening now! Migrating to self hosted Gitea for work. Super excited!

I think I agree if you don't have someone dedicated to running and maintaining Jenkins. If you have someone at your org that knows what they are doing though, I'd still take Jenkins over GHA any day.

We've started writing some new workflows against argo-workflows for the last like 9months or so. Especially for use cases where the actions are like self-service type automation.

Except anything I'd self host would be down much more often. I've hardly ever actually experienced down time issues with github when I needed it.

Then you would be exceptionally bad at selfhosting as Github has like one nine of uptime.

I've never understood this argument. Even if self-hosted things were offline more often (which I've never found to be the case, I've had Forgejo + runners running for a year now with no downtime), the real benefit is that you yourself can work to bring it back online when it does, rather than waiting on a large, slow-moving organization to figure out what slopped PR caused their global service serving ungodly amounts of RPS to go offline again.

gitlab self hosted though? no outage. i remember equivalent functionality at my last job taking this path

That's significantly more difficult to set up and it cost money. Github's big issues that it's free, Microsoft doesn't want to allocate enough budget to keep the thing running properly.

But for people who either don't pay anything at all or phenomenal amount one 9 of up time is all you need.

If you're actually trying to run a business I guess you can call and gitlab and get an Enterprise contract

A case study in vendor lock-in.

Pretty much, eh?

If I could right now:

1. go sign-up elsewhere 2. Log into GitHub and point Actions to that new host 3. All my actions files immediately worked without question

I'd probably give it a spin and make a wiki page explaining how to swap back and forth. No meetings. No design issues. No scheduling. Just a flip switch on who to pay for computers.

For what it's worth, we switched some of our actions to a self-hosted Woodpecker instance, and although there were a few kinks to iron out, it works better overall (for example, because of better caching on that single instance, our docker images build faster).

We self-host gitlab and it’s great. It’s not cheap, but it’s also never gone down.

Here's what I did.

Step one - migrate my build workflows to Docker.

My Github actions are now basically: "checkout / set env vars from secrets / docker-compose builder run make".

Step two:

And Github's self-hosted runners are actually surprisingly easy to set up and use. They are also somewhat more robust.

Step three:

Swap Github for something else.

Forgejo seems decent but I've only scratched the surface.

Codeberg ironed out a lot of issues in Forgejo

Hey, there's still plenty if you don't mind the leading 8

Never mind. It's happening now! Migrating to self hosted Gitea for work. Super excited!

Have they aimed for 86 uptime?

67 uptime should be within reach for them.

Pretty much, eh?

If I could right now:

1. go sign-up elsewhere 2. Log into GitHub and point Actions to that new host 3. All my actions files immediately worked without question

I'd probably give it a spin and make a wiki page explaining how to swap back and forth. No meetings. No design issues. No scheduling. Just a flip switch on who to pay for computers.

That's significantly more difficult to set up and it cost money. Github's big issues that it's free, Microsoft doesn't want to allocate enough budget to keep the thing running properly.

But for people who either don't pay anything at all or phenomenal amount one 9 of up time is all you need.

If you're actually trying to run a business I guess you can call and gitlab and get an Enterprise contract

Then you would be exceptionally bad at selfhosting as Github has like one nine of uptime.

It still has 2 9s according to the status page, but it is approaching Claude levels of downtime. :(

I wrote my reply before yours appeared but it's basically agreeing. A simple Forgejo hasn't given me any grief. It really seems simpler.

Maybe they're hosting in us-east-1 though :)

67 uptime should be within reach for them.

It still has 2 9s according to the status page, but it is approaching Claude levels of downtime. :(

I wrote my reply before yours appeared but it's basically agreeing. A simple Forgejo hasn't given me any grief. It really seems simpler.

Maybe they're hosting in us-east-1 though :)

Update - We are continuing to work on an issue affecting GitHub Actions. Webhook triggers remain throttled to aid recovery, so many push and pull request events are not triggering new workflow runs.

We identified runners being assigned jobs that are no longer valid and are deploying a change to address this issue. Both GitHub-hosted and self-hosted runners are affected.

Copilot code review, Copilot coding agent, and GitHub Pages may experience failures or delays. Migrations using GitHub Enterprise Importer have been paused to support mitigation efforts.

Aug 06, 2026 - 21:30 UTC

Update - We are continuing to work on an issue affecting GitHub Actions. Webhook triggers are currently throttled to help with recovery and and we are processing approximately 15% of webhooks, so many events such as pushes and pull requests are not triggering workflow runs. Of jobs queued, approximately 65% are succeeding, improved from a low of 30 to 40% earlier in this incident.

We have narrowed the remaining impact to runners that are stuck retrying jobs that are no longer available. Both GitHub-hosted and self-hosted runners are affected, and we are working to recover them.

Copilot code review, Copilot coding agent, and migrations using GitHub Enterprise Importer may also be affected.

Aug 06, 2026 - 20:34 UTC

Update - We are continuing to work on an issue affecting GitHub Actions.

Capacity remains constrained and jobs may still be delayed or fail while it recovers gradually. Customers using self-hosted runners may see errors or rate limiting when runners register.

Copilot code review, Copilot coding agent, and migrations using GitHub Enterprise Importer may also be affected. Webhook deliveries may be delayed.

Our engineers remain actively engaged.

Aug 06, 2026 - 19:43 UTC

Update - We are continuing to work on an issue affecting multiple GitHub services.

Workflow runs are still failing, and jobs may remain queued for an extended period before starting or may time out. Jobs using GitHub-hosted runners are particularly affected while capacity is constrained.

Customers using self-hosted runners may see errors or rate limiting when runners register.

Copilot code review, Copilot coding agent, and migrations using GitHub Enterprise Importer may also be affected. Webhook deliveries may be delayed.

Recovery is taking longer than we expected, and engineers remain actively engaged.

Aug 06, 2026 - 18:46 UTC

Update - We are continuing to work on an issue affecting multiple GitHub services.

Workflow runs are still failing or delayed in starting, and some queued jobs may time out.

Customers using self-hosted runners may see errors or rate limiting when runners register.

Copilot code review, Copilot coding agent, hosted runners, and migrations using GitHub Enterprise Importer may also be affected.

Webhook deliveries may be delayed.

Engineers have applied further mitigations and are continuing to work towards full recovery.

Aug 06, 2026 - 18:11 UTC

Update - We are continuing to work on an issue affecting multiple GitHub services.

Workflow runs are failing or delayed in starting, and some queued jobs may time out.

Copilot code review, Copilot coding agent, hosted runners, and migrations using GitHub Enterprise Importer might also affected.

Webhook deliveries may be delayed.

Engineers have applied a number of mitigations and are rolling out a further fix across all affected systems now.

Aug 06, 2026 - 17:40 UTC

Update - We are continuing to work on the issue affecting GitHub Actions.

Workflow runs are still failing or delayed in starting, and some queued jobs may time out.

Some requests to the Actions API are returning errors. Customers running migrations with GitHub Enterprise Importer may see failures.

Our engineers have applied several mitigations and are rolling out a further fix now.

Aug 06, 2026 - 17:02 UTC

Update - Actions and Pages are experiencing degraded availability. We are continuing to investigate.
Aug 06, 2026 - 16:33 UTC

Update - We are continuing to work on the issue affecting GitHub Actions.

Some workflow runs are still delayed or failing to complete, and some requests to the Actions API are returning errors.

Customers running migrations with GitHub Enterprise Importer may also see failures.

Engineers are actively working towards full recovery.

Aug 06, 2026 - 16:27 UTC

Update - Pages is experiencing degraded performance. We are continuing to investigate.
Aug 06, 2026 - 16:27 UTC

Update - Pages is operating normally.
Aug 06, 2026 - 16:19 UTC

Update - Pages is experiencing degraded performance. We are continuing to investigate.
Aug 06, 2026 - 15:53 UTC

Update - We are investigating errors affecting GitHub Actions. Some workflow runs are failing to start or failing partway through, and some requests to the Actions REST API are returning errors.

Some customers may also see unexpected rate limiting in their workflows.

Engineers have identified the source of the disruption and are actively working on a mitigation

Aug 06, 2026 - 15:45 UTC

Update - Actions is experiencing degraded availability. We are continuing to investigate.
Aug 06, 2026 - 15:41 UTC

Investigating - We are investigating reports of degraded performance for Actions
Aug 06, 2026 - 15:22 UTC

Hacker Times

Hacker Times

GitHub Is Experiencing Difficulties

Discussion

Discussion