I’ll admit I haven’t checked before posting, perhaps an admin can merge both submissions and change the URL on the one you linked to the one in this submission.

> Is there a mirror registry with all the same basic images?

https://gallery.ecr.aws/

I guess the best way would be to have a self-hosted pull-through registry with a cache. This way you'd have all required images ready even when dockerhub is offline.

Unfortunately that does not help in an outage because you cannot fill the cache now.

This is the way tho this can lead to fun moments like I was just setting up a new cluster and couldn't figure out why I was having problems pulling images when the other clusters were pulling just fine.

Took me a while to think of checking the docker hub status page.

In the case where you still have an image locally, trying to build will fail with an error complaining about not being able to load metadata for the image because a HEAD request failed. So, the real question is, why isn't there a way to disable the HEAD request for loading metadata for images? Perhaps there's a way and I don't know it.

You might still have it on your dev box or build box

  docker image ls
  docker tag name/name:version your.registry/here/name/name:version
  docker push your.registry/here/name/name:version

Yeah, this is the actual error that I'm running into. Metadata pages are returning 401 and bailing out of the build.

Per sibling comment, public.ecr.aws/docker/library/.... works even better

I’m able to pull by the digest, even images that are now missing a tag.

In our ci setting up the docker buildx driver to use the artifact registry pull through cache involves (apparently) an auth transaction to dockerhub which fails out

I think it was likely caused by the cache trying to compare the tag with Docker Hub: https://docs.docker.com/docker-hub/image-library/mirror/#wha...

> "When a pull is attempted with a tag, the Registry checks the remote to ensure if it has the latest version of the requested content. Otherwise, it fetches and caches the latest content."

So if the authentication service is down, it might also affect the caching service.

It's a bit stupid that I can't restart (on Coolify) my container, because pulling the image fails, even though I am already running it, so I do have the image, I just need to restart the Node.js process...

Quay.io is nice (but you have to memorize the spelling of its name)

same

I had some images in cache, but not all of them, and pull is failing

for example, i have redis:7.2-alpine in cache, but not golang:1.24.5-alpine

I needed the golang image to start my dev-backend

so i replaced FROM golang:1.24.5-alpine with FROM redis:7.2-alpine, and manually installed golang with apk in the redis container :)

Not Google Artifact Registry... Our Docker Hub pull-through mirror went down with the Docker Hub outage. Images were still there but all image tags were gone

CNCF has harbor [0], which I use at home and have deployed in a few clusters at work, and it works well as a pull through cache. In /etc/containers/registries.conf it's just another line below any registry you want mirrored.

    [[registry]]
    location = "docker.io"
    [[registry.mirror]]
    location = "core.yourharbor.example.com/hub"

Where hub is the name of the proxy you configured for, in this case, docker.io. It's not quite what you're asking for but it can definitely be transparent to users. I think the bonus is that if you look at a podspec it's obvious where the image originates and you can pull it yourself on your machine, versus if you've mutated the podspec, you have to rely on convention.

[0] https://goharbor.io/

Not Debian itself, but Red Hat's registry has them: https://quay.io/organization/lib

Someone mentioned Artifactory; but its honestly not needed. I would very highly recommend an architecture where you build everything into a docker image and push it to an internal container registry (like ecr; all public clouds have one) for all production deployments. This way, outages only affect your build/deploy pipeline.

The images I use the most, we pull and push to our own internal registry, so we have full control.

There are still some we pull from Docker Hub, especially in the build process of our own images.

To work around that, on AWS, you can prefix the image with public.ecr.aws/docker/library/ for example public.ecr.aws/docker/library/python:3.12 and it will pull from AWS's mirror of Docker Hub.

Another reply had some good insight: https://news.ycombinator.com/item?id=45368092

You pull the images you want to use, preferably with some automated process, then push them to your own repo. And anyways use your own repo when pulling for dev/production. It saves you from images disappearing as well.

GHCR authentication is just broken. They still require the deprecated personal access tokens.

Even cloud vendors can’t get distributed systems design right.

Nevermind, I used the terminal, docker ps to find the container and docker restart <container_id>, without going through Coolify.

Or start a pronunciation revolution and say "kway". It's all made up anyway ;-)

This saved me. I was able to push image from one of my nodes. Thank you.

What do you like using for your own repo? Artifactory? Something else?

I was publishing public containers on Docker Hub, and I'm publishing public containers on GHCR.

It _is_ pronounced "kway", and it _is_ a real word: https://www.merriam-webster.com/dictionary/quay !

Note, artifactory SaaS had downtime today as well.

I have experience with ECR. If you’re in the AWS ecosystem it does the job.

I know quay is a real word - it's not normally pronounced like "kway" but like "key". But only because enough people agree on that - that's what I mean by made up. The rules are just a majority agreement for both meaning and pronunciation.

It's pronounced keɪ (from your link - The spelling quay, first appearing in the sixteenth century, follows modern French. As noted by the Oxford English Dictionary, third edition, the expected outcome of Middle English keye would be /keɪ/ in Modern English). Or key (with modern spelling).

My french speaking partner recently informed me the quay (pronounced key) meant something like ‘dock’ when we were discussing the Florida Keys, and suddenly everything fell into place!