The Docker Hub 401 That Took Five Deploys to Kill
Some bugs are hard because the problem is complex. Others are hard because the problem is simple and you spend hours not believing it.
This one was the second kind.
It started the way these things always do: I glanced at our deployments dashboard and saw a wall of red. A bunch of customer builds had failed, and they all had the same error:
> [internal] load metadata for docker.io/library/python:3.12-slim
Error: failed to resolve source metadata for docker.io/library/python:3.12-slim:
failed to authorize: unexpected status: 401 Unauthorized
If you've ever shipped anything that builds Docker images, you already know what this is.
The villain: Docker Hub rate limits
Here's the deal. When you build a container, the very first thing it does is pull a base image — FROM python:3.12-slim, FROM node:22, whatever. That image lives on Docker Hub.
And Docker Hub, in its infinite generosity, lets anonymous users pull a whopping ~100 images per 6 hours, counted per IP address.
Now imagine a build farm. Lots of builds, sharing a small pool of egress IPs, all anonymously pulling python:3.12-slim over and over. You blow through 100 pulls before lunch, and every build after that gets a polite 401 Unauthorized.
The diagnosis took about five minutes. The errors were intermittent and clustered by time — the unmistakable fingerprint of a rate limit, not a code bug. Our own generated Dockerfiles were fine (we pull those from a mirror that isn't rate-limited). The failures were all from people who wrote their own Dockerfile pointing straight at Docker Hub. Nothing we could rewrite. We just had to authenticate the pulls.
Authenticated accounts get much higher limits. Easy. Create a read-only access token, hand it to the builder, done by dinner.
Reader, it was not done by dinner.
Fix #1: the one the docs told me to do
Our builds run on Depot, which uses remote builders. The standard way to authenticate is the same as everywhere else in Docker-land: drop a config.json with your credentials and the tooling picks it up.
So I did the modern, tidy thing — wrote the credentials to a temp directory and pointed DOCKER_CONFIG at it.
Deploy. Wait. Retry the build.
failed to authorize: unexpected status: 401 Unauthorized
Exactly the same. Not even a different flavor of failure. Byte for byte the same error.
Fix #2: okay, the other place
Fine. Maybe the tooling ignores DOCKER_CONFIG and only reads the default location, ~/.docker/config.json. The docs and a couple of blog posts both write there, so let me match that exactly.
Deploy. Wait. Retry.
failed to authorize: unexpected status: 401 Unauthorized
Now I was annoyed. Two different, both-supposedly-correct ways to provide credentials, and the builder was clearly using neither.
At this point the temptation is to keep guessing. Different file path. Different JSON key. Maybe the token is wrong?
I almost went down the "maybe the token is wrong" hole — which would have been hours of nonsense, because I stopped and tested the token directly against Docker Hub's auth endpoint first. It worked perfectly. The credentials were fine. The config file was fine. Something was just... not reading it.
The rule that saved me
When you've verified your assumptions twice and reality still disagrees, stop doing and start proving.
So instead of another blind fix, I shipped a deploy whose only job was to log the truth: does the build process actually have the credentials, and did it actually write the file?
The next build told me everything:
[depot-auth] Docker Hub creds configured (user=mcpuse) -> /root/.docker/config.json
> [internal] load metadata for docker.io/library/python:3.12-slim
Error: ... 401 Unauthorized
There it was, in black and white. The credentials were present. The file was written, at the exact default path the builder is supposed to read. And the builder pulled anonymously anyway.
So the bug was never "the file is in the wrong place." The bug was: the builder wasn't reading the file at all.
The actual cause
Here's the trap, and it's the kind of thing you'd never guess from the error message.
We push our finished images to a different registry, and we were handing the builder those push credentials through a pair of dedicated environment variables. Convenient, explicit, looked perfectly innocent.
Except: when those push-credential env vars are set, the builder authenticates the whole session with only those, and completely ignores ~/.docker/config.json.
So our beautifully-written Docker Hub pull credentials were sitting in a file the builder had decided not to open. Two systems for credentials, and turning one on silently turned the other off. No warning. No log line. Just a 401 from a place that had nothing to do with the real problem.
The fix, once I understood it, was almost boring: put both sets of credentials — the push registry and Docker Hub — into the same config.json, and stop using the env vars entirely. One source of truth. The builder reads the file, finds everything it needs, and authenticates both the pull and the push.
I validated the push side on a staging server first (you do not want to discover you've broken everyone's deploys after shipping to production), then rolled it out. The next build sailed past the base-image pull, built, pushed, and deployed. The wall of red went green.
What I actually learned
The Docker Hub part is useful trivia. The real lessons are the boring, portable ones:
1. A failing component is not always the broken component. The 401 came from Docker Hub, so I spent two rounds fixing Docker Hub credentials. The bug was in how a completely different feature (push auth) quietly disabled credential loading. The error points at the symptom, not the cause.
2. Config systems have hidden precedence. "Set this env var" and "write this file" looked like two independent ways to do the same thing. They weren't — one overrode the other, silently. Any time there are two ways to configure something, ask which one wins when both are present. The answer is rarely in the error message.
3. When stuck, replace guessing with logging. My first two fixes were guesses dressed up as reasoning. The thing that actually broke the loop was a deploy that did nothing but print whether my assumptions were true. One log line collapsed the search space from "infinite" to "oh."
4. Verify the cheap assumptions early. Testing the token directly took thirty seconds and saved me from a multi-hour wild goose chase into "maybe the credentials are wrong."
Five deploys. A handful of "this should definitely work" moments. And in the end, the fix was a few lines moving credentials from environment variables into a file.
The most expensive bugs aren't the complicated ones. They're the simple ones hiding behind a confident wrong assumption.
P.S. If you run builds on a shared farm and you're not authenticating your registry pulls yet — go do it now, before the wall of red. And if you ever find yourself fixing the same thing twice, that's your cue to stop fixing and start proving.