nginx in front of podman/docker containers: '[emerg] host not found in upstream "backend:3001"' at start, or 502 to the OLD IP after recreating a backend: nginx resolves upstreams once
Give nginx a resolver and mark upstream servers with resolve, so container names are looked up at runtime, not just at startup. Otherwise restart or reload nginx every time you recreate a backend container. Open-source nginx supports resolve in upstream blocks since 1.27.3:
resolver 10.89.0.1 valid=10s; # podman: the network's gateway IP (aardvark-dns); docker: 127.0.0.11
upstream python_backend {
zone python_backend 64k;
server realtime-python1:3001 resolve;
server realtime-python2:3001 resolve;
}Why: hostnames in upstream/proxy_pass are resolved when nginx loads its config. If a backend container is stopped (or the name is wrong, e.g. the compose service name node1 when the container is really called realtime-node1), nginx exits at startup with [emerg] 1#1: host not found in upstream "node1:3000" in /etc/nginx/nginx.conf:10. If you rebuild and recreate a backend, it usually gets a new IP (we saw 10.89.3.9 change to 10.89.3.19). nginx keeps the old one and logs connect() failed (111: Connection refused) while connecting to upstream ... upstream: "http://10.89.3.9:3001/health", so you get 502s that look like a backend bug.
Checked with nginx:alpine (nginx 1.31.8): nginx -t passes with server not-running-yet:3001 resolve; and fails with host not found in upstream without resolve. To avoid hard-coding the DNS IP, the official image can fill it in for you. Set NGINX_ENTRYPOINT_LOCAL_RESOLVERS=1, write resolver ${NGINX_LOCAL_RESOLVERS} valid=10s; in a /etc/nginx/templates/*.conf.template, and the entrypoint copies the nameservers from /etc/resolv.conf into it.