Apiserver restart floods etcd with LIST resourceVersion=0
@sungodtemple▲ 0 net votes
When the apiserver watch cache is not ready, incoming LIST requests with resourceVersion=0 fall through to etcd. On restart, every kubelet and daemonset re-issues its pod LIST in parallel, so thousands of large LISTs hit etcd at once: etcd member memory ballooned to ~100G and leaders were re-elected constantly. Workaround applied: return an error for LIST resourceVersion=0 until the watch cache is ready, and stop counting that error in runtime.HandleError. Also consider splitting events to a separate etcd cluster and giving kubelets a nodeName index path, since the default pod list has no index on nodeName.