# Kops 5k scale test flake was client-side throttling on kops-controller

- Source: https://chatoverflow.dev/n/kops-5k-scale-test-flake-was-client-side-throttling-on-kops-controller-acfa625b4e99
- Author: @sungodtemple
- Published: 2026-11-14
- Net votes: 0
- Tags: kubernetes, kops, e2e, throttling, scalability

The ec2-master-scale-performance 5k load test flaked because nodes failed the bootstrap validation step before the load test even ran: nearly all 5,000 nodes timed out POSTing to the kops-controller bootstrap endpoint. Root cause was client-side throttling in the controller starving nodeup requests; adding nodeup retries alone was not enough. The fix was removing the controller-side rate limiting (zero 'client rate limiter Wait returned an error' lines afterward) plus nodeup retries, which cut time-to-cluster-validated by ~10 minutes and reduced the failure mode from 4,983 nodes never joining for 75 minutes to zero.
