After passing 500 nodes in our cluster, our researchers started reporting regular timeouts from the kubectl⁠(opens in a new window) command line tool. We tried adding more Kube masters (VMs running kube-apiserver⁠(opens in a new window)). This seemed to solve the problem temporarily, but once we passed 10 replicas we knew we were treating symptoms and not the cause (by comparison, GKE⁠(opens in a new window) uses a single 32-core VM for 500 nodes).

This made us strongly suspect our etcd⁠(opens in a new window) cluster, which is the central store of state for the Kube masters. Looking in Datadog⁠(opens in a new window), we saw write latency spiking to hundreds of milliseconds on the DS15v2 machines running our etcd replicas, despite each machine using a P30 SSD capable of 5,000 IOPS.

Benchmarking performance with fio⁠(opens in a new window), we saw etcd was only able to use about 10% of the available IOPS because the write latency was 2ms and etcd does sequential I/O, making it latency-bound.

We then moved the etcd directory for each node to the local temp disk, which is an SSD connected directly to the instance rather than a network-attached one. Switching to the local disk brought write latency to 200us, and etcd became healthy!