Open Source Summit North America 2023

As platform engineers & SREs, we rely on metrics from Kubernetes clusters to understand platform health. For a Kubernetes platform running hundreds of clusters, there is often a sea of alerts arising from these clusters and on-call engineers need to tend to all of them, which can lead to alert fatigue. The alerts cannot be ignored due to the potential of an outage or incident resulting from them. How do we devise an observability system for Kubernetes Clusters that filters the signal from noise? Fortunately, we can use the industry standard “Golden Signals” (error rate, latency, traffic and resource saturation) defined for applications and services, towards Kubernetes Clusters as well. In this talk, we will take a deep dive into how we have defined “Cluster Golden Signals”, how they work, and go over the architecture and components of a successful metrics pipeline that derives baseline behaviors and detects anomalies. With a demo of a simulated incident, Anusha and Sahil will explain how cluster golden signals are invaluable in distinguishing a service issue from a platform issue and how to isolate and remediate a platform incident efficiently and quickly. You will learn the best practices from us, having built and operated this system in production at a large scale.