Open Source Summit North America 2023

A few months back, we discovered some applications had lower performance on a few Kubernetes clusters on a specific provider. After some investigation, we discovered network throughput was lower than expected because we were using a single transmit queue on the external interface of the instance. Probably just a configuration issue, either at the Operating System level or in our CNI plugin, right? This is when things started to get really interesting: we could not find any significant differences between the clusters that were affected and those that were not. At that point, we had no other choice than to dive deeper, trace packet paths within Linux itself and read kernel code. In this talk, we will focus on one of the most complex network performance issues we have faced in our Kubernetes environment. We will go through the debugging steps in detail and explain how we tracked the issue using bpftrace and ultimately uncovered a small kernel bug. Finally, we will discuss the upstream kernel fix and the ebpf-based mitigation we were able to deploy quickly with the help of the Cilium team.