Summary

Have you already tried to set up a Kubernetes cluster with metrics-server and wondered why everybody proposes to set the insecure flag in the configuration?

It is definitely not trivial to properly secure a Kubernetes cluster. Configuring metrics-server makes this obvious, but also shows that even experienced DevOps engineers treat the topic of security with negligence.

In this article, we want to raise awareness for this important topic and explain how to secure Kubernetes metrics-server.

Understanding the issue

To scrape metrics, the metrics-server uses the http endpoint of the kubelets. When using kubeadm in the standard way, the certificate which is used for TLS is self-signed. Therefore, metrics-server cannot validate it and throws an error. To be able to run metrics-server at all, the kubelet-insecure-tls flag needs to be set to true.

Apparently, the origin of the problem is not the metrics-server, but the configuration of kubeadm and the kubelets. A kubelet has two certificates and only the second is interesting for us:

• kubelet-cert: used for authentication of the kubelet with kube-apiserver
• kubelet-serving-cert: used as TLS certificate for its HTTP endpoint

In the following, we will find out how to create and approve actual Kubelet serving certificates.

Even big companies ignore this issue

A google search on the error the metrics-server throws when being unable to validate the certificate gives only a few results - all of them suggesting setting the insecure flag.

Even DigitalOcean used the insecure flag in their official repository of Kubernetes applications until end of March 2022.

The solution

It is non-trivial to solve this issue. There are two steps. First, we need to tell Kubernetes to use actual certificates for the kubelets. Second, we need to validate and approve them.

The first step is quite easy and is explained in the Kubernetes docs. We can set the following in the cluster configuration:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
serverTLSBootstrap: true

Alternatively, we can also set the --rotate-server-certificates flag in a kubelet, as described here.

Both approaches ensure that the Kubernetes API for CertificateSigningRequests (CSRs) is used to create CSRs for the kubelets’ certificates, which then have to be validated and approved. Kubernetes cannot do this for security reasons, as it cannot verify whether a CSR of a kubelet actually belongs to the cluster.

This is why we need step 2: an external controller that validates and approves CSRs. There are few projects that have implemented such a controller. If you are interested in Cluster API, then the Hetzner provider integration might be interesting for you, as it implements a CSR controller that validates and approves CSRs. OpenShift also has a solution for this.

A more general approach is the kubelet-csr-approver, a controller that can be configured in such a way that it suits your needs. This controller can be adapted for and included in already running cluster environments.

A note on the author

This article has been written by the maintainers of Cluster API Provider Hetzner. If you are interested in using Cluster API, have a look at this post. If you have any questions on how to run secure, production-ready, and easy maintainable clusters, then write us an email or visit our website https//syself.com.