Skip to main content

Expired certificates

Inspect 1.36

kubectl stops working and reports an expired certificate:

		$ kubectl get nodes
Unable to connect to the server: tls: failed to verify certificate:
x509: certificate has expired or is not yet valid
	

Two different certificates produce that error. The client certificate inside your kubeconfig is one. The API server's own certificate is the other. They need different fixes, so find out which one expired before you change anything.

Which certificate expired?

Run both checks from your own machine. Neither needs a working connection to the cluster.

The host and port are the API server endpoint, which your kubeconfig already holds:

		$ kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'
https://49.13.46.99:443
	

Check the certificate the API server serves, using that host and port:

		$ openssl s_client -connect <host>:<port> -servername <host> </dev/null 2>/dev/null \
  | openssl x509 -noout -dates -subject -issuer
notBefore=Mar 24 18:28:34 2025 GMT
notAfter=Mar 24 18:33:34 2026 GMT
subject=CN=kube-apiserver
issuer=CN=kubernetes
	

The endpoint goes through the control-plane load balancer, so this shows the certificate of whichever control plane answered. Each control plane has its own, so on an HA cluster check each node's address too.

Then check the client certificate in your kubeconfig:

		$ kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' \
  | base64 -d | openssl x509 -noout -dates -subject -issuer
notBefore=Mar 12 11:50:15 2025 GMT
notAfter=Mar 12 11:55:17 2026 GMT
subject=O=system:masters, CN=kubernetes-admin
issuer=CN=kubernetes
	

Compare each notAfter with today. Whichever is in the past is the one to fix.

Your kubeconfig expired

The kubeconfig you downloaded is a copy, and its client certificate is valid for one year. When it runs out, download a fresh copy from the management cluster:

		$ kubectl get secrets mycluster-kubeconfig -o=jsonpath='{.data.value}' \
  | base64 -d \
  > mycluster-kubeconfig.yaml
	

Point KUBECONFIG at the new file and carry on:

		$ export KUBECONFIG=mycluster-kubeconfig.yaml
	

The API server certificate expired

Control-plane certificates last a year, and they refresh whenever a control-plane node is replaced. That normally happens long before expiry: clusters move to newer Cluster Stack releases through the year, and even without an update Syself Autopilot rolls the control plane on its own once its certificates are within 21 days of expiry. Reaching expiry means that rollout could not run, most often because the pool's server type is no longer offered, so no replacement node can come up. See .

Renewing them means SSH on each control-plane node, so read first. You need kubectl access to the management cluster and the namespace your cluster lives in.

Warning

Do one control-plane node at a time. On a single control-plane cluster the restart in step 5 takes the API server down for a few seconds. The cluster is already unreachable at this point, so that is acceptable, but do not restart several nodes at once on an HA cluster.

Find the control-plane machines

		$ kubectl -n <cluster-namespace> get machines \
  -l cluster.x-k8s.io/cluster-name=<cluster-name>,cluster.x-k8s.io/control-plane=
	

Note each name. You repeat the steps below once per machine.

Get the node IP

		$ kubectl -n <cluster-namespace> get machine <machine-name> \
  -o jsonpath='{.status.addresses[?(@.type=="ExternalIP")].address}'
	

Connect to the node

		$ ssh -p 100 -i <your-key> -o IdentitiesOnly=yes root@<node-ip>
	

Renew the certificates

Check what expired, renew everything, then check again:

		$ kubeadm certs check-expiration
$ kubeadm certs renew all
$ kubeadm certs check-expiration
	

Every residual time should now read about a year.

Restart the containers

The files on disk are new, but the running containers still hold the old certificates. Stopping a container is enough, because the kubelet recreates it from /etc/kubernetes/manifests/ and the new one reads the renewed files:

		$ crictl stop $(crictl ps -aq)
	

This stops every container on the node, not only the four control-plane components. The rest come back the same way.

Wait for the API server, then confirm the whole control plane came back:

		$ until crictl ps --name kube-apiserver --state Running -q | grep -q .; do sleep 2; done
$ crictl ps
	

You want kube-apiserver, etcd, kube-controller-manager and kube-scheduler all running, each created seconds ago. A recent creation time is what tells you the container restarted and picked up the new certificates.

If one is missing it is either still starting or crash-looping. Check before you move on:

		$ crictl ps -a
$ crictl logs <container-id>
	

Exited entries in that list are normal. The containers you just stopped stay listed, and so do init containers, which run once and exit by design.

A container that keeps exiting means this node's certificates are not usable yet. Fix it before you touch the next node.

Check the node from the inside

		$ KUBECONFIG=/etc/kubernetes/admin.conf kubectl get nodes
	

Every node should be listed and no TLS error should appear. On an HA cluster, control planes you have not renewed yet may show NotReady until you reach them.

Leave the SSH session, then repeat from step 2 for the next control-plane machine.

Check it from outside

Once every control-plane node is done, check the served certificate again from your workstation:

		$ openssl s_client -connect <host>:<port> -servername <host> </dev/null 2>/dev/null \
  | openssl x509 -noout -dates
	

notAfter should be about a year from now, and kubectl get nodes should work again.

Keep it from happening again

Certificates refresh when a node is replaced, and nodes are replaced on every upgrade and every rollout, so a cluster that is upgraded through the year does not reach this state. See for how rotation works and .

The health daemon also warns you first. It raises CertRenewalFailing when a node certificate is within seven days of expiry and has not renewed, so alert on it. See . The autopilot.syself.com/certs annotation on each Node carries the same certificates with their notAfter dates, so you can alert before a renewal ever fails. It covers the node's own certificates, not the API server certificate.

Related: for a control plane that is unhealthy for other reasons, and for reaching a node.