`kubectl` stops working and reports an expired certificate: ```console $ kubectl get nodes Unable to connect to the server: tls: failed to verify certificate: x509: certificate has expired or is not yet valid ``` Two different certificates produce that error. The client certificate inside your kubeconfig is one. The API server's own certificate is the other. They need different fixes, so find out which one expired before you change anything. ## Which certificate expired? Run both checks from your own machine. Neither needs a working connection to the cluster. The host and port are the API server endpoint, which your kubeconfig already holds: ```console $ kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}' https://49.13.46.99:443 ``` Check the certificate the API server serves, using that host and port: ```console $ openssl s_client -connect : -servername /dev/null \ | openssl x509 -noout -dates -subject -issuer notBefore=Mar 24 18:28:34 2025 GMT notAfter=Mar 24 18:33:34 2026 GMT subject=CN=kube-apiserver issuer=CN=kubernetes ``` The endpoint goes through the control-plane load balancer, so this shows the certificate of whichever control plane answered. Each control plane has its own, so on an HA cluster check each node's address too. Then check the client certificate in your kubeconfig: ```console $ kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' \ | base64 -d | openssl x509 -noout -dates -subject -issuer notBefore=Mar 12 11:50:15 2025 GMT notAfter=Mar 12 11:55:17 2026 GMT subject=O=system:masters, CN=kubernetes-admin issuer=CN=kubernetes ``` Compare each `notAfter` with today. Whichever is in the past is the one to fix. ## Your kubeconfig expired The kubeconfig you downloaded is a copy, and its client certificate is valid for one year. When it runs out, download a fresh copy from the management cluster: ```console $ kubectl get secrets mycluster-kubeconfig -o=jsonpath='{.data.value}' \ | base64 -d \ > mycluster-kubeconfig.yaml ``` Point `KUBECONFIG` at the new file and carry on: ```console $ export KUBECONFIG=mycluster-kubeconfig.yaml ``` ## The API server certificate expired Control-plane certificates last a year, and they refresh whenever a control-plane node is replaced. That normally happens long before expiry: clusters move to newer Cluster Stack releases through the year, and even without an update Syself Autopilot rolls the control plane on its own once its certificates are within 21 days of expiry. Reaching expiry means that rollout could not run, most often because the pool's server type is no longer offered, so no replacement node can come up. See [A server that will not provision](/docs/hetzner/apalla/servers-and-nodes/access/server-not-provisioning). Renewing them means SSH on each control-plane node, so read [Emergency SSH access](/docs/hetzner/apalla/servers-and-nodes/access/emergency-ssh-access) first. You need `kubectl` access to the management cluster and the namespace your cluster lives in. > [!WARNING] > Do one control-plane node at a time. On a single control-plane cluster the restart in step 5 takes the API server down for a few seconds. The cluster is already unreachable at this point, so that is acceptable, but do not restart several nodes at once on an HA cluster. Find the control-plane machines ```console $ kubectl -n get machines \ -l cluster.x-k8s.io/cluster-name=,cluster.x-k8s.io/control-plane= ``` Note each name. You repeat the steps below once per machine. Get the node IP ```console $ kubectl -n get machine \ -o jsonpath='{.status.addresses[?(@.type=="ExternalIP")].address}' ``` Connect to the node ```console $ ssh -p 100 -i -o IdentitiesOnly=yes root@ ``` Renew the certificates Check what expired, renew everything, then check again: ```console $ kubeadm certs check-expiration $ kubeadm certs renew all $ kubeadm certs check-expiration ``` Every residual time should now read about a year. Restart the containers The files on disk are new, but the running containers still hold the old certificates. Stopping a container is enough, because the kubelet recreates it from `/etc/kubernetes/manifests/` and the new one reads the renewed files: ```console $ crictl stop $(crictl ps -aq) ``` This stops every container on the node, not only the four control-plane components. The rest come back the same way. Wait for the API server, then confirm the whole control plane came back: ```console $ until crictl ps --name kube-apiserver --state Running -q | grep -q .; do sleep 2; done $ crictl ps ``` You want `kube-apiserver`, `etcd`, `kube-controller-manager` and `kube-scheduler` all running, each created seconds ago. A recent creation time is what tells you the container restarted and picked up the new certificates. If one is missing it is either still starting or crash-looping. Check before you move on: ```console $ crictl ps -a $ crictl logs ``` `Exited` entries in that list are normal. The containers you just stopped stay listed, and so do init containers, which run once and exit by design. A container that keeps exiting means this node's certificates are not usable yet. Fix it before you touch the next node. Check the node from the inside ```console $ KUBECONFIG=/etc/kubernetes/admin.conf kubectl get nodes ``` Every node should be listed and no TLS error should appear. On an HA cluster, control planes you have not renewed yet may show `NotReady` until you reach them. Leave the SSH session, then repeat from step 2 for the next control-plane machine. Check it from outside Once every control-plane node is done, check the served certificate again from your workstation: ```console $ openssl s_client -connect : -servername /dev/null \ | openssl x509 -noout -dates ``` `notAfter` should be about a year from now, and `kubectl get nodes` should work again. ## Keep it from happening again Certificates refresh when a node is replaced, and nodes are replaced on every upgrade and every rollout, so a cluster that is upgraded through the year does not reach this state. See [Certificate expiry and rotation](/docs/hetzner/apalla/servers-and-nodes/access/certificate-expiry-and-rotation) for how rotation works and [Upgrade the cluster to a new Kubernetes version](/docs/hetzner/apalla/clusters/upgrades/upgrade-to-a-new-kubernetes-version). The health daemon also warns you first. It raises `CertRenewalFailing` when a node certificate is within seven days of expiry and has not renewed, so alert on it. See [Node health conditions](/docs/hetzner/apalla/reference/node-health-conditions). The `autopilot.syself.com/certs` annotation on each Node carries the same certificates with their `notAfter` dates, so you can alert before a renewal ever fails. It covers the node's own certificates, not the API server certificate. Related: [Control-plane and etcd recovery](/docs/hetzner/apalla/clusters/troubleshooting/control-plane-and-etcd-recovery) for a control plane that is unhealthy for other reasons, and [Emergency SSH access](/docs/hetzner/apalla/servers-and-nodes/access/emergency-ssh-access) for reaching a node.