Certificate expiry and rotation
Node certificates rotate on their own, so normally you do nothing. You are here because a CertRenewalFailing condition fired, or a certificate expired and connections are breaking.
Both mean the certificates stopped refreshing in time. Most often that is not a broken renewer at all, but a cluster that has not rolled a new node in about a year.
The most common cause: no rollout in about a year#
Node certificates refresh whenever a node is replaced, which happens on every upgrade and every rollout. So a certificate that actually expires almost always means the cluster has not rolled a new node in roughly a year. The usual reason the automatic rollout never happened is that it could not: the pool's server type was deprecated, or there was no cloud capacity in the region, so no replacement node could come up. Check for a stalled rollout first, see A server that will not provision . Once nodes can roll again, fresh certificates come with them.
How node certificates auto-rotate#
The kubelet is the agent on each node that runs pods and talks to the API server. It holds two certificates, and each renews a different way.
| Certificate | Proves | Renews | Approved by |
|---|---|---|---|
Client (/var/lib/kubelet/pki) | Who the kubelet is | The kubelet renews it itself (rotateCertificates: true) | The kubelet renews its client certificate automatically. |
Serving (/var/lib/kubelet/pki/kubelet-server-current.pem) | That other components can trust the kubelet | The kubelet requests a new one via CSR (serverTLSBootstrap: true) | Syself's management-cluster controller |
A CSR (certificate signing request) is how a node asks the cluster to sign a new certificate. kube-controller-manager refuses to auto-approve a serving CSR: it cannot verify the requested SANs (Subject Alternative Names, the hostnames and IPs a certificate covers), or confirm the node's identity.
Syself's management-cluster controller checks that instead, then signs with the cluster CA. This is also why metrics-server can verify the kubelet over TLS: it trusts that same cluster CA.
Note
Certificate signing (CSR) problems are worth bringing straight to us rather than working them alone. If a signing request is denied or stays stuck pending, reach out to Syself support ; that is what we are here for.
The tunnel agent does not hold a separate certificate for the reverse tunnel to the control plane. It re-reads the kubelet client certificate on every TLS handshake, so a fresh certificate takes effect on the next connection, with no restart needed.
But a node can only open a tunnel with a valid certificate. An expired client certificate does not just stop the kubelet's status updates, it can cut the node's only path back to the API server. That is what "connections are breaking" means here.
In normal operation there is nothing to do. Rotation is invisible.
The CertRenewalFailing condition#
CertRenewalFailing is an alert condition. It fires when a node certificate is getting close to expiry and has not renewed yet, with roughly a week of runway still left. Treat it as an early warning that catches a stuck renewer in good time. It is not an emergency on its own, unless it has been firing long enough that a certificate already expired.
A related signal, KubeletHeartbeatFailed, also an alert, means the kubelet stopped retrying node-status updates, so it lost its connection to the API server. An expired client certificate is one cause.
Note
CertRenewalFailing does not trigger automatic replacement by itself. Only a few health signals do that, and a certificate problem is not one of them, so it is always your call, covered next.
Read the certs report#
The health daemon writes a certificate report as a node annotation, with an updatedAt timestamp, the node role, and a list of certificates:
$ kubectl get node <node-name> -o jsonpath='{.metadata.annotations.autopilot\.syself\.com/certs}' | jq
Use it to see which certificate is close to expiry and whether the report is still being refreshed.
Tip
If updatedAt itself has stopped advancing, the health daemon is down, not just the renewer. That is a different problem: check that the node is Ready and that syself-agent is running before you chase a certificate.
When reprovision is the fix#
Important
The OS is immutable, so do not hand-patch a node's PKI. Replace the node instead. The replacement comes up with fresh certificates.
If the renewer is stuck, or a certificate already expired on a sealed node, replace the node. CertRenewalFailing does not replace the node for you, so this is your call: reprovision the affected node, then confirm its replacement is healthy.
Control-plane and etcd certificates#
A control-plane node also holds the cluster's certificate authorities (CAs). A CA is the key that signs other certificates. The provisioner generates all four on the first control-plane node:
| CA / key | Signs |
|---|---|
| cluster CA | the API server, kubelet, and client certificates |
| front-proxy CA | the aggregation-layer (extension API server) certificates |
| etcd CA | the etcd peer and client certificates |
| service-account signing key | the tokens pods use to talk to the API server |
The tamper monitor raises NodeTampered when a protected file changes, and a CA or signing-key change is one such file. Leaf certificates are excluded, so ordinary rotation does not trip a false alarm.
Treat two cases as an escalation: a CA-level problem, and a control-plane node whose certificates will not rotate. Collect what to send , then reprovision the node. Replace one control-plane node at a time to keep quorum, the majority of control-plane nodes that etcd needs to stay available.
Short-lived certificates, an approver that checks identity before it signs, and a monitor that flags any CA change: this rotation chain is part of the platform's zero-trust model . For the full picture of sealed layers, the host firewall, and tamper evidence, see Node and OS security .
Confirm rotation resumed#
After replacing a node, confirm the certs report is fresh (its updatedAt is advancing and the near-expiry entry is gone) and that CertRenewalFailing and KubeletHeartbeatFailed have cleared on the node:
$ kubectl get node <node-name> -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{"\n"}{end}'