Certificate expiry and rotation

Node certificates rotate on their own, so normally you do nothing. You are here because a CertRenewalFailing condition fired, or a certificate expired and connections are breaking.

Both mean the certificates stopped refreshing in time. Most often that is not a broken renewer at all, but a cluster that has not rolled a new node in about a year.

The most common cause: no rollout in about a year

Node certificates refresh whenever a node is replaced, which happens on every upgrade and every rollout. So a certificate that actually expires almost always means the cluster has not rolled a new node in roughly a year. The usual reason the automatic rollout never happened is that it could not: the pool's server type was deprecated, or there was no cloud capacity in the region, so no replacement node could come up. Check for a stalled rollout first, see A server that will not provision . Once nodes can roll again, fresh certificates come with them.

How node certificates auto-rotate

The kubelet is the agent on each node that runs pods and talks to the API server. It holds two certificates, and each renews a different way.

A CSR (certificate signing request) is how a node asks the cluster to sign a new certificate. Syself's management-cluster controller checks the node's identity before it signs the serving one.

A node also needs a valid certificate to keep its tunnel to the control plane open. An expired client certificate does not just stop the kubelet's status updates, it can cut the node's only path back to the API server. That is what "connections are breaking" means here.

The CertRenewalFailing condition

CertRenewalFailing is an alert condition. It fires when a node certificate is getting close to expiry and has not renewed yet, with roughly a week of runway still left.

A related signal, KubeletHeartbeatFailed, also an alert, means the kubelet stopped retrying node-status updates, so it lost its connection to the API server. An expired client certificate is one cause.

Read the certs report

The health daemon writes a certificate report as a node annotation, with an updatedAt timestamp, the node role, and a list of certificates:

		$ kubectl get node <node-name> -o jsonpath='{.metadata.annotations.autopilot\.syself\.com/certs}' | jq
	

Use it to see which certificate is close to expiry and whether the report is still being refreshed.

When reprovision is the fix

If the renewer is stuck, or a certificate already expired on a sealed node, replace the node. CertRenewalFailing does not replace the node for you, so this is your call: reprovision the affected node, then confirm its replacement is healthy.

Control-plane and etcd certificates

A control-plane node also holds the cluster's certificate authorities (CAs). A CA is the key that signs other certificates. The provisioner generates all four on the first control-plane node:

The tamper monitor raises NodeTampered when a protected file changes, and a CA or signing-key change is one such file. Leaf certificates are excluded, so ordinary rotation does not trip a false alarm.

Treat two cases as an escalation: a CA-level problem, and a control-plane node whose certificates will not rotate. Collect what to send , then reprovision the node. Replace one control-plane node at a time to keep quorum, the majority of control-plane nodes that etcd needs to stay available.

Confirm rotation resumed

After replacing a node, confirm the certs report is fresh (its updatedAt is advancing and the near-expiry entry is gone) and that CertRenewalFailing and KubeletHeartbeatFailed have cleared on the node:

		$ kubectl get node <node-name> -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{"\n"}{end}'
	

Related

• Node health conditions
• A server that will not provision
• Node and OS security
• Security model