Monitor cluster and machine health

Your cluster's health has a layer above the nodes: the Cluster API objects that describe it. The Cluster, Machine, and MachineDeployment records are what Syself uses to run and heal your cluster. Syself runs the management cluster and the controllers that turn your Cluster object into Machines and Nodes, and that heal them when they fail. You do not operate those controllers, but you have read access to the objects they reconcile, and watching those objects is how you catch a cluster that is not reaching the state you declared.

What the management cluster does for you#

You declare a Cluster (and its pools) once. In the management cluster Syself runs , the controllers reconcile that declaration continuously: they create Machines, provision servers, join Nodes, and step in when a node stops being healthy. On a cloud node that means a reboot, and if the node does not come back, replacing it with a fresh one from a clean image. On a bare-metal server it means rebooting the server. This is the self-healing that keeps the fleet running without you. The management cluster's own health is Syself's responsibility.

The objects you watch#

You have kubectl access to your objects in the management cluster, so you read their status directly:

		$ kubectl get clusters,machinedeployments,machines
	

• Cluster carries top-level conditions: whether the control plane is ready and the infrastructure is provisioned.
• MachineDeployment is a pool. Compare its ready replicas against its desired count; a gap that does not close means the pool cannot reach its target.
• Machine is one server on its way to becoming a Node. Its phase walks Provisioning to Running; a Machine stuck in Provisioning is the signal behind most cases where a cluster does not come up (see Cluster creation fails ).

What to alert on#

Watch for the states that mean your declared cluster and the real one have drifted apart:

• A Machine stuck in Provisioning past the normal few minutes: capacity, credentials, or a bad manifest.
• A MachineDeployment not reaching its replicas: it cannot get the servers it asked for (common on bare metal when no free host matches, or on cloud during a capacity shortage).
• Repeated remediation on one pool: Machines being rebooted or replaced in a loop points at a persistent fault, a bad image reference, or on bare metal a hardware fault a reboot does not clear.

To alert on these, run a kube-state-metrics custom-resource-state configuration (or a small exporter) that reads the Cluster API objects from the management cluster with your read credentials, and turn their conditions into metrics you can put a PrometheusRule on. Watch the Machine and Cluster events too; they narrate provisioning and remediation as it happens. The node health daemon also records what it finds on each node, as node conditions and as a health report in autopilot.syself.com/* node annotations, which help you diagnose why a node was rebooted or replaced.

Where the boundary sits#

Syself operates the management cluster and the healing loop; you observe the objects that loop reconciles. Syself watches the platform's own health; you watch whether your declared state (your pools, replicas, and versions) is actually met. When it is not, and it is not a capacity or manifest issue you can fix, that is when to open a support case. To run this watch across many clusters from one place, see Multi-cluster observability .