Skip to main content

Where to start troubleshooting

Inspect 1.36

Start with the object, then go to the tab that owns it#

When you are not sure what is wrong, start with the object that is failing and its events, then work the runbook in the tab that owns the fix. The event on the object tells you what a controller last tried and why it stopped. That is what points you at the right runbook.

Syself Autopilot repairs many faults on its own by replacing the node, so before you dig in, check whether the object is already recovering. See .

The general approach:

  1. Find the object that is unhealthy: the Cluster, a Machine, a Node, a Pod, or a PVC.
  2. Read its status and its events. The events name the reason and the last action.
  3. Match the symptom or the exact error string to its runbook.
  4. If the runbook does not clear it, collect what support needs and open a case.

Symptom to runbook#

Symptom Where it is handled
A new cluster is stuck in Provisioning
The first control plane comes up, but no other nodes join, and provisioning seems to take forever Custom API server domain: check the DNS records
No cloud capacity, provisioning will not start
A server never finishes provisioning
A node is NotReady, or a workload will not schedule
A network path or a Service is broken ·
Pods cannot resolve DNS
An upgrade is stuck or failed
A PVC is stuck, or a pod will not mount
A GPU pod stays Pending
A certificate is failing to renew, or has expired

Common error strings#

Error string What it means Where it is handled
DetectLinuxOnAnotherDisk A foreign OS or an existing Linux install on the disk blocks the bare-metal install. Wipe the disks.
server location disabled (also resource_unavailable) Hetzner cloud capacity ran out in the region, so no new server can be created. Move to bare metal or wait.
NotReady The node stopped reporting Ready to the API server. Check the node first; if the load balancer path is down, check networking. ·
CertRenewalFailing A node condition that a certificate is not renewing inside its window.

Custom API server domain: check the DNS records#

One provisioning trap looks much worse than it is. You set a custom domain for the API server, created a load balancer for it, and the first control plane came up fine. Then nothing else joins. The cluster looks like it is taking forever to provision.

The cause is almost always DNS. The other nodes reach the API server by that custom domain, so they cannot join until the domain resolves to the load balancer. If the DNS records were never created, or they were created but have not propagated yet, the join hangs.

Check, in order:

  1. The DNS records for the custom domain exist and point at the load balancer's IP.
  2. The records have propagated. Resolve the domain from a machine outside your DNS provider and confirm you get the load balancer's address.

Once the domain resolves, the waiting nodes join on their own. No other action is needed.

Read a Machine or Cluster event#

Cluster and Machine objects live on the management cluster. Their events show what the controller last tried on your cluster and its nodes.

Note

Run these against the management cluster kubeconfig, where the Cluster and Machine objects live. Node, Pod, and PVC objects live on the workload cluster.

  1. List the objects and their phase:

    		$ kubectl get cluster,machine -n <namespace>

  2. Describe the object and read the Events section at the bottom:

    		$ kubectl describe machine <name> -n <namespace>
# or
$ kubectl describe cluster <name> -n <namespace>

  3. Sort every recent event in the namespace by time:

    		$ kubectl get events -n <namespace> --sort-by=.lastTimestamp

  4. Note the exact Reason and Message, and the object they sit on. That string routes you to the runbook, and it is what support needs first.

Pause automation while you investigate#

Sometimes the automation gets in your way. You are looking at a failing Machine, and the controllers keep retrying, or self-healing reboots and replaces the node out from under you before you can read what went wrong. Pause it so it holds still.

Set the cluster.x-k8s.io/paused annotation on the object you want frozen:

		# Freeze one Machine (stops its controllers, including automatic remediation):
$ kubectl annotate machine <name> -n <namespace> cluster.x-k8s.io/paused=true
 
# Or freeze the whole cluster:
$ kubectl annotate cluster <name> -n <namespace> cluster.x-k8s.io/paused=true

While an object is paused, its controllers stop reconciling it. Automatic remediation and replacement do not run, so a paused unhealthy node stays put instead of being rebooted or replaced. Nothing is deprovisioned and no workloads move. The object simply stops changing while you look at it.

Resume by removing the annotation:

		$ kubectl annotate machine <name> -n <namespace> cluster.x-k8s.io/paused-
Warning

Pausing turns off self-healing for that object. It does not fix anything by itself, and a node left paused will not recover on its own. Pause to investigate, then resume as soon as you are done.

Pausing is different from maintenanceMode, which deprovisions a bare-metal host and reschedules its workloads. Use pause when you want the automation to stop touching a node without removing it. See for the maintenance-mode path and when each fits.

When to stop and contact support#

Stop working the runbook and open a case when:

  • You have worked the owning runbook and the object is still unhealthy.
  • The control plane or etcd is at risk (API unreachable, quorum in doubt) and you should not act alone.
  • There is a risk of data loss.
  • The fix needs Syself to act on the sealed base or the bare-metal server itself.

Before you write, gather the details in . Then reach Syself on your channel: see .

Get support

Find a fix fast, know when a case is the right move, and write a first message the Syself team can act on.

What to collect first

Gather these details and files before you write to support so the first reply moves your case forward.