Skip to main content

Offsite and immutable backups

Inspect 1.36

Ransomware does not just encrypt your running data. It hunts for the backups and deletes them first, because a working backup is the one thing that beats it. A stolen backup key or a disgruntled insider does the same. If a single credential can erase every copy you own, you do not have a backup you can trust. Two properties fix this: a copy that lives somewhere else, and a copy nobody can delete.

A second copy in another region#

One bucket is one failure domain. A region outage, an account lockout, or a fat-fingered delete takes the whole bucket, and your only backup with it. Replicate your backups to a second bucket in a different region, or with a different provider, so no single event reaches both.

Set up the second bucket to receive a copy of every backup, either by replication on the storage side or by a scheduled sync job. The goal is simple: losing the primary bucket never means losing the data.

Lock it so nobody can delete it#

Object-lock makes a stored object impossible to delete or overwrite until a retention period expires. This is WORM retention: write once, read many. Once written, the backup is frozen for the window you set, and no key, no admin, no attacker can remove it early.

Caution

Immutability is what defeats ransomware and rogue deletes. Encryption software and a malicious insider both try to erase your backups. An object-lock retention window makes that erase fail. Set the window to at least how long you need to notice an attack and recover, because you cannot shorten it once set, and you cannot delete a locked object before it expires.

Turn object-lock on when you create the backup bucket, because it usually cannot be added to an existing bucket. Then apply a retention period to every backup object as it lands.

Versioning as a safety net#

Versioning keeps the old copy when an object is overwritten, so a corrupted or maliciously replaced backup does not destroy the good one underneath. It is a lighter safeguard than object-lock: versioning protects against overwrite, object-lock protects against deletion. Use both. Versioning catches mistakes, object-lock stops attacks.

Separate credentials for the backup bucket#

The key your cluster uses for day-to-day storage must not be able to touch the backup bucket. If it can, one compromised cluster credential reaches your backups too. Give the backup bucket its own key, scoped to that bucket alone, and store it away from the cluster. The account that writes backups should not be the account that can delete them.

The 3-2-1 rule for Kubernetes#

The old rule still holds, translated to clusters:

  • 3 copies of the data: the live PVC, a Velero backup, and an offsite copy.
  • 2 different storage systems: the cluster's storage and object storage.
  • 1 copy offsite and locked: a second region with object-lock.
Copies Media or location Offsite
3: live PVC, Velero backup, offsite copy 2 systems: cluster storage and object storage 1 copy in a second region with object-lock

One number to weigh: cross-region traffic costs money on egress, and a second provider costs more still. Replicate what you cannot lose, not every scratch backup. Match the spend to the value of the data, not to a habit of copying everything.

This is the last line of the backup chain. The covers running your own S3-compatible object storage and scoping keys to a bucket, and shows the access side. When the primary is gone, an offsite immutable copy is what the restores from.

Tip

Rehearse a restore from the offsite copy, not just the primary bucket. The offsite copy is the one you reach for in a real disaster, so it is the one worth proving.