Safe, rolling upgrades for Talos Linux clusters

KangalPatch is a Kubernetes operator. Describe the upgrade in a PatchPlan and it drains, upgrades, reboots and verifies each node, group by group, within the failure limits you set.

helm install kangal-patch oci://ghcr.io/uozalp/helm/kangal-patch \
  --version 0.2.0 \
  --namespace kangal-patch \
  --create-namespace

MIT licensed. Installs with Helm or kubectl. Version checks cover the Talos and Kubernetes support matrix for Talos 1.10 to 1.14.

Watch a rollout

A simulated PatchPlan with the default order: control plane first, then workers. Pause the plan, or make the next node fail to see the Halt failure policy stop it.

Simulation of a PatchPlan upgrading three control plane nodes and six worker nodes to Talos v1.12.1. Use the buttons below to run, pause, restart, or make the next node fail.

  • Pending
  • Draining
  • Upgrading
  • Rebooting
  • Completed
  • Failed

Features

Everything a Talos upgrade needs, declared in one resource and enforced by the operator.

Safe by default

  • Preflight checksBlock a rollout before any node is touched: Kubernetes API readiness, Talos credentials, installer image and version support.
  • Safe drainingDrains enforce PodDisruptionBudgets, with timeouts for the drain and the reboot.
  • Failure budgetsHalt or pause the plan once the number of failed nodes you set is reached.

In the order you choose

  • Rollout groupsPatch databases, GPU nodes, workers and control plane in your order, each group with its own concurrency.
  • Maintenance windowsAllow patching only in set time windows and skip holidays with excluded dates.
  • Pause, resume, cancelStep in at any time with a single kubectl patch.

Hands-off when you want it

  • Auto-updateFollow upstream Talos releases, with a minimum release age before one is used.
  • Kubernetes upgradesPatch the kubelet, control plane static pods and kube-proxy without reboots.
  • Live statusTrack every plan, group and node. Finished PatchJobs are cleaned up after a retention period.

How it works

You create one resource. The operator does the rest.

A PatchPlan declares the target versions, node selection, rollout order, safety settings and schedule. It is the only resource you create.

For every node it schedules, the operator creates a PatchJob that moves that single node through the upgrade state machine.

PatchPlan
  +-- PatchJob / node-1
  +-- PatchJob / node-2
  +-- PatchJob / node-3
  +-- PatchJob / node-4
  1. Select and group nodes

    nodeSelector picks the nodes. Each one is assigned to the first matching group in strategy.order.

  2. Run preflight checks

    The checks run once. Nothing is scheduled until they all pass.

  3. Patch each node

    Cordon and drain, upgrade Talos, reboot, verify. If you set a Kubernetes target, the Kubernetes components are patched and validated too.

  4. Enforce your limits

    Per-group concurrency, timing and failure settings apply throughout.

  5. Stop when it matters

    If failures pass the threshold, the plan halts or pauses according to failurePolicy.

Every node passes through the same phases

A PatchJob tracks one node from start to finish.

  1. Pending
  2. Draining
  3. Upgrading
  4. Rebooting
  5. UpgradingKubernetes
  6. ValidatingKubernetes
  7. Completed

Dashed phases only run when target.kubernetesVersion is set. A job that fails ends in Failed.

Quick start

From an empty namespace to your first rolling upgrade in four steps.

  • A Kubernetes cluster running Talos Linux
  • kubectl configured for the cluster
  • Helm 3.x for the Helm install
  • Talos API client credentials: CA certificate, client certificate and key
  1. Install the operator

    helm install kangal-patch oci://ghcr.io/uozalp/helm/kangal-patch \
      --version 0.2.0 \
      --namespace kangal-patch \
      --create-namespace

    Two replicas with leader election are the default. Helm installs the CRDs on first install but never upgrades them, so apply the CRDs from the GitHub Release before upgrading the chart.

    # CRDs
    kubectl apply -k config/crd
    
    # RBAC and operator
    kubectl apply -k config/manager

    Check that the operator and its CRDs are there:

    kubectl -n kangal-patch get pods
    kubectl get crd patchplans.kangalpatch.ozalp.dk patchjobs.kangalpatch.ozalp.dk
  2. Create the Talos credentials Secret

    The operator reads the Talos API credentials from a Secret in its own namespace. Values can be raw PEM or base64-encoded, as in a talosconfig file.

    CA_CERT=$(base64 -w0 < /path/to/ca.crt)
    CLIENT_CERT=$(base64 -w0 < /path/to/client.crt)
    CLIENT_KEY=$(base64 -w0 < /path/to/client.key)
    
    kubectl create secret generic talos-credentials \
      --namespace kangal-patch \
      --from-literal=ca.crt="$CA_CERT" \
      --from-literal=tls.crt="$CLIENT_CERT" \
      --from-literal=tls.key="$CLIENT_KEY"

    Treat this Secret as sensitive. These credentials grant full administrative access to the Talos API. Restrict access with RBAC and consider an external secret manager.

  3. Create a PatchPlan

    This plan upgrades the control plane first, then the workers, one node at a time.

    apiVersion: kangalpatch.ozalp.dk/v1alpha1
    kind: PatchPlan
    metadata:
      name: simple-upgrade
    spec:
      target:
        talosVersion: v1.11.6
        source: factory
        installer: nocloud
    
      # Control plane first, then workers (this is the default order)
      strategy:
        order: [controlPlane, workers]
    
      # How many nodes of a group are patched at once
      groups:
        controlPlane:
          concurrency: 1
        workers:
          concurrency: 1
    
      # Timing
      delayBetweenNodes: 300s
    
      # Safety
      respectPDBs: true
      drainTimeout: 5m
      rebootTimeout: 10m
      failurePolicy:
        type: Halt
        maxFailures: 1
    
      # Talos API
      talosConfig:
        endpoints:
          - 10.0.0.10:50000
        secretRef:
          name: talos-credentials
          namespace: kangal-patch
    kubectl apply -f patchplan.yaml

    source: factory resolves the installer from Talos Image Factory and is required from Talos 1.14. Without a schematicID, each node keeps the schematic it runs today, so system extensions and kernel arguments survive the upgrade.

  4. Monitor progress

    $ kubectl get patchplan -w
    NAME             PHASE       TALOSTARGET   K8STARGET   TOTAL   RUNNING   COMPLETED   FAILED   AGE
    simple-upgrade   Completed   v1.11.6                   6       0         6           0        79m
    
    $ kubectl get patchjob

Recipes

Common PatchPlan settings, ready to adapt. More manifests live in config/samples.

Groups are label filters over the nodes your plan selects. They run strictly in strategy.order, and a node is patched at most once per plan: the first listed group it matches schedules it.

spec:
  nodeSelector:
    matchLabels:
      environment: production
  groups:
    database:
      selector:
        matchLabels: {workload: database}
      concurrency: 1
    kafka:
      selector:
        matchLabels: {workload: kafka}
      concurrency: 1
    gpu:
      selector:
        matchLabels: {accelerator: nvidia}
      concurrency: 2
    workers:        # built in, declared only to set its concurrency
      concurrency: 3
  strategy:
    order: [database, kafka, gpu, workers, controlPlane]
  failurePolicy:
    type: Halt
    maxFailures: 1
  retention:
    history: 168h

Patching pauses outside the windows and on excluded dates. All times are UTC, windows may span midnight, and omitting days matches every day.

spec:
  maintenance:
    excludeDates:             # holidays, blackout periods (YYYY-MM-DD)
      - "2026-12-24"
      - "2026-12-25"
      - "2026-12-31"

    windows:                  # when patching is allowed (UTC)
      - days: ["Monday", "Friday"]
        startTime: "01:00"
        endTime: "05:00"

      - days: ["Wed"]
        startTime: "22:00"
        endTime: "02:00"      # spans midnight

      - startTime: "03:00"    # every day
        endTime: "04:00"

The plan becomes a template in the Watching phase. Every checkInterval it creates a child plan for a new upstream Talos release, so each release has its own plan and history. Omit talosVersion on templates.

spec:
  target:
    source: factory       # required once releases reach Talos 1.14
    installer: nocloud
    autoUpdate:
      enabled: true
      allow: patch          # patch (default) | minor
      checkInterval: 30m    # default 1h, minimum 1m
      minReleaseAge: 72h    # default 0s

Set schematicID to change the extensions and customization of the Talos image on every node. Add secureBoot to use the secure boot installer.

spec:
  target:
    talosVersion: v1.12.1
    source: factory
    installer: aws
    schematicID: 376567988ad370138ad8b2698212367b8edcb69b5fd68c80be1f2ec7d603b4ba
    secureBoot: true                  # adds the -secureboot suffix
  groups:
    workers:
      concurrency: 2
  talosConfig:
    endpoints:
      - 10.0.0.10:50000
    secretRef:
      name: talos-credentials
      namespace: kangal-patch

A cancelled plan stops scheduling new nodes. PatchJobs already in progress run to completion, and setting cancelled back to false resumes the plan. To stop for good, delete the plan.

# Pause and resume
kubectl patch patchplan simple-upgrade --type merge -p '{"spec":{"paused":true}}'
kubectl patch patchplan simple-upgrade --type merge -p '{"spec":{"paused":false}}'

# Cancel
kubectl patch patchplan simple-upgrade --type merge -p '{"spec":{"cancelled":true}}'

Several minor versions behind? Upgrade in stages. Talos does not reliably support skipping minor versions, and a large jump can fail silently: the node reboots into its old partition and the PatchJob waits in Rebooting. For example, go from v1.11.x to v1.13.10, then to v1.14.0, running one PatchPlan per stage.