Talos Rolling Upgrades: When a PDB Blocks the Drain

Introduction

Upgrading Talos is normally the most boring part of running my homelab cluster. One command per node: talosctl upgrade cordons the node, drains it, installs the new image, reboots, and the node rejoins. I have a just recipe that walks all five nodes one at a time to keep etcd quorum. Fire and forget.

Except this time, node 4 refused to drain. The upgrade sat there retrying evictions for a few minutes, then aborted. The node stayed on the old Talos version, cordoned, with a SchedulingDisabled badge in kubectl get nodes - and my “fully automated” rolling upgrade needed a human.

The culprit was a PodDisruptionBudget doing exactly what I told it to do. This post walks through why a single-replica workload with a PDB makes a node impossible to drain, and how I made the upgrade automation route around it instead of fighting it.

Four Concepts Before the Story

If you already drain nodes for a living, skip ahead. Otherwise, four pieces of Kubernetes vocabulary carry this whole post:

Cordon - marking a node “unschedulable”. Nothing new gets placed on it; everything already running stays put. It’s the “we’re closing, no new customers” sign.

Drain - actually emptying the node: every pod on it is asked to leave so the machine can be rebooted, upgraded, or removed. Cordon happens first, then the pods are moved. This is what kubectl drain does - and what Talos runs internally before upgrading a node.

Evict vs. delete - the distinction everything hinges on. A drain doesn’t delete pods; it sends polite eviction requests through the Eviction API. An eviction asks: “may this pod go down?” A plain kubectl delete pod never asks - it just kills the pod, and its controller recreates it elsewhere. Same outcome, completely different rules.

PodDisruptionBudget (PDB) - the object that answers the eviction’s question. It declares “at least N copies of this app must stay available” (minAvailable). Kubernetes computes a live number from it: allowed disruptions = healthy pods - minAvailable. Every eviction decrements the answer; when it’s zero, evictions are refused. Voluntary disruptions (drains) respect it; direct deletes and crashes don’t.

That last sentence is the entire plot. Now, the story.

The Setup

Five Talos nodes (all control plane, workloads allowed), upgraded one at a time:

# just/talos.just
TALOS_VERSION   := "v1.13.3"
SCHEMATIC_ID    := "777e54a0...c3a79d"   # image factory: i915 + qemu-guest-agent

upgrade node version=TALOS_VERSION:
    talosctl upgrade --nodes {{ node }} --endpoints {{ node }} \
        --image "factory.talos.dev/metal-installer/{{SCHEMATIC_ID}}:{{ version }}"

Among the workloads: a CloudNativePG PostgreSQL cluster. And because this is a homelab, not a bank, it runs a single instance. One pod, postgres-1, backing every app that needs a database.

CNPG, being a well-behaved operator, automatically creates a PodDisruptionBudget for the primary:

$ kubectl get pdb -n postgres
NAME               MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
postgres-primary   1               N/A               0                     212d

Look at that third column. ALLOWED DISRUPTIONS: 0. That number is the entire story.

Why the Drain Can Never Succeed

A PDB is simple arithmetic: allowed disruptions = healthy pods - minAvailable. With one replica and minAvailable: 1, that’s 1 - 1 = 0. Forever. There is no point in time where evicting this pod is allowed.

Now the part I had half-forgotten: kubectl drain (and talosctl upgrade, which performs the same operation internally) doesn’t delete pods - it uses the Eviction API. And the Eviction API is precisely the thing PDBs guard. Every eviction request for postgres-1 comes back:

error when evicting pods/"postgres-1" -n "postgres" (will retry after 5s):
Cannot evict pod as it would violate the pod's disruption budget.

The drain retries, the PDB math never changes, the timeout hits, the upgrade aborts.

graph LR
    A[talosctl upgrade] --> B[cordon node]
    B --> C[drain: evict pods]
    C --> D{PDB check}
    D -->|allowed = 0| E[eviction denied]
    E --> C
    D -->|timeout| F[upgrade aborted
node cordoned]

    style E fill:#ff6b6b,stroke:#c0392b,color:#fff
    style F fill:#ff6b6b,stroke:#c0392b,color:#fff
  

To be clear: nothing here is buggy. CNPG is right to protect the primary from eviction. Talos is right to abort rather than force-kill a database. The PDB is right to say “you told me one copy must always exist.” Three correct components, one deadlock. The bug is in my topology: a single-replica stateful workload is a promise that every node maintenance event will need special handling.

The Options

Three ways out, from most to least correct:

1. Run PostgreSQL with 2+ instances. The PDB then allows one disruption; CNPG fails over gracefully during drains. This is the real fix, and it costs a second instance’s worth of RAM and storage - which is exactly why my homelab hasn’t done it yet.
2. Delete the PDB before maintenance, recreate after. Works, but now nothing protects the database from unplanned evictions during that window (and CNPG will fight to reconcile the PDB back).
3. Move the pod yourself, then drain an empty node. The Eviction API respects PDBs, but a direct pod delete does not. kubectl delete pod postgres-1 bypasses the budget entirely - CNPG immediately recreates the pod elsewhere. Controlled downtime (~30-60 s), and the drain that follows has nothing left to block on.

I went with option 3, automated.

Sequencing the Rolling Upgrade

The insight that makes option 3 clean: don’t just relocate the pod anywhere - upgrade the node hosting PostgreSQL last, and relocate the pod onto a node that has already been upgraded. Otherwise the pod lands on a not-yet-upgraded node and blocks the very next drain. This costs one relocation for the whole rolling upgrade instead of up to four.

The just recipe (trimmed to the interesting parts, IPs from the documentation range):

NODES="192.0.2.11 192.0.2.12 192.0.2.13 192.0.2.14 192.0.2.15"

# Which node hosts the postgres primary? Upgrade it last.
PG_NODE="$(kubectl get pod postgres-1 -n postgres \
    -o jsonpath='{.spec.nodeName}' 2>/dev/null || true)"
PG_IP="$(kubectl get node "$PG_NODE" \
    -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')"

# Build the order: every other node first, postgres node at the end
ORDER=""
for ip in $NODES; do [[ "$ip" != "$PG_IP" ]] && ORDER="$ORDER $ip"; done
ORDER="$ORDER $PG_IP"

for ip in $ORDER; do
    # Idempotent: skip nodes already on the target version
    cur="$(talosctl version --nodes "$ip" --endpoints "$ip" --short \
        | awk '/Tag:/{print $2}' | tail -1)"
    [[ "$cur" == "$VERSION" ]] && continue

    # Relocate postgres off this node before upgrading it
    if [[ "$ip" == "$PG_IP" ]]; then
        kubectl cordon "$PG_NODE"                 # nowhere to land but an upgraded node
        kubectl delete pod postgres-1 -n postgres # direct delete bypasses the PDB
        # wait for the pod to be Ready on its new node...
        kubectl uncordon "$PG_NODE"
    fi

    talosctl upgrade --nodes "$ip" --endpoints "$ip" --image "$IMAGE"
done

Details worth calling out:

• The cordon before the delete matters. When CNPG recreates postgres-1, the scheduler must not put it back on the node we’re about to upgrade. Cordoning first guarantees it lands elsewhere - and since every other node is already upgraded at this point, “elsewhere” is always safe.
• Idempotency is free and priceless. Checking the current version before each node means a run that aborted halfway (as mine did) can simply be re-launched; already-upgraded nodes are skipped in seconds.
• One node at a time is not optional. All five nodes run etcd, the database backing the Kubernetes API itself. A five-member etcd cluster needs three members alive to keep accepting writes (its quorum) - upgrading two nodes at once means one unexpected failure away from a frozen control plane. The version check makes the serialization cheap to live with.

Manual Recovery

If the upgrade aborted on the postgres node before you had the automation (my situation the first time), the node is cordoned and still on the old version. The by-hand version of the same idea:

kubectl cordon <other-not-yet-upgraded-nodes>  # force postgres onto an upgraded node
kubectl delete pod postgres-1 -n postgres      # direct delete bypasses the PDB
kubectl get pod postgres-1 -n postgres -o wide -w   # wait for Ready
kubectl uncordon --all                         # or list them explicitly
talosctl upgrade --nodes <postgres-node-ip> --endpoints <postgres-node-ip> --image "$IMAGE"

The Broader Lesson: Audit Your PDBs

This isn’t a Talos problem, or even a PostgreSQL problem. Any node maintenance that drains - cluster autoscaler scale-down, Kured reboots, managed-Kubernetes node pool upgrades - hits the same wall on the same math. One command tells you whether you have a landmine waiting:

kubectl get pdb --all-namespaces

Any row with ALLOWED DISRUPTIONS: 0 is a node that cannot be drained gracefully. Some operators (CNPG, some Kafka and MongoDB operators) create these PDBs automatically for single-replica deployments - protecting your data, and quietly signing you up for manual intervention on every drain.

Your options per row are always the same three: add a replica, plan a relocation step, or accept that maintenance on that node means downtime. What you don’t want is to discover the answer mid-upgrade.

Key Takeaways

Conclusion

The satisfying part of this failure is that every component behaved correctly - the deadlock emerged from an honest declaration (“this database has one copy and it must stay up”) meeting an honest operation (“this node must be emptied”). Kubernetes didn’t hide the contradiction; it surfaced it at the worst possible time.

The permanent fix is still on my list: a second PostgreSQL instance would let CNPG fail over during drains and delete this whole workaround. Until the homelab’s RAM budget allows it, the upgrade recipe knows about its one special pod - and kubectl get pdb -A runs before every maintenance window, not after.

Useful Resources:

• Kubernetes: Pod Disruption Budgets
• Talos: Upgrading Talos Linux
• CloudNativePG: Kubernetes Upgrade and Maintenance
• Talos Image Factory