2025-10-07

Stateful Apps on Kubernetes: Storage, Identity, and Upgrade Pace

A concrete guide to stateful applications on Kubernetes, covering storage choices, stable identities, rollout concerns, and failure handling.

Stateful applications are where Kubernetes stops feeling like simple container orchestration and starts feeling like real systems engineering. Databases, queues, and replicated stores need more than Pods. They need identity, storage, ordering, and a recovery plan.

What stateful workloads usually require

  • stable identity
  • persistent storage
  • ordered startup and shutdown
  • predictable replication behavior
  • realistic backup and recovery workflows

If any of those are missing, the workload may still start, but operating it will become painful quickly.

The usual building blocks

For many stateful systems, the baseline stack is:

  • StatefulSet
  • PVC-backed storage
  • StorageClass for provisioning policy
  • Headless Service for stable DNS
  • backup and restore workflow outside the workload itself

Read this page with StatefulSet, PV and PVC, StorageClass, and Headless Service.

Why stateful is harder than stateless

Stateless systems mostly care about healthy replacement.

Stateful systems care about:

  • which replica is which
  • where the data lives
  • how recovery happens
  • whether topology remains consistent during rollout

That is why the YAML is usually not the hardest part. Recovery time and operational correctness are harder.

Verify identity and storage with a disposable lab

This example does not run a database. It tests the two StatefulSet properties that should be understood first: stable Pod names and one PVC per replica. Save it as stateful-demo.yaml:

apiVersion: v1
kind: Service
metadata:
  name: stateful-demo
spec:
  clusterIP: None
  selector:
    app: stateful-demo
  ports:
    - name: peer
      port: 80
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: stateful-demo
spec:
  serviceName: stateful-demo
  replicas: 2
  selector:
    matchLabels:
      app: stateful-demo
  template:
    metadata:
      labels:
        app: stateful-demo
    spec:
      containers:
        - name: app
          image: busybox:1.36
          command: ["sh", "-c"]
          args:
            - 'echo "$(hostname)" > /data/identity; while true; do sleep 3600; done'
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi

With a default StorageClass available, run:

kubectl apply -f stateful-demo.yaml
kubectl get pod,pvc -l app=stateful-demo
kubectl exec stateful-demo-0 -- sh -c 'date -Iseconds > /data/marker'
kubectl delete pod stateful-demo-0
kubectl wait --for=condition=Ready pod/stateful-demo-0 --timeout=5m
kubectl exec stateful-demo-0 -- cat /data/identity /data/marker

The final command should show the stable stateful-demo-0 identity and the marker written before deletion. This proves that the recreated Pod reattached its PVC. It does not prove cross-node storage, database replication, or backup recovery.

Replication and topology questions to answer early

Before treating a stateful app as production-like, answer these:

  • is there one leader or multiple writable nodes?
  • how do clients discover the leader?
  • how do replicas catch up?
  • what happens during restart or node loss?
  • can old and new versions coexist during upgrade?

Kubernetes can run the workload, but it does not solve your replication semantics for you.

Storage planning is part of the app design

Each replica usually needs its own PVC. Shared storage may look simpler, but it often creates contention or correctness problems unless the app is explicitly designed for it.

That means capacity planning should happen per replica, not just per application.

Backups are not optional side notes

PVCs are not backups.

You still need:

  • logical backups or snapshots
  • restore drills
  • retention policy
  • off-cluster backup placement

The biggest stateful mistake is assuming persistence equals recoverability.

Upgrades need a slower mindset

Stateful upgrades are usually slower because you have more to protect:

  • membership ordering
  • replication lag
  • attach and mount time
  • warm-up and readiness time

If your rollout process only works for stateless Deployments, it is probably still too naive for a real stateful system.

Useful operating habits

  • start with one replica before scaling out
  • make readiness reflect true application availability
  • spread replicas across nodes when possible
  • monitor disk usage and replication lag
  • document failover and restore steps before incidents

Most real reliability comes from these habits, not from one magical controller flag.

Three production decisions

Q: Can I run databases on Kubernetes safely? A: Yes, but only if you treat storage, replication, and recovery as first-class concerns instead of assuming Kubernetes will handle them automatically.

Q: What is the biggest risk with stateful apps on Kubernetes? A: Usually the workload starts, but recovery takes longer or behaves differently than the team expected.

Q: Why should I rehearse restores before production? A: Because many stateful failures are not about whether backup files exist. They are about whether the team can restore them correctly and fast enough.

Connect the controller and storage chain

Minimum acceptance line

Complete Pod deletion, node maintenance, volume reattachment, and backup-restore drills, then record recovery time. Kubernetes organizes identity, placement, and persistence; application replication and data recovery still belong to the application, its operator, and the runbook.

References

View original