MySQL replication is not a matter of starting several Pods. Once a source and its replicas are involved, identity, storage, routing, lag, and failover must be verified together. This article uses MySQL 8.4 source / replica terminology; commands copied from older master / slave documentation may no longer exist.
The replication path
- the source accepts writes and produces the binary log
- replicas fetch and replay binary log events
- clients write to the source and read replicas only when the consistency model allows it
This sounds straightforward, but the operational difficulty is not the initial setup. It is what happens when lag grows, storage gets slow, or the primary disappears.
Why Kubernetes changes the picture
Replication on Kubernetes usually depends on:
- StatefulSet for stable ordinal identity
- Headless Service for Pod-level DNS
- PVC-backed storage per replica
- careful readiness and failover logic
Review StatefulSet, Headless Service, PV and PVC, and StorageClass before debugging database replication.
What to verify first in a replication setup
Before you even look at MySQL internals, confirm these basics:
- each replica has its own PVC
- Pod DNS identity is stable
- readiness reflects actual service availability
- clients know which endpoint is for writes and which is for reads
If those are wrong, MySQL replication problems will look worse and be harder to explain.
Why replication lag is the signal to respect
A cluster can look superficially healthy while replication is quietly degrading. Pods may all be running, but if lag grows, read-after-write behavior gets worse and failover risk increases.
That is why replication lag is one of the most important stateful signals to watch.
Common sources of lag and instability
- slow disk
- insufficient CPU
- network jitter between nodes
- large transactions
- bad readiness or promotion logic
The first instinct should usually be to inspect IO and resource pressure before tuning obscure database parameters.
Routing matters as much as replication
In many setups, the problem is not that replication itself broke. It is that clients are talking to the wrong place.
Typical pattern:
- a write endpoint for the primary
- a read endpoint for replicas
- direct Pod DNS for internal replication or admin flows
If Service selectors or DNS assumptions are wrong, the symptoms can look like inconsistent data even when replication is technically working.
A concrete operator checklist
When bringing up or reviewing a replication cluster, I would check:
- are primary and replicas clearly separated?
- does each replica own its own volume?
- is read traffic really going where expected?
- do we know how primary promotion would work?
- do we know how to rejoin an old primary after failover?
These are the questions that usually decide whether the system feels calm or fragile in production.
Quick commands that help early
kubectl get pods -n <ns> -o wide
kubectl get pvc -n <ns>
kubectl get svc -n <ns>
kubectl get endpointslice -n <ns> \
-l kubernetes.io/service-name=<svc> -o wide
kubectl describe pod -n <ns> <pod>
Enter the source first and confirm the actual server version and binary log coordinates:
kubectl exec -it -n db mysql-source-0 -- mysql -uroot -p
SELECT VERSION();
SHOW BINARY LOG STATUS\G
Then enter a replica:
kubectl exec -it -n db mysql-replica-0 -- mysql -uroot -p
SHOW REPLICA STATUS\G
Inspect at least Replica_IO_Running, Replica_SQL_Running, Seconds_Behind_Source, Last_IO_Error, and Last_SQL_Error. A value of 0 for Seconds_Behind_Source is only one sample, not proof of data consistency. A NULL value requires checking whether replication threads stopped. MySQL 8.4 removed SHOW MASTER STATUS and SHOW SLAVE STATUS; run SELECT VERSION() before choosing commands for an older server.
Failover is the real test
The most dangerous sentence in a replication setup is: “we assume failover will be fine.”
Replication is relatively easy to start. Promotion, rejoin, and client behavior during failure are what matter operationally.
At minimum, test in staging:
- time to promote a replica
- time to rejoin nodes afterward
- client behavior during the gap
- what data consistency guarantees you actually get
Three replication acceptance decisions
Q: Is replication enough to call MySQL highly available? A: Not by itself. Replication is a building block, but actual HA depends on failover behavior, client routing, and recovery discipline.
Q: What should I look at first if reads seem stale? A: Check replication lag, then storage and CPU pressure, then verify reads are really hitting replicas and not the wrong service path.
Q: Why does Kubernetes not solve the hard part automatically? A: Because Kubernetes manages workload lifecycle and placement, not MySQL consensus, leader promotion, or application-level correctness.
Complete the database path
- MySQL with Helm: verify whether chart defaults implement the expected topology.
- Stateful apps on Kubernetes: cover backup, recovery, and upgrade boundaries.
- MySQL on Kubernetes: compare against a single-instance baseline.
Minimum failover acceptance line
Stop the source in staging and record replica promotion time, write-endpoint switch time, client error window, old-source rejoin steps, and the transaction-loss boundary. Replica Pods in Running state do not prove failover.