CFN Cloud
2025-10-02

Kubernetes Architecture Explained: Control Plane, Nodes, and Reconciliation

Understand Kubernetes architecture from the control plane to worker nodes, including the API server, scheduler, controllers, kubelet, and reconciliation loops.

The fastest way to stop feeling lost in Kubernetes is to divide the system into two layers: the control plane makes decisions, and nodes execute them. That sounds simple, but it is the mental model behind most troubleshooting.

The short version

  • Control plane: decides what should happen.
  • Nodes: carry out that decision.
  • Controllers: keep comparing desired state with reality.
  • Reconciliation: the loop that keeps trying until the cluster converges.

If you remember only one thing, remember this: Kubernetes is not a single command executor. It is a distributed control system.

Control plane components

API server

The API server is the front door. kubectl, controllers, schedulers, and operators all talk to the cluster through it.

If writes are timing out, authentication is failing, or controllers seem blind, the API server is one of the first places to suspect.

etcd

etcd stores cluster state. It is the durable memory of the control plane.

When etcd is unhealthy, the whole cluster starts acting confused: writes stall, leader elections wobble, and control loops cannot reliably converge.

Scheduler

The scheduler looks for unscheduled Pods and chooses a node for them. It evaluates resource requests, taints and tolerations, affinity rules, topology, and other constraints.

If a Pod stays Pending, the scheduler or one of its constraints is usually involved.

Controller manager

Controllers are the part that makes Kubernetes feel declarative. They compare desired state with current state and keep pushing reality toward the spec.

Deployments, ReplicaSets, StatefulSets, Nodes, Jobs, and many more all rely on controllers.

Node components

kubelet

kubelet is the node agent. It watches for Pods assigned to that node and works with the container runtime to start them, monitor them, and report status back.

If a Pod is stuck in ContainerCreating, cannot pull images, or repeatedly fails probes, kubelet is in the story.

Container runtime

Usually containerd or CRI-O. This is the layer actually pulling images and starting containers.

Kubernetes decides what to run; the runtime turns that decision into running processes.

kube-proxy

kube-proxy maintains packet forwarding behavior for Services, usually through iptables or IPVS rules.

When a Service exists but traffic does not land where expected, kube-proxy may not be the root cause, but it is part of the traffic path.

CNI and CSI

CNI plugins handle networking. CSI drivers handle storage attachment and volume integration.

If Pods cannot get IPs, or volumes do not mount, you are now outside the core controllers and inside plugin territory.

The architecture in one diagram

kubectl
  |
  v
API Server ---> etcd
  |
  +---> Scheduler
  |
  +---> Controllers
  |
  +---> kubelet on nodes ---> container runtime
                       |
                       +---> CNI / CSI / kube-proxy

What really happens after kubectl apply

This is the event chain most people should keep in their heads:

  1. You send a manifest to the API server.
  2. The API server validates and stores it in etcd.
  3. The relevant controller notices a gap between desired and current state.
  4. That controller creates or updates objects such as ReplicaSets and Pods.
  5. The scheduler assigns unscheduled Pods to nodes.
  6. The kubelet on those nodes pulls images, mounts volumes, configures networking, and starts containers.
  7. Status updates flow back to the API server.
  8. Controllers keep reconciling until the result matches intent.

Once you understand this chain, architecture stops being a diagram and starts becoming a troubleshooting map.

Symptom-to-component map

Use symptoms to narrow the layer quickly.

Symptom Likely layer to inspect first
kubectl apply times out API server, auth, connectivity
Pod stays Pending scheduler, requests/limits, affinity, PVC binding
Pod stays ContainerCreating kubelet, image pull, CNI, CSI
Pod restarts repeatedly app process, probes, kubelet, runtime
Service exists but traffic fails selectors, endpoints, kube-proxy, policy
PVC not bound storage provisioner, StorageClass, topology

This is not perfect, but it is a much better starting point than random guessing.

Control plane problem or node problem?

Ask this early:

  • Is the cluster making the wrong decision?
  • Or is the node failing to carry out the right decision?

Examples:

  • Pod never gets a node -> scheduler or constraints.
  • Pod gets a node but image pull fails -> node-side execution issue.
  • Deployment spec changed but rollout never appears -> controller or API layer.
  • Service exists but has zero endpoints -> object relationship issue, not necessarily network plumbing.

Common misunderstandings

“Kubernetes runs my commands”

Not really. kubectl submits desired state or requests data, but controllers and kubelets do the real ongoing work.

“If I applied it successfully, it should already be running”

Also no. Created or configured means the request was accepted, not that the cluster has fully converged.

“Pending means the node is broken”

Sometimes, but often it means the scheduler cannot find a node that satisfies constraints.

Minimal lab sequence

Use a tiny deployment to watch architecture in motion:

kubectl create namespace demo
kubectl apply -f deployment.yaml -n demo
kubectl get deploy,pods -n demo
kubectl describe deploy -n demo app
kubectl describe pod -n demo <pod-name>
kubectl get events -n demo --sort-by=.metadata.creationTimestamp

That one loop shows the API, controller, scheduler, and kubelet handoff in practice.

What to check when things stall

If a Pod is Pending

  • kubectl describe pod
  • check requests and limits
  • check affinity / anti-affinity
  • check taints and tolerations
  • check PVC binding status

If a Pod is ContainerCreating

  • image pull errors
  • volume mount failures
  • CNI setup failures
  • node readiness

If a rollout is slow or stuck

  • Deployment conditions
  • ReplicaSet events
  • readiness probes
  • unavailable surge limits

Operational habits that make architecture easier to reason about

  • Keep manifests in version control.
  • Use a stable label scheme.
  • Do not leave requests and limits empty forever.
  • Turn on logs and metrics before the first incident.
  • Write down rollback steps before rollout steps.

These habits reduce ambiguity, and ambiguity is what makes distributed systems feel magical and hostile.

FAQ

Q: Why is reconciliation so important in Kubernetes? A: Because the cluster is always drifting: nodes disappear, containers crash, and config changes. Reconciliation is the mechanism that keeps pushing actual state back toward desired state.

Q: If kubectl apply succeeds, why can the workload still fail later? A: Because the request being accepted is only the first step. Scheduling, image pulls, volume mounts, probes, and controller progress all happen after that.

Q: What is the most useful architecture question during troubleshooting? A: Ask where the chain broke: API acceptance, controller creation, scheduling, node execution, or traffic exposure.

Next reading

  • Read kubernetes-quickstart-pod.md to understand the smallest schedulable unit.
  • Continue with kubernetes-quickstart-deployment-replicaset.md for controller behavior.
  • Then read kubernetes-quickstart-service.md to understand traffic flow.

References

View original