cgroups account for resources, assign relative shares, and enforce limits across a group of processes. Container runtimes, Kubernetes, and systemd all use them, but production debugging cannot stop at Pod requests and limits. The final evidence for throttling, reclaim, or OOM lives in the node’s cgroup files.

This guide focuses on that path: identify V1 or V2, locate the target cgroup, read CPU and memory counters, then map the evidence back to Kubernetes settings.

Identify the cgroup version first

stat -fc %T /sys/fs/cgroup
  • cgroup2fs: cgroup V2, normally with one unified hierarchy.
  • tmpfs: usually V1; inspect the controller mount points next.
mount -t cgroup,cgroup2
cat /proc/self/cgroup

Do not assume V2 merely because /sys/fs/cgroup exists. V1 and V2 use different filenames and, in several places, different semantics. Mixing commands produces either No such file or directory or a wrong diagnosis.

Key differences between V1 and V2

Area cgroup V1 cgroup V2
Hierarchy Controllers can use separate trees All controllers share a unified hierarchy
Process placement CPU and memory controllers may place one process at different paths A process has one position in the unified tree
CPU ceiling cpu.cfs_quota_us + cpu.cfs_period_us cpu.max
CPU weight cpu.shares cpu.weight
Memory ceiling memory.limit_in_bytes memory.max
Memory throttling memory.soft_limit_in_bytes, with weak semantics memory.high, which drives reclaim and throttling
IO blkio.* io.*
Pressure metrics Primarily global PSI Per-cgroup cpu.pressure, memory.pressure, and io.pressure

V2 also has the No Internal Process Constraint. For domain controllers, an internal node that enables child controllers normally cannot also contain ordinary processes. Delegation and threaded cgroups add exceptions, so “all processes must live in leaf nodes” is too broad.

Do not edit systemd- or kubelet-owned cgroups directly

Production cgroups are normally owned by systemd, the container runtime, or kubelet. Manual echo writes may be overwritten during the next reconciliation and can violate runtime assumptions.

For a local experiment, prefer a transient systemd scope:

systemd-run --scope \
  -p CPUQuota=50% \
  -p MemoryHigh=250M \
  -p MemoryMax=300M \
  bash

cat /proc/$$/cgroup

Write cgroup.subtree_control, cpu.max, or memory.max directly only inside a subtree that has been explicitly delegated to you.

CPU: separate relative weight from hard quota

cpu.weight is a relative share under contention, not a CPU ceiling. cpu.max is the V2 bandwidth ceiling and has this format:

$MAX $PERIOD

For example, allow 50ms of CPU time in each 100ms period:

echo '50000 100000' > cpu.max

The cgroup averages at most 0.5 CPU. Multiple runnable threads consume the same group quota in parallel, so they can exhaust it before the wall-clock period ends and then wait for the next period.

Start with cpu.stat:

cat cpu.max
cat cpu.weight
cat cpu.stat

Common V2 fields include:

  • usage_usec: accumulated CPU time.
  • nr_periods: quota periods observed.
  • nr_throttled: periods that experienced throttling.
  • throttled_usec: accumulated throttled time.

Do not alert merely because nr_throttled is non-zero. Measure deltas across the same window as the latency incident, then correlate them with request latency, runnable threads, GC, and batch work.

cpu.max.burst can permit bounded bursts, but it changes the quota model rather than creating free capacity. Validate neighboring workloads and total node contention before enabling it.

Memory: memory.high, memory.max, and OOM are different states

For V2 memory incidents, read at least:

cat memory.current
cat memory.high
cat memory.max
cat memory.events
cat memory.events.local
grep -E '^(anon|file|kernel|slab|sock|shmem) ' memory.stat
  • memory.current: memory currently charged to the cgroup.
  • memory.high: crossing it drives reclaim and throttling; it normally does not directly invoke OOM.
  • memory.max: hard ceiling. The kernel attempts reclaim first and invokes cgroup OOM only when usage cannot be reduced sufficiently.
  • memory.events: high, max, oom, and oom_kill counters, potentially including descendants.
  • memory.events.local: events local to this cgroup, without descendant aggregation.

Page Cache is charged as file memory, but it is normally reclaimable. When RSS is low but memory.current is high, compare anon, file, slab, shmem, and memory.events before deciding that the application leaked memory.

For a workload that should fail as one unit, evaluate memory.oom.group=1. It asks the kernel to kill the cgroup workload together during cgroup OOM instead of leaving a partially functioning service after one process dies.

IO and PID: use controller-specific evidence

V2 uses io.max to configure bandwidth or IOPS by device:

cat io.max
cat io.stat

A device is identified by major and minor number:

8:0 rbps=max wbps=10485760 riops=max wiops=max

Buffered-write timing, the filesystem, and the underlying device all affect observed behavior. Validate application latency, io.stat, and io.pressure together instead of inferring throughput from one configuration line.

The PID controller limits the number of tasks in a cgroup:

cat pids.current
cat pids.max
cat pids.events

At pids.max, process or thread creation may return EAGAIN. The max counter in pids.events proves that the controller rejected creations.

PSI: measure stalls caused by resource pressure

System-wide PSI lives under:

cat /proc/pressure/cpu
cat /proc/pressure/memory
cat /proc/pressure/io

cgroup V2 also exposes pressure per group:

cat cpu.pressure
cat memory.pressure
cat io.pressure

Example output:

some avg10=2.34 avg60=1.87 avg300=0.98 total=492817349
full avg10=0.41 avg60=0.20 avg300=0.09 total=91234567

some means at least one task is stalled by pressure. full means all non-idle tasks are stalled at once. Avoid a universal alert such as avg10 > 5; establish a service baseline and validate thresholds against SLOs, queue depth, and latency.

How Kubernetes maps resources to cgroups

On common Linux nodes:

  • A CPU request primarily affects relative weight; a CPU limit normally becomes a CFS bandwidth quota.
  • A memory limit normally becomes the cgroup hard memory ceiling.
  • Whether a memory request maps to memory.min or memory.low depends on Kubernetes, the runtime, and MemoryQoS configuration.
  • Init containers, sidecars, Pod-level resources, and QoS classification can affect the final hierarchy; the main container is not the whole picture.

Collect Kubernetes evidence first:

kubectl get pod -n <ns> <pod> -o wide
kubectl describe pod -n <ns> <pod>
kubectl get pod -n <ns> <pod> \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.containerID}{"\t"}{.lastState.terminated.reason}{"\n"}{end}'

Then use the container runtime to obtain the host PID, read /proc/<pid>/cgroup, and map it to the actual cgroup directory. Paths vary by runtime and cgroup driver; do not hard-code a kubepods.slice layout in operational scripts.

Two production runbooks

CPU latency spike

  1. Record deltas for nr_periods, nr_throttled, and throttled_usec across the incident window.
  2. Inspect cpu.pressure, runnable threads, and node CPU saturation over the same period.
  3. Separate hard quota throttling from node contention: the former uses cpu.max and throttling counters; the latter needs weight, PSI, and node-load evidence.
  4. Before changing the limit, verify concurrency, GC, batch work, and request bursts.

Pod OOMKilled

  1. Confirm the container exit reason, timestamp, and restart count from Pod status and Events.
  2. Read memory.events.local in the matching cgroup and check whether oom or oom_kill increased.
  3. Use memory.stat to separate anonymous memory, file cache, kernel memory, and shared memory.
  4. Inspect parent limits; a container can be below its own ceiling while a Pod or parent slice reaches another limit.
  5. Only then choose between fixing a leak, reducing cache or concurrency, or changing requests and limits.

Five-minute collection script

cg=${1:?usage: cgroup-snapshot.sh /sys/fs/cgroup/path}

for file in \
  cgroup.type cgroup.controllers cgroup.subtree_control \
  cpu.max cpu.weight cpu.stat cpu.pressure \
  memory.current memory.low memory.high memory.max \
  memory.events memory.events.local memory.pressure \
  io.stat io.pressure pids.current pids.max pids.events
do
  if [ -r "$cg/$file" ]; then
    printf '\n### %s\n' "$file"
    cat "$cg/$file"
  fi
done

The script is read-only and leaves systemd- or kubelet-owned state untouched. Compare the snapshot with application and node metrics from the same window to determine whether a limit is the cause, an effect, or unrelated evidence.

References