Linux Heap Memory: Arenas, Chunks, Bins, and Tcache

RSS that stays high after objects are freed does not automatically prove a memory leak. glibc may cache free Chunks, Arenas can retain fragmented pages, and anonymous mappings may remain in place. The reverse is also true: stable RSS does not prove the absence of a leak when the process is reusing pages it already owns.

This article skips outdated heap-exploitation recipes and answers the production questions that recur: where malloc obtains memory, why Arenas and Tcache affect RSS, what free actually changes, and how to build a verifiable evidence chain.

Separate four kinds of memory growth first

Classify the growth before changing code, allocator tunables, or container resources.

How malloc obtains memory from the kernel

glibc malloc primarily obtains address space from two kinds of regions:

• The main Arena can extend the traditional Heap, the program-break region shown as [heap].
• Larger allocations, secondary Arenas, and other cases can use anonymous mmap regions.

Do not put “allocations above 128 KiB always use mmap” in a runbook. mmap_threshold depends on the glibc version, allocation history, and tunables. Setting a fixed threshold also disables parts of the dynamic adjustment.

Inspect process mappings:

grep -E '\[heap\]|rw-p.*00:00' /proc/<pid>/maps
grep -E '^(Rss|Pss|Private|Anonymous|Swap):' /proc/<pid>/smaps_rollup

maps describes regions; smaps_rollup provides aggregate residency. Neither tells you which C or C++ objects are still live, so pair them with allocator or application evidence.

What Arenas, Chunks, Bins, and Tcache do

Arenas reduce allocation-lock contention

Each Arena has independent allocator state and locking. The main Arena commonly uses the traditional Heap, while secondary Arenas normally obtain memory through mmap. Threads are not permanently assigned one Arena each; glibc creates and reuses Arenas under limits influenced by the platform and tunables.

More Arenas can reduce lock contention while increasing fragmentation and resident memory. glibc.malloc.arena_max is an experiment for bounding Arena count, not a universal instruction to use 2 or 4.

Chunks are allocator-managed blocks

glibc stores Chunk metadata near user data: size, neighboring-state flags, and pointers needed while a Chunk is free. Allocated and free states reuse parts of the layout, so an overflow, Use-After-Free, or Double-Free can corrupt allocator state.

Modern glibc includes consistency checks and mitigations such as Safe-Linking. They can make some corruption fail earlier or become harder to exploit; they do not make memory-unsafe code safe.

Bins organize reusable free Chunks

Free Chunks can move through Tcache, Fastbins, the Unsorted Bin, Small Bins, or Large Bins depending on size and state. Exact boundaries, checks, and routing change with the architecture and glibc release. Production debugging should not depend on one fixed “Chunk size chart.”

Two behaviors matter most:

1. free normally returns a Chunk to the allocator before it returns pages to the kernel.
2. Later malloc calls can reuse those Chunks and avoid system calls while RSS remains unchanged.

Tcache is a per-thread small-allocation cache

Tcache lets common-size allocations and frees avoid the Arena lock. Long-lived threads each retaining a cache means thread count can affect reclaimable memory and RSS.

glibc.malloc.tcache_count=0 disables Tcache for an A/B comparison, but it may change throughput and lock contention substantially. Use it for diagnosis, not as the default response to high RSS.

Reproducible lab: why RSS can remain after free

Save this as heap-lab.c:

#define _GNU_SOURCE
#include <errno.h>
#include <malloc.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

static size_t parse_size(const char *text, size_t fallback) {
    if (text == NULL) {
        return fallback;
    }

    errno = 0;
    char *end = NULL;
    unsigned long long value = strtoull(text, &end, 10);
    if (errno != 0 || end == text || *end != '\0' || value == 0 || value > SIZE_MAX) {
        fprintf(stderr, "invalid positive integer: %s\n", text);
        exit(2);
    }
    return (size_t)value;
}

static void pause_at(const char *stage) {
    printf("pid=%ld stage=%s; press Enter\n", (long)getpid(), stage);
    fflush(stdout);
    (void)getchar();
}

int main(int argc, char **argv) {
    size_t count = parse_size(argc > 1 ? argv[1] : NULL, 1024);
    size_t block_size = parse_size(argc > 2 ? argv[2] : NULL, 65536);

    if (count > SIZE_MAX / sizeof(void *)) {
        fputs("allocation table is too large\n", stderr);
        return 2;
    }

    void **blocks = calloc(count, sizeof(*blocks));
    if (blocks == NULL) {
        perror("calloc");
        return 1;
    }

    for (size_t index = 0; index < count; index++) {
        blocks[index] = malloc(block_size);
        if (blocks[index] == NULL) {
            perror("malloc");
            return 1;
        }
        memset(blocks[index], 0xa5, block_size);
    }
    pause_at("allocated");

    for (size_t index = 0; index < count; index += 2) {
        free(blocks[index]);
        blocks[index] = NULL;
    }
    pause_at("half-freed");

    for (size_t index = 1; index < count; index += 2) {
        free(blocks[index]);
    }
    pause_at("all-freed");

    printf("malloc_trim released memory: %s\n", malloc_trim(0) ? "yes" : "no");
    (void)malloc_info(0, stdout);
    pause_at("after-trim");

    free(blocks);
    return 0;
}

Compile and run it. The defaults touch about 64 MiB of user data:

gcc -O2 -g -Wall -Wextra heap-lab.c -o heap-lab
./heap-lab 1024 65536

In another terminal, collect each stage:

pid=<heap-lab-pid>
grep -E '^(Rss|Pss|Private|Anonymous|Swap):' /proc/$pid/smaps_rollup
pmap -x $pid | tail -n 1

Typical observations:

• RSS rises after allocated because memset faults in the pages.
• After half-freed, free Chunks are reusable but fragmentation can prevent page release.
• After all-freed, active objects are gone while RSS may remain above its initial value.
• malloc_trim(0) may release free pages; a 0 return is not an error, only “nothing released this time.”

Do not call malloc_trim after every request. It changes latency and system-call behavior and belongs only at benchmarked batch points or in a diagnostic experiment.

Use tunables for comparisons, not direct production changes

Run the same workload three ways:

./heap-lab 1024 65536

GLIBC_TUNABLES=glibc.malloc.arena_max=2 \
  ./heap-lab 1024 65536

GLIBC_TUNABLES=glibc.malloc.tcache_count=0 \
  ./heap-lab 1024 65536

Compare:

• peak and steady-state RSS
• Arena and mapping totals from malloc_info
• request throughput and P50/P99 allocation latency
• CPU time and lock contention

If RSS drops while tail latency or CPU cost rises, the original configuration was trading memory for allocation concurrency. Whether that trade is acceptable depends on the container ceiling, node density, and service SLO.

Allocator counters are not total process memory

Useful interfaces include:

• malloc_info(0, stream): XML with Arena and mmap summaries.
• mallinfo2(): allocator counters without the integer-overflow problem of legacy mallinfo().
• malloc_stats(): allocator statistics written to standard error.
• malloc_trim(0): attempt to release reclaimable pages.

These APIs describe memory managed by glibc malloc, not thread stacks, JIT regions, other allocators, driver mappings, or all shared memory. Always compare them with RSS, smaps_rollup, and cgroup memory.current and memory.stat.

Use Sanitizers for memory errors instead of allocator guesswork

AddressSanitizer directly identifies heap overflows, Use-After-Free, and some Double-Free cases:

gcc -O1 -g \
  -fsanitize=address,undefined \
  -fno-omit-frame-pointer \
  app.c -o app-asan

ASAN_OPTIONS=detect_leaks=1 ./app-asan

When rebuilding is not possible, use Memcheck:

valgrind --leak-check=full \
  --show-leak-kinds=all \
  --track-origins=yes \
  ./app

Sanitizers and Memcheck belong in testing and reproduction environments, not as substitutes for production monitoring. They report invalid access and leak evidence; RSS retention still requires allocator-cache and fragmentation analysis.

Production debugging order

1. Identify the process, container, and cgroup responsible for growth; do not stop at the Pod total.
2. Use smaps_rollup to separate anonymous residency, file mappings, and Swap.
3. Use object metrics, a Heap Profile, or LeakSanitizer to determine whether live objects keep increasing.
4. Use malloc_info, thread count, and tunable A/B tests to determine whether Arenas or Tcache amplify RSS.
5. Verify request concurrency, cache capacity, queue depth, and object lifetimes against the design.
6. Change arena_max, Tcache, or trimming only when allocator evidence supports it.

Fix lifetime bugs and unbounded growth first. Allocator tuning changes caching, fragmentation, and lock contention; it cannot repair a real leak.

References

• glibc Memory Allocation Tunables
• malloc_info(3)
• mallinfo2(3)
• malloc_trim(3)
• Linux /proc Filesystem
• GCC Instrumentation Options
• Valgrind Memcheck