Why RAM Exhausts When Running Gemma 4 with llama.cpp

A model was loaded onto a GPU with 32GB of VRAM. There is still ample headroom in the VRAM usage, yet after sending only a few prompts, the process was forcibly terminated—the cause was not the GPU, but exhaustion of system RAM.

In overseas Reddit communities (r/LocalLLaMA), there has been discussion regarding an issue where running Gemma 4 31B using llama.cpp results in abnormal consumption of system RAM. A reporter noted that despite having a generous environment with 32GB VRAM and 64GB system RAM, llama.cpp was forcibly terminated by Linux’s OOM Killer. Multiple users have confirmed similar phenomena, and reports indicate the issue is reproducible even on RTX 5090 environments.

In the world of local LLMs, “whether VRAM is sufficient” often becomes the primary concern, but this case challenges that common assumption. This article delves into the cause of this phenomenon and organizes countermeasures available at present.

The Phenomenon of System RAM Exhaustion Instead of VRAM

First, let us accurately grasp the reported symptoms.

The Reddit poster’s environment consists of 32GB VRAM and 64GB system RAM (DDR5). Upon loading the Gemma 4 31B Unsloth quantized model (UD_Q5_K_XL) with a context length of 102,400 tokens (approx. 100k), the model itself fit within VRAM, and there was headroom in VRAM usage immediately after loading.

The problem manifests from this point onward. After sending several prompts, system RAM usage rises sharply to reach 63GB out of 64GB, resulting in Linux’s OOM Killer forcibly terminating the llama.cpp process.

An interesting detail is that the poster subsequently verified this on another PC equipped with 128GB RAM (DDR4). While it did not crash immediately there either, reports indicate that after processing just a few prompts of tens of thousands of tokens each, RAM usage rose to 80GB and was still trending upward.

Furthermore, lowering the quantization level does not resolve the issue. Switching to Q4 quantization reduced VRAM usage to approximately 23GB, but the abnormal consumption of system RAM remained unchanged. This fact suggests that the root cause lies in a memory area other than model weight size.

Before attempting countermeasures, please verify the following environment information first:

• OS: Linux (environment where OOM Killer activates) / Windows (manifests as swap bloat)
• GPU: Model name and VRAM capacity (confirmable via nvidia-smi)
• System RAM: Installed capacity and current usage (check with free -h)
• llama.cpp: Build date or commit hash (check with the --version option)
• Model Used: Model name, quantization format, and context length settings.

Why System RAM is Consumed in Large Amounts

The phenomenon of system RAM exhaustion despite having ample VRAM seems counterintuitive; the author believes this relates to llama.cpp’s memory management structure.

Relationship Between KV Cache and Context Length

When running an LLM with llama.cpp, memory is largely divided into two uses. One is model weights (parameters), which are primarily placed in VRAM. The other is the KV cache, a region used to retain past token information during inference.

KV cache consumption increases proportionally with context length. If the context length is set to 100k tokens, a correspondingly massive KV cache area becomes necessary.

A point raised in comments on Reddit is the existence of “context checkpoints.” In llama.cpp, checkpoints are created approximately every 8,192 tokens; for MoE models, this amounts to about 533MB per instance, and it may be even larger for Dense models. With a 100k context, roughly 12 checkpoints occur, calculating to occupy over 6GB of memory just from these alone. Moreover, there is a high probability that these accumulate on the system RAM side rather than VRAM.

Potential Differences Between Dense and MoE Behaviors

The Gemma 4 family includes two variants: the Dense-based 31B (the subject of this article) and an MoE-based 26B variant (activating 8 out of 128 experts). Memory behavior may differ depending on the architecture. In MoE, a structure exists where different experts (sub-networks) are activated based on input, requiring additional buffers for expert switching during inference. Conversely, Dense models pass all parameters every token, making memory pressure from KV cache the primary cause.

The Gemma 4 31B used by the Reddit poster is considered a Dense model; thus, it is highly likely that system RAM consumption was driven not by MoE-specific expert buffers but by the accumulation of KV caches and checkpoints.

However, there is currently insufficient backing from clear technical documentation regarding this point; it remains at the stage of speculation within the community.

Organizing Reproduction Conditions and Scope

Summarizing the reported conditions in a table reveals the contours of the problem.

There are three key points to note.

The first is that RAM consumption does not improve even when lowering the quantization level. Changing from Q5 to Q4 reduced VRAM usage to about 23GB, but abnormal system RAM consumption remained unchanged. This serves as evidence of factors independent of model weight size at work.

The second point is reproducibility across multiple machines. The reporter verified this on two PCs with different hardware configurations and confirmed identical behavior. It is likely not a specific hardware or driver issue, but rather stems from memory management within llama.cpp itself.

The third point concerns the current instability of Gemma 4 in general local deployment. On r/LocalLLAMA, reports have also surfaced regarding issues such as tool calling functions failing to operate correctly and “lazy” behavior during inference (e.g., skipping necessary external tool calls). While not directly causally linked to the RAM exhaustion issue, this serves as circumstantial evidence that the model itself is still in a developmental stage.

Countermeasures Available at Present

A fundamental fix requires waiting for the llama.cpp development team, but there are several measures users can attempt currently.

Guidelines for Context Length and RAM Capacity

The most reliable countermeasure is to shorten the context length (-c option). The reporter themselves stated that even with headroom in VRAM, they had no choice but to lower the context.

Please refer to the following figures as a guideline (these are estimates at present and may vary with future llama.cpp updates).

If you wish to fully utilize a 100k context, even 128GB of system RAM may be insufficient. In reality, keeping the context length around 32k ~ 64k would likely constitute a safe operational line.

The reporter’s usage parameters were -ngl 999 -c 102400 -fa on –cache-type-k q8_0 –cache-type-v q8_0. Among these, the most effective measure is lowering the value of -c. Try changing it to -c 32768 or -c 16384 first and check if RAM consumption stabilizes.

Options Available in llama.cpp Settings

Beyond shortening the context length, there are several settings worth trying.

1. Lower KV Cache Quantization Level: The reporter used –cache-type-k q8_0 –cache-type-v q8_0. Changing this to q4_0 may reduce memory consumption of the KV cache by approximately half. However, the impact on output quality varies depending on the model and task.
2. Adjust Batch Size: Reducing -b (batch size) or -ub (micro-batch size) may reduce temporary buffers during processing. Another option is to lower from default values to -b 512 or -b 256 and observe.
3. Expand Swap Area: While not a fundamental solution, adding a swap file on Linux can delay the activation of OOM Killer. However, inference speed will drop significantly if access to swap occurs; thus, this is merely an emergency measure.

Note that llama.cpp is a project under active development; multiple builds have been released as of April 2026, including b8838 (confirmed release on 4/18). Builds supporting various GPU accelerators like CUDA 12/13 and ROCm 7.2 are available for macOS, Linux, and Windows platforms, indicating vigorous development progress. Regarding this RAM exhaustion issue, discussions have already begun in llama.cpp’s GitHub Discussions, so there is a high possibility of improvement in future versions.

In our site’s verification environment, VRAM usage when running gemma4:26b (MoE variant) via Ollama was 14.8GB, with an inference speed of 36.6 tokens/sec recorded. However, this measurement is based on Ollama’s default context length and differs significantly from conditions where a 100k context is set in llama.cpp. If the context length is kept short, data suggests that even in a 16GB VRAM environment, inference for Gemma 4 MoE variants can operate at sufficiently practical speeds. Please note that the RAM exhaustion discussed in this article is a phenomenon prominent when operating Dense 31B models with long contexts and differs from conditions where it occurs with the MoE variant 26B.

How Much System RAM is Needed for Local LLM Environments?

This case contains an important lesson regarding hardware configuration for local LLMs. Focusing solely on “VRAM capacity” can lead to unexpected bottlenecks.

The memory in a local LLM environment should be understood as having essentially three layers:

Layer 1: VRAM (GPU Side) — Stores model weights and the main portion of KV cache. This is the most critical resource directly linked to inference speed.

Layer 2: System RAM (CPU Side) — Used for model loading processes, KV cache checkpoints, temporary buffers during inference, etc. It also serves as a spill-over destination when data overflows from VRAM.

Layer 3: Storage (Swap) — The final fallback location when RAM is insufficient. Speeds are orders of magnitude slower; relying on this drastically reduces practical usability.

The following table shows estimated system RAM guidelines by model size, assuming long-context operation. These estimates were derived by the author based on source information and general memory consumption trends; please note they may vary depending on model architecture and quantization format.

When considering the configuration of an AI PC, one should be mindful not only of GPU VRAM budgets but also investment in system RAM. DDR5 memory prices are on a downward trend; as of April 2026, a configuration with two sticks (totaling 64GB) can be obtained for around $130 USD (approx. ¥20,000). If planning to operate with long contexts, it is safer to install at least 64GB from the start. For details on VRAM selection, please refer to our separate article explaining basic knowledge of VRAM.

NVIDIA is also advancing technical development for improving VRAM efficiency; new technologies like RTX Neural Texture Compression are beginning to contribute to reducing VRAM usage. However, it must be distinguished that these have no direct effect on system RAM-side issues and remain technologies within the context of VRAM optimization.

Conclusion

The issue where running Gemma 4 with llama.cpp using long contexts leads to system RAM exhaustion is believed to stem primarily from KV cache checkpoints and buffers accumulating on the system RAM side rather than VRAM issues.

Organizing the priority of countermeasures yields the following flow:

1. First, shorten context length (-c) to 32768 or less and check if symptoms improve.
2. If not improved, lower KV cache quantization level to q4_0.
3. If system RAM is under 64GB, consider upgrading to 128GB.
4. Update to the latest llama.cpp build and wait for fixes in future versions.

In designing local LLM environments, attention often focuses solely on “whether VRAM is sufficient,” but system RAM is also a critical resource. Especially when handling models of 30B class or larger with long contexts, having at least 64GB of system RAM is becoming a de facto requirement.

Have you considered the usage of system RAM in your environment? The era where RAM becomes a bottleneck after VRAM may be approaching.

Frequently Asked Questions (FAQ)

Q: Does this problem occur only with Gemma 4?

A: While prominently reported for Gemma 4, it is a phenomenon that can occur with other models if combining large context sizes with large parameter models. There are indications that MoE architecture models have particularly high memory consumption for checkpoints; thus, caution is required for models sharing similar architectures.

Q: Does the same problem occur on Windows?

A: While forced termination by an OOM Killer like in Linux does not occur, RAM exhaustion itself can happen regardless of OS. On Windows, swap files (pagefiles) automatically expand, making immediate crashes less likely; however, inference speed drops drastically if access to swap increases. It is advisable to monitor memory usage via Task Manager while operating.

Q: Can this be avoided by using Ollama instead of llama.cpp?

A: Since the backend for Ollama utilizes GGML/llama.cpp technology, fundamental memory management behavior may share similarities. However, because Ollama is set with relatively short default context lengths, one would likely not encounter similar issues unless explicitly specifying extreme long contexts like 100k.

This site participates in the Amazon Associates Program. As an Amazon Associate, we earn from qualifying purchases.

This article was written by the AI Hardware Zukan Editorial Team based on information available at the time of writing. Evaluations may change due to product updates or fluctuations in third-party benchmarks, prices, and supported runtimes. We recommend re-evaluating content after a certain period has passed.

References

• llama.cpp Official: GitHub Repository
• llama.cpp Official: GitHub Discussions
• Hugging Face Official: Google Gemma Model Card
• Unsloth Official: Gemma Execution Guide
• Ollama Official: GitHub Repository