“How much GPU memory do we actually need?” sounds like a simple question. It isn’t, and getting the answer wrong in either direction is expensive: underprovisioned VRAM causes a training run to crash partway through or forces a model to split across more GPUs than necessary, while overprovisioned VRAM means paying for capacity a workload will never touch. This guide breaks down the actual formulas behind VRAM sizing, so a federal program can specify GPU memory based on real calculation rather than a vendor’s default recommendation.
VRAM usage breaks down into four distinct categories, and which ones matter most depends entirely on whether a workload is running inference or training.
As a practical rule of thumb, inference in FP16 precision requires approximately 2 GB of VRAM per billion model parameters, plus 15 to 20 percent overhead for KV cache, activations, and framework runtime. Quantized formats reduce that substantially, roughly 0.5 GB per billion parameters at INT4 precision, though at some cost to output quality.
Training requirements scale considerably higher. Fine-tuning with LoRA, a parameter-efficient technique, adds roughly 50 percent headroom over base inference requirements. Full training using the Adam optimizer requires approximately 16 bytes of memory per parameter, accounting for the model weights, gradients, and optimizer state combined, a dramatically higher requirement than inference alone.
Model Size | Inference (FP16) | Fine-Tuning / Full Training |
7B parameters | ~14-15 GB | Fine-tuning: ~21-22 GB; full training: well over 100 GB |
13B parameters | ~24-26 GB | Fine-tuning: ~35-40 GB; full training: ~97 GB or more |
70B parameters | ~140 GB (often split across multiple GPUs) | Full training: far beyond single-GPU capacity; requires multi-GPU or heavily quantized deployment |
These figures illustrate why a workload assessment matters more than a vendor’s default recommendation. A 70-billion-parameter model running inference and the same model undergoing full training represent an order-of-magnitude difference in memory requirement, and specifying hardware without first confirming which of those two scenarios actually describes the workload is one of the most common and costly mistakes in AI infrastructure procurement.
VRAM capacity sets the floor for whether a model fits on a given GPU at all. Memory bandwidth determines how fast that GPU can actually move data once the model is loaded, and for many AI workloads, bandwidth drives real-world throughput more than raw computational capability does. A GPU with higher theoretical compute but slower memory bandwidth can underperform a GPU with lower peak compute but faster memory access, particularly for large-model inference where the GPU spends much of its time moving data rather than computing on it.
Ace Computers’ workload assessment process calculates actual GPU memory requirements based on your program’s specific model size, precision requirements, and inference versus training needs, rather than defaulting to a standard configuration that may significantly over- or under-provision memory for your actual workload.
→ Contact Ace Computers Federal Sales Team
Approximately 14 to 15 GB in FP16 precision. Quantized formats can reduce this significantly, though with some tradeoff in output quality.
Training requires storing gradients and optimizer state alongside the model weights themselves, and activations for every layer must be retained for backpropagation. Using the Adam optimizer, full training can require roughly 16 bytes of memory per parameter, compared to roughly 2 bytes per parameter for FP16 inference.
Yes. The KV cache, which stores attention states during inference, scales directly with context length, batch size, and the number of attention layers in the model, meaning longer context windows require meaningfully more memory beyond the base model weight calculation.
Yes. Ace Computers’ workload assessment process calculates GPU memory requirements based on your program’s specific model size, precision needs, and whether the workload is inference, fine-tuning, or full training.
