NVIDIA's new Nemotron 3.5 Lightning 30B advertises a one million token context, but the BF16 checkpoint only delivers the full window across eight H100 GPUs with tensor and expert parallelism. On a single 80GB H100 you are validated to 256K, so plan your hardware around the context you actually need before you promise a million.
When NVIDIA released Nemotron 3.5 Lightning 30B this week, the headline number was a one million token context window. That number is real, but the fine print in the model card matters enormously for anyone planning to self host it. The full million token window on the BF16 checkpoint is an eight GPU configuration, not something a single card delivers, and teams that skip this detail will provision one H100, load the model, and discover their context silently caps far below what the marketing implied.
The concrete figures are these. On a single 80GB H100 in BF16, NVIDIA validates the model to a 256K token context, and it explicitly calls the single card configuration memory bound at that length. To reach the full 1048576 token window from the same BF16 weights, the card recommends eight H100 GPUs running tensor parallelism across all eight along with expert parallelism, or a larger card such as GB200 or B200. In other words, the jump from a quarter million to a full million tokens is a roughly eightfold increase in GPU count, not a configuration flag you flip on one machine.
This is a mixture of experts model with three billion active parameters out of thirty billion total, so the weights themselves fit comfortably on one card. The constraint is not the model, it is the key value cache that a million token context demands, which grows with sequence length and quickly exhausts a single 80GB card. That is why the fix is parallelism across more memory rather than a smaller quantization trick, at least for the BF16 checkpoint documented here. Quantized variants change the math, so if you need a long window on fewer cards, that is the path to investigate rather than forcing the BF16 weights onto hardware that cannot hold the cache.
The practical takeaway is to size hardware to the context you will actually use. A large share of real workloads, even demanding agent loops over sizable codebases, live comfortably under 256K tokens, and for those a single H100 running the BF16 checkpoint is the efficient choice. Reserve the eight GPU deployment for the genuine long context cases, retrieval over huge document sets or very long running sessions, where the full million token window earns its cost. Decide which regime you are in before you buy or reserve the cards, not after.
Try it today
- Read the model card's validated context line before provisioning: a single 80GB H100 in BF16 is memory bound at 256K tokens, not 1M.
- If your workloads stay under about 256K tokens, deploy the BF16 checkpoint on one H100 and keep the cost low.
- For the full 1M window from BF16 weights, launch with eight H100 GPUs using the recipe: --tensor-parallel-size 8, --enable-expert-parallel, --max-model-len 1048576 (or use a GB200 or B200 class card).
- If you need long context on fewer cards, evaluate a quantized variant rather than trying to force the full BF16 window onto a single H100.