Back to Blog
vramfit
A series in 6 parts, published from to .
I shrank a 93 GB model onto a 24 GB card by measuring which layers survive being crushed, instead of guessing.
·
I claimed a smaller card gives you a different quantization recipe, not the same one squeezed, and then didn't prove it.
·
Measure a model's per-layer quantization damage, solve for a recipe that fits your card, and check the result before you trust it.
·
The smallest 2-bit-labeled build of Nemotron 3.5 Lightning is 17.54 GiB — it doesn't fit a 16 GiB card, and its label names 12 of 417 tensors.
·
The official Q4_0 build of Gemma 4 31B is 16.44 GiB and fits a 24 GiB card with room to spare, so "it fits" was never the claim.
·
Download the published fit24gib pack, install llama.cpp at the build the numbers were measured on, and serve it at 86,016 tokens of text context or 73,728 with an image aboard.
·