alberto.codes

Gemma 4 read a love note as a scam. A rewritten question brought it within two messages of Jev.

Thu, 01 Oct 2026 00:00:00 GMT

"I want to show you the world, princess :) how about europe?"

That is a text message from a public scam dataset, and it is not a scam. I asked Gemma 4 31B, running on the graphics card under my desk, whether it was a scam, phishing or social-engineering attempt. It said yes, 99.98 percent sure.

Then I let an optimizer rewrite that one question. Asked the new way, Gemma put the same message under 0.001 percent. Across 158 messages it had never seen, it went from 131 right to 153, two short of Jev's 155.

The test

In the Jev calibration post I asked TypeSafe's Jev one yes-or-no question about text messages from DIFrauD, a public set labelled scam or not. In the last post I built typevet, which asks the same kind of question of Gemma 4 on my own card.

So I put Jev and the Jev at home side by side. Same question, same 158 messages, 36 of them scams:

Is this message a scam, phishing or social-engineering attempt?

Jev got 151 right. Gemma got 131. It missed no scams at all, but it flagged 27 ordinary messages as scams. Love notes. "Got meh... When?" "I fetch yun or u fetch?" Gemma read anything personal and a little cryptic as somebody working an angle.

Letting each model rewrite its question

The question is one sentence. Small wording changes move these models a lot, so I let an optimizer search for a better sentence, once for each model.

The optimizer is GEPA, run through gepa-adk, my package for it. It works like an editor with a red pen. The model answers a few practice messages. A second model, Qwen3.8-27B on the same desk, reads how those answers scored and proposes a rewrite. The winning rewrite is the one that scores best on 200 separate messages. Every rewrite had to fit in 94 characters, so it could not grow into an essay.

Each model got its own run with the same settings and data. Gemma's took 34 minutes on my card. Jev's took 27 minutes. Each run made 2,280 calls. The 158 messages in the test were never shown to either run.

The two rewrites went opposite ways

Gemma's rewrite narrowed the word "scam". Obvious deception only, and ignore personal text. With that wording, every love note in the test came back as not a scam.

Jev's rewrite widened it. It added unsolicited promotions, offers, calls and alerts. DIFrauD counts a lot of spam as scam, and Jev's rewrite suggests its optimizer picked that up. A real-estate promotion that ends "For Best Deal Call" is labelled a scam in the dataset. Jev moved it from 32 percent to 89.

The result

Judge	Right, same question	Right, own rewrite	Average gap, same question	Average gap, own rewrite
Jev	151 of 158	155 of 158	8.9 points	7.1 points
Gemma 4 31B on my RTX 4090	131 of 158	153 of 158	17.0 points	3.4 points
Gemma 4 31B on a rented H100	128 of 158	153 of 158	18.5 points	3.4 points

The average gap is the calibration measure from the Jev post: how far a model's percentages sit from what actually happened. Zero is perfect.

Jev still gets the most messages right. Gemma's percentages now mean more: on that measure they sit closer to what actually happened than Jev's do, an average gap of 3.4 points against 7.1. Jev keeps a small edge on the other scores in the study, and every gap here is small at this size.

Learned on my card, worked on the big one

The rewrite was learned on the 24 GiB pack on my RTX 4090, which stores most weights at 4 bits and some at 2 or 3. I then ran it on Google's full-precision Gemma 4 31B on a rented H100.

Same answer, scam or not, on all 158 messages. The same 153 right, the same 3.4-point gap. A wording found on a compressed model at home carried over to the full model on a datacenter card without losing a single message.

The H100 was also the fast one. A message came back in 0.17 seconds at the median there, 1.1 seconds on my desk, and 0.1 seconds from Jev's hosted service.

The five that are left

Two are promotions that DIFrauD calls scams and Gemma's narrower question now lets through, like the real-estate one above. The other three are these, labelled not a scam in the dataset:

"Send me your id and password"
"What's ur pin?"
"Perhaps * is much easy give your account identification, so i will tomorrow at UNI"

Gemma says scam to all three, and so would I. So did Jev with the original question.

Scope

158 messages with 36 scams, one run for each judge. At this size one message moves accuracy by more than half a point. Each rewrite is tuned to DIFrauD's idea of a scam, so it describes this dataset, not scams in general.

Where it lives

The study: Gemma 4 and Jev on DIFrauD, with the settings, every number above and the per-message receipts.
The optimizer: gepa-adk, Apache-2.0 licence.
The model: gemma-4-31B-it-fit24gib-GGUF, the 24 GiB pack the rewrite was learned on.
The release: typevet v0.5.0, the version that shipped this study: the Jev judge option for wording evolution, the held-out comparison harness, and native Gemma 4 framing for text runs.

One sentence, rewritten once on my own card, took Gemma from 131 to 153. The princess can go to Europe.

My Gemma was 99.96 percent sure the total matched. It had not seen the receipt.

Tue, 29 Sep 2026 00:00:00 GMT

The claim said the receipt came to 646,329. The receipt says 664,329. Two digits swapped, the kind of slip anyone makes typing a number in a hurry.

I asked Gemma 4 31B, running on the graphics card under my desk, whether the claim matched the receipt. When I sent the claim without the photo, it said the total matched, 99.96 percent sure. It had never seen the receipt. When I attached the photo, it said the total did not match, 99.999 percent sure.

Both answers came out of typevet, a library I made public today.

In the Jev calibration post I described a model you do not talk to. You write the test it takes, like a fill-in-the-bubble sheet, and it tells you how likely each bubble is. Jev is TypeSafe's hosted model, built for exactly that job. It also reads nothing but text. TypeSafe's docs say so plainly: "Jev accepts text only. ... Images, audio, and video are not supported (yet)."

The questions I wanted to ask were about pictures. So I built typevet to ask the same kind of question of an open model that can see, running on my own card. Call it Jev at home. Jev at home can look at a receipt.

What typevet does

typevet has three question types, the same three shapes Jev answers:

Yes or no: is this message a scam?
Pick one label: does the claim match the receipt, not match it, or is there not enough to tell?
Pick one level on a scale: how well does this answer follow the rubric?

typevet does not let the model write an answer and then try to parse it. Before the model writes anything, typevet reads how likely each allowed answer is as the very next word. The labels you did not offer get no vote. What comes back is a label with a probability for every option, or an error. It is never a paragraph you have to interpret.

Here is the question the receipt test asked, shortened where you see .... It is data, not a prompt I wrote around the model:

Choice(
    instructions=(
        "Look at the claimed amount in the expense claim before you look at "
        "the receipt. ... Compare with the receipt total only when the claim "
        "shows every digit of its amount. Otherwise choose insufficient_evidence."
    ),
    criteria={
        "insufficient_evidence": "The claimed amount is not fully readable ...",
        "mismatch": "The claim shows every digit of its amount and that amount differs from the receipt total.",
        "match": "The claim shows every digit of its amount and that amount equals the receipt total.",
    },
)

The photo rides along with the text. The model sees both, and typevet reads the three probabilities.

The receipt test

I wanted a test I could check by eye. CORD is a public set of real shop receipts, photographed and labelled with their totals. I took six of them and wrote three made-up expense claims for each:

one that states the receipt's real total,
one that states a wrong total, with two digits swapped,
one with digits smudged out, written as ?, so no total can be read.

That is 18 claims. I sent each one twice: once as text alone, and once with the receipt photo attached.

	Claim sent without the photo	Claim sent with the photo
Right total, said match	6 of 6	6 of 6
Wrong total, said does not match	0 of 6	6 of 6
Smudged total, said not enough to tell	6 of 6	6 of 6

Without the photo, the model said match to every claim that showed a full number, twelve of twelve, including all six wrong ones. On the wrong ones it was at least 99.96 percent sure. With the photo, it caught every swapped total, each at 99.998 percent or more.

It also refused to guess on the smudged claims in both runs. The question tells it that a ? is a lost digit, and it listened.

Why the photo is the whole point

A probability tells you how sure the model is. It does not tell you what the model looked at. Without the photo, that 99.96 percent rests on nothing but a tidy claim. With the photo, the 99.999 percent rests on the total printed on the receipt. A model that can see is the difference, and a text-only model cannot give you that.

typevet's tests make sure the photo is really there. In this run, attaching a receipt added between 228 and 1,108 prompt tokens, and the tests fail if the count does not grow.

Why Gemma 4, and why on my own card

Gemma 4 31B reads text and images. It is released under the Apache 2.0 licence, and it placed third among open models on LMArena when it launched. I chose it because it reads pictures, its licence lets me build on it, and it can be made to fit on one consumer card.

It fits on mine because of the 24 GiB pack I built for the last vramfit post. The file the receipt test loaded is byte for byte the one published on Hugging Face, served by llama.cpp on an RTX 4090. Nothing in the test left the machine.

I would rather not send a receipt to somebody else's server to find out whether two numbers agree.

The same questions on a rented H100

For hosted work, typevet talks to vLLM. On one rented H100 running the full-precision Gemma 4 31B, the receipt test got all 18 claims right with the photo. Reversing the order of the three answer options changed none of the 18 answers.

For speed, I sent 480 public bank support messages through it, two questions each. One at a time, a message came back in 0.24 seconds at the median. With 64 in flight, the server answered 39.6 messages a second, with no errors.

Who else does this

I am not the only person who wanted typed answers about pictures. Jev launched on 2026-09-15, and open takes on the idea followed fast.

TypeLLM is where typevet's text side comes from: how a JSON Schema becomes a set of decisions, and how each decision is scored from the next-word probabilities. TypeLLM added image input on 2026-09-24, on SGLang, tested with Qwen3.8-27B. It returns probabilities for yes-or-no and pick-one fields, and its README example reads a receipt too. typevet's image path was built separately and shares no code with it. I worked it out on llama.cpp's own request format, for Gemma 3 first and Gemma 4 the same day.
A Jev-like wrapper for LLMs, including vision models, a blog post from 2026-09-25, rebuilds Jev's three question types over webcam images in one script, with Gemma 4 12B on llama.cpp and a 24 GiB card.
VQAScore has read the probability of "Yes" from a vision model since 2024, to score whether an image matches a caption.

What typevet brings is a library, not a script. It runs the larger Gemma 4 31B on one 24 GiB card with llama.cpp, and the same code serves a rented H100 through vLLM. And its tests check that each image reached the model before any answer counts.

Scope

The receipt test is 18 made-up claims on six real receipts, one run on each server. It shows the photo reaching the model and deciding the answer. The probabilities are the model's own confidence, not a calibrated forecast. Checking calibration, as in the Jev post, takes hundreds of labelled examples.

Where it lives

The code: typevet on GitHub, MIT licence.
The docs: alberto-codes.github.io/typevet, including how the image reaches each server and the public receipt for the receipt test's pass.
The model: gemma-4-31B-it-fit24gib-GGUF, the 24 GiB pack the receipt test ran on.
The release: typevet v0.1.0, the first public version: yes-or-no, pick-one-label and scale questions over text and images, on llama.cpp and vLLM, plus JSON output checked against a schema you supply.

Same model, same claim, the same near-certainty both times. The photo is what made it right.

Jev looked underconfident on Banking77. Two of my labels were the reason.

Sun, 27 Sep 2026 00:00:00 GMT

When a weather forecast says 30 percent chance of rain, it should rain on about three of every ten days it says that. If it rains on nine of those ten days, the forecast is too cautious. If it rains on none, it is too alarmed. Forecasters call this being calibrated: the percentages mean what they say.

I wanted to know whether that holds for Jev, a model from TypeSafe that answers yes-or-no questions with a percentage instead of a yes or a no. So I asked it one question about a thousand public messages and compared its percentages with what the messages actually were.

On one set of messages, it looked badly miscalibrated. Where Jev said 20 to 50 percent, the real answer turned out to be yes almost nine times in ten. That looked like a forecaster saying "probably not" before a week of rain. Then I read the messages, and the mistake was mine.

What Jev says it does

Jev does not chat, write or explain itself. You give it some text and a question, and it returns a number. For a yes-or-no question, TypeSafe's docs define that number as the probability that the answer is yes.

When I explain it to people, I say you don't talk to Jev. You write the test it takes. Think of the fill-in-the-bubble exams from school, the Iowa tests or an SAT answer sheet. Your job is to write the question and the answer choices. Jev fills in the bubbles, except that instead of filling in one, it tells you how likely each choice is. A yes-or-no question is a test with two bubbles. With Jev, the work is in writing the question, not in a conversation.

TypeSafe says its models are "trained for calibrated decisions", and it is careful about what that promises: "Calibration is measured across groups of predictions; it does not guarantee that an individual answer is correct." One message scored at 30 percent can be yes or no. But across a large group of messages scored near 30 percent, about 30 percent should be yes. A group is something I can check.

The test

I used two free public datasets where people had already labelled every message.

DIFrauD: text messages, each labelled scam or not. I asked Jev: "Is this message a scam, phishing or social-engineering attempt?"
Banking77: short messages people send to a bank's support desk, sorted into 77 topics such as "card arrived" or "exchange rate". There is no scam label here, so I made one. I picked six topics that sounded like "someone took money I did not agree to" and counted them as yes. I asked Jev: "Does the customer report a transaction they did not authorize?"

Then I sorted Jev's answers into ten buckets: 0 to 10 percent, 10 to 20, and so on. For each bucket I compared the average percentage Jev gave with how often the answer was really yes. The average gap across all buckets is the calibration error. Zero would be perfect. I also measured something simpler: whether Jev at least scored the yes messages higher than the no messages. Call that the sorting score. 1.0 means every yes message scored above every no message.

The scam texts: close to the forecast

On the 500 text messages, Jev behaved like a good forecaster. The average gap was 7 percentage points, and the sorting score was 0.995.

The extremes were clean. Jev put 369 messages below 20 percent, and not one of them was a scam. It put 47 messages at 90 percent or above, and 46 of them were.

In the chart, Jev's percentage runs along the bottom and the real share of yes answers runs up the side. A dot on the dashed line means Jev's percentage matched reality for that bucket. A dot above the line means more yeses than Jev predicted. The left panel stays close to the line. The small wobbles in the middle come from small buckets: a few buckets hold only 7 or 8 messages, so one message moves the result by more than 10 points.

The bank messages: a forecaster saying "probably not" before the rain

On the 480 bank messages, the sorting score was still excellent at 0.986. Jev put the yes messages above the no messages almost every time. But the percentages themselves were off, with an average gap of 17 points, and always in the same direction. None of the no messages scored 50 percent or more. But 93 of the 240 yes messages scored below 50 percent. Jev looked too cautious, as the middle panel shows.

Splitting the yes messages by topic showed where the gap came from. Four topics matched what I had asked about. Two did not. Each topic had 40 messages:

Reading the messages

Here are six messages from those two topics. My answer key said yes to every one. Jev's percentage is beside each:

Someone who was charged twice agreed to the purchase. They are reporting a billing mistake, not a transaction they never made. Jev said "probably not", and for the question I asked, that is the right answer. My label said yes.

The "extra charge" topic is more mixed, and Jev kept up with it message by message. The two about a fee or purchase the customer does not recognise scored high. The two questions about a refund and about fees scored low. My label treated all 40 the same because they share a topic name.

In test terms, Jev answered the question printed on the page. The answer key I graded it with was written for a slightly different question.

The same answers, counted fairly

I kept every one of Jev's 480 answers and changed only my labels, counting those two topics as no. The average gap fell from 17 points to 6, which is better than on the scam texts. That is the right-hand panel of the chart.

I owe you one caveat about that 6. I chose to relabel those two topics after seeing the results, and I checked the fix on the same answers. It is not a fair new score for Jev. What it does show is that most of the 17-point gap came from my labels answering a different question from the one I asked Jev.

A third topic is borderline too. A compromised card does not always mean a transaction the customer did not make, and Jev put 10 of those 40 messages below 50 percent. I left that topic as yes. If I kept moving topics until the number looked good, I would be tuning my labels to flatter the model.

What the two scores told me

The only thing I changed was two topics' labels. Here is what that did to each score:

So the first result was not a measurement of Jev on its own. It measured Jev's answers against my idea of what the answers should be. On the scam texts, the labels came from the dataset's authors, answering nearly the same question I asked. On the bank messages they came from me, answering a slightly different question: which topics involve a charge the customer questions.

What this does not show

This is two groups of about 500 messages, one run each. Several buckets hold fewer than 20 messages, which is small.
The bank sample is half yes by design. In the full dataset, those six topics are 7.8 percent of messages, 240 of 3,080. A more natural mix would give a different number.
DIFrauD has several kinds of text. I used only the text messages.
The same messages sent three days apart gave nearly the same results: an average gap of 17 points both times on the bank messages, and 7 both times on the scam texts. A handful of messages moved between buckets.

For the curious

What	Detail
Model	TypeSafe `jev-1.13.0`, called on 2026-09-27
Client	judgevet 0.13.0, my Python client for Jev
Calls	980, all successful
Scam texts	DIFrauD, MIT licence: 500 messages from the SMS test split, 98 of them scams
Bank messages	Banking77, CC BY 4.0 licence: all 240 test messages from my six topics, plus 240 others chosen with a fixed seed
My six topics	`card_payment_not_recognised`, `direct_debit_payment_not_recognised`, `cash_withdrawal_not_recognised`, `compromised_card`, `extra_charge_on_statement`, `transaction_charged_twice`
Average gap	Expected calibration error (ECE), ten equal-width buckets weighted by size: 0.073 scam texts; 0.174 bank messages as first labelled; 0.056 relabelled
Sorting score	AUROC: 0.995 scam texts; 0.986 bank messages as first labelled; 0.972 relabelled

TypeSafe calls a yes-or-no question a Noul, so judgevet does too, and the answer's .noul field is Jev's percentage as a number between 0 and 1. One question is a few lines:

import os

from judgevet import HTTPSystemOneAdapter, Noul

question = Noul(instructions="Does the customer report a transaction they did not authorize?")

with HTTPSystemOneAdapter(api_key=os.environ["JEV_API__KEY"]) as client:
    response = client.system_one(
        state="I have multiple charges on the same transaction.",
        questions={"unauthorized": question},
    )

print(response.nouls["unauthorized"].noul)

The 17-point gap came apart when I stopped looking at the chart and read ten messages from the buckets where it was widest.

The scan lost one letter, and a derivative was crowned the mother sauce

Tue, 08 Sep 2026 00:00:00 GMT

Here is what saucier tree printed for the Espagnole family in the 1907 scan, before this week:

$ uv run saucier tree espagnole --source escoffier-1907
LENTEN ESPAGNOLE  [espagnole]
├── HALF GLAZE  (en)
│   ├── SAUCE BORDELAISE  (fr)
│   ├── BROWN CHAUD=FROID SAUCE  (en)
│   ├── DEVILLED SAUCE  (en)
│   ├── ITALIAN SAUCE  (en)
│   ├── LYONNAISE SAUCE  (en)
│   ├── MADEIRA SAUCE  (en)
│   ├── PERIQUEUX SAUCE  (en)
│   ├── PIQUANTE SAUCE  (en)
│   └── ROBERT SAUCE  (en)
├── ORDINARY POIVRADE SAUCE  (en)
└── POIVRADE SAUCE FOR VENISON  (en)

Twelve preparations, and the sauce at the head of them is a Lenten variation — the fish-day Espagnole, made when the ordinary one will not do. Escoffier's entry for it opens by doubting whether it needs to exist at all. It is a derivative wearing the crown of the base it derives from.

One letter

The heading it should have taken:

$ sed -n 1730p corpus/escoffier-1907.txt
22— BROWN  SAUCE  OR  ESPAQNOLE
$ sed -n 1392p corpus/escoffier-1909.txt
22—BROWN SAUCE OR ESPAGNOLE

ESPAQNOLE. The scanner read the G as a Q, which it does all through this witness: 16— POULTRY QLAZE at line 1539, 38— QENEVOISE SAUCE at 2150, 39— QRAND-VENEUR SAUCE at 2222, 46— PIQNONS SAUCE at 2300. Three of those have been sitting in the diff output since the scan was added, as rows labelled ocr-suspected, which is exactly what they are:

  ocr-suspected   piqnons-sauce ~ pignons-sauce           PIQNONS SAUCE / PIGNONS SAUCE
  ocr-suspected   qenevoise-sauce ~ genevoise-sauce       QENEVOISE SAUCE / GENEVOISE SAUCE
  ocr-suspected   qrand-veneur-sauce ~ grand-veneur-sauce QRAND-VENEUR SAUCE / GRAND-VENEUR SAUCE

Harmless, and useful: that label is what the second-copy post added the second witness to produce. ESPAQNOLE is not on that list. It is not anywhere in the diff, because entry 22's heading gives it a second name and both witnesses key it on that one:

$ uv run saucier show espaqnole --source escoffier-1907
BROWN SAUCE OR ESPAQNOLE
entry 22, line 1730, ocr of escoffier-1907
  term  BROWN SAUCE  [en]  brown-sauce
  term  ESPAQNOLE  [en]  espaqnole
  parent  brown-roux
  procedure  (unrecorded)

The base is there, one letter away, catalogued as brown-sauce in both printings and carrying the same brown-roux parent in both, which is why the diff has nothing to say about it. What entry 22 does not have is the word espagnole anywhere among its names. So when the catalogue was asked which preparation the mother concept espagnole names, every exact match failed, and the search fell back to the rule for partial matches: the concept has to appear as a whole run of words inside a name. In the 1907 witness exactly one name satisfies that.

LENTEN ESPAGNOLE   entry 24, line 1795

The rule had a premise it never checked

ADR-0008 says a mother binds to the first preparation, in source order, that answers to its name. The reasoning is a fact about the book: Escoffier presents a base before its derivatives, so among several names carrying the word, the earliest one is the base. That is true of this book, and it is the reason veloute reaches ORDINARY VELOUTÉ SAUCE at entry 25 rather than ALLEMANDE SAUCE OR THICKENED VELOUTÉ at entry 27.

The premise underneath it is that the base is among the candidates at all. The lookup sorted the hits by where they appear and returned the first one. It never asked whether the list it was sorting contained the thing it was looking for. When the scanner takes the base's only copy of the name, the list still has entries in it, the ordering rule still fires, and the code still returns something — with no less confidence than when it is right.

There is a second casualty in the same failure. Entry 24 opens like this:

$ sed -n 1797,1798p corpus/escoffier-1907.txt
Practical  men  are  not  agreed  as  to  the  need  of  Lenten
Espagnole.  The  ordinary  Espagnole  being  really  a  neutral

The first sentence states Espagnole on its own, because the run sits inside Lenten Espagnole, and the sentence after it names the ordinary Espagnole outright. The parent resolver reads exactly that kind of sentence and would have recorded espagnole as this entry's parent. It did not, because it discards a statement that names the entry doing the stating, and the mother lookup was telling it that espagnole was entry 24. So the same defect that gave Lenten Espagnole a crown also cost it the parent it plainly states. saucier show reported parent (unresolved) and stated no candidate for an entry whose first sentence names its base.

The proposed fix was aimed at ranking

The ticket that opened this proposed that a mother bind only on an exact name match. If nothing in the witness is named exactly espagnole, bind nothing, and the bad crown disappears.

It does. There are ten mother bindings across the two witnesses: five mothers — bechamel, espagnole, hollandaise, tomato, veloute — resolved once per witness. Exactly one of the ten is an exact name match, the 1909 ESPAGNOLE that Escoffier prints as the second term of entry 22's heading. The other nine reach their base through the run rule, because what the book prints is BÉCHAMEL SAUCE, HOLLANDAISE SAUCE, TOMATO SAUCE. Binding on exact names only removes nine bindings in order to remove one wrong one.

The census will not tell you that. Run the proposal and both witnesses come out at the counts they should — 151 / 57 / 94 and 140 / 51 / 89 — and Lenten Espagnole picks up the parent it states, because the self-reference that suppressed it is gone either way. What moves is which identity a parent names:

  escoffier-1909, binding on exact names only
- mornay-sauce  parent  bechamel
+ mornay-sauce  parent  bechamel-sauce

Eight parent values move like that, four in each witness: in 1909, Cream Sauce and Mornay off bechamel, Maltese and Mousseline off hollandaise; in 1907, Cream Sauce off bechamel, and Maltese, Mousseline and Noisette off hollandaise. Mornay is the asymmetry — the 1907 scan splits its heading into MORN AY SAUCE and its first input reads Bdchamel Sauce, which reaches no catalogued name, so that witness has no bechamel parent there to move, as the parent post worked through. saucier tree hollandaise then prints a bare heading with no children in either witness, and the family survives only under hollandaise-sauce, which is a heading and not a mother. That is ADR-0008's coalescing rule — names reaching one preparation coalesce under the mother concept — quietly reversed, and it is the rule the Marrow Sauce post spent two and a half thousand words getting right. Eight records that were reading the book correctly would have paid for one that was not.

The rule the proposal was aimed at turns out to decide almost nothing. Ranking by source order arbitrates two of the ten bindings, and both of them are velouté, the one mother in this book with three names carrying its word:

veloute, escoffier-1909   ORDINARY VELOUTÉ SAUCE
                          VELOUTÉ DE VOLAILLE
                          ALLEMANDE SAUCE OR THICKENED VELOUTÉ

Seven of the other eight bindings have a single candidate each, and the 1909 Espagnole binding never reaches the candidate list because its exact name wins first. Espagnole in 1907 had a single candidate too. Nothing was ranked, so no ranking rule could have prevented it. Measuring the proposal is what made that visible, and the measurement is the reason the actual defect got named: the run-match branch never tested whether the name it crowned belonged to the base or to a derivative of the base.

The sentence a chef would nod at

A mother does not bind by a name run to a preparation whose opening paragraph states that mother. An entry that says it is made from Espagnole is not Espagnole.

That is ADR-0018, and the test it applies is not new. It is the statement test ADR-0008 already used to read parents out of prose: a whole run of words, inside one sentence, of the opening paragraph. Two questions now share it, so it moved into domain/statement.py where neither caller owns it. Resolution asks which preparations an opening states. The lookup asks whether a candidate states the base it is being offered as. Same reading, opposite direction.

The opening paragraph is the right place to look because of how Escoffier writes. The first paragraph of an entry is its ingredient list. A sauce named there is a sauce this one is built from, not one it is being compared against — entry 22's own third paragraph, The time required for the despumation of an Espagnole at line 1755, is not a claim of derivation, and the guard does not read that far.

An exact catalogued name still wins outright, before the guard runs. A base that names itself exactly in its own heading keeps its own identity; the guard only ever removes run matches. Ordering is untouched, and the survivors still rank by source order. When nothing survives, the mother is uncatalogued in that witness, which is the honest outcome and not a fallback:

$ uv run saucier show espagnole --source escoffier-1907
no preparation named 'espagnole'
[exit 1]

One line of JSON

$ uv run saucier tree espagnole --source escoffier-1907
espagnole  [espagnole]
├── HALF GLAZE  (en)
│   ├── SAUCE BORDELAISE  (fr)
│   ├── BROWN CHAUD=FROID SAUCE  (en)
│   ├── DEVILLED SAUCE  (en)
│   ├── ITALIAN SAUCE  (en)
│   ├── LYONNAISE SAUCE  (en)
│   ├── MADEIRA SAUCE  (en)
│   ├── PERIQUEUX SAUCE  (en)
│   ├── PIQUANTE SAUCE  (en)
│   └── ROBERT SAUCE  (en)
├── LENTEN ESPAGNOLE  (fr)
├── ORDINARY POIVRADE SAUCE  (en)
└── POIVRADE SAUCE FOR VENISON  (en)

The tree heads on the bare concept, because the witness carries the derivations without carrying a usable name for what they derive from, and the twelve preparations that were beneath Lenten Espagnole are beside it now. tree lenten-espagnole prints one line, derives from espagnole, and no children.

The whole change to the parsed data is one line:

  data/escoffier-1907.json, line 126
-       "parent": null,
+       "parent": "espagnole",

That is entry 24 recording the base its first sentence names. It is the only parent value that moves in either witness, and it is the 51st derived sauce in the 1907 catalogue:

escoffier-1909  151 sauces, 57 derived, 94 unresolved
escoffier-1907  140 sauces, 51 derived, 89 unresolved

140 / 50 / 90 became 140 / 51 / 89. The 1909 census does not move, and its JSON and JSONL come out byte-identical to the previous release — the guard does remove one 1909 candidate, VELOUTÉ DE VOLAILLE, whose opening states velouté, but it was already losing to entry 25 on source order, so nothing downstream notices. Two candidates removed across both witnesses, one binding changed, nine correct bindings untouched. The diff summary moves from 11 unmatched, 19 parent-changed, 36 ocr-suspected to 11 unmatched, 18 parent-changed, 35 ocr-suspected. One row leaves and both counters drop, because that row carried both labels: it is Lenten Espagnole's own, marked parent-changed, ocr-suspected and reading lenten-espagnole (none) / espagnole. The 1909 witness always resolved that parent, and the two printings now agree on it.

ADR-0008 stays accepted with its mother-binding clause amended, and everything else in it intact. Its subject, shadow, ambiguity, and cycle rules behave as before. Names reaching one preparation still coalesce under the mother concept, so Mornay still reads parent bechamel and the record of that edge still stands. A fix that is one sentence long and moves one field is what you get when the sentence is about the book rather than about the code.

Where this guard runs out

It reads the witness's opening prose, so it is only as good as that prose survived. In 1907 the VELOUTE DE VOLAILLE opening does not state velouté, although the 1909 opening does — which is why the guard removes that candidate in one witness and not the other. Damage to a base's heading and to a derivative's opening paragraph would defeat it exactly the way the heading alone defeated the old rule.

The guard also reads the true base's own opening, and a base that stated its own name there would remove itself and hand the heading to the next match in source order. None do, which is what it means that the eight name-run bindings survived the test: Escoffier opens a base with its quantities, not with its name. The 1909 ESPAGNOLE binding is an exact catalogued name and returns before the guard runs, so it never faced the test at all.

Entry 22 comes within a paragraph of a different trap. It names Espagnole at line 1755, in its third paragraph, discussing how long despumation takes, and nothing would hold it back from stating itself: an uncatalogued mother is keyed by a bare concept that no entry holds, so no entry is held out from stating it. Had that paragraph opened the entry, though, entry 22 would have recorded no parent at all. Three lines on, the same paragraph names a second mother — the Mirepoix and the tomato are inserted from the first, at line 1758 — and ADR-0008 takes exactly one stated candidate or no parent, so two of them resolve to none. What catches the near-miss is a rule that already refuses to guess, not the cycle check: nothing reaches the cycle check, and it reads heading lines, so it could not have caught a parent that names no entry.

And the crown is the only thing that got fixed. The scan still reads ESPAQNOLE, entry 22 is still catalogued as brown-sauce, and tree brown-sauce --source escoffier-1907 still prints one line with nothing under it, while thirteen preparations hang off a concept whose heading the witness cannot supply. ADR-0013 repairs the punctuation that delimits a record, never the characters inside one, and one Q is a character inside one. ADR-0014 says a damaged witness cannot establish absence. This says the neighbouring thing: a damaged witness cannot establish identity by promoting a stated derivative when the base's name disappears. Neither one repairs anything. They both decline to conclude.

Everything here reproduces from the tag:

$ git clone https://github.com/Alberto-Codes/saucier
$ cd saucier && git checkout v0.7.0
$ uv sync && uv run saucier parse
$ uv run saucier diff escoffier-1907 escoffier-1909
$ uv run saucier tree espagnole --source escoffier-1907
$ uv run saucier show espaqnole --source escoffier-1907
$ uv run saucier show lenten-espagnole --source escoffier-1907
$ sed -n 1730p corpus/escoffier-1907.txt
$ sed -n 1795,1800p corpus/escoffier-1907.txt

If a heading in either witness is being read as something it is not, or if the guard rejects a candidate you think belongs, the issue template asks for the entry number and the source lines. Line 1730 is where I would start.

The release: saucier v0.7.0 — the guard: a mother no longer binds by a name run to a preparation whose opening paragraph states that mother, with ADR-0008's statement test moved into the shared saucier.domain.statement so both callers apply the same one. In the 1907 scan the mother stays uncatalogued rather than falling back to a derivative, saucier show espagnole --source escoffier-1907 refuses and exits 1 where it used to print a Lenten sauce, and one field of the parsed data moves. MIT.

Serve Gemma 4 31B on a 24 GiB card with the context it was packed for

Sun, 06 Sep 2026 00:00:00 GMT

Who this is for

You have a 24 GiB card and you want to run Gemma 4 31B on it with as much context as the card will hold. You have read, or do not care about, the argument for why a 14.92 GiB pack beats the 16.44 GiB official build on this card. You want the server up.

Assumed: a Linux box with a 24 GiB NVIDIA card, a working driver, and a terminal. Not assumed: any quantization background. The why is the explanation post, and the evidence is on the model card. This post repeats none of it.

Every number below is published, and the serve boundaries were measured on an RTX 4090 running llama.cpp b10362 on 2026-08-31. Numbers from another frame, the H100 build, the multi-image ladder, say so where they appear. Your card is a different box, and the last section is about what to do when a number does not reproduce.

Before you start

Disk. The decoder is 14.92 GiB and the projector sidecar is 629 MiB. The llama.cpp build or tarball adds a few GiB. Call it 20 GiB free.

Two files, one artifact. The decoder alone serves text. Images need the sidecar too. Both download in one command below.

The license. The weights carry the Gemma 4 license note. Read it before you serve them to anyone but yourself.

The build is part of the claim. The boundaries on the card were measured at llama.cpp b10362. A newer build allocates the KV cache its own way and may land a rung higher or lower. Pin the build first, reproduce the boundary, then move if you want to.

1. Download the pack

The repo is public. No token needed.

uv tool install "huggingface_hub[cli]"
mkdir -p ~/models && cd ~/models
hf download Alberto-Codes/gemma-4-31B-it-fit24gib-GGUF \
  gemma-4-31B-it-fit24gib.gguf gemma-4-31B-it-mmproj-q4km.gguf \
  --local-dir .

What lands:

File	Bytes	Size
`gemma-4-31B-it-fit24gib.gguf`	16,015,862,144	14.92 GiB
`gemma-4-31B-it-mmproj-q4km.gguf`	659,537,504	629 MiB

Check the decoder before you trust it. The card publishes the hash:

sha256sum gemma-4-31B-it-fit24gib.gguf
# 2a7bd7a7be6979c858258618ab576db573a7b671b45ee5e9785247341b8c3b1e

2. Install llama.cpp b10362

Two ways. The first is the same build number and the same backend the card's boundaries were measured on. The second is the path the repo's rented-H100 how-to takes.

Prebuilt Vulkan tarball. The b10362 release ships a Linux Vulkan build:

cd ~
curl -LO https://github.com/ggml-org/llama.cpp/releases/download/b10362/llama-b10362-bin-ubuntu-vulkan-x64.tar.gz
tar xzf llama-b10362-bin-ubuntu-vulkan-x64.tar.gz
export BIN=~/llama-b10362    # the tarball unpacks flat into this directory
"$BIN/llama-server" --version

Build from source with CUDA. The release ships no Linux CUDA tarball, so CUDA means a build. Pin the commit the tag points at:

git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp
cd ~/llama.cpp
git fetch --tags origin
git checkout 4801e3c56    # the commit tag b10362 points at
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
export BIN=~/llama.cpp/build/bin
"$BIN/llama-server" --version

Either way the version line reads version: 10362 (4801e3c56). On the rented H100 that build took 429 seconds. CUDA is a different runtime from Vulkan, so treat the boundaries below as the rung to test first, not a promise. Step 5 shows how.

3. Serve text at the measured boundary

Note what is free before you load. The card's ladders ran with 23,629 to 23,631 MiB free on a 24,564 MiB device, under a desktop.

nvidia-smi --query-gpu=memory.total,memory.free --format=csv

Then the command from the card, verbatim:

M=~/models
"$BIN/llama-server" -m "$M/gemma-4-31B-it-fit24gib.gguf" \
  -c 86016 -ngl 99 -np 1 --port 8991 > server.log 2>&1 &

Three flags carry the claim.

-c 86016 is the measured text boundary. The next rung, 90,112, fails to load.
-ngl 99 offloads every layer. Any layer left on the CPU frees VRAM and invalidates the comparison.
-np 1 is one slot. The b10362 server defaults to four, which on this geometry adds about 2,400 MiB of sliding-window cache and fails loads that fit at one.

Wait for model loaded in the log, then hit the health route. This build logs all slots are idle only at trace verbosity, so do not wait on it: a check on that line waits forever with the server healthy.

while pgrep -x llama-server > /dev/null && ! grep -q "model loaded" server.log; do sleep 2; done
curl -s localhost:8991/health

The reply is {"status":"ok"}. If the loop returns before model loaded appears, the server exited, and the tail of server.log says why; see "If it does not fit" below.

A loaded server is not a server on the GPU. Confirm the offload before you trust any number below:

grep -i offloaded server.log
grep -i "gpu-layers option will be ignored" server.log

The first prints offloaded N/N layers to GPU, both numbers equal and neither of them zero. The second prints nothing. offloaded 0/N means -ngl 99 was parsed and then discarded, and the second grep says why: the build has no backend for your card, a CPU-only archive or a driver it cannot see. Nothing else looks wrong — the server loads, the health route answers ok, requests come back — and every layer runs on the CPU at a fraction of the speed, while every boundary in this post assumes the card is doing the work. Fix step 2 before you measure anything.

4. Send one request

The server speaks the OpenAI chat shape. One text turn:

curl -s localhost:8991/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "messages": [{"role": "user", "content": "In one sentence, what is a sliding-window attention layer?"}],
    "max_tokens": 64
  }' | python3 -c 'import json,sys; print(json.load(sys.stdin)["choices"][0]["message"]["content"])'

That answers from inside the 86,016-token envelope. It does not tell you what a full envelope costs to decode: the card's boundary check decoded five tokens, and throughput at the boundary is unmeasured. The throughput the card does publish was taken at 8,192 tokens of context on the same 4090, 47.8 tokens per second for this pack against 43.3 for Google's Q4_0.

5. Read the VRAM numbers back

While the server is up:

nvidia-smi --query-gpu=memory.used,memory.free --format=csv

At the 86,016 boundary on the 4090 the load passed with 143 MiB free. That is what a fit bar looks like: the card is full, and the next rung is the one that fails. If you see a few gigabytes free, one of the three flags above is not doing what you think, most often a default -np 4 from a wrapper script.

To confirm the boundary is real on your box, stop the server and load one rung higher:

pkill -x llama-server
until ! pgrep -x llama-server > /dev/null; do sleep 1; done
"$BIN/llama-server" -m "$M/gemma-4-31B-it-fit24gib.gguf" -c 90112 -ngl 99 -np 1 --port 8991

If it fails, you have confirmed the boundary; if it loads, stop it with Ctrl-C and keep climbing a rung at a time until one fails; that rung is yours. A higher rung means your box idles with more VRAM free than the card's did. The card calls the tuple of box, build, backend, and free VRAM before load the frame, and it prints the frame beside every boundary, because the same file served 81,920 for text three days earlier in a frame with less idle VRAM. Write yours down the same way. It is the only way two boundaries compare.

6. Serve images

Images need the sidecar and two more flags. Stop whichever server is still holding the card, start the image server from the card, and wait for model loaded the same way as in step 3:

pkill -x llama-server
until ! pgrep -x llama-server > /dev/null; do sleep 1; done
"$BIN/llama-server" -m "$M/gemma-4-31B-it-fit24gib.gguf" \
  --mmproj "$M/gemma-4-31B-it-mmproj-q4km.gguf" \
  -c 73728 -ngl 99 -np 1 --mtmd-batch-max-tokens 264 \
  --port 8991 > server.log 2>&1 &
while pgrep -x llama-server > /dev/null && ! grep -q "model loaded" server.log; do sleep 2; done

-c 73728 is the measured one-image boundary. The next rung, 77,824, loads and then fails at encode time.
--mtmd-batch-max-tokens 264 caps the encode batch at one 1280×720 image, which is 264 image tokens. Without it the server packs up to 1,024 image tokens into one encode graph, two images share a graph, and the graph asks for 328 MiB against a 150.63 MiB one-image reserve. On 2026-09-02 that crashed the server on the second image at both configurations. With the cap, the same ladder filled the window to a clean context refusal.

Keep about 200 MiB free beyond the load. The image encode allocates at request time, and this build crashes on that failure instead of refusing.

An image request is the same chat shape with an image_url part carrying a base64 data URL. Write it to a file first: a base64 screenshot is bigger than the single-argument limit a Linux shell allows, and an inline -d fails with "Argument list too long".

python3 - <<'PY'
import base64, json
img = base64.b64encode(open("screenshot.png", "rb").read()).decode()
json.dump({"max_tokens": 64, "messages": [{"role": "user", "content": [
    {"type": "text", "text": "What is on this screen?"},
    {"type": "image_url", "image_url": {"url": "data:image/png;base64," + img}},
]}]}, open("req.json", "w"))
PY
curl -s localhost:8991/v1/chat/completions -H 'Content-Type: application/json' \
  -d @req.json | python3 -c 'import json,sys; print(json.load(sys.stdin)["choices"][0]["message"]["content"])'

One 768×768 image costs 256 decoder tokens on this pack, measured at the server. The 1280×720 screenshot above is 264 image tokens, 271 once its wrapper is counted, which is why the cap is 264 and the decoder sees 271.

What you should see

The card's serve ladders, measured 2026-08-31, RTX 4090, llama.cpp b10362 Vulkan, -ngl 99 -np 1, KV cache f16, 4,096-token rungs:

Serving shape	This pack	Google's QAT Q4_0
Text only, max load	86,016 (fails at 90,112)	65,536 (fails at 69,632)
One image aboard, max load	73,728 (encode fails at 77,824)	49,152 (encode fails at 53,248)

The gain is 20,480 tokens of text and 24,576 with an image, on that frame. Two of the three things that set a boundary, the build and the box, are yours.

Quality is not the subject of this post. The card's five held-out benchmarks put this pack at four ties and one win against Google's build. Read the tables there, not here.

If it does not fit

Work down this list in order. Each item is a boundary the card already crossed.

Drop one rung. The rungs are 4,096 tokens. Try 81,920 for text, 69,632 with an image. The 81,920 text rung reproduced on 2026-08-31 with 465 MiB free at load, so it is the safer first stop under a heavier desktop.
Check -np. A wrapper that sets slots for you costs about 2,400 MiB of sliding-window cache on this geometry. One slot, or nothing here holds.
Check what else holds the card. The ladders ran with 23,629 to 23,631 MiB free before each load. A rung is about 320 MiB on this geometry, so a browser or a second model can cost one.
Serving images: keep the encode-batch cap. Every crash on the card's multi-image ladder traced to two images sharing one encode graph. --mtmd-batch-max-tokens 264 is the fix, not a tuning knob.
A different backend is a different frame. CUDA, ROCm, or a newer llama.cpp allocates differently. Measure your own ladder with step 5 and report the frame with the number.

If you have more card than this, the repo has a how-to for serving the pack on a rented H100. It is the instrument behind the card's real-GUI campaign: a CUDA build of the same b10362 tag, the same two files, and context 8,192 for single-image evaluation. The 24 GiB boundaries and the encode-batch flag belong to the 4090 frame and do not carry over.

What you have at the end is not the card's number. It is a boundary measured on your own box, with its frame written beside it, which is the only kind of number the card ever claimed.

Where the numbers come from

The pack and its card: gemma-4-31B-it-fit24gib-GGUF. The serve commands, the ladders, the reproduction traps, the hashes.
The map behind it: gemma-4-31B-it-sensitivity-maps, published 2026-09-04, so vramfit plan can re-solve this model for a different budget.
The baseline: google/gemma-4-31B-it-qat-q4_0-gguf, 16.44 GiB.
The tool: vramfit v0.5.0 — the tag this guide serves from: the thin gguf extra that reads a pack without torch, pack comparing its packed bytes against the recipe's prediction, and every refusal raised under one root. MIT.
The argument: the explanation post. The pipeline that built the file is fit a model to the GPU you actually have.

The parent finally has a verb. The book still will not say how many minutes.

Sat, 05 Sep 2026 00:00:00 GMT

Here is what saucier show mornay printed at v0.5.0, cut to the two lines that matter:

  parent  bechamel

Boil one pint of Béchamel Sauce with one-quarter pint of the _fumet_

The record says Mornay derives from Béchamel. The sentence underneath, at line 2439 of the 1909 text, two lines below the heading of entry 91, says what is done with the Béchamel, and the record does not read it. The Marrow Sauce post spent two and a half thousand words getting the parent edge right, and the edge it got right is an input with the operation stripped off. A parent says what a sauce is built from. It does not say boil, reduce, or finish, and it does not say how much or how far.

v0.6.0 reads that sentence, for one preparation, by hand. The census does not move: 151 sauces, 57 derived, 94 unresolved in the 1909 text, and 140, 50, 90 in the 1907 scan, the same census as the stream post, whose 151 and 140 these split. What moves is the record underneath one of them.

Six operations, in the book's order

$ uv run saucier show mornay
MORNAY SAUCE
entry 91, line 2437, transcription of escoffier-1909
  term  MORNAY SAUCE  [en]  mornay-sauce
  parent  bechamel
  procedure  6 operations, recorded by hand
    Boil      Béchamel Sauce [fr] 1 pint, fumet [fr] 1/4 pint
    Reduce    criterion: by a good quarter (unresolved)
    add       Gruyère [fr] 2 oz., Parmesan [en] 2 oz.
    Put       duration: a few minutes (unresolved), on the fire again
    stirring  instrument: small whisk, criterion: the melting of the cheese (unresolved)
    Finish    butter [en] 2 oz., away from the fire, added by degrees

And the eight lines it was read from, lines 2439 to 2446 of the committed corpus, exactly as Project Gutenberg transcribed them:

Boil one pint of Béchamel Sauce with one-quarter pint of the fumet of the fish, poultry, or vegetable, which is to constitute the dish. Reduce by a good quarter, and add two oz. of Gruyère and two oz. of grated Parmesan.

Put the sauce on the fire again for a few minutes, and ensure the melting of the cheese by stirring with a small whisk. Finish the sauce away from the fire with two oz. of butter added by degrees.

Every line on the right is a run of words on the left. Boil takes two inputs, each with the quantity the text gives: one pint, one-quarter pint. Reduce takes nothing and carries a criterion, the words the reduction is carried to. add takes two cheeses at two ounces each. Put carries a duration and a constraint, on the fire again. stirring carries an instrument and a criterion. Finish takes the butter and two constraints, away from the fire and added by degrees. That is the whole entry, and nothing in it is inferred. ADR-0017 is the record of the shape, and the glossary gains ten terms for it, because this project uses one word per thing and this release needed ten new things: procedure, operation, wording, input, parameter, criterion, constraint, instrument, recorder, unrecorded.

Three slots the text left empty

Look at the three (unresolved) marks. Béchamel gets 1 pint. Gruyère gets 2 oz.. The reduction gets by a good quarter, and no number. The return to the fire gets a few minutes, and no number. The stirring gets the melting of the cheese, which is a criterion with no number in it at all.

That is deliberate, and it is the same rule this series has applied to parents since the first post. A parameter holds the words, the number the words give, and the unit they name. one-quarter pint records the fraction one over four and the unit pint. a few minutes records the unit minutes and no number, because the text gives none. by a good quarter names a degree and not a quantity, so it records no number either. No code fills the slot. Anyone who has cooked a Mornay knows roughly how many minutes "a few" is, and a model would happily write 3. The parser writes nothing, because Escoffier wrote nothing, and the moment this record holds a number the book does not the record stops being the baseline a model has to beat.

The stream post argued that null in a parent field means the source declined to state one, never that the sauce has none. This is the first time the same absence is recorded against something other than a parent. A duration with no number is a fact about the text.

The record cannot say what the entry does not

The six operations are Python literals in an adapter, written by hand against lines 2439 to 2446. A hand can misquote. So the command never prints a procedure it has not first found in the body.

Each operation carries its wording, the whole clause it was read from. Each input, criterion, duration, and constraint carries its own wording, and the entity refuses an element whose words do not lie inside its operation's wording. Then, before show prints anything, it collapses whitespace on both sides and looks for each operation's wording in the body, in order, each one after the last. An operation the body does not carry, or carries out of order, is reported, and the command exits 2 with nothing on standard output:

$ uv run saucier show mornay   # with a hand record that says "by a good half"
saucier: MORNAY SAUCE at line 2437 of escoffier-1909 does not state 'Reduce by a good half'
[exit 2]

The first version of this printed the title, the reference, the terms, and the parent before it ran the check, so a piped reader got half a record and then a non-zero exit. Review caught it, and the check now runs before the first byte. The stream post's first reader accepted a file missing its last line. This release's first show printed half a preparation beside a refusal. Same lesson, one layer up: a check that runs after the output has started is a comment.

What the check proves is narrow, and the ADR says so. It proves the words are there and in that order. It does not prove the reader parsed the clause correctly. Three choices in the Mornay record are the reader's, and they are written down so a disagreement has something to point at. ensure the melting of the cheese by stirring with a small whisk records the verb as stirring, with the melting as the criterion and the whisk as the instrument. the sauce and the cheese name the preparation in progress and are not inputs. Put the sauce on the fire again names no heat, so no heat is recorded.

The scan keeps its verb and loses its parent

The 1907 witness carries the same entry at line 2864, and the release records a second procedure for it, in the scan's words, because a procedure quotes its witness and the two witnesses do not read alike:

$ uv run saucier show morn-ay-sauce --source escoffier-1907
MORN AY SAUCE
entry 91, line 2864, ocr of escoffier-1907
  term  MORN AY SAUCE  [en]  morn-ay-sauce
  parent  (unresolved)
  stated  no candidate
  procedure  6 operations, recorded by hand
    Boil      Bdchamel Sauce [fr] 1 pint, fumet [fr] 1/4 pint
    Reduce    criterion: by a good quarter (unresolved)
    add       Gruy^re [fr] 2 oz., Parmesan [en] 2 oz.
    Put       duration: a few minutes (unresolved), on the fire again
    stirring  instrument: small whisk, criterion: the melting of the cheese (unresolved)
    Finish    butter [en] 2 oz., away from the fire, added by degrees

The heading reads MORN AY SAUCE, so show mornay finds nothing in the scan. The first input reads Bdchamel Sauce, whose folded form reaches no catalogued name, so the scan's Mornay has no parent and no candidate. The text resolves it. The verb is on the record in both. Gruy^re stays Gruy^re, and the Reduce clause carries the running page header 40 GUIDE TO MODERN COOKERY inside its wording, because the scan carries it there and ADR-0013 repairs the punctuation that delimits a record, never the characters inside one. The Périgueux page break from the stream post is the same running header, ten pages earlier, doing the same damage.

The one difference that is not the scanner

The second-copy post ended on a sentence I have repeated in every post since: not one confirmed editorial difference between the 1907 and 1909 printings. Every candidate was the scanner or the reader. The README said so at v0.5.0.

Writing the 1907 procedure by hand meant reading line 2867 of the scan against line 2440 of the text, word by word, because the check would refuse anything else. The fumet is of that fish which is to constitute the dish in 1907. It is of the fish, poultry, or vegetable, which is to constitute the dish in 1909. No scanner adds two nouns and three commas. Between the first printing and the revised edition the sentence widened, from fish to fish, poultry, or vegetable. Whose hand widened it, the two texts do not say.

That is one difference, confirmed by hand, on two lines anyone can open. The diff command has still confirmed none, and the README now says exactly that: the diff has confirmed nothing, and one difference has been confirmed by a reader with both texts in front of them. It took recording a procedure to find it, because a procedure quotes the witness and a parent only names it. bechamel is the same concept in both books. of that fish and of the fish, poultry, or vegetable are not the same words.

What I am not claiming

One preparation is recorded, once per witness, and a test pins the count at one. Two procedures written by hand is not extraction. The rule that would read a second one out of an entry does not exist yet, and until it does the count stays where it is. What does exist is the port the hand record enters through, and the line recorded by hand in the output is the recorder naming itself. A rule reader, or a model, is another implementation behind the same port, and whatever it records will say who read it. Cardinal Sauce, entry 69, still boils Béchamel and finishes with lobster butter and still records no parent, because two catalogued names sit in its opening paragraph and a resolver may refuse, never rank. The verb that would tell those two names apart, one boiled and one added at the finish, is exactly what a procedure carries. Reading it by rule is the record after this one, and the ten sauces the line-1437 post lost to a butter are waiting on it.

The procedure is not stored and the interchange does not carry it. show fetches it beside the preparation, checks it, and prints it. The JSON files under data/ did not change, saucier/1 did not change, and a consumer of the stream sees no operation. ADR-0016 says a new field earns a new schema version, and one preparation does not earn one.

And the parent field is untouched. The procedure sits beside it and never writes it. Mornay's first operation boils Béchamel, which is its parent, and that agreement is a check on the resolver, not a replacement for it.

Everything here reproduces from the tag:

$ git clone https://github.com/Alberto-Codes/saucier
$ cd saucier && git checkout v0.6.0
$ uv sync && uv run saucier parse
$ uv run saucier show mornay
$ uv run saucier show morn-ay-sauce --source escoffier-1907
$ sed -n 2437,2446p corpus/escoffier-1909.txt
$ sed -n 2864,2878p corpus/escoffier-1907.txt

Six operations, five quantities the book gives, three it withholds, and one sentence Escoffier changed between two printings. If the record says something lines 2439 to 2446 do not, or reads one of the three choices above differently than you would, the issue template asks for the entry number and the source lines. Line 2440 is where I would start.

The release: saucier v0.6.0 — the first recorded procedure: saucier show mornay prints six operations in the book's own words, five quantities the text gives and three it withholds left unresolved rather than guessed, every word checked against the source before it prints. MIT.

I cut the last sauce off the file. Every line still parsed.

Fri, 04 Sep 2026 00:00:00 GMT

I deleted the last line of a 293-line file and fed the rest to a reader I had written for it that week. It rebuilt two catalogues, printed a census one sauce short, and exited zero. Every line it read was valid JSON. The line I had deleted was the scan's Strawberry Sauce, entry 2417 of the 1907 text, at line 41807, and nothing in the stream knew it was gone.

The file exists because, after the line-1437 post, I wanted one list: every sauce that sits on half glaze, in both copies of the book, side by side, asked from a shell and not from inside my own package. I could not ask. The catalogue was two JSON files, one for each copy of the book the project reads, and their shape was whatever my dataclasses were on the day they were written. So v0.5.0 adds two commands. saucier export prints both catalogues to standard output as one stream, one complete JSON record per line. saucier import --check reads that stream back, rebuilds every catalogue in memory, prints the census, and writes nothing. The JSON files stay as they were, and parse, show, tree, and diff do not know the stream exists.

One line, one record

Line 9 of the export is entry 25 of the 1909 text, the sauce Escoffier prints at line 1467:

Every line says what schema it follows, what kind of record it is, and which catalogue it belongs to, so a line can be read on its own. The two catalogue records come first, and each states how many preparations follow it.

Every line still parsed

Delete the last line of the export and feed the rest to the reader:

$ uv run saucier export | head -n -1 | uv run saucier import --check

The first reader I wrote rebuilt two catalogues, printed the census with the scan at 139 sauces instead of 140, and exited zero. Every line that survived was complete JSON, and my reader asked nothing of the stream except that each line parse.

That is why a catalogue record states how many preparation records belong to it. The reader counts them in, and the line number in its refusal points at line 2, the record that made the promise, not at the gap. The count was not in the stream until a review deleted a line and the reader said nothing. It was the second of three things a stream of well-formed records could lie about and the first reader could not see.

what the stream did	the first reader	the reader at `v0.5.0`
arrived empty	rebuilt nothing, exit 0	`interchange carries no catalogues`, exit 2
lost its last line	census one short, exit 0	`line 2: ... states 140 preparations, the stream carries 139`, exit 2
repeated `parent` on line 9	Velouté moved to unresolved, exit 0	`line 9: object repeats a key: ['parent']`, exit 2

The empty stream matters because of the pipeline the command exists for, saucier export | saucier import --check. Without pipefail a pipeline reports its last command's exit status. If the export died before writing a byte, the first reader saw an empty stream, accepted it, and the pipeline would have finished with exit zero and the failed export behind it.

The repeated key is a property of Python's JSON parser, which keeps the last value of a duplicated key and says nothing. Append a second "parent":null to line 9 and an ordinary decoder moves Velouté from derived to unresolved with the census off by one and nothing raised. The reader still uses the standard parser but gives it a hook that watches the keys arrive and refuses a repeat. Nobody found that one. I went looking once the first two had taught me to.

The one-liner, and the Périgueux row

Now the list I wanted, in one line of jq that imports nothing of mine:

$ uv run saucier export \
  | jq -r 'select(.type == "preparation" and .parent == "half-glaze")
           | "\(.catalogue)  line \(.ref.line)  \(.title)"'

Arranged into two columns:

escoffier-1909, the text	escoffier-1907, the scan
line 1680 SAUCE BORDELAISE	line 2057 SAUCE BORDELAISE
line 1708 BROWN CHAUD-FROID SAUCE	line 2085 BROWN CHAUD=FROID SAUCE
line 1750 DEVILLED SAUCE	line 2132 DEVILLED SAUCE
line 1841 ITALIAN SAUCE	line 2233 ITALIAN SAUCE
line 1874 LYONNAISE SAUCE	line 2268 LYONNAISE SAUCE
line 1886 MADEIRA SAUCE	line 2280 MADEIRA SAUCE
	line 2314 PERIQUEUX SAUCE
line 1930 PIQUANTE SAUCE	line 2327 PIQUANTE SAUCE
line 1994 ROBERT SAUCE	line 2398 ROBERT SAUCE

Eight sauces in the text, nine in the scan, and two rows that are findings. BROWN CHAUD=FROID is the scanner reading a hyphen as an equals sign at line 2085, and it stays as recorded: ADR-0013 repairs the punctuation that delimits a record, never the characters that constitute one, and = is inside the title.

The unpaired row is the one I care about. In the 1909 text Périgueux refuses to resolve, because its opening paragraph names two catalogued sauces, half glaze and Madeira, and the resolver may refuse but never rank. The scan records half glaze, confidently. Same sentence, same two names, and a page break between them:

$ uv run saucier show perigueux-sauce
PÉRIGUEUX SAUCE
entry 47, line 1920, transcription of escoffier-1909
  term  PÉRIGUEUX SAUCE  [fr]  perigueux-sauce
  parent  (unresolved)
  stated  half-glaze, madeira-sauce

The resolver reads the opening paragraph only. In the scan that paragraph ends at the blank line on 2319, the running header sits on 2321, and "per quart of Madeira Sauce" is on 2324, outside what the resolver reads. One name left, so it answers. That is the Aurore shape from the second-copy post, where a damaged witness resolves what a clean one honestly cannot, and the line-1437 post already counted Périgueux among three rows of it. What is new is where I was standing when I saw it: in a shell, with a stream and jq. grep -c '"parent":null' on the same stream says 184, which is the 94 unresolved sauces of the text and the 90 of the scan.

What I am not claiming

This stream is not a database. It indexes nothing, keeps no history, and answers no question about the graph of sauces. The JSON snapshot is still the working store behind every other command, and the stream carries records between processes and stops there. It is also less of a stream than the word suggests: the reader consumes one line at a time, but a catalogue is validated whole, so rebuilding one holds all of its preparations in memory first.

Two exports cannot be concatenated, because each carries every configured catalogue and the reader stops at line 294 pointing back to line 1. A catalogue id is a source id, which names a work and an edition, so two scans of one printing would collide on every id. saucier/1 carries one catalogue per source id and does not pretend the identifiers can tell those texts apart. ADR-0016 records those limits and why the interchange is not the store.

And there is nothing here about what Escoffier changed between 1907 and 1909. The Périgueux row is a fact about a page header and a resolver that reads one paragraph, surfaced from a shell.

Everything here reproduces from the tag:

$ git clone https://github.com/Alberto-Codes/saucier
$ cd saucier && git checkout v0.5.0
$ uv sync && uv run saucier parse
$ uv run saucier export | uv run saucier import --check
$ uv run saucier export | head -n -1 | uv run saucier import --check

The first pipeline prints 151 sauces and 140, then 2 catalogues and 291 preparations rebuilt. Nothing written. The second stops at line 2. Between them is a 293-line file that anyone with jq can ask about half glaze, and that lists a Périgueux the scan resolved and the text refused. If you can find a line in that stream that lies about the book and the reader lets through, the issue template asks for the entry number and the source lines. Line 2314 of the scan is where I would start.

The release: saucier v0.5.0 — the stream and the reader that refuses it: saucier export writes both witnesses as saucier/1 records, 293 lines to the same SHA-256 every run, and saucier import --check rejects the empty file, the truncated one, the repeated key and the rest, each with the line number where there is a line to name — the empty stream has none. MIT.

The book spells it out at line 1437. Two of my posts said it never did.

Thu, 03 Sep 2026 00:00:00 GMT

Two earlier posts in this series carry a figure of the same four sauces: the one where ice cream got into the catalogue, and the one where Marrow Sauce finally got a parent. At the top of that figure is Espagnole. A dashed arrow runs from it down to half glaze, and the arrow is labelled assumed, not stated — half glaze, the caption says, is "a reduction of Espagnole the book never spells out." The Marrow Sauce post goes further in prose: a demi-glace is an Espagnole reduction, "which every reader of Escoffier knew and the book therefore never says."

saucier is a parser that reads Escoffier's 1909 Guide to Modern Cookery and records only what the book states. Here is entry 23 of that text, at line 1437, in the chapter Escoffier titles The Leading Warm Sauces:

HALF GLAZE. This is the Espagnole sauce, having reached the limit of perfection by final despumation. It is obtained by reducing one quart of Espagnole and one quart of first-class brown stock until its volume is reduced to nine-tenths of a quart.

The book says it. It says it in the first sentence of a numbered entry, in the sauce chapter, nine entries before Bordelaise. I published two posts and two figures asserting the book never says it, because my parser could not see entry 23, and I believed my parser over the book.

The parser could not see it because the rule that decides what counts as a sauce had two tests, and the second could overrule the first. This release deletes the second test. Here is what that does to the census at v0.4.0:

$ uv run saucier parse
escoffier-1909  New and Revised Edition, January 1909 (impression: January 1920)
                transcription of Project Gutenberg 71395
                mothers: bechamel, espagnole, hollandaise, tomato, veloute
                151 sauces, 57 derived, 94 unresolved
escoffier-1907  no edition stated, copyright 1907
                ocr of Internet Archive cu31924000610117
                mothers: bechamel, espagnole, hollandaise, tomato, veloute
                140 sauces, 50 derived, 90 unresolved

At v0.3.0 the 1909 line read 124, 50, 74. Twenty-seven sauces entered. Unresolved rose by twenty, and every one of the twenty is something the parser could not see before: an entry my rule hid, or an ambiguity that was invisible while the entry was hidden. The honest number got worse because the instrument got better. And derived rose by seven, which is the misleading number, for the best reason in the release.

The clause that vetoed Escoffier

Escoffier opens the chapter at line 1246 by saying what his sauces are:

Warm sauces are of two kinds: the leading sauces, also called "mother sauces," and the small sauces, which are usually derived from the first-named, and are generally only modified forms thereof.

The ice cream post introduced ADR-0007: the source decides what counts as a sauce, because deciding for ourselves is what put vanilla ice cream in the catalogue. Under that record an entry enters on two kinds of evidence. Its heading says "sauce". Or the source filed it in a chapter Escoffier titles as sauces — and its heading also names one of the five mothers.

That second test has two clauses. The chapter clause reads the source's own classification. The mother clause adds a test of mine on top of the reading, and when the two disagree, mine wins. A second test on a classification the source has already made is a veto, not a check.

I wrote the mother clause to keep LENTEN ESPAGNOLE and VELOUTÉ DE VOLAILLE, two derivatives in the sauce chapters whose headings never use the word. It kept them. ADR-0015 counts what it cost, in the three sauce chapters of the 1909 text:

	entries
numbered entries in the three sauce chapters	139
heading lacks the word "sauce"	29
admitted on the mother clause	2
vetoed	27

The 27 are the three roux, half glaze, two gravies, a lobster method that Escoffier numbered as its own entry, whisked mayonnaise, various cullises, and 18 compound butters from the chapter titled Cold Sauces and Compound Butters. Every one of them is an entry the source classified. My rule classified them again and lost.

Here is what that looked like on screen, at v0.3.0:

$ uv run saucier show robert-sauce --chars 260
ROBERT SAUCE
entry 52, line 1994, transcription of escoffier-1909
  term  ROBERT SAUCE  [en]  robert-sauce
  parent  (unresolved)

Finely mince a large onion and put it into a stewpan with butter. Fry
the onion gently and without letting it acquire any colour. Dilute
with one-third pint of white wine, reduce the latter by one-third,
add one pint of half-glaze, and leave to simmer for twen...
[378 more characters, raise --chars to read them]

The record and the sentence that contradicts it, four lines apart, for three releases. Eleven sauces name half glaze in their opening paragraph. Five name a roux. The book writes the chain in full — brown roux, Espagnole, half glaze, Robert — and the catalogue had dropped the middle link.

The new rule is one sentence: an entry inside a sauce chapter qualifies on the chapter, and an entry outside one qualifies on its heading alone. The function that decides admission went from five lines to one:

return names_a_sauce(title) or in_sauce_chapter

The mothers are still read, for the catalogue and for resolution, and they take no part in admission. Nothing is hand-excluded to make the result tidy: VARIOUS CULLISES is entry 144, numbered inside a sauce chapter, and it is in the catalogue now. An admitted entry that reads oddly as a preparation is a finding, not a special case in the parser.

Twelve gained, and one of them was the bar

With half glaze and the roux in the catalogue, the chain resolver from the Marrow Sauce post has something to resolve against. Twelve sauces now record the parent Escoffier wrote, and none of them needed a new rule to do it:

resolves to	sauces
half glaze	Bordelaise, Brown Chaud-froid, Devilled, Italian, Lyonnaise, Madeira, Piquante, Robert
brown roux	Espagnole
pale roux	Velouté
white roux	Scotch Egg
manied butter	Mousseuse

Bordelaise is on that list, and the Marrow Sauce post made a point of Bordelaise. Its opening names half a pint of half glaze and never says Espagnole, so it stayed unresolved — "not a gap in this release," I wrote, but "the bar the model post will have to clear." The bar is cleared, and no model cleared it. A chapter test cleared it, by letting the resolver see an entry that was in the book the whole time.

$ uv run saucier tree espagnole
BROWN SAUCE OR ESPAGNOLE  [espagnole]  derives from brown-roux
├── HALF GLAZE  (en)
│   ├── SAUCE BORDELAISE  (fr)
│   │   └── MARROW SAUCE  (en)
│   ├── BROWN CHAUD-FROID SAUCE  (en)
│   ├── DEVILLED SAUCE  (en)
│   ├── ITALIAN SAUCE  (en)
│   ├── LYONNAISE SAUCE  (en)
│   ├── MADEIRA SAUCE  (en)
│   ├── PIQUANTE SAUCE  (en)
│   └── ROBERT SAUCE  (en)
├── LENTEN ESPAGNOLE  (fr)
│   └── GENEVOISE SAUCE  (en)
├── ORDINARY POIVRADE SAUCE  (en)
└── POIVRADE SAUCE FOR VENISON  (en)

That heading line is new. Escoffier names five mothers, and that does not change. But Espagnole opens with "one lb. of brown roux dissolved in a tall, thick saucepan with six quarts of brown stock," and the catalogue can now see the roux, so a mother may state a parent. It cannot see the stock: brown stock is entry 7, in Fonds de Cuisine, a chapter Escoffier does not title as sauces. The chapter decides, and Chapter I decided stocks.

Ten lost, and they are the finding

Derived rose from 50 to 57, and seven is three numbers: twelve sauces gained a parent, five admitted entries state one of their own, and ten sauces that were resolved at v0.3.0 are unresolved now. Eighteen of the 27 admissions are compound butters, and seven of the ten losses are sauces that now see a compound butter beside their old parent. The butters that came in are the butters that took Cardinal's parent away.

Here is Cardinal, which is the shape of all ten, with a line that did not exist at v0.3.0:

$ uv run saucier show cardinal-sauce
CARDINAL SAUCE
entry 69, line 2192, transcription of escoffier-1909
  term  CARDINAL SAUCE  [en]  cardinal-sauce
  parent  (unresolved)
  stated  bechamel, lobster-butter

And the paragraph beneath it, with the two verbs the resolver cannot read:

Boil one pint of Béchamel, to which add one-half pint of fish fumet and a little truffle essence, and reduce by a quarter. Finish the sauce, when dishing up, with three tablespoonfuls of cream and three oz. of very red lobster butter (No. 149).

At v0.3.0 Cardinal recorded bechamel, and that was correct. Lobster butter was on the page then too, at entry 149, but the mother clause had kept it out of the catalogue, so as far as the resolver could see it was an ingredient with no entry. Now it is catalogued, the opening paragraph states two catalogued names, and ADR-0012 says what happens then: the resolver refuses. It may not rank. So Cardinal is unresolved, and so are Nantua, Noisette, Diplomate, Joinville, Herb and Ravigote, each with at least one butter beside its old parent. Périgueux, Reform and Chasseur lose theirs the same way to half glaze, which now sits beside the base each of them names.

The refusal is correct under the rule as written, and the rule as written is wrong about this sentence. The source stated one base and one finish. Any cook reading the paragraph knows which is which. The resolver reads names. It cannot tell a base from a finish, because it has never been asked to read the verb a name sits inside — and until it can, the honest answer is the refusal.

I could recover all ten this afternoon. A rule that prefers a mother when one is stated would put Cardinal back on Béchamel and the derived count back near 67. I am not doing it, and the reason is the second-copy post's Aurore sauce. The candidate rule is not tuned to save the number. Every tuning of it that saves Cardinal is a choice made by me rather than a statement made by the book, and the whole value of the unresolved count is that it has never contained one of those.

What the release does instead is make the refusal readable. That stated line names every catalogued candidate the opening paragraph states, in the order the paragraph states them. The adjudication this tool cannot perform still exists only in me. But the evidence I would adjudicate from is now on screen next to the refusal, which is the most the parser is entitled to do.

One of the twelve gains belongs in this section too. MOUSSEUSE SAUCE resolves to manied butter, because its opening calls for "one-half lb. of stiffly-manied butter" and manied butter is now a catalogued entry. The same paragraph continues: "This preparation, though classified as a sauce, is really a compound butter." Escoffier classified it as a sauce and said in the same breath that the classification was a courtesy. The catalogue reads the chapter and records the sauce. The reader reads the sentence and knows better. It stays as recorded. I would rather carry a visible oddity than an invisible rule.

What it did to the scan

The 1907 witness moves from 115 sauces, 36 derived, 79 unresolved to 140, 50, 90. Twenty-five entries entered, not 27. MONTPELLIER BUTTER is on the page at line 3782 of the scan, and HAZEL-NUT BUTTER at 3816:

$ sed -n '3782p;3816p' corpus/escoffier-1907.txt
IS3— MONTPELLIER  BUTTER
15s— HAZEL-NUT  BUTTER

IS3 for 153, 15s for 155. The entry pattern wants digits and never matches, so both sit inside the 284-entry blind spot the second-copy post measured. Repairing them means deciding that S is 5 from outside the document, and ADR-0013 makes no such decision.

The scan also reads two entry numbers twice, 138 and 63, and until this release the catalogue used the number as a preparation's identity, so the later entry's bookkeeping overwrote the earlier one's. A preparation is now identified by the line its heading sits on, which is unique in both witnesses and is the field a reader checks by hand.

The diff moves from 9 unmatched, 18 parent-changed, 35 ocr-suspected to 11, 19, 36. Three of the new rows are the Aurore shape again: HERB, RAVIGOTE and PÉRIGUEUX state two candidates in the proofread text and refuse, the scan hides one candidate in each, and the scan answers. A damaged witness confidently resolving what a clean witness honestly cannot was described once as a finding. It has now happened four times.

Worse than the scanner, in one specific way

The second-copy post gave this series its second rule, observed, never assumed, and wrote it down as ADR-0014: a damaged witness cannot establish absence.

I wrote that record about a scanner. Ten days later the same failure turned up on the clean witness, in a proofread transcription with no OCR damage anywhere in it. The blind spot was a clause I wrote, and the absence it produced went into two posts, two figures, and the published claim that the book never spells out what it spells out at line 1437. The project had 243 tests when I found it, and every one of them was green.

The scan's blind spot could be measured, because there was a clean witness to measure it against. The mother clause's blind spot had no second witness to reveal it, because both copies of the book went through the same rule.

A second copy of the book cannot catch an error in the reader. Only a second reader can, and the second reader here was a person following the chain Robert states, by hand, and asking why the middle of it was not in the catalogue.

Both earlier posts now carry a dated note beside their figure. The figures are left as printed, because what they show is genuinely what the parser recorded at those tags. What they say about the book was wrong.

What I am not claiming

Not one of the ten lost derivations is a finding about Escoffier. Every one is a finding about a resolver that reads names and not verbs. And the 27 admitted entries are a claim about where Escoffier put them, not about what a sauce is — if a lobster method numbered as its own entry in the sauce chapter is not a sauce, the argument is with the book's table of contents.

Everything here reproduces from the tag:

$ git clone https://github.com/Alberto-Codes/saucier
$ cd saucier && git checkout v0.4.0
$ uv sync && uv run saucier parse
$ uv run saucier show cardinal-sauce
$ uv run saucier tree espagnole

That prints 151 sauces and 140, the chain from brown roux to Robert that the book has always written in full, and a stated line under Cardinal naming the two things it says, in the order it says them. If you can tell me a rule that reads "finish the sauce with" as a finish and not a base, without reading anything the sentence does not contain, the issue template asks for the entry number and the source lines. Entry 69, line 2192 is a good place to start.

The release: saucier v0.4.0 — the mother clause deleted: the second test is the chapter clause alone and the heading test still stands beside it, the 27 entries Escoffier filed among his own sauces come back, ten derivations are given up on purpose, and a preparation is identified by the line its heading sits on rather than by an entry number the scan reads twice. MIT.

Google's 4-bit Gemma already fit my 24 GiB card. I wanted the 20,000 tokens it left on the table.

Wed, 02 Sep 2026 00:00:00 GMT

Gemma 4 31B ships with something the last two models didn't have: an official quantization. Google publishes a quantization-aware-trained Q4_0 GGUF, trained toward 4-bit deployment on purpose, at 16.44 GiB. On a 24 GiB card it loads with seven and a half gigabytes to spare.

So for the first time in this series, "it fits" isn't a claim anyone needs to make. The question is what those spare gigabytes are worth — and on this model, they're worth more than usual. If you've been splitting a long transcript into pieces to fit a 24 GiB card, this one is for you. The only way to price the answer was to climb a ladder of context sizes until the card said no.

What a gigabyte of weights buys here

Everything a card holds past the weights goes to the KV cache, the per-token memory that grows with context. Gemma 4 31B's decoder has 60 layers, and 50 of them are sliding-window: each one sees the last 1,024 tokens and no more, so once a sequence passes that window its cache stops growing. Only the 10 global layers keep growing with context — on a runtime that honors the sliding window, as llama.cpp does. One that allocates full-context cache for every layer would price this model six times higher, and the trade below would vanish.

That makes the price of context small and flat. Measured at the runtime — llama.cpp b10362, not the config file — a token of context costs 81,920 bytes once the windows saturate, plus a fixed 1,200 MiB pool per sequence for the fifty windowed layers. (The config's arithmetic says half that per token, because Gemma stores one tensor for both keys and values; the runtime allocates both anyway, and the runtime is what runs.) A gigabyte of weights freed is about 13,100 tokens of context bought.

The budget is the claim

That inverts how vramfit had been thinking. Publications #1 and #2 asked how few bytes a model could survive. This one asks the other direction: pick the context you want, subtract, and give the weights whatever's left. Nine gigabytes of KV headroom on a 24 GiB card leaves 15 GiB for weights — call that arm kv9. That's the one that shipped. An eleven-gigabyte headroom leaves 13 GiB, and that arm is the more interesting story, because it lost.

Measuring a model that only speaks in turns

Before either arm could be solved, the meter had to be fixed. This checkpoint is instruction-tuned and quantization-aware-trained, and it is channel-locked: it has only ever seen text inside its chat frame. Feed it raw prose and it prices it as noise. On the calibration corpus, the same tool read a perplexity in the thousands on bare text and 37.39 on the identical text wrapped in the model's turn markers.

A sensitivity map measured on the raw distribution would price every layer against text no user ever sends. So every measurement in this campaign — the scan, the importance matrix, the perplexity and divergence readings — ran inside one fixed model-turn frame: 357 blocks of public-domain text, 182,404 tokens, the turn markers parsed as the control tokens they are. One trap that step caught: the bf16 conversion had emitted <|turn> as an ordinary token, and only a token count exposed it.

The map itself is keyed per decoder layer, 60 layers and the token embedding, because this is a dense model and a layer is the unit the pack addresses. The vision tower never entered it. A text-measured map licenses no vision claim, and that gap gets its own measurement below.

The recipe: cheap early, protected deep

The solver walked the map greedily by damage per byte saved, under the 15 GiB weight budget, over Google's QAT-unquantized checkpoint — the weights Google annealed toward 4-bit before quantizing them. What came out:

6 layers at Q2_K — layers 1 through 5, and 11. The map priced the early layers cheapest.
9 layers at Q3_K — layer 0, 6 through 10, and 12 through 14.
45 layers at Q4_K — layers 15 through 59. The solve spent its budget protecting depth.
The token embedding and the output head at Q4_K.

The packed decoder is 14.92 GiB, 86.08 MiB under budget, with the recipe's 81-step trace beside it. Google's build is one preset applied uniformly; this one buys forty-five protected layers by spending fifteen cheap ones, and still lands a gigabyte and a half lighter.

Images need a second file: the projector, the encoder that turns pixels into tokens the decoder can read. It ships beside the decoder as a sidecar, priced below.

The metric that pointed backwards

The 13 GiB arm, kv11, was the aggressive point, and on the lingua-franca metric it won. Perplexity ratio against the bf16 reference, in frame:

Arm	Weights	PPL / bf16 ↓	Mean KLD ↓	Same top ↑
kv11 (13 GiB)	12.92 GiB	1.0390	0.1352	86.35%
kv9 (15 GiB, shipped)	14.92 GiB	1.0681	0.0446	92.04%
Google's QAT Q4_0	16.44 GiB	1.1043	0.0420	92.32%

Perplexity ratio ranks the three packs in one order; every fidelity metric ranks them in the opposite order. kv11 has the best perplexity ratio and 3.2 times the divergence of Google's build. A perplexity-only scoreboard would have shipped the wrong file. I don't know why perplexity pointed backwards on this model, and this post doesn't pretend to.

The held-out benchmarks sided with divergence. Five tasks fixed before any run, full evaluation splits, every arm on the same lane. A delta inside the combined standard error is a tie:

Task	kv9 (shipped)	Google's QAT Q4_0	Verdict
MMLU (5-shot)	71.36	70.20	win (+1.15, σ 0.53)
GSM8K (5-shot)	92.34	92.42	tie
HellaSwag (10-shot)	58.71	59.34	tie (−0.63, σ 0.69)
Winogrande (5-shot)	68.27	68.03	tie
ARC-Challenge (25-shot)	61.77	61.09	tie

Four ties and a win, from the lighter file. kv11 took the same slice and lost HellaSwag outright, −2.33 points against a combined σ of 0.69, with four ties around it. Its divergence showed up on exactly one task, which is one more than the shipping bar allows. kv11 would have served the most context of the three, and it stays on the record as the arm that measured too much damage to publish.

Two honest asymmetries travel with the first table. Google's build holds a slightly better mean KL divergence and top-token agreement than the shipped pack, and the benchmarks are what settle that disagreement. And that table is measured on the pack's own calibration frame — the same corpus its importance matrix consumed — which leans the in-frame numbers toward my pack. The held-out slice is the check on that.

What the bytes bought, served

Computed capacity is arithmetic. Served capacity is the ladder: load the file at a context size, step up 4,096 tokens, repeat until the load fails. Both packs ran the ladder on the same RTX 4090, the same day, llama.cpp b10362 Vulkan, one sequence:

Serving shape	This pack	Google's QAT Q4_0	Gain
Text only	86,016 tokens	65,536	+20,480 (+31.25%)
One image aboard	73,728 tokens	49,152	+24,576 (+50%)

The image row moves two variables at once — my decoder and my converted projector — so the card also prints the one-variable version: behind Google's own BF16 projector, this pack serves an image at 69,632 tokens, still +41.7%. The projector conversion adds the last 4,096.

fit24gib is a contract, not a boast, and the boundary has edges:

It moves with the frame — in this project's vocabulary, the exact conditions a number was measured under: the box, the build, the day, the idle VRAM. The 2026-08-28 ladder found 81,920 for text; three days later, in a different frame, the same file passed 86,016. The boundary moves with the box's idle VRAM share. Both are real, and the card names both.
Pass -np 1. The server defaults to four slots, which adds about 2,400 MiB of sliding-window cache on this geometry and fails loads that fit at one.
Keep about 200 MiB free when serving images, and cap the encode batch at one image. The image-encode buffer allocates at request time, and this server build crashes on that failure instead of refusing.
The ladder is a fit bar, not a speed bar. The boundary check decoded five tokens.

Throughput at the boundary is unmeasured, but throughput at a working context is not, and it's the number I refused to print last time. Both quantized arms ran the same 20-task subset at 8,192 tokens of context on the 4090, the target card: 47.8 tokens per second for this pack against 43.3 for Google's. On an H100 the order flipped, with Google's build 15% ahead. Twenty generations, one slot, my pack's answers averaging 20 tokens to Google's 48 — a measured number on the card the claim is about, with its caveats attached.

The vision claim had to be earned separately

The map measured text, and a card that infers image quality from text damage would be guessing. So the campaign measured it: a BF16 reference decoder generated greedily over ten held-out 768×768 images, and each quantized arm was teacher-forced along the reference's sequence, reading a truncated top-20 KL divergence at the server boundary (the server caps its probability list at 20). Positions that carry image content are scored apart from the chat-frame policy tokens that dominate the average.

Content-class results, 120 positions:

Arm	Mean KLD ↓	p95 ↓	Same top ↑
This pack, as shipped	0.0050	0.0193	99.2%
This pack, Google's BF16 projector	0.0045	0.0239	99.2%
Google's QAT Q4_0, Google's BF16 projector	0.0373	0.1928	97.5%

7.5 times below Google's build, 47 times above the instrument's noise floor (1.07e-4, measured by scoring an arm against its own greedy output). The reference ran on CPU — a second instrument — so these read as divergence from the reference, never as same-instrument damage; only the pack-against-Q4_0 comparison shares one instrument.

That measurement is also what priced the projector. Google's ships in BF16 at 1,145 MiB. Run through llama-quantize's Q4_K_M recipe it lands at 629 MiB — and holds not one Q4_K tensor, because every quantizable tensor in it falls back on this geometry to Q5_0 or Q8_0. The recipe name labels the command, not the contents. The cost was 0.0045 to 0.0050 on the content mean; the return was 482 MiB at load and one rung of context. It shipped.

Then a second campaign put the bound somewhere real: 1,349 GUI screenshots at 1280×720 from a public computer-use dataset, three arms on one H100. On image-content positions this pack diverges less than Google's build — 0.0973 against 0.1143 at the mean, and a median 3.6 times lower. Read that as a bound, not a separation: this campaign measured no noise floor, and a prompt-prefix difference between the three files rides inside the mean margin. On a task-identification score, no arm separates: the BF16 reference itself scored 49.9%, my pack 51.2%, Google's 51.4% (from a 512-token regeneration, after every one of its 48-token answers ended mid-thought), all inside a 6.9% judge noise floor. One screenshot against a whole task's name is a ceiling the reference can't clear either.

What I'm still not claiming

The sensitivity map and the importance matrix are not published. Publications #1 and #2 shipped their maps as datasets; this one ships the recipe, the run log, the evaluation sidecars, and both campaign records, but the map and the matrix stay in the run archive. The recipe replays the type placement from the base checkpoint. Without the map, vramfit plan cannot re-solve this model for a different budget, and without the matrix a rebuild reproduces the placement but not this file's exact bytes.

Note 2026-09-04: both have since been published. The sensitivity-map dataset carries the map, the framed importance matrix, and both calibration texts, so neither limit above still holds.

Google's build is the comparator, not the shelf. It is the vendor's own artifact and the obvious baseline. No claim here covers the other GGUFs of this model on the Hub.

The vision numbers bound divergence, not safety. Quantization compresses every tensor with one lossy procedure and can shift any behavior. The tables are the measured bound on that shift over text, ten images, and 1,349 screenshots. Read the card as a damage disclosure. Deploy the pack with whatever protections you'd give the base model.

Damage values don't travel. The map's 0.184 predicted damage is one scan's number in one frame. It ranks layers within this map and nothing else.

Where it lives

The model: gemma-4-31B-it-fit24gib-GGUF — two files, one artifact. The decoder serves text alone; add the projector sidecar for images. The card carries the recipe, the serve ladders, the eval sidecars, both campaign records, and the reproduce commands. Apache 2.0, under the Gemma 4 license note.
The tool: vramfit v0.4.0 — what this campaign forced: per-layer KV geometry priced at the runtime's measured allocation, a capacity readout that turns a packed recipe back into tokens, the projector sidecar and vision line, and the framed-calibration script. MIT.
The policy: ADR-0030 — how a vision tower enters the budget, and what a text-measured map may claim.
The how-to: fit a model to the GPU you actually have — the command-by-command version.

Last time the villain was a label that hid a fallback. This time there was no villain. Google's build is good, it fits, and it's the first comparator in this series I'd happily run. The only thing wrong with it was the seven and a half gigabytes it wasn't using — and the only way to find out what they were worth was to measure, solve, and serve the ladder until it failed.

I added a second copy of the same book. It told me nothing about the book.

Tue, 01 Sep 2026 00:00:00 GMT

I put a second copy of Escoffier into the corpus this week. The project had been reading one text of A Guide to Modern Cookery since its first commit, and there is a second one available — the 1907 first printing, photographed and machine-read by the Internet Archive. Two witnesses of one book, and a command that compares them.

Two things came out of it that are worth having. The catalogue now reads its own edition out of the book's front matter, rather than taking it from the name of a file on my disk — which is infrastructure the rest of this series will stand on, and which did not exist a week ago. And the 1907 printing, which the parser could not read at all when it arrived, now yields 115 sauces.

And then the result, which is the reason for the post: it told me nothing about the book. Not one confirmed editorial difference between the two printings. Every apparent difference I have found so far has turned out to be a property of my instruments — a wrapped heading, a broken separator, a corrupted digit, a gap in my own comparison code. The release produced no facts about a cookbook and four classes of fact about the tools reading it.

That is a better outcome than it sounds. The failure underneath it has a shape worth naming before the cookbook details start, because it is not about cookbooks: an instrument reported an absence it had no way to observe, through a blind spot nobody had measured. It said a book lacked something the book contains. Then, having been corrected, it did it again, and the second time the gates were just as green as the first.

If you run anything that compares two versions of a document and reports what changed, that failure is available to you today, and the question this post is really asking is whether your pipeline could tell you how much of its input it could not read.

First, the label was wrong

Before any of that: this project had been citing an edition it had never parsed.

The corpus file was called escoffier-1907. It is not the 1907 edition. Here is the printing history, transcribed in the file itself, forty lines above the first recipe:

$ sed -n '119,126p' corpus/escoffier-1909.txt
        _First Printed, May 1907
     Second Impression, December 1907
  New and Revised Edition, January 1909
 New Impressions, August 1911, May 1913,
        March 1916, January 1920._


_Copyright 1907 by William Heinemann._

The book is the New and Revised Edition of January 1909, in its January 1920 impression. The only 1907 on the page is a copyright line and the date of a first printing this file is not. I named the file after the most 1907-looking thing in view, and thereafter the code read the filename.

The claims survive this. Every entry number, every line number, every derivation was correct and still reproduces. What was wrong was the label on the book they cite — which, for a record whose whole pitch is go and check, is not a small field to get wrong. It is also ADR-0007 broken by the person who wrote it. That rule says the source decides what counts as a sauce, because deciding for ourselves is what put vanilla ice cream in the catalogue in post #1. A book states its edition more plainly than it states anything else. I read the filename anyway, in the same release that introduced the rule.

So a source now reports the edition it states, and the source_id derives from that reading. Four facts come out of the front matter and are kept apart, because one string cannot carry four: the edition statement, its year, the last impression, and the copyright year. With no edition stated, the copyright year decides — a first printing has no printing history to print. A source that states neither raises rather than guessing.

{ "source_id": "escoffier-1909", "work": "escoffier",
  "origin": "Project Gutenberg 71395", "fidelity": "transcription",
  "edition": { "statement": "New and Revised Edition, January 1909",
               "stated_year": 1909, "impression": "January 1920",
               "copyright_year": 1907 } }

Which frees the name for the real 1907, and both published posts in this series get a correction: they describe the book as Escoffier's 1907 Guide to Modern Cookery, and it is the 1909 revision. I have corrected the prose in both, the way the August 11 post was fixed twice — the canonical post is the one that gets corrected. Each also carries a dated note beside its console block, because the identifier those posts printed is genuinely what the code printed, and a reader running their pinned commands will still see it. The counts, entries and line numbers are unaffected and reproduce exactly as published.

Twenty, then eight, then none

Now the second witness, and the arc that is the actual story.

saucier diff compares two stored catalogues and puts a cause on every row where they disagree. The first time it ran, it reported twenty sauces the 1909 revision had added — twenty preparations present in the later catalogue and absent from the earlier one.

All twenty are printed in the 1907 book. I checked every one by hand. They were invisible to the parser because the entry pattern is ^(\d{1,4})—(.+)$, and a photographed page defeats it three ways: the digits come through as letters, the em dash comes through as something else, or a space lands in the middle.

Six of them, quoted from corpus/escoffier-1907.txt with their line numbers and arranged into a table:

line	as the scan has it	line	as the scan has it
3003	`loi— POULETTE SAUCE`	2072	`33- CHASSEUR SAUCE (Escoffier's Method)`
3147	`Ill— WHITE WINE SAUCE`	3345	`126-- MAYONNAISE SAUCE`
2560	`6s— BERCY SAUCE`	3172	`1 12- APPLE SAUCE`

33- CHASSEUR SAUCE has perfectly legible digits and an ordinary hyphen. The parser wanted an em dash, so the sauce did not exist, and the diff reported that Escoffier had added it in 1909.

Repairing those separators took two rounds and moved the 1907 census from 102 sauces to 113, then to 115. The twenty claimed additions fell to eight — plus one claimed removal, which had the same cause pointing the other way.

Then the third round, which is the one that matters. Eight was still a claim of absence, and absence is not a thing this instrument can observe. So the cause is gone. There is no added row against a scanned witness any more:

  9 unmatched, 18 parent-changed, 35 ocr-suspected
  entries read  2679 of escoffier-1907, 2963 of escoffier-1909, a blind spot of 284
  A witness is ocr. An unmatched row says the diff found no
  counterpart, never that the printing lacks one.
  No row is adjudicated. An ocr-suspected row is a suspicion, and
  separating a scan artefact from a revision needs both lines read.

Twenty, then eight, then none. Three rounds of being confidently wrong about the same question, on a project with 223 tests, 100% coverage, and fourteen decision records — and every one of the three was caught by a person reading output, never by a test. The gates were green the entire time. They were green for the twenty, and they are green now.

Two more of the same species turned up alongside, and they are worth a sentence each because they came from opposite ends of the pipeline. An entry heading that wrapped onto a second line lost its tail, and since the two printings are set to different column widths, they wrapped in different places — so the two witnesses recorded visibly different titles for a heading Escoffier printed identically, and the diff blamed the scan for a difference my own regular expression had manufactured. At the other end, the comparison was intersecting its two indexes, so any preparation whose names had been matched across the witnesses never had its parent compared at all. Six real disagreements were invisible. Neither defect failed anything.

Aurore Sauce, and the failure worth being frightened of

Everything above is an instrument reporting something that is not there. Here is the other direction, which is worse.

Entry 60 is Aurore Sauce. Here is its opening, as the 1909 transcription gives it:

Into one-half pint of boiling velouté put the same quantity of very red tomato purée (No. 29), and mix the two.

Two candidates are named there: velouté and tomato. Post #2 built its whole argument on what the parser does with that situation — two candidates, so the resolver refuses and records unresolved, because the source named two and chose neither. In the 1909 transcription that is exactly what happens.

The scan carries the same sentence with two words damaged. purée comes through as pur^e, which changes nothing, because tomato is the candidate and it survives intact. And velouté comes through as velout^.

$ uv run saucier show aurore-sauce --source escoffier-1907
AURORE SAUCE
entry 60, line 2503, ocr of escoffier-1907
  term  AURORE SAUCE  [en]  aurore-sauce
  parent  tomato

One candidate becomes unreadable. The other survives. The resolver now sees a single unambiguous statement and records tomato — confidently, and wrongly.

The scan did not add noise to that record. It removed the ambiguity that was the reason for the honest answer, and converted a correct refusal into a confident wrong claim. And nothing downstream can tell: the record is well formed, it validates, it carries a source id and an entry and a real line number you can go and read. It is exactly the kind of output post #1 warned about — plausible, checkable-looking, and false — arriving this time with no model anywhere near it.

Now notice what I just did to tell you that. I read velout^ as velouté, and I did it using French orthography and the other witness — evidence from outside the document being read, which is exactly the evidence this project forbids its own code to use. I cannot prove from the scan alone that Escoffier printed velouté there in 1907. I am confident he did, and my confidence is a reader's judgement rather than a record's.

That judgement is the adjudication this post keeps saying the tool cannot perform. It turns out to exist. It exists in me, it is not written down anywhere, and nothing in the catalogue is entitled to it.

There is a one-character fix available and the project refuses to make it. Repairing velout^ back to velouté requires evidence from outside the document being read: French orthography, or the other witness. The separator repairs were allowed precisely because their evidence is internal — a line reading 126-- MAYONNAISE SAUCE looks exactly like the thousands of undamaged headings in the same file, and the repair changes no recorded byte, because the separator never reaches a term. QRIBICHE SAUCE is still recorded as QRIBICHE SAUCE. That is ADR-0013: repair the punctuation that delimits a record, never the characters that constitute one.

The guard on that rule was measured rather than chosen, which I want to note because it is the part people skip. Requiring the whole title in capitals was tried first and held out two real headings, including the Chasseur one above. Four opening capitals admits 57 lines in the scan, and every one of them is a heading.

A second rule, and it is the transferable one

Until this week the series had exactly one epistemic rule: stated, never inferred. The catalogue records what the source says, an unresolved parent stays unresolved rather than being guessed, and everything in the first two posts descends from that.

It now has a second one, and it is independent rather than a refinement: observed, never assumed. ADR-0014 puts it plainly — absence is only observable through an instrument that can see everything present, and where the instrument has a measured blind spot, absence is unobservable and is not reported as observed.

The two rules govern different questions. The first is about what you may read from a source. The second is about what you may conclude from what you read. A comparison involving a scanned witness now reports unmatched, which says the diff found no counterpart and says nothing about what either book contains. Between two proofread texts, added and removed survive — the rule is about damage, not about comparison.

And the blind spot is printed beside the counts rather than in a footnote, so no reader sees how many rows the diff found without also seeing how much of the source it could not read. Today that is 284 entries of the 1907 scan that the parser still cannot see, against 2,679 it can.

That number is the honest reason the next release exists, and it is the thing that has to shrink before an absence claim comes back. Restoring added for a scanned witness is not a matter of a better comparison or a kinder threshold. It requires reading the entries the parser currently cannot read, and then measuring what is left.

What I am not claiming

Eighteen rows report a parent disagreement between the two witnesses. None of them is a finding. Every one carries ocr-suspected, which says a lost candidate explains it as well as a revision does, and the diff adjudicates none of them. Adjudicating one means putting a human eye on both lines — and for the disputed spans, on the photographed page itself, because the question is what the ink says rather than what the text file says. That capability does not exist here yet.

So there is no claim in this post about what Escoffier changed between 1907 and 1909. There is a catalogue of two witnesses, a diff that names what it cannot distinguish, and a measured statement of how much of one witness remains unread.

What breaks next

Post #2 promised the storage layer failing, and this is not that post either. The reason is the same as the reason for everything above: a record that named the wrong book had to stop naming the wrong book before anything got built on top of it.

The bill keeps growing while I do not pay it. There are two catalogues now, saucier diff loads both of them in full to compare them, and closing that 284-entry blind spot means reading a witness against its own page images — which is a third artefact per entry, and the JSON file is still the entire storage layer.

Everything here reproduces from the tag:

$ git clone https://github.com/Alberto-Codes/saucier
$ cd saucier && git checkout v0.3.0
$ uv sync && uv run saucier parse
$ uv run saucier diff escoffier-1907 escoffier-1909

That prints two witnesses of one book — 124 sauces and 115 — twenty-seven rows where the catalogues disagree, a blind spot of 284 entries, and not one adjudicated difference between two printings of A Guide to Modern Cookery. If you can read both lines of a disputed row and tell me which is the scanner and which is Escoffier changing his mind, the issue template asks for the entry number and the source lines. It is the same thing I would have needed to catch the twenty, and the eight.

The release: saucier v0.3.0 — the second witness and the diff that reads it: the 1907 first printing beside the 1909 revision, 9 unmatched rows, 18 parent-changed, 35 ocr-suspected and not one of them adjudicated, and a 284-entry blind spot the tag admits to. MIT.

The 2-bit label was 4.5 bits inside. My 16 GiB card could tell.

Sat, 22 Aug 2026 00:00:00 GMT

I wanted NVIDIA's Nemotron 3.5 Lightning — a 30-billion-parameter mixture-of-experts model — running entirely on a 16 GiB card. The smallest GGUF on the shelf is labeled IQ2_XXS, a 2-bit-class quantization, about as aggressively crushed as llama.cpp gets.

It is 17.54 GiB. It does not fit.

That's not sloppiness, and it's not anyone's dishonesty. The label is doing the only thing it can: naming 12 of the file's 417 tensors.

A label is not a recipe

Last time, the villain was the preset — a fixed rule that never measures the model in front of it. (That post also builds quantization from first principles; if "GGUF" and "2-bit" are new words, start there.) This time the preset doesn't even get to run. llama.cpp's compact quant types work on rows whose width 256 divides, and this model's routed-expert tensors are 2688 and 1856 columns wide. 256 divides neither. So the quantizer falls back, silently, and rewrites every one of those tensors to IQ4_NL at 4.5 bits per weight.

This model stores the 128 routed experts of one projection in one layer as a single tensor — an expert stack. Twenty-three MoE layers, an up and a down projection each: 46 stacks, and those 46 stacks hold 93% of the parameters. The fallback quietly rewrites all 46. Ask for the most aggressive 2-bit build and you get 4.5-bit experts wearing a 2-bit name tag.

It's your abuelita's recipe again, one town over: the jar says what the jar has always said, and the kitchen has been quietly substituting an ingredient for years because the real one doesn't fit the pan. The dish still works. But if your table only seats 16 GiB, the label will not warn you — the smallest thing on the shelf simply doesn't fit, and the shelf's lower-labeled rungs are made of the same locked-out types, so they land at essentially the same size.

The person who published that build, bartowski, did nothing wrong — the fallback belongs to the quantizer, and it runs without asking. This post exists because the way past it is the same move as last time: stop reading labels, start measuring.

Measure the stacks, not the layers

vramfit built a sensitivity map keyed to what this model actually is. Not per-layer — per expert stack, because a stack is the unit a GGUF pack assigns a precision to. Each of the 46 stacks was quantized alone, at each candidate precision, and the damage recorded: how far the model's output distribution shifts from the bf16 reference when only that group is crushed. 46 stacks × 2 candidate widths = 92 measured cells, each with its run-log line — wall clock, memory high-water mark, the number itself.

Two full scans ran, and the difference between them matters later: one weighted its 4-bit fits with the same importance matrix the shelf's build used, and one ran unassisted. Both maps are published, run logs beside them.

The dense weights — attention, Mamba-2, shared experts, embeddings, the output head — never entered the map at all. They're 7% of the parameters, which makes generosity nearly free: the recipe pins every quantizable dense class at 8-bit and spends the real decisions where 93% of the model lives.

The recipe a 16 GiB budget buys

The solver walks the map greedily by damage per byte saved, under the budget. What came out:

11 expert stacks at 2.25 bits per weight (Q2_0) — all of them down_proj stacks, the ones the map priced cheapest, landing spread across the depth from layer 22 on rather than clustered. The figure marks each one; the recipe records them exactly.
The other 35 stacks at Q4_0 — 4.5 bits, same real rate as the shelf's fallback.
118 dense groups at Q8_0, and 46 groups passed through at F16 — the Mamba-2 convolutions and router gates, classes llama.cpp's quantizer never touches.

Q2_0 is the reason this recipe is possible at all: a plain block format — one scale per block, no super-block, no 256-wide requirement — that llama.cpp merged on 2026-07-07. It gives the solver a real 2.25-bit rung on tensors the compact types refuse. (It also means the file needs a llama.cpp build that carries that merge — a build older than it refuses the file. The serve test ran b10326.)

The recipe isn't a magic constant. It records the full solve — the budget bytes, the nine pins, the 11-step demotion trace — so the solve replays from the artifact, and the pack rebuilds this file's bytes on one machine from the base checkpoint. Across machines the byte count can differ by tens of bytes (the GGUF metadata stores the imatrix path); the recipe, not the checksum, is the identity.

fit16gib is a contract, not a boast

The name on the repo says fit16gib, and I want to be precise about what that claims, because "it fits" is exactly the kind of unweighed statement this project exists to replace.

The claim is: this file loads fully offloaded on a 16 GiB card, holds 16k of context, and generates. It's a measured serve result under a stated configuration, not a promise about every runtime.

The test: a hard ballast cap held an RTX 4090 to 16,383 MiB visible, and llama.cpp b10326 (Vulkan) loaded the file with every layer offloaded — 53 of 53, 15,774.00 MiB of weights on the device, with the 357.00 MiB token embedding host-mapped, as llama.cpp always keeps it. llama-server answered a completion request from inside that envelope: 16,157.88 MiB of device buffers against the 16,383 visible. The buffers total more than the 0.224 GiB reserve because the server ran four slots and recurrent state grows per sequence — which is exactly the growth the margin exists to absorb. The published build cannot take this test: its 17.54 GiB of weights exceed the card before the first buffer allocates.

And if that 0.224 GiB reserve looks suspiciously small beside the multi-GiB context reserve of publication #1: this model is a Mamba-2 hybrid with six attention layers, so 16k of KV cache is 96.00 MiB. The reserve is measured buffers rounded up, not a guessed overhead.

The contract's edges, stated plainly:

The margin absorbs recurrent-state growth to 8 parallel sequences. Above 8, the claim is off.
The test's 16,383 MiB is a ballast-capped 4090, not your card. A 16 GiB card driving a display, a different backend, or a runtime with different buffer sizes moves the envelope — the claim is the stated configuration, and the card states it.
No tokens-per-second figure appears here or on the card. The serve ran on a VRAM-capped 4090, and a decode number from that method would read 1.4 to 3.5 times higher than real 16 GiB silicon delivers. Publishing it would be publishing a flattering lie.
A 16 GiB owner can already run bigger builds today by spilling part of the weights to CPU and accepting slower decode. That is a real alternative, and the project has not measured its speed cost. This pack is the option that keeps every weight on the card; whether the trade favors it on your workload is genuinely unsettled.

The scoreboard

Both damage metrics were ruled before the measurement ran, read together from one instrument: the perplexity ratio (how much worse than the f16 original the pack predicts held-out text) and mean KL divergence (how far its output probabilities drift from the original's). Held-out WikiText-2, 594 chunks — my 15.76 GiB pack against the shelf's 17.54 GiB build:

Model	PPL / f16 ↓	Mean KLD ↓	Same top ↑
This pack	1.1611	0.2043	83.13%
`IQ2_XXS` (the shelf)	1.3209	0.3703	76.09%

Both ruled metrics at once: a perplexity ratio 0.16 lower and 44.8% lower mean KL divergence, at 1.78 GiB fewer bytes. Top-token agreement rides along uninvited — it wasn't ruled, and in publication #1 it was the metric that beat me — so it reports here, but it doesn't rank. Last time the win was 2.9% on one metric at matched size. This one isn't close.

Then the task slice — five benchmarks fixed before any run, full evaluation splits, both models on the same lane. A delta inside the combined standard error reports as a tie; Winogrande clears that bar by a whisker:

Task	This pack	The shelf's build	Verdict
MMLU (5-shot)	0.7651	0.6848	ahead, +8.0 points at 16.1σ
HellaSwag (10-shot)	0.8038	0.7652	ahead (6.7σ)
GSM8K (5-shot)	0.7839	0.7627	ahead (1.3σ)
Winogrande (5-shot)	0.7443	0.7261	ahead (1.04σ, barely)
ARC-Challenge (25-shot)	0.6630	0.6715	tie (0.4σ)

Four leads and a tie, from the smaller file. The one nominal deficit prints with its error bar.

One cell I won't fill: the shelf build's bits-per-parameter. Mine is 4.287; its bytes include a block mine omits (next section), so the division would run over different weights. An empty cell is more honest than a wrong one.

The handicap ran the wrong way

The cleanest objection: you used a different importance matrix. No — both packs consumed the same one, bartowski's, 185 entries over 822 chunks. It stays in his repository, linked from my card at a pinned revision with its hash, because it's his work and a link with credit is the right way to use it.

And the assistance ran against me. His build quantizes 91.53% of its bytes with the matrix's help. Mine: 74.44% — no type takes an assisted fit at 2.25 bits on these row widths, and the 8-bit quantizer discards the matrix outright. The pack wins carrying less of the shared advantage.

Which raises the attribution question the second scan exists to answer: is the win the map's, or the matrix's? The unassisted map — measured without the importance matrix — agrees with the assisted one through every rank the solve reads, and yields the identical placement. The win credits the damage ranking, not the borrowed weighting.

Placement was a decision, not a default

One more claim earned its keep before publication: that where the 11 cheap stacks land matters, and the map knows where.

The same campaign packed and measured nine placements of the identical width mix — the map-ranked one, a spread-map probe, three blind draws, a spread-matched control, a measured-map arm, a class-wise arm, and one deliberately inverted to sit on the stacks the map prices dearest. Same 11-at-2.25, 35-at-4.5 spend, nine different answers to "which eleven." The map-ranked placement measured the least damage of the nine; the worst arm measured 2.7 times as much.

Same bytes, same widths, 2.7× spread in damage. Allocation decides — and the map ranked it correctly before any of the nine files existed.

What I'm still not claiming

This pack is missing a block the shelf's build has. The base checkpoint ships multi-token-prediction layers; my conversion dropped them (--no-mtp), so speculative decoding off that block is unavailable from this file. The comparator carries its MTP block at Q4_0 — which is also part of why it's bigger. If MTP-based speculation matters to your serving stack, that's a real difference, not a detail.

No speed claim survives this post. Not mine — the capped-4090 method inflates it. Not the CPU-offload alternative's — nobody measured it. The honest sentence is: this is the only fully-on-card option I can prove exists at 16 GiB and 16k context, and how much that's worth over offloading is an open question I've chosen to leave open in public rather than settle with a number I don't have.

Don't take the damage numbers on tour. The sensitivity values are one scan's measurements in one frame. They rank stacks within this map, and that's all they do — comparing them across scans, calibration sets, or models is meaningless, and the dataset card says so in bold. Rank packed models by measured quality at a fixed model and budget, never by raw damage.

And the ledger keeps growing. Every claim above traces to the evaluation record, the same public trail the first publication started — including the arms that lost. The going standard for a published quant is still a checksum and a file size. A checksum proves the file is the file. These documents are the argument that it's any good.

Where it lives

The model: NVIDIA-Nemotron-3.5-Lightning-30B-A3B-fit16gib-GGUF — a quantization of NVIDIA's Nemotron 3.5 Lightning 30B-A3B, under OpenMDW 1.1. The card carries the recipe, the serve log, the eval sidecars, and the reproduce commands.
The measurements: the sensitivity-map dataset — both scans, run logs beside them. Solve your own card's budget against it.
The how-to: fit a model to the GPU you actually have — the command-by-command version, for your own card and model.
The tool: vramfit v0.3.0 — the tag that solved and packed this build: the stack-keyed scan, the spread placement rule, and the q0/q0-imx meters. MIT.

A week ago I argued that a different ceiling is a different recipe, on paper — and that for the 49B at 16 GiB, the honest answer was no dish at all. That model still doesn't fit a 16 GiB card. This is a different model, solved for that smaller ceiling from the start: a locked-out quant family, a stack-keyed map, and a recipe that looks nothing like last time's, because the model is nothing like last time's. The method didn't change. Measure, then solve.

Marrow Sauce finally has a parent. The parser gave up three answers to earn it.

Fri, 21 Aug 2026 00:00:00 GMT

Post #1 ended with a promise: the next post would be the JSON file failing, because that is the only way I am willing to introduce a database. This is not that post. The chain resolver was promised in the same breath, it costs no infrastructure, and it cut in line. The storage failure is still coming — and by the end of this post it will be closer than it was, for reasons the resolver itself just demonstrated.

Here is what changed, in one sentence: a preparation's parent may now be any catalogued preparation, not only one of the five mothers. Here is what that did to the census, at the tag that carries it, v0.2.0:

$ uv run saucier parse
source      escoffier-1907
mothers     bechamel, espagnole, hollandaise, tomato, veloute
sauces      124
derived     50 linked to a stated parent
unresolved  74 state no base in their prose

Correction, 2026-09-01: that escoffier-1907 is the identifier this release printed, and it was wrong. The file is the New and Revised Edition of January 1909, and its own title page says so. Everything else here is unaffected and still reproduces at v0.2.0. Post three is the correction.

Twenty-nine became fifty. Still no model, no API key, no network — a regular expression got better at reading, nothing more. And the number I want to show you first is not the 24 derivations it added. It is the three it took away.

The promise, kept literally

Post #1 left Marrow Sauce as the standing embarrassment. Entry 45 opens:

Follow the proportions as indicated under "Sauce Bordelaise" (No. 32) for the necessary quantity of this sauce, the Marrow Sauce being only a variety of the Bordelaise.

An author stating a derivation in plain English, and a record answering parent: null — because Bordelaise is not a mother, and the old rule only resolved mothers. The fix was named in that post: resolve to any catalogued sauce and walk the chain. Deterministic, checkable line by line, no model required.

That rule now exists, and:

$ uv run saucier tree bordelaise
SAUCE BORDELAISE  [bordelaise]
└── MARROW SAUCE  (en)

Readers of post #1 have seen this picture before. One arrow in it is different now.

Correction, 2026-09-03: this figure and the paragraph beneath it say the book never spells out that half glaze is a reduction of Espagnole. It does, at entry 23, line 1437, in the first sentence of an entry the parser at this tag could not see. The figure is left as printed, because it shows what the parser recorded. The post about line 1437 is the correction.

Notice what did not change. Bordelaise's own parent is still null. Its opening names shallots, red wine, mignonette pepper, thyme, bay, and half a pint of half-glaze — and a demi-glace is an Espagnole reduction, which every reader of Escoffier knew and the book therefore never says. The resolver could have special-cased it in one line. It did not, because a term that encodes a derivation is not a statement of one, and the moment this parser records one inference it stops being the baseline the eventual model gets measured against. Bordelaise staying unresolved is not a gap in this release. It is the bar the model post will have to clear, set in print for the second time.

Widening the net without catching ice cream

The obvious risk here has a precedent. Post #1's original census put vanilla ice cream in the sauce catalogue because a substring test matched hollandaise inside BOMBE HOLLANDAISE. The old parent rule had five candidate names to match. This one has every name in the catalogue — well over a hundred, in two languages, many of them containing each other. Widening the candidate set widens the ways a match can be wrong. The whole job of this release was to take the wider set without the wider wrongness.

Three rules do that work, and each one exists because the corpus punished its absence (ADR-0008):

A statement is a whole run of words inside one sentence. Folding an opening paragraph flattens punctuation, and a matcher that joins words across a full stop is the ice cream defect wearing a new costume. tomatoes is still not tomato, and a name split across two sentences is not a name.

A run inside the entry's own name states an ingredient, not a parent. Entry 138, HORSE-RADISH SAUCE (line 3091), opens with "finely-rasped horse-radish" — words that match the catalogued HORSE-RADISH OR ALBERT SAUCE (entry 119, line 2804). It is naming its own subject. It stays unresolved. Mothers are exempt from this rule, because a mother is never an entry's own subject — which is why LENTEN ESPAGNOLE still resolves to Espagnole.

A longer stated name shadows the shorter one inside it. Entry 38, Genevoise (line 1769), opens with "add one pint of Lenten Espagnole". The word "Espagnole" sits inside that name. Reading both would turn one clear statement into a fake ambiguity; reading only the longer one records what the author wrote. Under the old rule Genevoise resolved straight to the mother Espagnole — which sounds right and is wrong. The source says Lenten Espagnole, entry 24, line 1449, a catalogued preparation with its own stated parent. So the record now holds a chain, every derivation in it stated:

$ uv run saucier tree espagnole
BROWN SAUCE OR ESPAGNOLE  [espagnole]
├── LENTEN ESPAGNOLE  (fr)
│   └── GENEVOISE SAUCE  (en)
├── ORDINARY POIVRADE SAUCE  (en)
│   └── REFORM SAUCE  (en)
└── POIVRADE SAUCE FOR VENISON  (en)

Escoffier's structure was never a five-way star. The tree finally has the depth the book always had.

The three answers it gave back

Ambiguity still resolves to nothing: exactly one candidate in the opening paragraph, or no parent. Post #1 made that rule sound like modesty. This release shows what it is actually for — because with more candidates in play, the rule started dissolving derivations the old parser had recorded.

ANDALOUSE SAUCE (entry 122, line 2855) was on the books as derived from the mother Tomato. Its opening reads:

Take the required quantity of Mayonnaise sauce (No. 126) and add to it the quarter of its volume of very red and concentrated tomato purée…

The old rule could only see mothers, found the word "tomato", and recorded a derivation — read off an ingredient purée. The new rule sees two candidate names, Mayonnaise and tomato, and two candidates means no parent. The wider net did not just add derivations. It exposed one of the old ones as exactly the grilled-tomatoes defect post #1 was about, one rung further up. Chaud-froid au vert-pré (entry 75, line 2258) went the same way: its opening names the velouté and the white Chaud-Froid sauce, the source chose neither, so neither does the record.

The third one took a human. Post #1's census bugs were caught by a review tool; this release's was caught by a person reviewing the pull request, and it is the same species of defect one layer deeper. The resolver bound each mother to a catalogued entry by preferring the least qualified name — which bound the mother velouté to THICKENED VELOUTÉ, an alias of Allemande (entry 27), instead of ORDINARY VELOUTÉ SAUCE (entry 25). Ordinary Chaud-froid (entry 73, line 2242) opens with "substituting Allemande Sauce for the velouté" — two stated candidates, which the bad binding collapsed into one, so the record answered velouté where the ambiguity rule owed it nothing. The fix binds a mother to the first preparation in source order that answers to its name, because the source states a base before its derivatives — and entry 73 dissolved into the unresolved column, moving the census from 51 to 50 before either number ever got published.

A resolver that can only add derivations is a ratchet. This one handed back three confident answers because the evidence for them stopped clearing the bar, and that — more than the 24 additions — is why the additions are believable.

SHRIMP SAUCE (entry 80, line 2322) still names fish velouté "or, failing this, Béchamel", still gets no parent, and is still the correct answer.

Every changed record, with receipts

Post #1 counted twenty-seven unresolved preparations that name another catalogued sauce in their opening. The resolver records twenty-four new derivations — the difference is the gap between naming and stating, which is precisely the gap the three rules above enforce. Add the three dissolved derivations and Genevoise's corrected parent and 28 records changed, every one checkable with sed -n against the committed corpus:

entry	line	preparation	before	after
38	1769	GENEVOISE SAUCE	espagnole	lenten-espagnole
45	1895	MARROW SAUCE	unresolved	sauce-bordelaise
47	1920	PÉRIGUEUX SAUCE	unresolved	madeira-sauce
59	2087	ANCHOVY SAUCE	unresolved	normande-sauce
68	2185	CAPER SAUCE	unresolved	butter-sauce
70	2202	MUSHROOM SAUCE	unresolved	allemande-sauce
73	2242	ORDINARY CHAUD-FROID SAUCE	veloute	unresolved
75	2258	CHAUD-FROID SAUCE, AU VERT-PRÉ	veloute	unresolved
82	2347	DIPLOMATE SAUCE	unresolved	normande-sauce
83	2354	HERB SAUCE	unresolved	white-wine-sauce
84	2362	GOOSEBERRY SAUCE	unresolved	butter-sauce
86	2390	OYSTER SAUCE	unresolved	normande-sauce
88	2406	JOINVILLE SAUCE	unresolved	normande-sauce
90	2428	MARINIÈRE SAUCE	unresolved	bercy-sauce
94	2468	MUSTARD SAUCE	unresolved	butter-sauce
100	2560	ORIENTAL SAUCE	unresolved	american-sauce
103	2587	REGENCY SAUCE	unresolved	allemande-sauce
107	2661	VENETIAN SAUCE	unresolved	white-wine-sauce
108	2672	VILLEROY SAUCE	unresolved	allemande-sauce
109	2681	VILLEROY SOUBISEE SAUCE	unresolved	allemande-sauce
114	2749	CELERY SAUCE	unresolved	cream-sauce
116	2777	FENNEL SAUCE	unresolved	butter-sauce
119	2804	HORSE-RADISH OR ALBERT SAUCE	unresolved	butter-sauce
120	2823	REFORM SAUCE	unresolved	ordinary-poivrade-sauce
122	2855	ANDALOUSE SAUCE	tomato	unresolved
137	3083	OXFORD SAUCE	unresolved	cumberland-sauce
2412	38683	SAUCE ORANGE	unresolved	apricot-sauce
2414	38696	GREENGAGE OR MIRABELLE SAUCE	unresolved	apricot-sauce

Read down the after column and the book's real middle layer appears: plain Butter Sauce quietly picks up five children — caper, gooseberry, mustard, fennel, and the Albert sauce. Normande gains four. And down in the dessert chapter, Sauce Orange and the Greengage resolve to Apricot Sauce — which is itself unresolved, so the tree now roots in a preparation that states no base of its own:

$ uv run saucier tree apricot-sauce
APRICOT SAUCE  [apricot-sauce]
├── SAUCE ORANGE  (fr)
└── GREENGAGE OR MIRABELLE SAUCE  (en)

That render is not an edge case someone forgot to reject. It is the data model saying what the source says: these two derive from that one, and that one keeps its counsel. Unresolved is a fact about the text, not a hole in the tree.

Two smaller guarantees ride along. Every new derivation carries the same provenance as every old claim — source id, entry, line — which is what made the table above checkable in an afternoon. And a preparation can never become its own ancestor: a cycle is cleared entirely rather than broken by choosing a derivation to keep, because choosing would be an arbitrary decision wearing the costume of determinism, and this project already shipped one of those (SHRIMP SAUCE, resolved to Béchamel because B sorts before V, corrected in post #1).

What this prices

The case for a model was priced in post #1 at 95 unresolved preparations. It is now priced at 74, and the discount came from the cheapest possible vendor: a stricter reading of what the source already says.

That matters for how the eventual model gets judged. Every derivation a deterministic pass can recover is one the model no longer gets credit for recovering. What remains in the 74 is the genuinely hard residue — Bordelaise, where the derivation hides inside the term half-glaze; Shrimp, where the author offered two bases and committed to neither; and the long tail of preparations that simply never say. When a model finally reads those, its score will mean something, because everything a regular expression could honestly claim has already been claimed.

What it cost to write 28 fields

One more number, and it is the ending post #1 actually promised. Changing 28 parents meant rewriting the entire catalogue — all 124 records, serialized back out as one JSON file, because that is the entire storage layer. This release felt the cost that post #1 only predicted: the write is all-or-nothing, the diff is the whole file, and every future rule that touches a handful of records will pay the same full-file toll. The next post is that seam finally tearing, and what replaces it — which is still the only way I am willing to introduce a database.

Everything here reproduces from the tag:

$ git clone https://github.com/Alberto-Codes/saucier
$ cd saucier && git checkout v0.2.0
$ uv sync && uv run saucier parse

124 sauces. 50 derived, every derivation stated. 74 that keep their counsel, on purpose. If you find a stated parent the resolver missed — or an inferred one it should never have recorded — the issue template asks for the entry number and the source lines, because a claim about this project should be checkable the same way the project's own claims are.

The release: saucier v0.2.0 — the resolver this post argues for: a parent may be any catalogued preparation, derived goes from 29 to 50, three derivations the old rule recorded are given back, and Bordelaise stays unresolved on purpose. MIT.

There is no model in this parser. It still told me ice cream was a sauce.

Wed, 19 Aug 2026 00:00:00 GMT

Here is something I wanted to know. It is 2026, every conversation that starts with "I have a pile of documents" ends with which model to point at them, and I had a century-old cookbook sitting on my disk. How much of it can a regular expression read?

Not as a stunt, and not as an argument against models — later posts hand the same book to one and find out what it does better. Just as a question worth answering first, because it costs an afternoon.

So: no model, no API key, no GPU, no network. git clone, git checkout v0.1.0, uv sync, uv run saucier parse, and you are standing where I am standing. Every post in this series adds one ingredient and everything runs at every tag — a sauce should taste like something at every stage of its reduction, and this is the earliest stage there is. The checkout matters: main keeps moving, and the numbers in this post are the numbers at v0.1.0.

The useful part of it is not what the parser got right. It is what it got wrong, how badly, and what caught it — because the thing that caught it is the one property I would keep if I had to throw away everything else in the repo.

Reading a book literally

Sauces specifically, rather than the whole book. A sauce is the most process-dense thing in a kitchen — reductions, emulsions and suspensions, with temperature and timing and order that genuinely matter — and Escoffier's mothers-and-derivatives scheme is already a taxonomy of them. The structure is in the book. The only question is how much of it comes back out.

Quite a lot, because Escoffier structured his own work. Every preparation in A Guide to Modern Cookery is numbered and titled — 2,963 of them — and he names his five base sauces in a sentence you can point at:

The basic sauces: Espagnole, Velouté, Béchamel, Tomato, and Hollandaise.

So the five mothers are not in the source code. A regular expression reads them out of the book:

MOTHERS = re.compile(r"basic sauces?:\s*(.+?)\.", re.IGNORECASE | re.DOTALL)
"""The source names its own base preparations; it is not our place to guess them."""

Hardcoding those five strings produces identical output today and a lie the first time the same code meets Fannie Farmer. Run the whole thing and it reports:

$ uv run saucier parse
source      escoffier-1907
mothers     bechamel, espagnole, hollandaise, tomato, veloute
sauces      124
derived     29 linked to a mother
unresolved  95 state no base in their prose

Correction, 2026-09-01: that escoffier-1907 is the identifier this release printed, and it was wrong. The file is the New and Revised Edition of January 1909, which says so on its own title page. The counts, entries and line numbers here are unaffected and still reproduce at v0.1.0. Post three is the correction.

29 of 124 is a coverage number that would embarrass most extraction demos. Hold that thought.

The first census was worse than embarrassing. It was wrong.

Before those numbers, this project published different ones: 166 sauces, 64 derivations. They were in the README, on the landing page, in the CLI output, and in the argument for why a model comes later. They were also nonsense.

An entry qualified as a sauce if its heading said "sauce", or if the folded heading contained a mother concept anywhere inside it. That second rule is a substring test, and two of the five mothers Escoffier names — tomato, velouté — are ordinary words in a cookery book. The catalogue filled up accordingly: 25 velouté soups from the soup chapter, eight tomato dishes and preserves including TOMATO JAM and TOMATO SALAD, six fish and meat dishes including SOLE A LA HOLLANDAISE, and BOMBE HOLLANDAISE, which is vanilla ice cream in a mould.

Roughly forty of the 166 were not sauces. Thirty of the 64 recorded derivations belonged to entries that are not sauces. And the record it produced looked like this:

{ "title": "GRILLED TOMATOES", "parent": "tomato",
  "ref": { "entry": 2263, "line": 36212 } }

A grilled tomato, recorded as a derivative of the mother sauce Tomato, because the word "tomatoes" occurs in its prose. An absence of evidence turned into a derivation and published with a line number.

There was a second defect underneath it. When an entry's opening paragraph named two mothers, resolve_parent sorted the candidates and took the first alphabetically. It decided three entries that way and was wrong in all three. SHRIMP SAUCE says "fish velouté or, failing this, Béchamel" — the source named both and chose neither, and the parser answered Béchamel because B sorts before V. That is an arbitrary choice wearing the costume of determinism.

Determinism bought reproducibility, not accuracy

The argument for writing a parser before reaching for a model usually runs like this: a model asked for JSON returns well-formed JSON, with a plausible parent for every entry, and those outputs pass validation, read correctly, and are wrong. Confidently wrong output that validates is the failure mode you cannot see.

Every word of that is true, and none of it is a property of models. My parser did it. No weights, no sampling, no temperature. It produced a well-formed, schema-valid, internally consistent catalogue in which vanilla ice cream was a sauce and grilled tomatoes descended from a mother sauce. It did so reproducibly, which meant only that it was wrong the same way every time.

A deterministic system is not a system that cannot be wrong. It is a system whose wrongness holds still while you look at it. That turns out to be worth a great deal — but only if something makes you look.

Who caught it

Not a test. No reasonable test asserts that a cookbook parser hasn't found ice cream.

Copilot did, reviewing the pull request:

is_sauce treats any title containing a mother concept id as a sauce. In the committed corpus this pulls in clear non-sauce entries such as 390—MOCK TOMATOES (folds to mock-tomatoes, which matches tomato) and dish headings like 829—SOLE A LA HOLLANDAISE. This will inflate the catalogue and break the documented/published counts.

It flagged the alphabetical parent bug in the same review, and it was right about both.

Two separate properties of this release made that review possible, and they are worth pulling apart, because only one of them is the one people skip.

The first is that the output is legible. The catalogue is a JSON file of titles a person can read, and BOMBE HOLLANDAISE sitting in a list of sauces is wrong on sight — no line number required. That is why this release writes JSON files rather than a database: while a human is still verifying a parser, inspectable by eye is the correct storage format.

The second is provenance, and it is what turned a suspicion into a finding. Every claim names an entry and a line in a file that ships with the repo:

$ sed -n '1680p' corpus/escoffier-1907.txt
32—SAUCE BORDELAISE

Notice what the review quoted: 390—MOCK TOMATOES, 829—SOLE A LA HOLLANDAISE. Entry numbers, because entry numbers were there to quote. That is the difference between "this rule looks too loose" and a report you can act on in an afternoon — and it is what made the damage countable afterwards rather than merely regrettable: forty of 166, thirty of 64.

Neither property has anything to do with determinism. A model that emitted the same catalogue with the same fields would have been exactly as catchable. One that emitted bare strings with no references would not have been, and neither would this parser.

Four things in this release do that job, and they are the four I would keep if I started over on a different book tomorrow:

Commitment	Costs	Buys
Every claim carries source id, entry, and line	Two integers per record	A wrong answer becomes a findable one
Terms carry a language tag, never a translation	One field	Nixtamal survives as nixtamal, not "corn"
Unresolved is recorded as unresolved, never as "none"	A worse-looking coverage number	A later stage can fill it without overwriting a fact
Ambiguity resolves to nothing	Three fewer derivations	No arbitrary choice is ever recorded as a reading

None of those require a model to be absent. They are what makes a model addable later without destroying the record — extraction sits behind a port, so the thing that fills parent can change while every guarantee about traceability holds.

Who gets to decide what a sauce is

The fix is not a better substring test. It is a rule about authority.

An entry now enters the catalogue on evidence the source supplies. Its heading uses the singular word "sauce" before any "with" — so SOUBISE SAUCE WITH RICE is a sauce served with something, and ASPARAGUS WITH VARIOUS SAUCES is something served with a sauce. Or its heading names a mother and Escoffier filed it in one of the three chapters he titled THE LEADING WARM SAUCES, THE SMALL COMPOUND SAUCES, and COLD SAUCES AND COMPOUND BUTTERS.

Reading those chapter titles is the same move as reading the mothers out of the text. Deciding for ourselves that a velouté soup is not a sauce would not be, however obviously true it is. The distinction sounds pedantic until you notice that the first rule was exactly that kind of self-authored judgment, and it put ice cream in a sauce catalogue.

That is ADR-0007. The census now lives in one place that the tests and the documentation both read, so the next time a number moves it cannot move in only three of the four places that publish it.

What is left: 95

Which brings back the number that was the point of all this. 95 preparations name no mother where the parser looks.

Not all of them are silent. Twenty-seven name another catalogued sauce in their opening paragraph — Allemande, Normande, Bordelaise, Bercy, Madeira, plain Butter Sauce — just not one of the five. Entry 45, Marrow Sauce, opens:

Follow the proportions as indicated under "Sauce Bordelaise" (No. 32) for the necessary quantity of this sauce, the Marrow Sauce being only a variety of the Bordelaise.

An author stating a derivation, in plain English, with a cross-reference number. parent: null, because Bordelaise is not a mother. Escoffier's structure is not a five-way star; it is a graph with depth, and resolving to any catalogued sauce and walking the chain is a legitimate next rule — deterministic, checkable line by line, no model required.

Which leaves roughly 68 that genuinely say nothing — and they are the interesting ones. Bordelaise is among them. Its opening specifies shallots, red wine, mignonette pepper, thyme, bay, and half a pint of half-glaze, and never uses the word Espagnole, because a demi-glace is an Espagnole reduction and the reader knew that. The derivation is encoded in a term rather than a sentence.

Recovering knowledge an author assumed is exactly the work a language model can do and a regular expression cannot. That is the case for adding one, and it is now a priced case rather than an assertion: a model here has to beat 29, against a source where I can tell you precisely which 68 entries it would have to read correctly and which line each one is on.

What breaks next

A JSON file. That is the entire storage layer — parse the book, rewrite the file, read it back to print a tree. It stops working the first time appending one record means rewriting 124, or a question needs answering without loading all of it. The next post is that failure and what replaces it, which is the only way I am willing to introduce a database.

Then one rung per post. A process graph instead of a flat record. NLP over the 68. A local model, then a frontier one, with the cost difference measured rather than assumed. Video, where this schema has to survive input that cannot be checked by eye — which is the real reason text came first, because text is where you can still tell a schema bug from an extraction bug by reading.

Every one of those replaces something in this release. The regex gets replaced. The line numbers do not.

None of which means a regular expression is the answer to your pile of documents. It worked here because Escoffier numbered his entries, titled his chapters, and named his own mothers — and even then it found ice cream. A book that does none of that gives a parser nothing to read, and the first move there is a different one.

The repo is Alberto-Codes/saucier and the parse takes about a second. The documentation site carries the tutorial, the data model, the glossary, and the seven decision records — the API reference on it is generated from the docstrings, so it cannot drift from the code.

If you find an entry the parser should have linked — or another ice cream — there is an issue template for exactly that. It asks for the entry number and the source lines, because a claim about this project should be checkable the same way the project's own claims are.

The release: saucier v0.1.0 — the parser exactly as this post leaves it: 124 sauces, 29 linked to a mother, 95 that state no base in their prose, and every one of those claims citing a line you can open with sed. MIT.

Fit a model to the GPU you actually have

Sat, 15 Aug 2026 00:00:00 GMT

Who this is for

You have a GPU with a fixed amount of memory. You want to run a model that does not fit. You have already tried an off-the-shelf quantization and want one fitted to your card instead of to the average card.

Assumed: comfortable with a terminal, a CUDA GPU. Not assumed: any quantization background — that is the explanation post.

Every command below was run start to finish on a clean rented 4090 with nothing else on it. The output is pasted as it came back.

Before you start

Install. The base install is small on purpose, so the planning step runs on a laptop with no GPU. The heavy parts are extras (ADR-0005):

pip install vramfit           # plan and budget only, no torch
pip install "vramfit[scan]"   # adds torch + transformers, for scan and validate
pip install "vramfit[pack]"   # adds what llama.cpp's converter needs

That base install is nine packages — vramfit, typer, structlog, and typer's own dependencies:

Pygments, annotated-doc, markdown-it-py, mdurl, rich,
shellingham, structlog, typer, vramfit

>>> import torch
ModuleNotFoundError: No module named 'torch'

Two things that catch people. vramfit validate needs [scan], not just scan — it replays a whole recipe through the same meter. And [pack] does not give you llama.cpp; it provisions the interpreter its converter script needs. You build llama.cpp yourself and point at the checkout with --llama-cpp (ADR-0012).

Check that torch can actually see your card before anything else. This is the first thing that bites, and it isn't vramfit's doing. pip resolves a torch build for whatever CUDA it feels like, and on a clean 4090 box with a 12.8 driver I got a 13.0 build:

$ python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
2.13.0+cu130 False

nvidia-smi says CUDA Version: 12.8, torch was built for 13.0, and so the GPU may as well not exist. The fix is to name the index that matches your driver:

pip install --force-reinstall torch --index-url https://download.pytorch.org/whl/cu128

$ python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
2.11.0+cu128 True

Sixty seconds of checking against half an hour of a scan silently running on CPU.

Disk. More than you expect, because the pipeline works through an uncompressed intermediate — an f16 GGUF that everything quantizes from.

	3B walkthrough	49B run
model weights	5.8 GiB	92 GiB
f16 GGUF	5.75 GiB	92.89 GiB
packed output	2.41 GiB	20.36 GiB
llama.cpp build	~2 GiB	~2 GiB

Budget roughly 20 GiB free for a 3B and 115 GiB for a 49B. The f16 conversion is the step that surprises people, not the model download.

Time. Measured, not estimated:

step	3B on a 4090	49B on a 4090
scan	52 min (148 cells)	37.0 hours (328 cells)
plan	seconds	seconds
f16 convert	2 min 12 s	~1 hour
quantize	24 s	16 min 34 s
smoke test	seconds	seconds

The 49B scan streams weights from host RAM, which is most of why it is 40 times slower per cell than a model that fits resident.

1. Work out your real budget

The number that matters is not your card's size. It is your card's size minus what the KV cache will need at the context length you plan to serve. vramfit budget does that arithmetic from the model's own config.json, which matters here because this model's attention blocks are NAS-pruned and heterogeneous — you can't eyeball the cost per token from the parameter count.

$ vramfit budget --vram 24GiB --context 16384 --model-config config.json
attention layers      49  (KV 200704 bytes/token, fp16)
VRAM total            24.00 GiB
- KV cache            3.06 GiB  (16384 tokens x 1 seq)
- runtime overhead    2.00 GiB
= weight budget       18.94 GiB

That remainder is what the solver targets, to the byte.

Do this before anything else, because it can end the job right here. Solving is cheap — seconds, no GPU — so you can find out whether your ceiling is even reachable before spending 37 hours on a scan. Run against a published price list and the answer comes back immediately:

error: no recipe fits the 12.47 GiB weight budget
       — minimum achievable is 15.31 GiB (2.85 GiB over)

That is a 16 GiB card being told this model will not fit, at any precision this price list allows. Better to find out now than after the download.

If your budget does land on a stock preset's size, take the preset — this whole pipeline buys you the most when your ceiling sits between the sizes the shelf stocks. The companion post works three ceilings side by side, including where the recipe's priorities reorder rather than just shrink.

2. Scan

Quantize one layer group at a time, measure how far the output distribution moves, write the per-layer price list. This is the expensive step, and it produces the artifact worth keeping.

You need a calibration text — a few hundred kilobytes of prose the meter pushes through the model to see what moves. Any representative text works. For a run you can compare against mine, take the one published beside the 49B maps:

curl -LO https://huggingface.co/datasets/Alberto-Codes/Llama-3_3-Nemotron-Super-49B-v1_5-sensitivity-maps/resolve/main/calibration.txt

That file is 754 KiB of WikiText-2, and it's the same one every number on this page was measured with.

Here it is on Qwen2.5-3B-Instruct, which is small enough to finish while you have lunch:

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
vramfit scan /path/to/Qwen2.5-3B-Instruct \
  --calibration calibration.txt \
  --max-tokens 32768 \
  --precisions 8,4,3,2 \
  --group-by layer \
  --out sensitivity.json

[3/148] model.embed_tokens @ 3-bit damage 0.249366
[8/148] model.layers.0 @ 2-bit damage 0.905969
[20/148] model.layers.3 @ 2-bit damage 2.908759
...
[147/148] model.layers.35 @ 3-bit damage 0.077071
scanned 37 groups x 4 precisions over 32768 tokens -> sensitivity.json

It prints a line per cell, so you can tell it's alive. What you want to see is 8-bit damage in the thousandths and 2-bit damage one to three orders higher — that spread is the signal. If every precision costs about the same, your calibration text is too short to have converged.

37 groups, 4 precisions, 148 cells, 52 minutes. Look at what it found before you go further: model.layers.3 costs 2.9088 at 2-bit and 0.0011 at 8-bit, a factor of about 2,600 between the cheapest and dearest thing you can do to one layer. That spread is the entire reason this tool exists.

The map is a property of the model, not of your machine. Scan once and you can solve it against any ceiling later, on a laptop, without a GPU — which is what makes step 3 cheap.

The memory cap matters on bigger models. The 3B above fits a 4090 resident and needs no cap at all. The 49B needs --gpu-memory 15GiB on the same card, and dies at 17:

error: scan halted at model.embed_tokens 8-bit: CUDA out of memory.
Tried to allocate 1.96 GiB ... (checkpoint keeps 0 cells)

The cap is lower than the card because the meter needs headroom beside the weights for the reference activations. expandable_segments is in the command above for the same reason — without it, fragmentation kills a cell that would otherwise fit. A halted scan checkpoints, so --resume picks up where it stopped.

Calibration length is the setting people get wrong. 8k tokens is a pilot — not a scan. Re-planning the same budget on a 32k map instead of an 8k one flipped 41 of 82 assignments on the 49B, and predicted damage went from 0.4949 to 0.0940. 32k suffices at 3-bit and above. (The 3B walkthrough above used 32k for the same reason.)

3. Plan

Solve the price list against your ceiling. Seconds, no GPU.

vramfit plan sensitivity.json --vram 4GiB --kv-headroom 1.5625GiB --out recipe.json

The thing worth understanding is that this does nothing interesting until your ceiling actually squeezes. Same 3B map, four cards:

card	weight budget	result	downgrades
12 GiB	9.44 GiB	3.07 GiB, every group at 8-bit	0
6 GiB	3.44 GiB	3.07 GiB, every group at 8-bit	0
4 GiB	2.44 GiB	2.42 GiB, 19 at 8-bit and 18 at 4-bit	18
3 GiB	1.44 GiB	1.43 GiB, 17 at 4-bit and 20 at 3-bit	57

A 3B on a 12 GiB card is not a problem anyone has. Run this on a model that genuinely doesn't fit, or you'll conclude the solver does nothing — it just has nothing to decide. Predicted damage across those four rows runs 0.0340, 0.0340, 0.0861, 0.4973.

Two flags deserve plain explanations, because they're what closed the gap on the published 49B artifact.

--protect "glob=bits" holds specific tensors at a precision floor inside their group (ADR-0022). A layer group is one assignment, but a layer isn't uniform — the attention value projection can be the thing that breaks while the rest of the layer is fine at 3-bit. Protection buys that one tensor back without paying for the whole group. The published recipe carries 48 such pairs.

--exclude-imatrix "glob" quantizes a matched protected tensor without its importance-matrix rows (ADR-0023). It sounds backwards, and it's a remedy rather than a default: occasionally the importance weighting makes a specific tensor's fit collapse, and the reconstruction check below is what tells you which one.

4. Pack

vramfit pack recipe.json \
  --model /path/to/Qwen2.5-3B-Instruct \
  --llama-cpp /path/to/llama.cpp \
  --smoke-text smoke.txt \
  --out qwen3b-fit4gib.gguf

converting /path/to/Qwen2.5-3B-Instruct -> qwen3b-f16.gguf (minutes at 3B scale)
packed 37 groups -> qwen3b-fit4gib.gguf (2.41 GiB), weight budget 2.44 GiB, margin 24.23 MiB under
smoke test: perplexity 16.0119 over 2 chunks, ceiling 1000 — passed

The run log records what actually happened, which is what you want six months later:

[gguf_converted] bytes=6178317216, reused=False, seconds=131.529
[model_packed]   base_type=Q4_K_S, output_tensor_type=q8_0, overrides=36, seconds=23.705
[size_checked]   fits=True, margin_bytes=25402464, weight_budget_bytes=2617245696
[smoke_tested]   chunks=2, passed=True, perplexity=16.0119, threshold=1000.0

Note base_type. The recipe drives every override, but the floor still maps to a stock ftype, so a pack that goes wrong tends to go wrong by quietly falling back to that floor rather than by failing.

On bigger models the importance matrix stops being optional. The 3B pack above ran without one — imatrix=None in that log. At 3-bit on the 49B it was the single largest contributor to closing the gap with the baseline — the scoreboard puts it at about 81 % of the recipe's deficit. Pass it with --imatrix, which also lets pack run the per-tensor reconstruction check below.

5. Check it before you trust it

Non-negotiable, and the reason is a real incident: a recipe once predicted damage 1.44 and the packed artifact was destroyed — perplexity around 10⁶, top-token agreement 0.3 %. Nothing between plan and pack would have caught it.

The smoke test (--smoke-text) runs a short perplexity pass on the packed file and fails above a ceiling (ADR-0017). The 3B run above scored 16.0119 against a ceiling of 1000.

That number is not a quality claim, and you shouldn't read it as one. The smoke test runs two chunks on the packed file and nothing else. It has no f16 reference to compare against, so it can tell you the artifact still produces language — it cannot tell you how much you lost. The ceiling looks absurdly loose because it's a corpse detector: the 10⁶ artifact would have tripped it instantly, and that is the whole job. Measuring what you actually lost is vramfit validate and a real evaluation, which is the subject of the last post.

The reconstruction check runs when you pack with --imatrix and protections. It re-quantizes each protected tensor and measures how far it moved, naming any tensor whose fit collapsed (ADR-0022). On the 49B it took 4 min 36 s and passed. When it doesn't pass, the tensor it names is the input to --exclude-imatrix from step 3.

Neither check tells you the artifact is good. They tell you it isn't broken, which is the more urgent question.

When this is the wrong tool

Off-the-shelf quantizations are good, they're free, and they're one download. This is worth it when your budget is unusual, when you need to know the artifact has no cliffs, or when you want the evidence.

The four-ceiling table in step 3 is the sharpest version of that test. If your ceiling leaves the model comfortable, the solver has nothing to decide and you should take a preset. The further your real ceiling sits from the sizes the shelf happens to stock, the more this pays — and the companion post works that through on a model where the shelf runs out entirely.

A different ceiling is a different recipe. I finally checked.

Sat, 15 Aug 2026 00:00:00 GMT

Cost a braise for a forty-dollar plate and you're refining: better stock, finish the sauce properly. Cost the same braise for twenty-eight and you're not shaving every ingredient by thirty percent. You drop the saffron entirely and keep the technique — a thin version of the dish is worse than a different dish done right.

Cost it for fifteen and there's no version. The professional answer is to say so before anyone shops.

I wrote something in the last post that I believed but hadn't checked:

A 16 GiB card, or the same card serving twice the context, is a different ceiling and therefore a different recipe — not the same recipe squeezed.

Reasonable. Unproven. Only one ceiling had receipts, so I moved on. This is me going back — and the sentence turns out to be too smooth. A different ceiling isn't only a different recipe. Sometimes it's a question with no answer at all.

Why this is cheap to check

The expensive half of measuring a model is the scan. Crush one layer group at a time, measure how far the output moves, write down the price. On a 49B that's 37 hours.

But the scan doesn't know your card. It produces a price list — what each layer group costs you at each precision — and that list is the same whatever you plan to run it on. This is mise en place. Solving that list against a memory ceiling is a separate step, and it takes seconds on a laptop with no GPU.

So — one price list, three ceilings. The list is published, so you can run this yourself.

Your card's size isn't your budget

What the solver targets is what's left after the model's conversation memory is reserved. On this model that's genuinely hard to eyeball — the attention blocks are NAS-pruned and heterogeneous, so you can't infer the cost per token from the parameter count.

$ vramfit budget --vram 24GiB --context 16384 --model-config config.json
attention layers      49  (KV 200704 bytes/token, fp16)
VRAM total            24.00 GiB
- KV cache            3.06 GiB  (16384 tokens x 1 seq)
- runtime overhead    2.00 GiB
= weight budget       18.94 GiB

Three ceilings: the card I've got at the context I published against, a smaller card, and my card serving twice the context.

What came back

	A: 24 GiB, 16k	B: 16 GiB, 16k	C: 24 GiB, 32k
weight budget	20.47 GiB	12.47 GiB	17.41 GiB
recipe	20.46 GiB	none exists	17.13 GiB
predicted damage	0.1215	—	0.2212
bits used	56x2, 13x3, 8x4, 5x8	—	71x2, 4x3, 6x4, 1x8

Column A reproduces the recipe I shipped, to the fourth decimal of predicted damage — that is the control. Without it the other two columns are just numbers I generated.

The most useful column is the empty one

A 16 GiB card can't run this model. Not "runs badly." There is no recipe:

error: no recipe fits the 12.47 GiB weight budget
       — minimum achievable is 15.31 GiB (2.85 GiB over)

This is the part I didn't anticipate when I wrote the original line.

A preset can't tell you this. Presets are fixed rules — you browse file sizes, find one under your number, download it. It will load. Whether the thing inside still works is something you discover later — subjectively, over days of it being slightly off.

The solver's answer is arithmetic. Here is the smallest arrangement this price list permits, here is your budget, the first is bigger by 2.85 GiB. One command, no download.

Being told no quickly is most of what I want from a tool that costs 37 hours when the answer is yes.

If that's your card, you have three real options and the tool won't pick for you. Run a smaller model — a 27B or 32B at these precisions fits 16 GiB comfortably. Rent a bigger card for the work that needs this one. Or accept partial offload and a much slower model, which llama.cpp will do and vramfit doesn't plan for. What you shouldn't do is download a preset that fits and assume the fit means it works.

What actually changed between A and C

Same card, twice the context. My claim was "a different recipe, not the same recipe squeezed." The data says both, and neither.

21 of 82 assignments differ. 61 are identical. So it is not a different recipe. It is also not the same one.
The level drops, as you'd expect: 56 groups at 2-bit becomes 71.
The shape changes, which I didn't expect. Of the 13 groups holding 4-bit or better at 16k, only 7 keep it. The biggest mover is the embedding table, which falls from 8-bit to 3-bit. It is the largest single object on the card, and the solver sells it to keep the layers alive. Layers 76 through 78, the deepest three, drop from 4-bit to 2-bit together.

There is the saffron. At the tighter ceiling the solver doesn't shave everything down a notch — it re-decides what's worth protecting at all, and the first thing it gives up is the most expensive item on the counter.

I'd have guessed the opposite. I'd have guessed it protects the big table and squeezes the many small layers, because that is what "make it smaller" feels like it should mean. The measurement disagrees, and the measurement has receipts.

What a preset user does at each ceiling

This is what decides whether any of it is worth an afternoon.

At A, presets are fine. Honestly. Q3_K_S is 20.45 GiB against a 20.47 GiB budget — it fits with 20 MiB to spare, and the measured recipe beats it by a margin you need instruments to see. That was the whole subject of the last post.

At C the shelf runs out. Every published preset I measured for this model overflows a 17.41 GiB budget. The smallest, IQ3_XXS, is 18.18 GiB and misses by 0.77 GiB. So you take the next one down and wear the gap — at these sizes, most of a bit per weight.

At B nothing works — and you learn it in one command instead of after a download.

The pattern is simple enough: the further your real ceiling sits from the sizes the shelf happens to stock, the more measuring pays. If your budget lands on a preset, take the preset.

One caveat, and it's the honest kind

Column C only exists because that solve was allowed to use 2-bit.

The artifact I published was planned on a price list with the 2-bit column removed, under a rule that bars 2-bit until it's been priced in the runtime rather than in the measurement frame (ADR-0021 decision 4). On that list both B and C are infeasible — the floor is 20.06 GiB, and 32k of context leaves 17.41 GiB.

So under the policy the shipped artifact was actually built with, this model serves 16k of context on a 24 GiB card and nothing else. Doubling the context isn't a tuning exercise. It is a question about whether you trust 2-bit, and on a different model that question has since been measured and answered no, at 4.1 times the reference perplexity.

I'd rather show you the column and tell you what it costs than quietly plan with a width I've argued against everywhere else.

What I'd say now

The original sentence wasn't wrong — it was underspecified in a way that hid the good part.

"A different ceiling is a different recipe" sounds smooth — turn the dial, get a different answer. What happens is that a quarter of the assignments move, the solver's priorities reorder, and at some point the dial runs out of travel and the honest output is an error message.

If I were writing that line again: a different ceiling is a different question, and sometimes it has no answer.

Part of a series on measuring quantization damage. Start with I couldn't tell my quantized model from the baseline, which is where the idea comes from. If you want to run this yourself, the walkthrough takes the whole pipeline from install to a packed file. The full scoreboard behind these numbers is still coming.

I couldn't tell my quantized model from the baseline. The instruments could.

Tue, 11 Aug 2026 00:00:00 GMT

I asked two compressed copies of the same model fifteen questions each. Same questions, same settings, no randomness.

They scored 19 out of 25. Both of them. And they didn't just tie — they agreed. Both got the same four facts wrong. Both wrote the same function with the same bug in it. One file was mine, built by measuring the model layer by layer. The other was the standard off-the-shelf quantization at that size, bartowski's Q3_K_S.

Chat with both and you'd find nothing, and conclude my work bought nothing.

It bought something. You just can't get at it by chatting — and why not is worth more than the result.

The problem, if you've never met it

Nemotron Super 49B is about 93 GB at full precision. An RTX 4090, a very good consumer card, has 24 GB. The model has to fit entirely inside that to run well.

So you compress it. The weights are billions of numbers stored at 16 bits each; store them at 4 bits, or 3, and the file shrinks. That's quantization, and it's why you can run useful models at home at all.

The catch: crushing the numbers damages the model, and not all parts damage equally. Most layers take 3 bits without complaining. A few — the value projections at the front of the stack, and the very first block — need more, and crushing those degrades the whole thing in ways the file size never warns you about.

Here's what surprised me. Essentially every GGUF you can download picks precision from a small family of presets — fixed rules that branch on a few architectural details, but never measure this model's fragility or solve against your memory ceiling. (Some other formats do measure. EXL2 runs a per-layer error pass, though it targets an average bit rate rather than a hard VRAM budget.)

Those presets are your abuelita's recipe. A handful of this, cook it till it looks right. They genuinely work, they're the product of real craft, and the reasons they work were never written down — nobody checks the rule against the model in front of them, because the rule has always been good enough.

I wanted the America's Test Kitchen version. Same dish, weighed in grams, tested in kitchens that aren't hers, with the steps that matter separated from the ones that don't. Not a better rule. No rule at all.

What compression actually takes away

One idea about how these models work explains everything else. It's the only technical thing in this post.

A language model doesn't produce a word. It produces a probability distribution over every possible next word. Tens of thousands of candidates, each with a score. "The capital of France is ___" might come out as Paris 94%, the 2%, a 1%, and a long tail of everything else. Only then does something pick one.

Now the trap is obvious. My fifteen questions used greedy decoding — always take the highest-scoring word. That reads only the tip of the peak. Two copies can agree on the top word often enough that fifteen questions cannot separate them, while the shape behind it has drifted.

Conversation samples the peak. Damage lives in the shoulders.

The instrument that sees the shoulders is KL divergence — how much two probability shapes differ. Point it at the original and a compressed copy across hundreds of pages, and you get one honest number: how far did this copy drift from what it was made from?

That's the number I optimize. Not vibes, and not a preset.

Measure, then solve

vramfit crushes one layer group at a time to build a price list for this specific model, solves that against your memory ceiling, checks the real damage against its own prediction, and only then builds the file.

The output isn't a preset. It's a recipe fitted to one model and one card — and the "one card" part is where this really differs.

Your card's size is not your budget. A 24 GiB card doesn't give you 24 GiB for weights: the model also needs room for the conversation it's holding, and reserving 16k of context left me 20.47 GiB. That remainder is what the solver targets, to the byte.

A 16 GiB card, or the same card serving twice the context, is a different ceiling and therefore a different recipe — not the same recipe squeezed. With presets you browse fixed sizes and take whichever fits under your number, wearing whatever gap is left over. Here the number goes in the front.

At the same file size, the measured recipe drifts less from the original — a 2.9% smaller gap, tiny-sounding but far outside the measurement noise. It also beats the three i-quants that fit the same budget, and ties the baseline on five capability benchmarks, so the closer fit costs nothing the benchmarks can see.

There's a sharper payoff than a better average. An earlier candidate of mine passed every check and looked clean — until a page-by-page comparison found one page out of 564 where it fell off a cliff, diverging more than fifty times worse than the baseline on that text. One page, and nothing had flagged it. The published artifact is the first with no such page anywhere in the 564.

That's the foolproof part, and the real return on weighing your ingredients. Not a better score — a known shape.

The part nobody publishes

Here's what I actually think the edge is, and it isn't the 2.9%.

Five losses, and what each one bought

Before that recipe won, it lost five times.

Full-loop attempt	Date	Result
First real attempt	2026-07-29	Lost — badly
Rematch, handicap removed	2026-07-29	Lost again, by less, for a different reason
Better-converged measurements	2026-07-31	Lost
More honest price list	2026-08-02	Lost
Most honest price list yet	2026-08-06	Lost by the most since the handicaps came off
Ban the most aggressive setting	2026-08-06	Tie on one metric, behind on the other
Pipeline builds its own winner	2026-08-09	Won

Those are the full-loop runs. Several hand-built probes in between are left out, and several of those lost too.

Read the table as a diagnosis, not a confession — each loss eliminated a suspect. The first proved most of my deficit came from a weighting file the competition used and I didn't, not from my recipe at all. The second proved damage doesn't simply add up: crushing two layers together can hurt far more than crushing each alone, which broke an assumption at the center of my solver. The fifth killed my favorite theory — three rounds of better measurements produced three steps backwards, because the frame I measured in and the frame I shipped in disagreed.

Each loss was me finding a step nobody had written down. The weighting file and the way damage compounds were both things the preset had been quietly getting right for years without anybody saying so — the pinch of salt your abuelita never mentions because her hand does it automatically. The difference between an accident and a technique is whether somebody writes it down.

The recipe I did not expect

I assumed measuring would produce something exotic — a wild, uneven allocation no rule of thumb would ever guess. It did, and those were the recipes that lost.

Given honest prices, the solver walked back to almost exactly the shape the preset already had. Your abuelita was right about the dish. What the measurement found was the one step she never mentioned — a single tensor inside the layers, kept at higher precision, in 47 places. That strip is the entire difference between what I published and what everyone downloads.

So I didn't beat the preset by out-thinking it. I beat it by finding the one thing it was missing, and only after five losses told me where not to look. Tracking that down meant working out why the same tensor was fitted ten times worse in my files than in the competition's. I suspected my settings, then my toolchain version. Both innocent. The real cause was upstream and subtle, in how the compression maths behaves when the weighting file has extreme values. Not my bug — but mine to find.

Why the losses are public

Every one of those losses is dated and numbered, with its receipts. Seventeen data points. Architecture decision records for the choices, an issue trail for the arguments, provenance and evaluation sidecars on the published files.

I don't think that's normal, and I think it should be. The going standard for a published quantized model is a checksum, a file size, and sometimes a single quality number. A checksum proves the file is the file. It proves nothing about whether the file is any good — and nothing about the five artifacts that didn't make it.

Weigh the ingredients. Write down the voyage.

What I'm still not claiming

The winning run was not a cold solve. By the time the pipeline built the published artifact, I had hand-discovered three things and handed them to it as inputs: ban the most aggressive setting, protect one tensor across 47 layers, exclude four tensors from the weighting file. The solver reproduced the rest of the layout and made one forced trade of its own. "Measure instead of guessing" is true. "The machine worked it out by itself" is not — the measurements found those rules across five losing attempts, but a human read the measurements.

The baseline beats me on one metric — how often the top word matches the original — by half a point, and that has never flipped. I let KL carry the ranking anyway, because the top word is a one-bit summary that ignores how wrong the model is when it misses, while KL weights the whole shape. If you always decode greedily, take that half point seriously. If you sample, take the KL.

And one control was missing when I first published this. When both models got the same four facts wrong, the natural reading is that the mistakes come from the original rather than from either compression. Natural isn't measured. The 93 GB original had never answered those fifteen questions, because it doesn't fit the card. I said so here, tracked it in the open, and said that if the original got them right, that would be more interesting than the result I had.

It got them wrong. I ran it the same night. 93 GB still doesn't fit 24 GB, so most of the model ran on the CPU — about one word every three or four seconds, thirty-five minutes for fifteen answers. It missed the same four facts. It wrote the same function with the same bug, under the same comment promising it had avoided that exact bug. The mistakes come from the original. Neither compression caused them.

The original scored 20 out of 25 — one point above both copies. Each compression gave up exactly one further point, and they turned out to be the two points that already separated the copies from each other: one flubbed a size parser, the other expanded an acronym wrong. The original got both right.

One point is not nothing, and I won't round it to zero. Compression may have cost each copy that point. Fifteen questions, asked once, cannot tell you whether it did — one point out of twenty-five sits inside what the choice of questions alone can move. That's the same weakness that made the 19-19 tie a weak result, and it cuts the same way in both directions. The conversation can't see the difference. The measurement can.

I'd rather publish the gap named than the story clean. Naming it is also what made it cheap to close.

Where it lives

The tool: vramfit — scan, plan, validate, pack. MIT.
The model: the packed 49B — a quantization of NVIDIA's Nemotron Super 49B v1_5, under the NVIDIA Open Model License. Built with Llama.
The measurements: the sensitivity-map dataset — the per-layer price list the recipe was solved from.
The full ledger: all the data points, every number and every loss.

Coming next: how to fit a model to the card you actually have, and why some layers break and others don't. It's also the second time I've come at quantization from the measurement end — the first was TurboQuant on a vision model, where the first output was garbage for a reason no benchmark would have told me.

"Two patterns that make production database changes boring"

Tue, 02 Jun 2026 00:00:00 GMT

Imagine a restaurant where changing the daily special requires a full kitchen renovation. New permits. Contractor review. A week of construction. The fish market calls at 6 AM with beautiful halibut, but the chef can't put it on the board until the renovation crew finishes, the inspector signs off, and the general manager reviews the blueprints.

That's how some teams treat production configuration changes. A prompt template needs a tweak? Pull request. Code review from an engineer who doesn't evaluate prompts. CI pipeline. Deployment. Change request ticket. The works. By the time the change is live, the moment has passed — like telling the chef the halibut is approved three days after it stopped being fresh.

The daily special exists because restaurants figured out something important: the kitchen infrastructure and the menu are two different things. You build the kitchen once, to code, with proper ventilation and fire suppression and health inspections. Then the chef changes the specials board whenever the market delivers something worth serving. The kitchen doesn't change. The specials board does.

Production configuration works the same way. Application logic, database schemas, API contracts — those are the kitchen. Prompt templates, feature flags, business rules, agent skill configs — those are the daily special. And regulated industries figured out how to change the daily special safely decades before most of us wrote our first SELECT statement.

Two patterns do most of the work: SCD Type 2 and maker-checker.

The two problems people conflate

When someone says "put it in code so it goes through the SDLC," they're conflating two different concerns:

Change safety: preventing bad changes from reaching production
Change auditability: knowing who changed what, when, and why

Code review solves both for code. But for runtime configuration — the daily specials — it solves neither well. A prompt template buried in a YAML file gets the same review process as a critical algorithm change. The reviewer is checking for merge conflicts and syntax, not whether the prompt actually classifies orders better. And the audit trail is git history, which tells you when the file changed but not which version was active at 3:47 PM on Tuesday when the system started behaving strangely.

The SDLC is the right process for building the kitchen. It's the wrong process for deciding what goes on the specials board.

This isn't theoretical:

In 2008, Jérôme Kerviel cost Société Générale €4.9 billion by accumulating enough system entitlements to approve his own trades — one person, both maker and checker. A database constraint enforcing that the proposer and approver must be different people would have made that physically impossible.
In 2012, Knight Capital lost $440 million in 45 minutes because a configuration deployment reactivated dormant code and no second person signed off on the change.
And the classic that every DBA has nightmares about: an UPDATE without a WHERE clause on Black Friday that zeroed out a production table and cost $47M — because the old values were gone the moment the UPDATE committed. With SCD2, there is no UPDATE. Every change is a new row. The previous values are always there.

The litmus test: kitchen vs. specials board

Here's the question I use: does this thing change on the same cadence as a code deployment?

If yes — it's part of the kitchen. Put it in your repo. Review it. Deploy it.

If no — it's on the specials board. Put it in a database with SCD2 and maker-checker.

Thing	Changes with deploys?	Verdict
Application logic	Yes	Kitchen
Database schema	Yes	Kitchen (migrations)
API contracts	Yes	Kitchen
Prompt templates	No — tuned weekly, sometimes daily	Specials board
Feature flags	No — toggled at runtime	Specials board
Business rules / thresholds	No — adjusted by business users	Specials board
Agent skill configs	No — updated as capabilities evolve	Specials board
Notification templates	No — marketing changes them	Specials board
Pricing tiers	No — product changes them quarterly	Specials board

When a prompt engineer needs to tweak a classifier prompt on Thursday afternoon, they shouldn't need a PR, a code review from an engineer who doesn't understand prompts, a CI pipeline, and a production deployment. They should propose the change, have another prompt engineer approve it, and have the system apply it — with a full audit trail, instant rollback, and zero downtime.

That's not less rigorous than the SDLC. It's more rigorous — because the controls are systemic, not procedural. The database constraint enforces dual authorization whether it's Tuesday morning or 2 AM on a holiday. A code review process depends on humans following the rules every time.

Pattern 1: SCD Type 2 — the chef's notebook

Slowly Changing Dimension Type 2 comes from data warehousing, but the pattern applies anywhere you need a complete history of changes to a record. The idea is simple: you never update or delete. Every change inserts a new row.

Think of it as the chef's notebook — the one where every daily special ever served is recorded with the date, what was in it, and why it was chosen. The chef doesn't erase yesterday's entry when today's special changes. They write a new line. Six months from now, when someone asks "what were we running the week that got all those compliments," the answer is right there.

CREATE TABLE prompt_templates (
    id              SERIAL PRIMARY KEY,
    prompt_key      TEXT NOT NULL,
    template_body   TEXT NOT NULL,
    version         INT NOT NULL,

    -- audit columns (inherited from base model in practice)
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    created_by      TEXT NOT NULL,
    approved_by     TEXT,
    approved_at     TIMESTAMPTZ,
    change_reason   TEXT NOT NULL,

    -- temporal columns
    start_date      TIMESTAMPTZ,            -- set on approval
    end_date        TIMESTAMPTZ,            -- NULL = current
    is_current      BOOLEAN NOT NULL DEFAULT FALSE
);

When you change a prompt, you don't UPDATE. You propose a new row, a second person approves it, and the system expires the old version and activates the new one:

-- Step 1: Propose (created_by fills in, approved_by stays NULL)
INSERT INTO prompt_templates
  (prompt_key, template_body, version, created_by, change_reason)
VALUES
  ('order_classifier', 'Classify the following order...', 3,
   'gordon', 'Reduced false positives on international orders');

-- Step 2: Approve (a different user)
BEGIN;
  UPDATE prompt_templates
     SET end_date = now(), is_current = FALSE
   WHERE prompt_key = 'order_classifier' AND is_current = TRUE;

  UPDATE prompt_templates
     SET approved_by = 'marco', approved_at = now(),
         start_date = now(), is_current = TRUE
   WHERE prompt_key = 'order_classifier' AND version = 3;
COMMIT;

What you get for free:

Full audit trail: every version that was ever active, who created it, when it was live, and why it changed. Not in a log file somewhere — in the data itself.
Point-in-time queries: "What prompt was active at 2026-05-15 14:30 UTC?" is a one-liner: WHERE prompt_key = 'order_classifier' AND start_date <= '2026-05-15 14:30+00' AND (end_date IS NULL OR end_date > '2026-05-15 14:30+00').
Instant rollback: expire the current row and copy any previous version forward. One transaction. No deployment pipeline.
Zero data loss: nothing is ever physically deleted. Every state the system has ever been in is recoverable.

Compare this to the git-based approach: to answer "what prompt was active during that incident at 3:47 PM," you'd need to correlate git commit timestamps with deployment timestamps with the actual moment the running application picked up the new config. With SCD2, the answer is one query.

When today's special isn't working: rollback

Thursday's halibut special isn't moving. The chef doesn't tear out the stove and reinstall last week's kitchen. They flip back two pages in the notebook and bring back Tuesday's short rib that sold out by 7 PM.

With SCD2, rollback works the same way. Say version 3 of your prompt is producing bad classifications. You want to go back to version 2 — the one that was working fine last week. Every previous version is still in the table:

BEGIN;
  -- Expire the current version
  UPDATE prompt_templates
     SET end_date = now(), is_current = FALSE
   WHERE prompt_key = 'order_classifier' AND is_current = TRUE;

  -- Bring back version 2 as a new entry
  INSERT INTO prompt_templates
    (prompt_key, template_body, version, created_by, approved_by,
     approved_at, start_date, is_current, change_reason)
  SELECT
    prompt_key, template_body, 4, 'gordon', 'marco',
    now(), now(), TRUE,
    'Rollback to v2: v3 increased false positives on international orders'
  FROM prompt_templates
  WHERE prompt_key = 'order_classifier' AND version = 2;
COMMIT;

Version 3 isn't deleted — it's preserved with timestamps showing exactly when it was active and when it was rolled back. Version 4 is a copy of version 2's content with a new change_reason explaining why. If an auditor asks what happened, you have the full story: v3 went live at 14:00, caused issues, rolled back at 15:22, and here's what v3 was trying to do.

No deployment queue. No waiting for CI. No merge conflicts because someone else pushed to main in the meantime. The application picks up the rolled-back config on its next read — the way a restaurant picks up the new specials board without rebuilding the dining room.

Pattern 2: Maker-checker — the sous chef and the head chef

Also called the four-eyes principle, maker-checker splits every sensitive operation into two steps: one person proposes a change, a different person approves it. The system physically prevents single-actor completion regardless of permission level.

Every kitchen runs on a version of this. The sous chef writes the special on a ticket. The head chef tastes the dish, approves it, and then it goes on the board. Two sets of hands, not because the sous chef can't cook — because the kitchen is designed so nothing reaches the dining room without a second check.

You don't need a separate approvals table. The maker-checker columns live on the same row as the data — the same prompt_templates table from above. A row with approved_by IS NULL is a pending proposal. A row with approved_by filled in and start_date set is live. The database enforces the constraint:

CONSTRAINT different_actors
    CHECK (created_by != approved_by OR approved_by IS NULL)

That's the whole point. The database itself enforces that the maker cannot be the checker. This isn't a policy in a wiki that people follow when they remember — it's a constraint that the system cannot violate.

The workflow maps to the columns directly:

Propose: created_by and created_at fill in. approved_by stays NULL. The row exists but isn't active.
Review: a different user approves — approved_by and approved_at fill in.
Activate: start_date is set, is_current flips to TRUE, and the previous version gets an end_date.

This is how most banks process wire transfers. How most hospitals manage medication orders. The pattern is so well-established that regulators don't just recommend it — they mandate it.

What the regulators actually require

If the instinct is "but compliance won't allow database changes in prod," it's worth checking what the compliance frameworks actually say — because these patterns are exactly what they describe.

Health codes don't require a renovation to change the menu — they require that the kitchen is built to code, and the daily decisions flow through it safely.

Regulatory frameworks work the same way:

Standard	What it requires	Retention	What SCD2 + maker-checker gives you
SOX §302/§404	Internal controls, audit trails for financial systems	7 years	Immutable history + segregation of duties
HIPAA (21 CFR Part 11)	Audit trails: who modified data, when, what changed	6 years	`created_by`, `approved_by`, `start_date`, `change_reason`
PCI-DSS v4.0	Logging of all changes to system components	12 months	Approval workflow + full version history
NIST 800-53 AU	Event logging: who, what, when, where, outcome	Policy-defined	Append-only records, no physical deletes
EU DORA	Change control with full audit trails for ICT systems	Per-contract	Maker-checker workflow + SCD2 trail
GDPR Article 30	Records of processing activities: what and why	Duration of processing	`change_reason` field + complete history

The pattern across all of them: who changed what, when, and why — with immutable records and independent verification. That's SCD2 + maker-checker by construction.

Putting it together

The chef's notebook after a week of specials:

v	special	created	approved	start	end	cur	reason
1	Pan-seared halibut, lemon beurre blanc	Gordon	Marco	Mon	Wed	no	Fresh halibut from market
2	Wagyu beef tartare, truffle vinaigrette	Gordon	Marco	Wed	Fri	no	Wagyu delivery came in
3	Lobster risotto, saffron cream	Marco	Gordon	Fri	Sat	no	Weekend prix fixe
4	Wagyu beef tartare, truffle vinaigrette	Gordon	Marco	Sat	—	yes	Rollback: risotto undersold

Four rows. Every special that ran, who wrote it, who approved it, when it was live, and why it changed. Saturday's special is Wednesday's wagyu brought back — the risotto didn't sell, so they flipped the notebook back two pages. Nothing erased. The health inspector can read it top to bottom and see the whole week.

The specials board changes. The kitchen stays.

What's next: making the SQL disappear

The raw SQL in this post is worth understanding — you should know what's happening at the database level. But in a real application, nobody should be writing expire-and-insert transactions by hand.

In a follow-up, I'll show how to wrap these patterns in a domain layer using SQLModel, the Unit of Work pattern, and ports and adapters. The idea: a base model carries all the audit and temporal columns — created_at, created_by, approved_by, start_date, end_date — and your domain models inherit from it. PromptTemplate just adds prompt_key and template_body. The SCD2 versioning, maker-checker enforcement, and approval workflow happen automatically in the infrastructure layer. Your application code never writes a temporal query directly — the same way a well-run kitchen handles food safety without the chef thinking about health codes on every plate.

Key takeaways

SCD Type 2 gives you a complete, immutable history of every configuration change — who, what, when, why — queryable by timestamp.
Maker-checker enforces dual authorization at the database level, not as a policy people can skip.
Together, these patterns map directly to the audit trail requirements of SOX, HIPAA, PCI-DSS, NIST 800-53, DORA, and GDPR — giving you the building blocks by construction, not by procedure.
The litmus test: if it changes on a different cadence than your code deployments, it's data, not code. Treat it accordingly.
Rollback is one transaction, not a revert-rebuild-redeploy cycle.
The fear of production isn't solved by avoiding production. It's solved by building the kitchen to code so the daily specials can change safely.

"From one model to seven: Making TurboQuant model-portable"

Tue, 31 Mar 2026 00:00:00 GMT

A compression algorithm that only works on one model isn't a tool — it's a demo. When turboquant-vllm v1.0.0 shipped, it was validated on exactly one architecture: Molmo2. The algorithm worked, the numbers were real (3.76x KV compression, ~97% cosine similarity), but every model has its own attention geometry. Head dimensions vary. Some layers use sliding windows. Triton kernels crash when you hand them a dimension that isn't a power of two.

v1.3.0 validates seven model families: Molmo2, Llama 3.1, Mistral 7B, Qwen2.5, Phi-3-mini, Phi-4, Gemma-2, and Gemma-3. Getting there required two things: fused kernels that actually perform well in production (v1.2.0), and kernel-level changes to handle the architectural diversity across those families (v1.3.0).

Fused kernels: decompress and attend in one pass

The v1.1.0 architecture had a clean separation: decompress the KV cache from TQ4 to FP16 in HBM, then run standard attention on the decompressed data. Clean, but wasteful — every decode step wrote decompressed values to HBM just to read them back immediately.

v1.2.0 introduced fused paged TQ4 kernels that eliminate that round trip. The decode kernel reads compressed blocks directly from vLLM's page table, decompresses in SRAM (nibble unpack → centroid gather → norm scale), and computes Q@K^T with online softmax — all in a single Triton kernel. No HBM writes of decompressed cache.

The concrete impact: HBM traffic drops from 1,160 to 136 bytes per token — an 8.5x reduction. A separate INT8 tensor core prefill kernel handles the prefill path with the same fusion strategy.

This wasn't just a performance optimization. It was the foundation for everything that followed. CUDA graph buffer pre-allocation eliminated kernel launch latency for decode. Feature gating allowed the fused path to coexist with the original decompress-first path as a fallback. And when container benchmarks (Experiments 022–023) surfaced OOM bugs in the scratch buffers, the fixes (v1.2.1, v1.2.2) landed in hours because the architecture was clean enough to patch confidently.

The portability problem: Triton and power-of-two

Here's what I didn't appreciate until I tried running TQ4 on Phi-3-mini: Triton's tl.arange requires power-of-two ranges. Molmo2 has head_dim=128 — a power of two. Phi-3-mini has head_dim=96. The kernels crashed at compile time.

The fix sounds simple — pad to the next power of two and mask the boundary. In practice, it touched all five Triton kernels: flash_attention, flash_attention_tq4, flash_attention_tq4_kv, tq4_compress, and tq4_decompress. Each needed a _next_pow2 helper, HEAD_DIM_PAD/HALF_D_PAD constants, and d_mask boundary guards to prevent out-of-bounds reads during the fused decompression step.

Gemma-2 and Gemma-3 added another dimension — literally. head_dim=256 required tuning the flash attention autotune search space (adding BLOCK_M=32 for SRAM optimization). The kernel works, but the autotune cost was real.

The throughput penalty for non-pow2 dimensions is ~5–15%, which is honest and documented. For head_dim=128 models — the majority — there's zero penalty.

Sliding window attention: not all layers compress equally

Gemma models use mixed attention: some layers are global (full context), others use sliding windows (fixed context window with cache eviction). Compressing a sliding window layer's cache breaks the eviction semantics — the cache needs to discard old entries, not keep them in compressed form.

The solution is a bypass. When CompressedDynamicCache encounters a layer with is_sliding=True (from HuggingFace's DynamicSlidingWindowLayer), it skips compression entirely and delegates to the original cache update. Global layers compress normally.

The implementation required None padding in the compressed key/value lists for SWA gaps (to keep layer indices aligned), SWA-aware guards in get_compressed, get_seq_length, and compression_stats, and a diagnostic warning when a Gemma-family config is detected but the cache was created without the model config (which means no SWA metadata). Thirteen new tests cover the bypass, warnings, and downstream guards.

Verification at scale

Model-by-model validation needed a repeatable process, not ad hoc benchmarking. The verify CLI (python -m turboquant_vllm.verify --model <name> --bits 4) loads any HuggingFace model, runs TQ4 compression on random Gaussian input, and reports per-layer cosine similarity against the uncompressed output. Pass threshold: 0.99.

All eight regression models pass:

Molmo2-4B — VLM, head_dim=128, cosine 0.9951
Llama 3.1 8B — text, head_dim=128, lossless at temperature=0
Mistral 7B — text, head_dim=128, lossless at temperature=0
Qwen2.5-3B — text, head_dim=128, cosine ≥0.99
Phi-3-mini — text, head_dim=96 (non-pow2), cosine 0.9952
Phi-4 — text, head_dim=128, cosine ≥0.99
Gemma-2-2b — text, head_dim=256 + SWA, cosine 0.9951
Gemma-3-4B-it — text, head_dim=256 + SWA, cosine 0.9951

Experiment 024 pushed further: Llama 3.1 and Mistral 7B through the full vLLM backend with zero code changes. Short prompts, 960-token passage comprehension, 5-turn conversations with 1,200+ KV tokens — all 6/6 PASS on both models. KV capacity: 1.88x advantage over FP8 baseline. At 16K context, TQ4 serves 6x concurrent requests versus baseline's 3x.

What this means

turboquant-vllm is no longer a single-model proof of concept. If your model uses head_dim 64–256 and runs on vLLM, there's a reasonable chance TQ4 compression works out of the box. The verify CLI takes thirty seconds to check.

The fused kernels, the non-pow2 padding, the SWA bypass — these are the kind of changes that don't show up in a changelog but determine whether a tool survives contact with the real model ecosystem. v1.0.0 proved the algorithm. v1.3.0 proved the engineering.

What's next

Upstream vLLM contribution — the fused paged kernel architecture is the candidate for contribution.
Flash Attention kernel fusion — full multi-layer correctness for the fused path, reducing decode overhead further.
VL-Cache stacking — combining TQ4 KV compression with token pruning for multiplicative savings on VLMs.

Note 2026-09-06: upstream has since shipped TurboQuant in vllm-project/vllm#38479, and the feature request, vllm-project/vllm#38201, is closed.

PyPI | Docs | GitHub

Serve compressed VLM inference from a container

Sat, 28 Mar 2026 00:00:00 GMT

The first turboquant-vllm release proved the algorithm works — pip install, one flag, 3.76x KV cache compression. But if you've ever set up a GPU inference environment from scratch, you know the real friction isn't the model or the framework. It's the CUDA toolkit version, the driver compatibility matrix, the pip packages that refuse to coexist.

v1.1.0 ships a Containerfile that eliminates that entire setup. Build the image once, and every run starts from a known-good state — vLLM, CUDA runtime, and the TQ4 compression plugin verified at build time.

This guide walks through building the container, serving a vision-language model with compressed inference, verifying it works, and optionally running it as a persistent systemd service.

Prerequisites

An NVIDIA GPU with drivers installed (tested on RTX 4090, 24 GB). AMD ROCm also works — adjust the device flag.
Podman or Docker. Commands below use Podman. For Docker, swap podman for docker.
Enough VRAM. Molmo2-8B needs ~24 GB at 6K context with --gpu-memory-utilization 0.90. Molmo2-4B fits with longer contexts on the same card.

Build the image

Clone the repo and build:

git clone https://github.com/Alberto-Codes/turboquant-vllm.git
cd turboquant-vllm
podman build -t vllm-turboquant -f infra/Containerfile.vllm .

The Containerfile does two things: installs turboquant-vllm from PyPI into the official vLLM image, then verifies the plugin entry point registered correctly. If the entry point check fails, the build fails — you won't discover a misconfigured plugin at runtime.

FROM docker.io/vllm/vllm-openai:v0.18.0
ARG TURBOQUANT_VERSION=1.1.0

RUN pip install --no-cache-dir "turboquant-vllm[vllm]==${TURBOQUANT_VERSION}"

RUN python3 -c "\
import importlib.metadata; \
eps = [e for e in importlib.metadata.entry_points(group='vllm.general_plugins') \
       if e.name == 'tq4_backend']; \
assert len(eps) == 1, 'TQ4 entry point not found'; \
print(f'turboquant-vllm {importlib.metadata.version(\"turboquant-vllm\")} — plugin verified')"

The TURBOQUANT_VERSION build arg defaults to 1.1.0. Override it for future versions without touching the file.

Start the server

podman run --rm \
  --device nvidia.com/gpu=all \
  --shm-size=8g \
  -v vllm-models:/root/.cache/huggingface \
  -p 8000:8000 \
  vllm-turboquant \
  --model allenai/Molmo2-8B \
  --attention-backend CUSTOM \
  --dtype auto \
  --max-model-len 6144 \
  --max-num-batched-tokens 6144 \
  --enforce-eager \
  --gpu-memory-utilization 0.90

One flag does all the work: --attention-backend CUSTOM. This tells vLLM to use the TQ4 backend instead of its default attention implementation. Everything else — model loading, tokenization, the OpenAI-compatible API — stays exactly the same.

The named volume (vllm-models) caches model weights between container restarts. Multi-gigabyte checkpoints download once.

Verify compression is active

Watch the container logs for the backend confirmation:

INFO [cuda.py:257] Using AttentionBackendEnum.CUSTOM backend.

If you see FLASH_ATTN or XFORMERS instead, the plugin didn't register. Rebuild the image and check the entry point verification passed.

You can also confirm from inside a running container:

podman exec <container-id> python3 -c "
from turboquant_vllm.vllm import TQ4AttentionBackend
import importlib.metadata
v = importlib.metadata.version('turboquant-vllm')
print(f'turboquant-vllm {v} — plugin loaded')
"

Query the API

The container exposes the standard vLLM OpenAI-compatible API. Nothing changes on the client side:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "allenai/Molmo2-8B",
    "messages": [{"role": "user", "content": "Describe this scene"}],
    "max_tokens": 256
  }' | python3 -m json.tool

Clients don't know — and don't need to know — that the KV cache is 3.76x compressed behind the API.

Persistent deployment with Quadlet

For production, Quadlet manages the container as a systemd service. Create ~/.config/containers/systemd/vllm-turboquant.container:

[Container]
Image=localhost/vllm-turboquant:latest
ContainerName=vllm-tq
SecurityLabelDisable=true
ShmSize=8g
AddDevice=nvidia.com/gpu=all
Exec=allenai/Molmo2-8B \
    --attention-backend CUSTOM \
    --dtype auto \
    --max-model-len 6144 \
    --max-num-batched-tokens 6144 \
    --enforce-eager \
    --gpu-memory-utilization 0.90
Volume=vllm-models.volume:/root/.cache/huggingface
PublishPort=8000:8000
HealthCmd=bash -c 'echo > /dev/tcp/localhost/8000'
HealthInterval=30s
HealthTimeout=10s
HealthRetries=5
HealthStartPeriod=300s

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=default.target

Reload and start:

systemctl --user daemon-reload
systemctl --user start vllm-turboquant

The health check gives the model up to five minutes to load weights before marking the service unhealthy. Restart=always handles crashes and GPU driver hiccups automatically.

Troubleshooting

Build fails at entry point verification. The vLLM base image version may not match turboquant-vllm's requirements. Check PyPI for supported vLLM versions.

Container starts but uses FLASH_ATTN instead of CUSTOM. Confirm --attention-backend CUSTOM is in your run command. In Quadlet, it goes in the Exec= line after the model name.

OOM during prefill. TurboQuant compresses the KV cache, not model weights or activations. Peak memory during prefill is activation-dominated — compression savings appear during generation. Lower --max-model-len or use a smaller model variant.

What you learned

You built a container image with turboquant-vllm baked in, served a vision-language model with 3.76x KV cache compression, verified the plugin was active, and optionally deployed it as a persistent systemd service — all from one Containerfile and one CLI flag.

The full API reference and additional usage guides are on the documentation site.

"Paper to PyPI in 72 hours: Building the first TurboQuant vLLM plugin"

Fri, 27 Mar 2026 00:00:00 GMT

There's a difference between nailing a recipe at home and running it on a restaurant line. At home you control the heat, the timing, the single plate going out. On the line, you need it to work with different stoves, multiple tickets firing at once, and a kitchen that wasn't built around your dish. The first post was the home kitchen version — implementing TurboQuant from the paper, finding what works and what breaks. This post is about getting it on the line.

Google published the TurboQuant paper on March 24, 2026. By March 27, turboquant-vllm was on PyPI serving compressed video inference through vLLM's OpenAI-compatible API. One flag to enable:

pip install turboquant-vllm[vllm]
vllm serve allenai/Molmo2-8B --attention-backend CUSTOM

3.76x KV cache compression. Near-identical output quality. No code changes.

This post is about the production journey — the decisions that turned a research implementation into a pip-installable plugin in 72 hours, and why nobody else has tested TurboQuant on vision-language models.

The gap nobody filled

Every other TurboQuant implementation I could find — and there are several — tests exclusively on text models: Qwen, Gemma, Mistral, Llama. Google's own paper benchmarks on Gemma, Mistral, and Llama-3.1-8B. Text only.

Vision-language models are a harder test case. A 12-second video clip through Molmo2-4B produces ~11,000 visual tokens — 10x longer than typical text prompts. That means 10x more KV cache memory, 10x more opportunities for precision bugs to compound across 36 transformer layers.

The existing VLM KV cache compression literature takes an entirely different approach: token pruning and sparsification (VL-Cache, Dynamic-LLaVA, ZipVL). These methods decide which tokens to discard. TurboQuant compresses the tokens you keep. They're complementary — you could stack TurboQuant on top of pruned caches for even greater savings.

Nobody had validated whether TurboQuant's vector quantization survives the visual token regime. Now someone has.

What shipped

turboquant-vllm 1.0.0 is a vLLM plugin, not a fork. It registers via vllm.general_plugins entry points — the same mechanism vLLM uses for official backends. Install it, pass --attention-backend CUSTOM, and the TQ4 backend handles everything:

Compress — Each new KV vector is rotated by a fixed orthogonal matrix, quantized to 4-bit Lloyd-Max centroids, and nibble-packed (two indices per byte)
Store — Compressed pages use 68 bytes per token per head, vs 256 for FP16
Decompress — Only new tokens are decompressed per decode step (incremental dequantization)
Attend — Standard Flash Attention runs on the decompressed cache

For HuggingFace users, CompressedDynamicCache wraps DynamicCache and compresses transparently on every cache.update().

The numbers

Molmo2-4B on RTX 4090, 11K visual tokens from a Seinfeld video clip:

Metric	Baseline	TQ4 Compressed
KV cache	1,639 MiB	435 MiB (3.76x)
Output quality	Detailed scene description	Near-identical (100+ tokens match word-for-word)
Decode overhead	—	1.78x

Molmo2-8B: same 3.76x compression ratio, correctly identifies all Seinfeld characters. Full 23-minute episode processed across 4 clips at 24 tok/s.

Design decisions that mattered

Plugin, not fork

Other vLLM TurboQuant efforts are forks (brittle, hard to update) or monkey-patches (fragile, version-dependent). turboquant-vllm uses vLLM's official plugin entry point:

[project.entry-points."vllm.general_plugins"]
tq4_backend = "turboquant_vllm.vllm:register_tq4_backend"

pip install registers the backend. --attention-backend CUSTOM activates it. No patching, no forking, no maintenance burden when vLLM updates.

Incremental dequantization

The naive approach decompresses the entire KV cache at every layer at every decode step. For 11K tokens across 36 layers, that's 3.36x overhead.

The fix: decompress only the 1 new token per step, append it to a running buffer, let standard Flash Attention handle the rest. Overhead drops to 1.78x. This optimization isn't in the Google paper — it's what makes TQ4 practical for production serving.

Cross-platform Triton

The fused kernels (compress, decompress, Q@K^T, Flash Attention + TQ4) run on both NVIDIA CUDA and AMD ROCm without code changes. I validated on a Radeon 890M iGPU — 84 of 84 GPU-parametrized tests pass with bit-identical math.

KV cache compression is most useful on memory-constrained hardware — exactly where AMD's consumer GPUs sit.

Validation depth

The v1.0.0 release includes:

180+ tests across 9 test files, 95%+ coverage
16 GPU experiments — each building on the last, documenting failures alongside successes
Cross-platform validation — NVIDIA RTX 4090 + AMD Radeon 890M (ROCm)
Production container test — installed from PyPI into stock vllm/vllm-openai:latest, served Molmo2-8B video inference with zero errors
100% docstring coverage enforced by docvet

The experiment logs document failure modes nobody else has published: the fp16 norms trap at 10K+ tokens, QJL correction being invisible in standard attention, and multi-layer precision drift in fused kernels. These are landmines in every other implementation that hasn't hit 10K+ visual tokens.

What's next

Upstream vLLM contribution — there's an open feature request for TurboQuant support. The plugin is a staging ground.
Flash Attention fusion — the fused Triton kernel achieves 17.8x on the Q@K^T micro-benchmark but needs full softmax+V fusion for multi-layer correctness
Stacking with token pruning — combining TurboQuant compression with VL-Cache-style sparsification for multiplicative savings on VLMs

Note 2026-09-06: upstream has since shipped TurboQuant in vllm-project/vllm#38479, and the feature request, vllm-project/vllm#38201, is closed.

The full implementation, 16 experiment logs, and architecture docs are at github.com/Alberto-Codes/turboquant-vllm.

pip install turboquant-vllm[vllm]
vllm serve your-model --attention-backend CUSTOM

I ran TurboQuant on a vision model. The first output was garbage.

Thu, 26 Mar 2026 00:00:00 GMT

The paper

Google published TurboQuant at ICLR 2026 — a method for compressing transformer KV caches from FP16 down to 3-4 bits per coordinate with near-optimal distortion. The theory is elegant: use Lloyd-Max optimal codebooks to quantize each coordinate independently after a random orthogonal rotation that spreads information across dimensions.

The paper validates on text-only LLMs — Gemma and Mistral. Other implementations have since appeared targeting LLMs on consumer hardware: turboquant-pytorch on an RTX 3060, turboquant_plus on Apple Silicon, and a native llama.cpp integration. But as of March 2026, none of them target vision-language models or video input. I wanted to know if TurboQuant works on Molmo2 — Allen AI's open-source VLM — analyzing Seinfeld clips on a single RTX 4090. The TechCrunch coverage hit on March 25. My first commit was 11 PM that night. By 1:33 AM I had 3.76x compression validated on video.

Why KV cache matters for video

When a vision model processes video, the KV cache grows fast. Molmo2 tokenizes each frame into ~81 visual tokens. A 30-second clip at 2fps produces ~11,000 tokens before the model generates a single word. At FP16, that's 1.6 GB of KV cache across 36 layers.

On a 24 GB RTX 4090, that 1.6 GB is budget you can't spend on longer clips, larger models, or higher frame rates. Compression directly translates to capability.

Experiment 001: garbage

I built the full TurboQuant pipeline — Lloyd-Max codebook solver, two-stage quantizer, and a drop-in DynamicCache wrapper that compresses on write and decompresses on read. The paper recommends TurboQuantProd for keys: 2 bits for MSE quantization, 1 bit for QJL sign correction.

First test: ask Molmo2-4B "What is 2+2?"

	Output
Baseline	"Four."
TurboQuant	"The number 222 is is a number..." (garbled repetition until max tokens)

What the paper assumes you have

You have a 3-bit budget per coordinate. The paper's recommended approach, TurboQuantProd, splits it: 2 bits for MSE quantization, 1 bit for QJL sign correction. That split makes sense when you have a custom attention kernel that uses estimate_inner_product() — QJL correction enables unbiased dot product estimation directly on compressed data.

But in drop-in mode, standard attention decompresses the keys first and computes Q @ K.T on the full vectors. QJL is invisible to standard attention. It's like spending a third of your ingredient budget on a garnish that never leaves the kitchen — the diner never sees it, and the dish suffers because you skimped on the protein.

That means TurboQuantProd at 3 bits actually gives you 2-bit MSE reconstruction — roughly 87% cosine similarity. Compounded across 36 layers of autoregressive generation, 87% per layer cascades into noise.

The fix was one line: switch key compression from TurboQuantProd to TurboQuantMSE. Full 3-bit MSE gives ~95% cosine similarity. The same prompt now returned "Four." — identical to baseline. On an 11,000-token Seinfeld clip, both baseline and compressed produced coherent scene descriptions with only 1.3x overhead.

The compression pipeline

Each KV vector gets its norm extracted (stored as fp32), gets rotated by a shared random orthogonal matrix to spread information across dimensions, then each coordinate is independently quantized using a Lloyd-Max optimal codebook. At 4 bits, two indices pack into one byte — standard bit-shift operations, no custom kernels.

Nibble packing: the practical sweet spot

3-bit compression stored as uint8 gives only 1.94x — one 3-bit index per byte. Packing across byte boundaries requires custom kernels that don't exist in PyTorch or Triton. 4-bit is trivial: two indices per byte via bit shift. The extra bit also improves reconstruction quality.

Mode	Bytes/token	Compression	Cosine similarity
FP16 baseline	256	1.0x	—
TQ3 (unpacked)	132	1.94x	~95%
TQ4 (nibble-packed)	68	3.76x	~97%
TQ3 (bit-packed)	52	4.92x	~95%

TQ4 is the sweet spot: 3.76x compression with better quality than TQ3, using only standard PyTorch operations.

Cutting the overhead in half

My first TQ4 implementation hit 3.76x compression but at 3.36x speed overhead — the cache wrapper re-dequantized all 11,000+ tokens at every layer at every decode step. That's like re-plating the entire service every time a new dish comes off the line. An 88 MB allocation plus a 128x128 rotation matmul, repeated 36 times per generated token.

The fix: maintain a running decompressed buffer and only dequantize the 1 new token per step — plate the new dish, leave the rest on the pass. Prefill still decompresses everything once, but decode drops from O(seq_len) to O(1) per step.

Metric	Full dequant	Incremental
Tokens/sec	8.9	16.9
Overhead	3.36x	1.78x
Compression	3.76x	3.76x

The first 100+ tokens match word-for-word between compressed and baseline output.

Four things I learned the hard way

FP16 norms are a trap. At 10K+ tokens across 36 layers, fp16 norm precision loss compounds and flips low-confidence logits. Always use fp32 — it costs 2 extra bytes per vector.
QJL is invisible in drop-in mode. Without a fused attention kernel that operates directly on compressed data, the QJL correction does nothing. Give all bits to MSE.
Peak VRAM is activation-dominated. KV cache is ~9% of peak VRAM during prefill. During decode, the ratio shifts as the cache grows — that's where compression pays off. The savings are real in permanent storage but invisible to torch.cuda.max_memory_allocated() until sequences grow beyond prefill.
Cache your Lloyd-Max codebooks. A 36-layer model creates 64 compressors. Without @lru_cache on the scipy integration, startup takes 2+ minutes.

From garbage to 3.76x

Two hours earlier, this pipeline produced "The number 222 is is a number..." Now it compresses 1.6 GB of vision-language KV cache into 435 MB with near-identical output quality. The fix that mattered most wasn't an optimization — it was understanding that QJL wastes a bit when you don't have a custom kernel.

A fused Triton kernel that computes Q @ K.T directly on compressed data already shows 17.8x speedup on micro-benchmarks. Multi-layer integration is blocked on full Flash Attention-style fusion — computing Q@K^T, softmax, and @V in a single kernel to avoid fp32/bf16 divergence compounding across 36 layers. That's the next milestone.

The code is MIT-licensed and works with any HuggingFace model:

from transformers import DynamicCache
from turboquant_consumer import CompressedDynamicCache

cache = DynamicCache()
compressed = CompressedDynamicCache(cache, head_dim=128, bits=4)  # nibble-packed, 3.76x compression
# Pass cache to model.generate() — compression is transparent

The full implementation — 62 tests, five experiment logs, and a benchmark harness — is at github.com/Alberto-Codes/turboquant-consumer.

I asked an AI to explain boto3. Then I fixed the docstrings.

Mon, 23 Mar 2026 00:00:00 GMT

The experiment

boto3 is the most downloaded package on PyPI — 43 million installs a day. Every AI coding assistant that helps you write AWS code reads its docstrings. But how good are those docstrings, and does it matter?

I ran an experiment. Clone boto3 twice at the same commit (04dfc51, v1.42.73). Leave one copy untouched. Run docvet on the other — 336 findings across 39 files, 50.3% docstring coverage — and fix every finding. Then ask a fresh AI agent to generate ARCHITECTURE.md from each copy, with no knowledge of what changed.

Same codebase. Same model. Same prompt.

The only difference: docstring quality.

What docvet found

The audit surfaced gaps across every layer of boto3's documentation:

147 missing docstrings — half of all public symbols had no documentation at all
82 missing Returns sections — functions that return values but don't say what
27 missing Attributes sections — classes with undocumented public attributes
13 missing Raises sections — exceptions thrown but never mentioned in docs
45 missing Examples — public classes with no usage examples

After a second pass with strict configuration (ignore-private = false, ignore-magic = false), docvet also surfaced gaps in private methods like _register_default_handlers — the architectural keystone that wires boto3's entire event-driven customization system.

Two architecture documents, one codebase

	Without docvet	With docvet
Lines	618	484
Mermaid diagrams	13	15
Resource factory	Covered	Covered
Action execution	Covered	Covered
Collection pagination	Covered	Covered
Session lifecycle	Covered	Covered
Event-driven customization	Not covered	Full flowchart
CRT transfer backend	Not mentioned	Dedicated diagram
DynamoDB pipeline	Basic	3 diagrams deep
Docs generation system	Absent	Flowchart
Exception hierarchy	Not covered	Class diagram

The untouched agent went deep on the subsystems it could reverse-engineer from code — resource factory internals, action execution flows, collection pagination. But it missed the subsystems that depend on documentation to discover: the event-driven customization wiring, the CRT backend, the documentation generation pipeline.

The docvet-fixed agent covered everything the first agent did, plus five additional subsystems — in 134 fewer lines. It didn't need to spend tokens reverse-engineering what the docstrings already explained.

The difference is clearest in the event system. The agent working with the original docstrings noted: "Session._register_default_handlers() has no docstring at all. This is arguably the most architecturally important method in the codebase. I had to read every register() call and trace the lazy_call targets to understand the customization architecture."

The docvet-fixed agent didn't complain about that method. It diagrammed it — because the docstring told it what was there:

Why docstrings change AI comprehension

The agents got the same facts right. Both understood that boto3 wraps botocore, that the resource factory dynamically generates classes from JSON, that collections handle pagination transparently. The difference was in how they got there.

Without docstrings, the agent reverse-engineers. It reads function bodies, traces imports, follows call chains. This works — AI models are remarkably good at it — but it's expensive in tokens and narrow in scope. The agent spends its budget understanding how individual functions work and runs out before mapping how modules connect.

With docstrings, the agent comprehends. It reads the module docstring, follows the See Also cross-references, checks the Returns and Raises sections, and moves on. It spends less time on each function and more time on the architecture. The result is broader coverage in fewer lines.

This aligns with what the research shows. Macke & Doyle (NAACL 2024) found that incorrect documentation degrades LLM task success by 22.6 percentage points — while missing documentation has no statistically significant effect on accuracy. The AI gets the answers right either way. But the path matters: reverse-engineering is slower, narrower, and misses the connections between modules that docstrings make explicit.

The pop quiz

I tested this with targeted questions across multiple models. The cleanest example: I asked Sonnet what exceptions S3Transfer.upload_file() raises and how to handle them.

Without docvet — the agent reported: "The docstring says only 'Upload a file to an S3 object' — zero mention of type validation or failure behavior." It found the right answer (ValueError and S3UploadFailedError) by reading the method body.

With docvet — the agent reported: "The Raises: section names both S3UploadFailedError and ValueError, which is the right starting point." Same correct answer, found in the documentation instead of the code.

I saw this pattern across every model I tested — Opus, Sonnet, Haiku, GPT-4o, GPT-4.1. The answers converged. The sources diverged. In my testing, the docvet-fixed sessions consistently finished faster — the agents spent less time searching code for answers the docstrings already provided.

What about wrong docstrings?

This experiment only tested missing documentation — I added docstrings where none existed. But the more dangerous case is stale documentation: a docstring that used to be correct but drifted from the code.

This is where docvet's freshness checks matter. The stale-signature rule detects functions whose signatures changed but whose docstrings weren't updated. stale-body catches implementation changes without corresponding doc updates. The new extra-param-in-docstring and extra-raises-in-docstring rules catch docstrings that claim behavior the code no longer exhibits.

boto3 ships new versions almost daily to track AWS API changes. In a codebase that moves that fast, freshness isn't a nice-to-have — it's the difference between documentation that helps your AI tools and documentation that actively misleads them.

What this means for your codebase

boto3 is maintained by Amazon. It has 50.3% docstring coverage. If the most downloaded Python package has 336 documentation gaps that affect how AI understands its architecture, your codebase almost certainly has more.

The fix isn't writing docstrings for the sake of coverage metrics. It's writing docstrings that tell AI agents what they need to know: what a function returns, what it raises, how modules connect, and where to look next. docvet identifies exactly where those gaps are — and with the right configuration, it catches them in private methods and magic methods too.

pip install docvet
# or
uv add docvet --dev

docvet check --all --verbose

Full documentation: alberto-codes.github.io/docvet

The AI reading your code will find the answers either way. The question is whether it finds them in your documentation or reverse-engineers them from your implementation. One path is faster, broader, and produces better results. docvet makes sure the documentation is there when the AI comes looking.

Both architecture documents — original and docvet-fixed — are available for anyone who wants to compare them side by side.

When docstrings lie, your AI tools pay the price

Sun, 22 Mar 2026 00:00:00 GMT

The problem isn't missing docstrings

Most docstring linters answer one question: does this function have a docstring? That's useful, but it's the wrong question for 2026.

A 2024 study by Macke & Doyle found that incorrect documentation degrades LLM task success by 22.6 percentage points — while missing documentation has no statistically significant effect. Read that again. Your AI coding assistant performs worse when your docstrings are wrong than when they don't exist at all.

The reason is straightforward. When a docstring says a function accepts timeout but the parameter was renamed to max_wait three commits ago, every tool that reads that docstring — Copilot, Claude, your IDE's autocomplete — generates code that passes an argument the function doesn't accept. Missing docs let the model fall back on reading the code. Wrong docs actively mislead it.

docvet 1.14 is built around this insight. Instead of just checking whether docstrings exist, it checks whether they match the code.

What bidirectional verification means

Before 1.14, docvet's enrichment checks worked in one direction: they looked at the code and asked "did the docstring mention this?" If your function raised ValueError but the docstring had no Raises: section, docvet flagged it. Useful — but only half the picture.

The other half is the reverse question: does the docstring claim something the code doesn't actually do?

A docstring that says a function raises FileNotFoundError when the function never raises anything isn't just stale — it's a trap. Callers write try/except FileNotFoundError blocks that never trigger, or worse, they restructure their error handling around phantom exceptions. AI tools generate defensive code for errors that can't happen.

v1.14 adds three reverse enrichment rules:

extra-raises-in-docstring — documents exceptions the function doesn't raise
extra-yields-in-docstring — documents yields in a non-generator function
extra-returns-in-docstring — documents return values the function never returns

Combined with the existing forward checks, docvet now performs full bidirectional verification: every claim in the docstring has a corresponding behavior in the code, and every behavior in the code has a corresponding claim in the docstring. With this release, docvet reaches 31 rules across six quality layers — and truthfulness is now the central theme.

Parameter agreement: where drift actually lives

The most impactful new feature in 1.14 is parameter agreement checking. Two rules — missing-param-in-docstring and extra-param-in-docstring — compare the Args: section against the function signature, parameter by parameter.

This catches the single most common form of documentation drift: renamed, added, or removed parameters where the docstring wasn't updated. It happens constantly in active codebases. You rename retries to max_retries across a refactor, update every call site, and forget the one place that still says retries — the docstring.

The checks handle real-world Python signatures: positional-only parameters (PEP 570), keyword-only parameters, self/cls exclusion, and optional *args/**kwargs filtering. Both Google and Sphinx docstring styles are supported.

Here's what a finding looks like:

src/client.py:47: missing-param-in-docstring Function 'connect' has parameters not documented in Args: max_retries [required]

That's a one-line signal that would otherwise become a user-filed bug report, or worse, silently wrong AI-generated code.

Catching docstrings that say nothing

Not all low-quality docstrings are wrong — some are just empty calories. The new trivial-docstring rule detects summary lines that restate the symbol name without adding information:

def get_user():
    """Get user."""

This docstring technically exists. It passes every presence check. But it tells you nothing you couldn't read from the function name. docvet decomposes both the symbol name and the summary into word sets (handling CamelCase and snake_case), filters stop words, and flags cases where the summary is a subset of the name.

The rule skips @property and @cached_property — those frequently have legitimate one-line summaries that mirror the attribute name.

Deprecation and constructor documentation

Two more rules round out the release:

missing-deprecation detects functions that use warnings.warn(DeprecationWarning) or the @deprecated decorator (PEP 702) without mentioning deprecation anywhere in their docstring. The match is intentionally loose — the word "deprecated" anywhere in the docstring satisfies it, because there's no single standard format for deprecation notices.

undocumented-init-params catches classes whose __init__ accepts parameters but neither the class docstring nor the __init__ docstring has an Args: section. This one defaults to off — it's opt-in for teams that want it.

Design tradeoffs

A few deliberate choices worth calling out:

Reverse checks use recommended severity, not required. Forward checks ("you raise but don't document it") have low false-positive rates because the code is the source of truth. Reverse checks ("you document but don't raise it") have higher false-positive risk — a function might delegate to a helper that raises, or a parent class might document exceptions from subclass overrides. We chose recommended to surface these without blocking CI pipelines.

Two rules default to off. missing-return-type and undocumented-init-params are opt-in. Not every team wants to enforce return types in docstrings when they already have type annotations, and not every class needs __init__ parameter docs in the class body. Progressive adoption matters.

Parameter agreement checks support both directions simultaneously. Rather than shipping "check if params are documented" and "check if documented params exist" as a single rule, they're separate rules gated by the same require-param-agreement config key. This means you can see exactly which direction the drift went — missing documentation vs. stale documentation — in your CI output.

Who this is for

If you maintain a Python library that other developers (or AI tools) consume, v1.14 catches the class of documentation bugs that cause the most downstream damage. Parameter mismatches, phantom exceptions, and trivial summaries are the documentation equivalent of type errors — they compile fine but break at runtime.

If you're using docvet in CI already, upgrade and enable the new checks:

pip install docvet
# or
uv add docvet --dev

The param agreement checks are on by default. Reverse enrichment checks are on by default. missing-return-type and undocumented-init-params are opt-in — enable them in pyproject.toml when you're ready:

[tool.docvet.enrichment]
require-return-type = true
require-init-params = true

What's next

v1.14 brings docvet to 31 rules across six quality layers. The next focus area is expanding what "truthfulness" means — moving beyond structural checks into semantic verification. The question isn't just "did you document the parameters?" but "is what you said about them accurate?"

Every function with a stale Args: section is a wrong answer waiting to happen — in your IDE, in your code review, in the pull request your AI assistant is about to generate. docvet 1.14 catches them before they ship.

Encrypt ADK Sessions in 5 Minutes

Fri, 06 Mar 2026 00:00:00 GMT

In Part 1, I showed that Google ADK stores everything your agent knows — tool calls, user messages, conversation context — in plaintext SQLite. If that made you uncomfortable, this post fixes it.

This is the recipe card. Ingredients, steps, done. The explanation of why the soufflé rises comes later in Part 3.

Prerequisites

Python 3.12+
An existing ADK agent using DatabaseSessionService (or a willingness to create a minimal one)

No system libraries, no C compilation, no Docker. The library is pure Python with two runtime dependencies: google-adk and cryptography. A short ingredient list.

Step 1: Install

pip install adk-secure-sessions

Or with uv:

uv add adk-secure-sessions

Verify the install:

python -c "import adk_secure_sessions; print('OK')"

Step 2: Swap the Import

Your agent code probably has something like this:

# Before — ADK default (unencrypted):
from google.adk.sessions import DatabaseSessionService

session_service = DatabaseSessionService(
    db_url="sqlite+aiosqlite:///sessions.db"
)

Replace it with:

# After — encrypted:
from adk_secure_sessions import EncryptedSessionService, FernetBackend

session_service = EncryptedSessionService(
    db_url="sqlite+aiosqlite:///sessions.db",
    backend=FernetBackend("your-secret-passphrase"),
)

Two changes: the import line and the constructor. Everything else in your agent stays the same — create_session, get_session, list_sessions, delete_session, append_event — the full ADK session lifecycle, identical behavior. The difference is what hits the disk.

Step 3: Use the Async Context Manager

For proper connection cleanup, wrap the service in async with. Here's a complete, runnable script:

import asyncio
from adk_secure_sessions import EncryptedSessionService, FernetBackend


async def main():
    backend = FernetBackend("my-secret-passphrase")

    async with EncryptedSessionService(
        db_url="sqlite+aiosqlite:///sessions.db",
        backend=backend,
    ) as service:
        # Create a session with sensitive state
        session = await service.create_session(
            app_name="my-agent",
            user_id="user-123",
            state={
                "patient_name": "Jane Doe",
                "diagnosis_code": "J06.9",
                "api_key": "sk-secret-key-12345",
            },
        )
        print(f"Created session: {session.id}")

        # Retrieve — state is automatically decrypted
        session = await service.get_session(
            app_name="my-agent",
            user_id="user-123",
            session_id=session.id,
        )
        print(f"Decrypted state: {session.state}")

        # List sessions for this app/user
        response = await service.list_sessions(
            app_name="my-agent",
            user_id="user-123",
        )
        print(f"Sessions found: {len(response.sessions)}")

        # Clean up when you're done
        await service.delete_session(
            app_name="my-agent",
            user_id="user-123",
            session_id=session.id,
        )
        print("Session deleted")


asyncio.run(main())

Copy this into a file and run it. The API behaves identically to ADK's DatabaseSessionService — same methods, same signatures, same return types. The only difference is what's stored on disk: switching from a glass jar to a lockbox. Same ingredients go in, same ingredients come out, but nobody can peek inside without the key.

Step 4: Verify the Encryption

Trust but verify. Open the SQLite database directly and confirm the data is actually encrypted.

Using the sqlite3 CLI:

sqlite3 sessions.db "SELECT state FROM sessions LIMIT 1;"

You'll see a base64-encoded string — the encrypted envelope — not readable JSON:

AQFnQUFBQUJuVm1Gc2RX...

Using Python:

import sqlite3

conn = sqlite3.connect("sessions.db")
row = conn.execute("SELECT state FROM sessions LIMIT 1").fetchone()
print(row[0][:60])  # First 60 chars of the encrypted envelope
conn.close()

What you won't see: {"patient_name": "Jane Doe", "diagnosis_code": "J06.9"}. That's the point. With DatabaseSessionService, anyone with file access reads your mise en place. With EncryptedSessionService, they see noise.

For a more convincing demo, run the basic usage example from the repo — it runs a real multi-turn ADK agent with Ollama and then inspects the raw database to prove no plaintext leaks. After a three-turn conversation about patient intake, the database contains zero occurrences of "Jane Doe" or "headache."

Step 5: Manage Your Passphrase

The passphrase is the only secret. Never hardcode it.

import os
from adk_secure_sessions import EncryptedSessionService, FernetBackend

backend = FernetBackend(os.environ["SESSION_KEY"])

Set it in your environment, your .env file, or your secrets manager. The library handles everything else — FernetBackend derives a cryptographic key using PBKDF2-HMAC-SHA256 with 480,000 iterations. You don't need to generate, store, or rotate raw key material.

If you use the wrong passphrase to read a session encrypted with a different one, you get a clear DecryptionError — never garbage data, never silent corruption.

What You Just Built

Five steps, plaintext to encrypted-at-rest:

Installed — pip install adk-secure-sessions
Swapped — one import, one constructor change
Ran — same API, encrypted storage
Verified — the database contains ciphertext, not JSON
Secured — passphrase in the environment, not the codebase

Your agent still works the same way. Your tests still pass. But the SQLite file is now useless without the key — like a walk-in freezer with a combination lock. Nothing changes about how the food is stored or retrieved, but the back door isn't open anymore.

Error Handling

When things go wrong, the library tells you what happened:

ConfigurationError — raised at startup if the backend is misconfigured. You'll catch this before any data is written.
DecryptionError — raised if you read a session with the wrong key. The library never returns garbage.

from adk_secure_sessions import (
    ConfigurationError,
    DecryptionError,
    EncryptedSessionService,
    FernetBackend,
)

try:
    async with EncryptedSessionService(
        db_url="sqlite+aiosqlite:///sessions.db",
        backend=FernetBackend("correct-passphrase"),
    ) as service:
        session = await service.get_session(
            app_name="my-agent",
            user_id="user-123",
            session_id="some-session-id",
        )
        if session is None:
            print("Session not found")
except ConfigurationError:
    print("Backend doesn't conform to EncryptionBackend protocol")
except DecryptionError:
    print("Wrong key — cannot decrypt session data")

Key Takeaways

One install, one import change — pip install adk-secure-sessions, swap the constructor, done
Full ADK lifecycle — create_session, get_session, list_sessions, delete_session, and append_event all work identically
Verify it yourself — inspect the SQLite file to confirm ciphertext, not plaintext
Passphrase management — use environment variables, never hardcode secrets
Clear errors — DecryptionError for wrong keys, ConfigurationError for bad setup

Stop Writing AI Agent Prompts by Hand

Thu, 05 Mar 2026 00:00:00 GMT

You write a prompt. You test it. It's okay but not great — the agent misses edge cases, ignores context, or produces output that's technically correct but tonally wrong. So you tweak a sentence. Test again. Better on one example, worse on another. You rewrite the whole thing. Test again. Repeat until you're tired or the deadline hits, whichever comes first.

This is how most people build AI agents. It's also how most people season food before they learn to taste as they go — a pinch of this, a dash of that, hope for the best. It works, kind of. But it doesn't scale, it doesn't generalize, and it leaves performance on the table that you'll never find through manual iteration.

What if an algorithm could do this loop for you — and do it better?

The Problem With Human Prompt Engineering

Manual prompt engineering has a ceiling. Humans are good at writing instructions that sound clear to other humans, but LLMs don't process text the way we do. The prompt that reads best to you isn't necessarily the prompt that produces the best output from the model.

There are a few specific failure modes:

Local optima — you find a prompt that works well enough and stop iterating, never discovering the dramatically different phrasing that scores higher
Narrow testing — you test against two or three examples, miss the edge cases, and discover them in production
Instinct over data — you change what feels wrong instead of what measurably underperforms
Single-objective thinking — you optimize for one quality (accuracy, tone, format) while degrading others you're not watching

These aren't character flaws. They're the natural limitations of a human trying to search a vast space of possible text through trial and error. The space of possible instructions for even a simple agent is effectively infinite, and your ability to explore it is limited by time, patience, and cognitive bias.

Evolutionary Optimization: The Core Idea

Evolutionary prompt optimization borrows from genetic algorithms. Instead of you guessing at better prompts, an algorithm systematically explores the space:

Evaluate — run the agent on a batch of examples, score the outputs
Reflect — analyze what worked and what didn't across all examples
Mutate — an LLM proposes improved instructions based on the analysis
Select — if the new instruction scores better, keep it; otherwise, discard it

Repeat until the scores plateau or you hit an iteration limit.

The key insight is step 3. The mutation isn't random — it's informed. A reflection model sees the agent's actual outputs, the scores, and the feedback, then proposes specific text changes to address the weaknesses it observed. It's less "random mutation" and more "systematic recipe refinement" — tasting every dish that comes out of the kitchen, identifying exactly what's off, and adjusting the technique accordingly.

This is the approach described in the GEPA paper (Genetic-Pareto prompt optimizer). GEPA treats prompt components as evolvable genes and uses LLM-powered reflection instead of random crossover. The result is an optimizer that can improve agent instructions in ways that surprise even the person who wrote the original prompt.

What This Looks Like in Practice

gepa-adk is a Python library that brings evolutionary optimization to Google ADK agents. You give it an agent, training examples, and a critic — it gives you back a better prompt.

Here's the simplest possible example. Start with a greeting agent that has a generic instruction:

from google.adk.agents import LlmAgent
from gepa_adk import evolve_sync, EvolutionConfig, SimpleCriticOutput

# The agent to evolve — starts with a vague instruction
agent = LlmAgent(
    name="greeter",
    model="gemini-2.5-flash",
    instruction="Greet the user appropriately.",
)

# A critic that knows what "good" looks like
critic = LlmAgent(
    name="critic",
    model="gemini-2.5-flash",
    instruction="Score for formal, Dickens-style greetings. 0.0-1.0.",
    output_schema=SimpleCriticOutput,
)

# Training examples covering different social contexts
trainset = [
    {"input": "I am His Majesty, the King."},
    {"input": "I am your mother."},
    {"input": "I am a close friend."},
]

result = evolve_sync(agent, trainset, critic=critic)
print(f"Score: {result.original_score:.2f} -> {result.final_score:.2f}")
print(result.evolved_components["instruction"])

The original instruction — "Greet the user appropriately" — is five words. The evolved instruction might be three paragraphs of specific guidance about formality levels, period-appropriate language, honorific handling, and tonal variation. You would never write that prompt yourself. Not because you couldn't, but because you wouldn't think to be that specific in those particular ways.

Why a Critic Changes Everything

The critic is what separates evolutionary optimization from "just running the prompt a bunch of times." Without a critic, you're scoring outputs manually or relying on the agent to self-assess (which is like asking the cook to rate their own soup — helpful but biased).

A critic agent is a separate LLM that evaluates outputs against explicit criteria. It returns a score and feedback:

class SimpleCriticOutput(BaseModel):
    score: float = Field(ge=0.0, le=1.0)
    feedback: str

The feedback is the ingredient that makes reflection work. The reflection model doesn't just see "this scored 0.4" — it sees "this scored 0.4 because the greeting was too casual for royalty and used modern slang." That specific diagnosis drives specific mutations.

You can also use multi-dimensional scoring — rate outputs on clarity, accuracy, tone, and format independently. GEPA tracks a Pareto frontier across dimensions, so you don't have to collapse everything into a single number and lose information about what's actually improving.

The Evolution Loop, Unpacked

What actually happens inside evolve():

Iteration 1: The agent runs on all training examples with its original instruction. The critic scores each output. Average score: 0.35. The reflection model reads every output, every score, every piece of feedback. It proposes a new instruction that addresses the most common failure patterns.

Iteration 2: The agent runs again with the proposed instruction. Scores improve to 0.62. The reflection model sees what got better and what's still weak. It proposes another refinement.

Iteration 3: Scores hit 0.78. The reflection model notices diminishing returns and makes a smaller, more targeted adjustment.

Iteration 4: Scores reach 0.81. No improvement over the previous best after the patience window. Evolution stops.

The EvolutionConfig controls this process:

config = EvolutionConfig(
    max_iterations=5,     # Upper bound on iterations
    patience=2,           # Stop if no improvement for N iterations
    reflection_model="gemini-2.5-flash",  # Model for generating mutations
)

Each iteration makes multiple LLM calls — the agent runs on every training example, the critic scores every output, and the reflection model analyzes everything. This is why gepa-adk recommends local models via Ollama for development. Evolution is compute-hungry by nature, and local inference keeps the iteration loop fast and free.

What Can Evolve

Instructions are the most common target, but they're not the only evolvable component. gepa-adk can also optimize:

Output schemas — the Pydantic model that structures the agent's response
Generation config — LLM parameters like temperature and top-p
Multi-agent systems — evolve instructions across multiple agents simultaneously, optimizing how they work together

That last one is where things get interesting. In a multi-agent pipeline, the instructions of one agent affect the inputs to the next. Evolving them together means the optimizer can find coordination patterns that no amount of individual prompt tuning would discover — like a kitchen brigade where the saucier and the grill cook learn to time their dishes together instead of each optimizing in isolation.

When to Use This

Evolutionary optimization isn't for every prompt. It shines when:

You have measurable quality criteria — if you can define what "good" looks like in a critic, evolution can optimize for it
Your agent runs on diverse inputs — evolution generalizes across training examples instead of overfitting to one
Manual tuning has plateaued — you've tweaked the prompt as far as your intuition goes
You're building for production — the difference between a 0.65 and a 0.82 score matters when the agent runs thousands of times

It's less useful for one-off prompts, creative tasks without clear quality metrics, or situations where the "right answer" changes too frequently for a training set to be meaningful.

Key Takeaways

Manual prompt engineering has a ceiling — human intuition can't efficiently search the space of possible instructions
Evolutionary optimization uses LLM-powered reflection — not random mutation, but informed analysis of what's working and what isn't
A critic agent is the key ingredient — structured scoring and feedback drive meaningful improvements
The evolved prompts are often surprising — the algorithm finds instruction patterns you wouldn't write yourself
Get started now: pip install gepa-adk

Give Your AI Agent a Docstring Quality Tool

Wed, 04 Mar 2026 00:00:00 GMT

Your AI coding agent can read your code, run your tests, and search your repo. But can it check whether your docstrings actually match what the code does?

In a previous post, I made the case that stale docstrings actively mislead AI agents — in controlled tests, incorrect documentation drops LLM task success by 22.6 percentage points. Agents given wrong docstrings performed worse than agents given no docstrings at all. docvet was built to catch those gaps: 19 rules that check whether your docstrings actually match what the code does.

Since v1.8, docvet ships an MCP server. That means any MCP-aware editor — VS Code, Cursor, Claude Code, Windsurf, Claude Desktop — can give its AI agent direct access to docstring quality checks. Structured JSON, not CLI text parsing. One config block, zero install.

What Your Agent Gets

Two tools appear in the agent's toolbox:

docvet_check — Run checks on any Python file or directory. The agent receives structured findings:

{
  "findings": [
    {
      "file": "src/pipeline/extract.py",
      "line": 42,
      "symbol": "extract_text",
      "rule": "missing-raises",
      "message": "Function 'extract_text' raises ValueError but has no Raises section",
      "category": "required"
    }
  ],
  "summary": {
    "total": 3,
    "by_category": {"required": 2, "recommended": 1},
    "files_checked": 8
  }
}

docvet_rules — List all 19 rules with descriptions and categories. Useful when the agent needs to explain a finding or decide what to fix first.

No CLI output to parse. No regex. The agent gets typed fields it can reason about directly.

Why MCP, Not CLI?

Without MCP, your agent shells out to docvet check and has to make sense of this:

src/pipeline/extract.py:42: missing-raises - Function 'extract_text' raises ValueError but has no Raises section
src/pipeline/extract.py:87: missing-param - Parameter 'timeout' not documented
src/utils/cache.py:15: stale-signature - Docstring parameters don't match function signature

That works, but the agent is regex-matching strings instead of reasoning about typed data. With MCP, it gets the structured JSON shown above — fields it can filter, sort, and act on directly.

Setup: One Block of JSON

The MCP server runs on stdio via uvx — no pip install in your project, no virtual environment pollution, no global packages. uvx downloads and runs docvet in an isolated environment automatically. You add the config and it just works.

VS Code

Add to .vscode/mcp.json in your project root:

{
  "servers": {
    "docvet": {
      "type": "stdio",
      "command": "uvx",
      "args": ["docvet[mcp]", "mcp"]
    }
  }
}

Note: VS Code uses "servers" as the top-level key, not "mcpServers".

Cursor

Add to .cursor/mcp.json (project-level) or ~/.cursor/mcp.json (global):

{
  "mcpServers": {
    "docvet": {
      "command": "uvx",
      "args": ["docvet[mcp]", "mcp"]
    }
  }
}

Claude Code

One command, no JSON editing:

claude mcp add --transport stdio --scope project docvet -- uvx "docvet[mcp]" mcp

Use --scope user to make it available across all projects.

Windsurf / Claude Desktop

Same mcpServers pattern — see the full editor guide for copy-paste configs. There's also a dedicated VS Code setup tab if you just want the one-click config.

What Happens Next

Once configured, your AI agent can invoke docvet_check on any file it's working with. A typical workflow:

Agent opens a Python file to modify
Agent runs docvet_check on the file
Findings come back as structured JSON — missing Raises sections, stale signatures, undocumented attributes
Agent fixes the docstrings alongside the code change

The feedback loop becomes automatic — like a line cook who taste-tests every dish before it leaves the pass. The agent checks docstring quality as part of its normal workflow, not as an afterthought.

Try It

If you have VS Code with MCP support enabled:

Add the .vscode/mcp.json block above
Open a Python file with a known docstring gap (a function that raises an exception but has no Raises: section)
Ask your AI agent to check the file with docvet
Watch it get structured findings and fix the docstring

Now your agent can check whether your docstrings actually match what the code does. One config block, zero install.

docvet is on PyPI, GitHub, and the MCP Registry. The full editor setup guide is at alberto-codes.github.io/docvet/editor-integration.

Your AI Agent's Memories Aren't Encrypted

Sun, 01 Mar 2026 00:00:00 GMT

You built an AI agent. It calls tools, remembers conversations, and tracks state across sessions. It works. You deploy it. Users start talking to it — sharing API keys, asking about internal systems, passing along customer data through tool calls.

Every word of that is sitting in a SQLite file, in plaintext, readable by anyone with file access.

This isn't a vulnerability you introduced. It's the default behavior of Google ADK's session persistence. And if you're building agents that handle anything beyond demo data, it's a problem you need to solve before production.

The Problem You Don't Know You Have

Google's Agent Development Kit gives you DatabaseSessionService — a session persistence layer that stores agent state in a relational database. It handles the hard parts: session lifecycle, event ordering, state reconstruction across turns. You configure it with a connection string and move on to the interesting work of building your agent.

from google.adk.sessions import DatabaseSessionService

session_service = DatabaseSessionService(db_url="sqlite+aiosqlite:///sessions.db")

That sessions.db file now contains everything your agent processes. Every user message. Every tool call and its arguments. Every function response. Every piece of state your agent carries between turns. All of it stored as structured data, fully readable, with no encryption layer between the file system and your users' data.

Open the file. Run a query. Read the JSON. There's nothing stopping you — and nothing stopping anyone else who gains access to that file.

What's Actually in a Session?

ADK sessions aren't just chat logs. They're structured records of everything your agent does:

User messages — the full text of every prompt your users send
Tool calls — function names and their complete argument payloads
Function responses — the raw data your tools return, including API responses and database query results
Agent state — accumulated context the agent carries across turns
Event metadata — timestamps, ordering, session identifiers

Think about what flows through a real agent. A user asks your agent to look up a customer record — the tool call contains the query, and the response contains the customer's data. A user passes an API key so the agent can call an external service — that key is now stored in the session. A user discusses internal business logic — the agent remembers it for context, and the database remembers it forever.

And none of this is hypothetical. If you open that SQLite file with any database browser, every session row contains JSON you can read without any special tooling. No keys, no decryption step, no access control. The file is the data, and the data is in the clear.

The session database isn't a chat transcript. It's a complete operational record of everything your agent knows about your users and what it did on their behalf. It's like a kitchen where every order ticket, every ingredient substitution, and every customer allergy note gets pinned to the wall in plain sight — and the back door is unlocked.

Why This Matters Now

When agents live in notebooks and demos, plaintext sessions are fine. Nobody's passing real credentials to a prototype. But agents are leaving the sandbox. It's the difference between cooking at home and running a restaurant — in your own kitchen, nobody cares if the pantry is unlocked. The moment you're serving customers, health inspectors show up and every storage decision matters. Google released ADK to help developers build production-grade agents, and teams are taking that seriously — building agents that handle real workflows with real data.

Production agents process real user data. They call real APIs with real keys. They handle internal workflows where the conversation itself is sensitive. The moment your agent crosses from "demo" to "deployed," the session database becomes a data-at-rest liability.

Compliance frameworks are explicit about this:

SOC 2 requires encryption of sensitive data at rest
HIPAA mandates encryption for protected health information in storage
GDPR expects appropriate technical measures to protect personal data

These aren't edge cases. If your agent processes user data and you're pursuing any compliance certification, an auditor will ask how session data is protected at rest. "It's in a SQLite file" is not an answer that passes review.

"The agent works" is necessary. "The agent is secure" is what lets you ship it. And right now, ADK gives you persistence but not protection. That gap between working and secure is exactly where production agents get stuck.

The Fix Is One Import Away

adk-secure-sessions is an encrypted session storage layer for Google ADK. It's a drop-in replacement — same session interface, same behavior, encrypted storage.

from adk_secure_sessions import EncryptedSessionService, FernetBackend

session_service = EncryptedSessionService(
    db_url="sqlite+aiosqlite:///sessions.db",
    backend=FernetBackend("your-secret-key"),
)

Your agent code doesn't change. Every call to create, get, list, or delete sessions works the same way. The difference is what happens at the storage layer: session data is serialized, encrypted, then written. On read, it's decrypted and deserialized. The SQLite file contains encrypted blobs instead of readable JSON.

The encryption is Fernet — AES-128-CBC with HMAC-SHA256 for authenticated encryption, meaning data is both confidential and tamper-evident. Key derivation uses PBKDF2-HMAC-SHA256 with 480,000 iterations, so your passphrase becomes a strong cryptographic key without managing raw key material.

Two runtime dependencies: google-adk and cryptography. Nothing exotic. Nothing heavy. No C extensions to compile, no system libraries to install.

The library extends ADK's BaseSessionService directly rather than wrapping DatabaseSessionService. That's a deliberate architectural choice — and one that came from a failed first attempt with the decorator pattern. That story, and the generalizable lesson about when composition breaks down, are coming in Part 4 of this series.

What Changes, What Doesn't

What stays the same:

Your agent code — no changes to how you interact with sessions
Session behavior — create, read, update, delete all work identically
The database — still SQLite, still a single file, still portable

What changes:

Session data at rest is encrypted — the file is useless without the key
Error messages are precise — DecryptionError and ConfigurationError tell you exactly what went wrong
You need a passphrase — managed via environment variable, not hardcoded

import os
from adk_secure_sessions import EncryptedSessionService, FernetBackend

session_service = EncryptedSessionService(
    db_url="sqlite+aiosqlite:///sessions.db",
    backend=FernetBackend(os.environ["SESSION_KEY"]),
)

That's the production pattern. The passphrase lives in your environment, the encrypted data lives in your database, and the two never meet in your codebase. Standard secrets management — nothing new to learn, nothing unusual to maintain.

With DatabaseSessionService, an attacker with file access sees everything. With EncryptedSessionService, they see noise.

Key Takeaways

Google ADK stores session state in plaintext by default — every tool call, user message, and piece of agent context is readable in the database
Sessions contain more than chat history — tool arguments, API responses, and accumulated state are all persisted
Production agents need encryption at rest — compliance frameworks require it, and good engineering practice demands it
adk-secure-sessions is a drop-in replacement — one import change, same API, encrypted storage
Get started now: pip install adk-secure-sessions

Your AI Reads Your Docstrings. Are They Right?

Wed, 25 Feb 2026 00:00:00 GMT

You're pair-programming with an AI agent. It reads your codebase, finds the function you're extending, checks the docstring, and generates code that calls it with the old parameter names. You debug for twenty minutes before realizing the docstring was updated six months ago—wait, no. The function was updated six months ago. The docstring still describes the version before the refactor.

This is the software equivalent of a recipe card that says "bake at 350°F for 30 minutes" when someone already modified the dish to be pan-fried. A cook following that card doesn't just get a mediocre result—they ruin the dish entirely. Outdated instructions are worse than no instructions, because they create false confidence.

Your AI coding agent reads your docstrings. Every time it generates code, it uses those docstrings as context for understanding your functions, classes, and modules. When those docstrings are stale, incomplete, or wrong, the agent doesn't know to distrust them. It follows the recipe card.

The Research Confirms What You Already Suspect

This isn't just an intuition. Multiple studies have quantified how documentation quality affects LLM performance on code tasks:

Incorrect documentation degrades LLM task success by 22.6 percentage points compared to correct docs (Macke & Doyle, 2024). Critically, incomplete or missing docs didn't cause nearly the same harm—wrong docs are uniquely toxic.
Comment density improves code generation by 40–54% across benchmarks (Wei et al., 2024). More documentation means better AI output, but only when that documentation is accurate.
Misleading comments reduce LLM fault localization accuracy to 24.55% (Jia et al., 2025). When the comments lie, the AI can't find bugs.
Performance drops substantially without docstrings, and intent-aware inference is needed to compensate (Li et al., 2025).

The 2025 DORA report puts it bluntly: "AI doesn't fix a team; it amplifies what's already there." If your docstrings are accurate, AI makes you faster. If they're stale, AI confidently generates the wrong code—and you trust it because the agent seemed so sure.

Stale docstrings don't just fail to help. They actively mislead. Incorrect documentation is worse than no documentation at all.

Think of it like a pantry with mislabeled jars. A cook with an empty shelf knows to go find ingredients. A cook with a jar labeled "cumin" that's actually filled with cinnamon? That cook seasons the chili, tastes nothing wrong until it's too late, and serves a dish that's subtly, confusingly off. That's what stale docstrings do to your AI agent.

The Gap in Your Toolchain

You probably already have docstring tooling. Most Python projects use some combination of:

ruff (D rules) — checks how your docstrings look. Formatting, style conventions, section order.
interrogate — checks if docstrings exist. Coverage percentage across your codebase.

These are layers 1 and 2 of docstring quality. Necessary, but not sufficient. They answer "is there a docstring?" and "is it formatted correctly?" They don't answer the question that actually matters for AI agents:

Is the docstring right?

Docstring quality has six distinct layers:

Layer	Question	Tool
1. Presence	Does it exist?	interrogate
2. Style	Is it formatted correctly?	ruff D rules
3. Completeness	Does it document all sections?	gap
4. Accuracy	Does it match the current code?	gap
5. Rendering	Will mkdocs render it correctly?	gap
6. Visibility	Will mkdocs even see the file?	gap

Layers 1–2 are table stakes. Layers 3–6 are where your AI agent's understanding lives or dies—and until now, no tool covered them.

It's like a restaurant kitchen where health inspectors check that you have a recipe binder (presence) and that the cards are legible (style), but nobody ever verifies that the recipes match what the cooks are actually preparing. The card says "sear for 2 minutes per side" but the chef switched to a 4-minute sear last month. The binder looks great. The food is wrong.

Enter docvet

docvet fills layers 3–6 with 19 rules across four checks:

Enrichment (10 rules) — completeness. Your function raises ValueError but the docstring has no Raises: section? Your dataclass has five attributes but no Attributes: section? docvet catches it. It reads your AST—the actual code structure—and compares it against what the docstring claims to document.

Freshness (5 rules) — accuracy. This is the killer feature. docvet uses git diff and git blame to detect when code changes but docstrings don't. Changed a function's signature last week? The docstring still describes the old parameters? That's a stale-signature finding, severity HIGH.

Griffe (3 rules) — rendering compatibility. If you publish docs with mkdocs, docvet catches griffe parser warnings before they silently break your documentation site.

Coverage (1 rule) — visibility. Missing __init__.py files make entire packages invisible to documentation generators. docvet finds them.

One line to try it:

pip install docvet && docvet check --all

Example output on a real codebase:

src/pipeline/extract.py:42: stale-signature Function 'extract_text' signature changed but docstring not updated [required]
src/models/customer.py:15: missing-attributes Dataclass 'CustomerRecord' has no Attributes: section [required]
src/utils/validate.py:88: missing-raises Function 'validate_schema' raises ValueError but has no Raises: section [required]

3 findings (3 required, 0 recommended)

Each finding tells you exactly what's wrong, where it is, and whether it's required or recommended. No configuration needed to start—docvet runs with sensible defaults out of the box.

The Feedback Loop

The tagline "better docstrings, better AI" isn't marketing—it's a literal feedback loop:

AI reads your docstrings to understand your code
docvet ensures those docstrings are complete and accurate
AI writes better code because its context is trustworthy
Repeat

If your AI agent is your sous chef, your docstrings are the recipe cards pinned above each station. docvet is the head chef who walks the line before service, pulls down every card, and checks it against what's actually in the pan. No stale cards make it to service.

Key Takeaways

Your AI coding agent reads your docstrings as primary context for understanding your codebase—stale docs mean bad AI output
Incorrect documentation is worse than no documentation—research shows a 22.6 percentage point drop in LLM task success with wrong docs
Existing tools cover style and presence (layers 1–2) but not completeness, accuracy, rendering, or visibility (layers 3–6)
docvet fills the gap with 19 rules across four checks: enrichment, freshness, griffe, and coverage
Freshness detection uses git history to catch code-docstring drift—no manual review needed
One command to try it: pip install docvet && docvet check --all

Entity Resolution is Recipe Matching

Thu, 05 Feb 2026 00:00:00 GMT

Birria in Jalisco is goat, slow-braised in dried chiles and served in a bowl with consommé. In Tijuana, it's beef, crisped on a plancha and folded into a taco. At Chipotle, it's a menu item with queso. Three different presentations, three different proteins, three different contexts—but you recognize them as the same dish.

You're not matching on a single ingredient. You're weighing evidence across multiple dimensions: the chile base, the braising technique, the way it's served with its cooking liquid.

Entity resolution works the same way. When you have two database records—"José García, 123 Main St, DOB 1985-03-15" and "GARCIA, JOSE M., 123 Main Street, DOB 03/15/1985"—you need to determine if they represent the same person. No single field gives you certainty, but together, the evidence points strongly toward a match.

I solved this problem on a project in financial services: deduplicating customer records, linking transactions across systems, matching entities without shared keys. It's harder than it looks, and most engineers reach for exact matching first—which works until it doesn't.

Why Exact Matching Fails

The instinct is to write rules:

If name matches AND DOB matches AND address matches → same person

This is deterministic matching, and it breaks the moment it touches real-world data.

Consider what customer data actually looks like:

Field	System A	System B
Name	José García	GARCIA, JOSE M.
Address	123 Main St	123 Main Street, Apt 2B
DOB	1985-03-15	03/15/1985
Email	jose.garcia@gmail.com	jgarcia85@yahoo.com

Exact matching finds zero matches here. Every field differs in format, abbreviation, or completeness. But a human reviewer immediately recognizes this as likely the same person.

You could write transformation rules—normalize names to uppercase, expand abbreviations, standardize date formats. This helps, but it's a game of whack-a-mole:

"José" vs "Jose" vs "Joe"
"García" vs "Garcia"
"123 Main" vs "123 Main St" vs "123 Main Street"
Typos: "Josè Garçia" → "José García"

Deterministic matching forces a binary decision: match or no match. Reality is probabilistic—some pairs are definite matches, some definite non-matches, and a large gray zone requires weighing evidence.

Probabilistic Matching: Weighing Evidence

Probabilistic record linkage flips the question. Instead of asking "do these records match?" it asks:

How likely is it that these records refer to the same entity, given the evidence?

Each field comparison contributes evidence:

Comparison	Evidence
Exact name match	Strong positive
Fuzzy name match (Jaro-Winkler > 0.9)	Moderate positive
DOB matches	Strong positive
DOB off by one digit	Weak positive (likely typo)
Completely different DOB	Strong negative

The accumulated evidence produces a match score.

This mirrors how you recognize birria. If two recipes both have dried chiles, braised meat, and a rich cooking liquid served alongside, that's strong evidence they're the same dish. If one uses guajillo and the other uses ancho, that's weak variation—regional preference, not a different dish. If one serves it in a bowl and the other crisps it in a taco, that's presentation, not identity.

But if one is a quick stovetop stew without the slow braise? That's evidence you're looking at something else entirely—maybe carne guisada, but not birria.

The Fellegi-Sunter Model (Without the Math)

The math behind probabilistic matching is the Fellegi-Sunter model, developed in 1969 and still the foundation of modern record linkage.

Fellegi-Sunter assigns each field comparison a weight based on two probabilities:

Probability	Question
m-probability	If two records truly match, how often would this field comparison look like this?
u-probability	If two records don't match, how often would this field comparison look like this by chance?

The ratio of these probabilities determines the weight:

Names match exactly: Common among true matches (high m), rare among random pairs (low u) → positive weight
Names completely differ: Rare among true matches (low m), common among random pairs (high u) → negative weight

Common vs. Rare Values

Here's the key insight: common values provide weaker evidence than rare values.

"María García" matching "María García" is less compelling than "Xiomara Tlapoyawa" matching "Xiomara Tlapoyawa." The first could happen by coincidence in any large dataset; the second almost certainly indicates the same person.

Think of it like identifying a mole. If two recipes both contain chiles, that tells you almost nothing—every mole has chiles. But if both call for chocolate, banana, and chipotle in specific proportions, you're probably looking at variations of mole negro from Oaxaca.

The rare, specific ingredients carry more weight than the ubiquitous ones.

Splink handles all of this automatically. You define what fields to compare and how (exact match, fuzzy match, within a date range), and Splink estimates the m and u probabilities using unsupervised learning—no labeled training data required.

The Blocking Problem

There's a computational catch. If you have a million records, comparing every pair means:

1,000,000 × 999,999 / 2 = 499,999,500,000 comparisons

That's 500 billion comparisons. Not feasible.

Blocking solves this by only comparing records that could plausibly match. You define rules like:

Only compare records where first name initial AND birth year match
Only compare records in the same city
Only compare records with similar phone area codes

This is mise en place for data matching—organizing your workspace before you start cooking. A chef doesn't wander the entire kitchen looking for ingredients during service. They set up their station with everything they'll need in reach. Blocking sets up your comparison space so you're only looking where matches are likely to be found.

The art is in the balance:

Problem	Cause
Missing matches	Blocking too aggressive—true pairs never compared
Slow performance	Blocking too loose—comparing pairs that obviously don't match

Why Splink?

You could implement Fellegi-Sunter from scratch, but Splink handles the hard parts:

Feature	What it does
Unsupervised training	Estimates m/u probabilities without labeled data (Expectation-Maximization)
Multiple backends	DuckDB (laptop), Spark (cluster), AWS Athena (serverless)
Fuzzy matching	Jaro-Winkler, Levenshtein, phonetic matching built-in
Term frequency	Automatically down-weights common values like "José García"
Visualization	Inspect match weights, debug blocking rules, understand model behavior

It's open source, built by the UK Ministry of Justice, and battle-tested on national-scale datasets including the 2021 Census.

Why This Matters in Financial Services

Entity resolution isn't an academic exercise. In financial services, getting it wrong has real consequences:

Use Case	Cost of Getting It Wrong
KYC/AML compliance	Fragmented risk profiles, regulatory failures
Fraud detection	Missed patterns across linked identities
M&A data migration	Duplicate customers in merged systems
Regulatory reporting	Incorrect unique customer counts

The cost of false negatives (missed matches) is fragmented data and compliance risk. The cost of false positives (incorrect matches) is merged records for different people—a data quality nightmare.

Probabilistic matching lets you tune the tradeoff for your specific use case.

Key Takeaways

Entity resolution determines if records refer to the same real-world entity when no shared key exists
Exact matching breaks on real-world data—typos, format variations, and abbreviations defeat deterministic rules
Probabilistic matching weighs evidence across fields, producing a match score rather than a binary decision
Common values (María, García) provide weaker evidence than rare values (Xiomara, Tlapoyawa)
Blocking reduces the comparison space from n² to something tractable
Splink handles the math, scales from laptops to clusters, and requires no labeled training data

Lineage IDs in Multimodal AI Pipelines

Wed, 04 Feb 2026 00:00:00 GMT

Multimodal pipelines look clean on a whiteboard:

pull images + companion JSON
render images → video
send video + JSON to an AI API
store outputs

Then you ship it and reality shows up: retries, partial failures, parallel workers, version bumps, and the occasional "why does this output not match the input we think it does?"

This is where lineage IDs earn their keep.

If you’ve ever worked a dinner rush, it’s the same problem. Tickets keep printing, stations work in parallel, and sometimes you have to refire a dish. If your containers aren’t labeled and your tickets aren’t tracked, you end up staring at the pass thinking: is this plate actually for table 12?

What “lineage” means (in practice)

Lineage is just the ability to answer “where did this come from?” at every step.

Which image set produced this video.mp4?
Which video.mp4 + metadata produced this model response?
Which response produced this final.json we shipped downstream?
If we retried, which attempt “won” and why?

Queue dashboards and worker metrics tell you what ran. Lineage tells you what the run touched — the digital equivalent of a ticket number plus labels on every prep container.

Rule of thumb: if you can’t answer “which exact bytes went into this model call?”, you don’t have lineage — you have hopeful logging.

The shape of a lineage-aware pipeline

This is what you're aiming for: a run_id that propagates through workers, durable receipts that link inputs/outputs, and provider request IDs that let you correlate with the API vendor later.

The minimum viable lineage model

Keep it boring. You only need a few identifiers.

Name	What it identifies	Why it matters	Kitchen analogy
`run_id`	One end-to-end workflow instance (created once at ingest)	Correlates everything across fan-out and retries	Ticket number
`step`	A named transformation stage (`fetch_images`, `render_video`, `call_ai`, `persist_outputs`)	Lets you reason about progress and partial failures	Station (prep, sauté, expo)
`attempt`	The retry count for a step	Distinguishes first-run vs refires	Refire count
`artifact_id`	A stable identifier for bytes (often a content hash)	Prevents “same path, different bytes” confusion	Labeled batch / container

If you already use a task queue, you’ll also have a task_id per task execution. That’s useful for ops, but it’s not a workflow lineage ID by itself.

Lineage vs idempotency vs tracing

These concepts are related, but they solve different problems:

Lineage — "What produced this?" — artifacts and transformations across the whole workflow
Idempotency — "What if this runs twice?" — make retries and duplicates harmless
Tracing — "Where did the time go?" — timelines, spans, latency, bottlenecks

In practice, you'll often reuse the same run_id as a correlation handle across all three, but keep the semantics separate:

Don't use a queue task_id as your lineage key.
Don't use an idempotency key as your run identifier (it's usually step-scoped).
Don't assume provider request IDs replace your own identifiers (they're necessary, not sufficient).

A file-first implementation (works before you have a DB)

You can get 80% of the value with a directory per run and a couple JSON “receipts”.

runs/
  2026-02-04T18-41-12Z_d5c1.../          # run_id (timestamp + UUID, for easy sorting)
    manifest.json                        # one place to start
    inputs/
      frames/0001.jpg
      frames/0002.jpg
      metadata.json
    steps/
      010_fetch_images/
        receipt.json
      020_render_video/
        video.mp4
        receipt.json
      030_call_ai/
        request.json
        response.json
        receipt.json
      040_persist/
        outputs.json
        receipt.json

The only “rule” is: every step writes a receipt that declares its inputs and outputs.

Here’s what a step receipt can look like:

{
  "run_id": "2026-02-04T18-41-12Z_d5c1f0b4e6b84aef9a7be8d07f2c3a1b",
  "step": "render_video",
  "attempt": 1,
  "started_at": "2026-02-04T18:41:15Z",
  "finished_at": "2026-02-04T18:41:22Z",
  "inputs": [
    {"path": "inputs/frames/0001.jpg", "sha256": "…"},
    {"path": "inputs/frames/0002.jpg", "sha256": "…"},
    {"path": "inputs/metadata.json", "sha256": "…"}
  ],
  "outputs": [
    {"path": "steps/020_render_video/video.mp4", "sha256": "…", "media_type": "video/mp4"}
  ]
}

That's lineage: a tiny, append-only audit trail you can reconstruct into a graph later.

Each step's receipt links inputs to outputs via content hashes — forming a chain you can traverse in either direction.

Why content hashes beat "whatever the filename was"

When you’re debugging, the most painful failure mode is “file A got overwritten by a retry, and now A points to different bytes than it did yesterday.”

In kitchen terms: someone reused the same unlabeled container, and now “sauce” could mean three different things depending on who touched it last.

If your artifact_id is a content hash (ex: SHA-256), you get:

Deduplication for free (same bytes → same ID)
Cheap integrity checks (hash mismatch = corruption/overwrite)
A stable join key across systems (filesystem, object storage, DB, logs)

You can still store by filename; just record the hash in the receipt so you have a stable identifier.

Here’s a tiny standard-library helper for hashing and run IDs:

import hashlib
import uuid
from datetime import datetime, timezone
from pathlib import Path


def new_run_id() -> str:
    ts = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H-%M-%SZ")
    return f"{ts}_{uuid.uuid4().hex}"


def sha256_file(path: str | Path) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1024 * 1024), b""):
            h.update(chunk)
    return h.hexdigest()

For large video files, hashing adds I/O overhead (you're reading every byte). For most pipelines the auditability is worth it — but if it becomes a bottleneck, faster non-cryptographic hashes like xxHash or BLAKE3 work fine for artifact IDs.

Propagating lineage through a task queue

The most important operational discipline is simple:

Generate run_id once, then pass it to every task and log line.

In task-queue terms, that usually means your task signature always includes run_id, and every task writes its receipt under runs/<run_id>/steps/<step>/….

Retries are expected in at-least-once systems (I wrote more about that in Task Queues, Idempotency, and AI Pipelines). Lineage doesn’t prevent duplicates — it makes duplicates understandable.

Structured logs (the breadcrumbs between receipts)

Receipts are your durable audit trail. Logs are your high-volume timeline.

The simple rule: every log line emitted by workers should include run_id, step, and attempt. Then add fields that make debugging cheap, like:

task_id (from your queue)
artifact_id / sha256 for important inputs/outputs
provider request/response IDs for model calls

If you use structured logging (for example structlog), you’re aiming for events that look like this:

import structlog


log = structlog.get_logger().bind(
    run_id=run_id,
    step=step,
    attempt=attempt,
    task_id=task_id,
)
log.info("step_started")
log.info("step_finished", output_sha256=output_sha256, duration_ms=duration_ms)

{
  "event": "step_finished",
  "run_id": "2026-02-04T18-41-12Z_d5c1f0b4e6b84aef9a7be8d07f2c3a1b",
  "step": "render_video",
  "attempt": 1,
  "output_path": "steps/020_render_video/video.mp4",
  "output_sha256": "…",
  "duration_ms": 7421
}

Correlating AI API calls (OpenAI or otherwise)

When you call an AI API, you want to keep three IDs tied together:

For every model call, tie together: your run_id, their request ID, and the exact bytes you sent and received.

A concrete OpenAI example:

Send your own ID via the X-Client-Request-Id request header (ASCII, ≤ 512 chars).
Log/store the server-generated x-request-id response header.

A practical pattern is:

steps/030_call_ai/request.json: the request payload you sent (or a redacted version)
steps/030_call_ai/response.json: the raw response body you received
steps/030_call_ai/receipt.json: a summary that links inputs/outputs + provider IDs

When the API supports it, also include your identifiers in the request (for example: request metadata, a user field, or custom headers in your own gateway). The goal is that support tickets and logs can be traced with either your run_id or their request/response ID. OpenAI documents this under Debugging requests.

For a provider call receipt, you’re usually trying to capture something like:

{
  "run_id": "2026-02-04T18-41-12Z_d5c1f0b4e6b84aef9a7be8d07f2c3a1b",
  "step": "call_ai",
  "attempt": 1,
  "provider": "openai",
  "client_request_id": "run=2026-02-04T18-41-12Z_d5c1f0b4e6b84aef9a7be8d07f2c3a1b;step=call_ai;attempt=1",
  "provider_request_id": "…",
  "provider_response_id": "…",
  "inputs": [{"path": "steps/020_render_video/video.mp4", "sha256": "…"}],
  "outputs": [{"path": "steps/030_call_ai/response.json", "sha256": "…"}]
}

Even if you later move storage to S3/GCS and logs to Splunk/Datadog, that “receipt” pattern still holds. It’s just a join table you can grep.

What you get from doing this early

Debugging that scales — "this output is wrong" becomes "this run_id, this step, this exact artifact hash"
Safer retries — you can see which attempt produced which bytes, and detect overwrites
Replayability — pick a run_id, re-run starting at a step, and keep the old receipts for comparison
Auditing — you can answer "what did we send to the model?" without guessing

Common pitfalls

Treating a queue task_id as a workflow ID (great for ops, terrible for lineage).
Writing outputs to fixed paths without attempt or hashes (retries overwrite history).
Adding correlation IDs to some steps but not all (you break the chain).
Logging raw prompts/inputs instead of receipts + hashes (you create privacy risk and still can’t reliably join artifacts).

Key takeaways

Create a run_id once at ingest and propagate it everywhere (tasks, logs, artifacts).
Treat every step like a station: write a receipt that names exact inputs and outputs.
Prefer content hashes for artifact_id so retries and overwrites don’t blur history.
Capture provider request/response IDs for model calls so you can correlate later.
Start file-first; you can always move the same receipts into a DB or a catalog later.

Closing thought

You don't need a data catalog or a dedicated lineage platform to start. A run_id, a couple receipts, and disciplined propagation through your tasks will save you days of debugging — and a lot of unnecessary model spend — once your pipeline hits the dinner rush.

Task Queues, Idempotency, and AI Pipelines

Mon, 02 Feb 2026 00:00:00 GMT

If you run AI workloads through a task queue, duplicates are not a bug. They are a feature of the delivery guarantee you almost certainly chose. Understanding why they happen, and designing for them, is the difference between a pipeline that wastes money on redundant LLM calls and one that shrugs off failures gracefully.

Think of it like a busy kitchen. A ticket printer spits out orders, and line cooks pull them. If a cook drops a plate halfway through and nobody crosses the ticket off, the expeditor calls it again. The kitchen makes the dish twice. That's at-least-once delivery: better to send an extra plate to the pass than to lose an order entirely.

Why Exactly-Once Delivery Is So Rare

Most engineers want exactly-once delivery: every message processed one time, no more, no less. The problem is that exactly-once requires coordination between the broker and the worker that is extremely hard to achieve in practice. The broker needs to know the worker finished, and the worker needs to know the broker won't send the message again. In a distributed system where either side can crash, restart, or lose a network connection at any moment, that coordination breaks down.

What you actually get in most systems is one of two alternatives:

At-most-once: the broker deletes the message before the worker confirms success. If the worker crashes, the message is gone. Simple, but you lose work.
At-least-once: the broker keeps the message until the worker explicitly acknowledges it. If anything goes wrong, the message gets redelivered. You never lose work, but you get duplicates.

At-least-once is the default in SQS, Celery with acks_late=True, Google Cloud Tasks, and most durable workflow engines like Temporal. It's the pragmatic choice for pipelines where losing a task is worse than doing it twice.

How Duplicates Actually Happen

Duplicates are not some theoretical edge case. Three scenarios produce them regularly:

Worker crash mid-processing. A worker picks up a message, starts an expensive OCR extraction, and gets OOM-killed halfway through. The broker never received an acknowledgment, so it redelivers the message to another worker. Now two attempts exist: one partial (dead), one fresh.

Visibility timeout expiry. In SQS, when a worker receives a message, that message becomes invisible to other consumers for a configurable window (the visibility timeout). If your AI pipeline task takes longer than expected—say a large PDF triggers a slow embedding model—the timeout expires and SQS hands the message to a second worker while the first is still running.

Late acknowledgment. The worker finishes the job and sends an ack, but the network blips. The broker assumes the worker failed and redelivers. The work was already done, but the queue doesn't know that.

All three scenarios result in the same task being executed more than once.

Idempotency: Making Duplicates Harmless

Idempotency means that running an operation multiple times produces the same result as running it once. Back to the kitchen: if you've already plated the risotto and it's sitting under the heat lamp, you don't fire a second one just because the ticket printed again. You check the pass first. Same idea in code.

The most common pattern is an idempotency key—a unique identifier derived from the input that you check before doing expensive work:

def process_document(task: dict):
    idempotency_key = f"doc:{task['document_id']}:v{task['version']}"

    if store.exists(idempotency_key):
        return  # Already processed, skip

    chunks = split_and_embed(task["content"])  # Expensive LLM call
    vector_db.upsert(task["document_id"], chunks)

    store.set(idempotency_key, ttl=86400)  # Mark as done

The key is derived from the document ID and version so that a legitimate re-processing (new version) still runs, but a duplicate delivery of the same version gets caught. The TTL on the key keeps your idempotency store from growing forever.

This is not clever. It's a few lines of code. But it's the difference between paying for one embedding call and paying for three because a worker got slow under load.

The Redelivery Lifecycle

When you combine at-least-once delivery with retry policies and a dead-letter queue, message flow looks like this:

The dead-letter queue (DLQ) is not where messages go to die. It's a throughput protection mechanism. Without it, a poison message—a malformed PDF, an input that consistently crashes your OCR service—cycles through the queue forever, consuming worker capacity and blocking healthy tasks behind it. The DLQ catches these after a configured number of retries and moves them aside so the rest of the pipeline keeps flowing. You review them later, fix the root cause, and replay if needed.

Every kitchen has an 86 board—the list of items that are cut from service because you're out of an ingredient or a piece of equipment is down. You don't keep trying to plate a dish you can't finish. You pull it and deal with it. A DLQ is your 86 board for tasks that can't be completed right now.

Why Retries Cost More in AI Pipelines

In a traditional web backend, retrying a database insert is cheap. In an AI pipeline, retries hit your wallet directly.

Consider a document ingestion pipeline: receive a PDF, OCR it, chunk it, generate embeddings, then extract structured fields with an LLM. If extraction fails and the task retries from the top, you're paying for OCR and embeddings again. At scale, this adds up:

Embedding generation: a 50-page document might produce 200 chunks at ~500 tokens each. One unnecessary retry means 100,000 extra tokens through your embedding model.
LLM extraction: a structured extraction call with a long context window can cost $0.05–$0.20 per call. Ten retries across a batch of 1,000 documents is an extra $500–$2,000.
Tool calls: if your agent pipeline triggers external APIs (geocoding, enrichment, web search) on every retry, you're multiplying both cost and rate-limit pressure.

Idempotency keys at each stage let you skip work that already completed successfully. The OCR result is cached. The embeddings are already in the vector store. The retry only re-executes the extraction step that actually failed. This is sometimes called partial progress recovery, and it turns an expensive full retry into a cheap targeted one.

Key Takeaways

At-least-once delivery is the default in most task queue systems. Duplicates are expected, not exceptional.
Idempotency keys are cheap to implement and directly reduce wasted LLM and embedding spend on retries.
Dead-letter queues protect pipeline throughput by isolating poison messages instead of letting them loop forever.
AI pipelines should cache intermediate results (OCR output, embeddings) so retries only redo the step that failed.
Design your idempotency keys from the input data (document ID + version), not from internal state that might change between attempts.

Why I Chose Reflex for My Portfolio Site

Sun, 01 Feb 2026 00:00:00 GMT

When I decided to build a personal site, the default answer was obvious: grab a React template, maybe Next.js, throw it on Vercel, and call it a day. That's what most developer portfolios run on. But I'm not a frontend engineer. I'm a Python engineer who works on AI systems, and I wanted my site to reflect that.

So I built it with Reflex.

What Is Reflex?

Reflex is a Python framework for building full-stack web applications. You write Python — components, state management, routing, everything — and Reflex compiles it to a React frontend with a FastAPI backend. No JavaScript required (unless you want it).

Here's what a simple page looks like:

import reflex as rx

def hello() -> rx.Component:
    return rx.container(
        rx.heading("Hello, world"),
        rx.text("Built with Python."),
    )

That's real, working code. No JSX, no useState, no npm install. Just Python.

Why Not Next.js?

There's nothing wrong with Next.js. It's battle-tested, well-documented, and has a massive ecosystem. But for me, it came down to a few things:

Context switching is expensive. My day job is Python — agents, pipelines, data engineering. Dropping into TypeScript and the React ecosystem for a side project meant learning (or re-learning) a different set of patterns, tooling, and conventions. With Reflex, I stay in the language I think in.

I don't need the React ecosystem. A portfolio site doesn't need server components, incremental static regeneration, or a component library with 200 options. It needs a few pages, some text, and maybe a blog. Reflex handles that without the overhead.

The site itself is a signal. If I'm positioning myself as a Python and AI engineer, it makes sense for the site to be built with Python. It's a small thing, but it's consistent.

What Reflex Gets Right

Components feel natural. If you've written Python, you can read Reflex code. Components are functions that return component trees. Styling uses keyword arguments. There's no template language to learn.

State management is simple. Reflex uses Python classes for state. You define variables, write event handlers as methods, and bind them to components. It's closer to how you'd think about state in a backend application.

Deployment is straightforward. reflex run --env prod gives you a production build. You get a FastAPI server you can deploy anywhere Python runs — no Node.js runtime needed in production.

It includes what you need. Routing, SEO meta tags, responsive breakpoints, markdown rendering, sitemaps — these come built in or as first-party plugins. I didn't have to research and install ten packages to get a basic site working.

What to Watch Out For

Reflex isn't perfect, and I'd be doing you a disservice if I didn't mention the tradeoffs:

Smaller community. When you hit an issue, there are fewer Stack Overflow answers and blog posts to lean on compared to React or Next.js. The Discord community is active, but you'll sometimes need to read source code.

Component library is growing but limited. You won't find the equivalent of every Radix or shadcn component. Reflex wraps Radix Themes and gives you a solid set of primitives, but if you need something niche, you may need to write a custom component or wrap a React library yourself.

Performance ceiling. For a portfolio site, performance is fine. For a complex, highly interactive application with thousands of concurrent users, you'd want to evaluate more carefully. The Python backend handles state, which adds a round-trip that a pure client-side React app avoids.

It's still maturing. Reflex is under active development. APIs occasionally change between versions. This is improving, but it's worth noting if stability is your top priority.

Who Should Consider Reflex?

If you're a Python developer building something like:

A personal site or portfolio
An internal tool or dashboard
A prototype or MVP
A data-driven app where Python is already doing the heavy lifting

Reflex is worth a serious look. You'll ship faster by staying in one language, and the result is a real web app — not a Streamlit dashboard with limitations.

The Bottom Line

I chose Reflex because it let me build a professional site using the tools I already know, without compromising on the result. The site is fast, responsive, and does everything I need. And when I want to add features — like the blog you're reading right now — I'm writing Python, not context-switching into a different ecosystem.

If you're a Python engineer who's been putting off building a personal site because you don't want to deal with JavaScript tooling, give Reflex a try.