Preprint · a hypothesis with measured corollaries
Order, calibration, and externalised knowledge in small language models — a hypothesis with measured corollaries
This paper proposes and partially tests the Dose-Window
Hypothesis: for a model of size N and a behaviour B,
there is an interval [W_install, W_harm] in effective training
exposure — examples × epochs, measured against the model's existing format
lock — below which B does not install, and above which it saturates
into template memorisation and erodes complementary behaviours, nearest
neighbours first. The hypothesis emerged from an eight-week program training
small language models (0.3B–8B) to investigate — answer from evidence,
challenge false premises, abstain when evidence is absent, cite what each
claim rests on — on a single 6 GB laptop GPU, with every score produced by
frozen, deterministic instruments and paired controls.
Four measured corollaries support it and stand independently of it. (1) Order beats ingredients: the same category data cost 0.081 mixed into a base mixture and gained 0.016 as an isolated pulse followed by a format re-lock, at identical compute. (2) The quantisation floor is calibration, not bits: two 2-bit builds of one model, identical in file size and bit width, scored 0.014 and 0.708. (3) Knowledge divides cleanly: a static, runtime-editable fact table supplies the content half — lifting premise correction 4–5× at zero training, with a single 8B reaching 0.959 on the bench and 0.90 on a blind holdout — while the polarity half remains the model's job and fails below roughly 2B regardless of what the table supplies. (4) Interfaces leak: giving the model a retrieval tool scored 0.510–0.633 against 0.939 for retrieving before inference, through three separable failure modes.
The hypothesis is stated with its untested predictions; section 7 gives the applications at scale and the open agenda, and section 9 states what remains unestablished. The evaluation harness and CLI are released as open source.
Every score in this paper comes from a frozen instrument with a deterministic grader and a paired control. This section defines what that means, how answers are graded, and how the reported numbers are bounded.
A question set with a deterministic grader, written and sealed before any training run and never trained on. Four disciplines apply: the file, its generator, and its build seed are committed together and never rebuilt in place; training corpora are checked for shared gold values against every frozen set; every item's gold target grades against itself at 1.0 by construction; and compared conditions share item identities exactly, so the comparison is paired.
shortcut_check runs on every new premise or entailment set
before it is trusted. It fits simple baselines — length ratio, lexical
overlap, answer position — and fails the set if any exceeds 0.90. Backfilled
across eight frozen instruments, the values were 0.783–0.867.
Every condition reported in this paper is paired with its control on identical items — an uplift without its baseline is not a result. Training corpora are re-graded for instrument overlap before every run, and the office evaluation instruments are deliberately not published, because a published evaluation set becomes a training set.
The production grader is strictly additive across three stages: exact and
substring gold-value matching under a rejection lexicon, with unit
normalisation; the same test applied to a tail window of the raw answer; and an
NLI entailment check with the gold fact as premise and the answer as hypothesis,
requiring P(entail) ≥ 0.80. Calibration: 0.991 entailment on a tight
paraphrase, 0.715 on a vague one, correctly declined. Tests pin the bar at
exactly 0.8.
For independent bounded scores, simultaneous two-sided intervals for each of
m axis means use Hoeffding's inequality with a union bound:
At α = 0.05, all four intervals jointly cover their population
means with probability ≥ 0.95. Dependence between axes is permitted; independence
of sampled questions is required. This is a conservative finite-sample bound, not
a power calculation — with a perfect mean it needs ≥ 254 items per axis for the
lower endpoint to reach 0.90. For paired comparisons on identical items,
D_i = candidate_i − baseline_i in [−1, 1], and the radius doubles
because the range width doubles.
Two further constraints on inference were measured. Identical
prompt_sha256 produced only 3 of 30 byte-identical answers across
server restarts, giving a ≈2-item noise band on n = 30 across server lifetimes;
within-server comparisons are robust, absolute values are ±0.07. And scores are
compared within a backend only, because kernel numerics can flip an argmax at the
margin — cross-backend cells agree within ≤2 items, so curves are read
within-sweep.
The order in which a behaviour is trained matters more than the data used to train it. Installing one behaviour by mixing its data into the base mixture cost score twice; applying the same data as an isolated pulse followed by a format re-lock produced the program's best result.
W_install the
target behaviour does not install; above W_harm it saturates into
template memorisation and erodes the protected axes, nearest neighbours first.
Measured arms are placed by their construction.| arm | construction | overall | Δ baseline |
|---|---|---|---|
| v18 | format-locked baseline | 0.804 | — |
| v19 | 34% SQuAD-v2 + premise, mixed into the main mix | 0.768 | −0.036 |
| v20 | 12% abstain-only, mixed into the main mix | 0.723 | −0.081 |
| v21 | same category data, staged pulse → format re-lock | 0.820 | +0.016 |
For a model of size N and behaviour B, there exists an
interval [W_install, W_harm] in effective exposure — examples ×
epochs, measured against the model's existing format lock — such that below
W_install the behaviour does not install, and above
W_harm it saturates into template memorisation and erodes
complementary behaviours, nearest neighbours first.
Status of its predictions. P1, windows widen with scale: supported at one size pair (0.6B → 1.2B, same recipe), untested beyond. P2, damage is dose-driven rather than data-driven: supported (v19/v20 vs v21; swapped-order control collapsed to 0.497). P3, recovery asymmetry between formats and behaviours: confirmed (Table 2). The mechanism — gradient-level interference between category batches — is unmeasured; the correlation structure in Figure 3b is descriptive only. The hypothesis is therefore a hypothesis: two confirmed predictions, one partially confirmed, mechanism open.
A deliberately over-dosed abstention pulse (300 steps, ≈7 epochs over 343 examples) produced a clean damage map. One epoch-equivalent of re-locking then repaired the structural damage while the newly installed behaviour survived:
| axis | before | after over-dose | after re-lock |
|---|---|---|---|
| grounding | 0.887 | 0.025 | 0.863 |
| premise | 0.733 | 0.533 | — |
| abstention | 0.633 | 0.567 | 0.833 |
| provenance | 0.980 | 0.940 | held |
Grounding was erased and restored almost completely while the newly installed abstention behaviour survived. At 1.2B the same recipe held abstention at 1.000 and grounding at 0.975 simultaneously — a combination no 0.6B arm reached. The window widens with scale.
Inherited from MiniCPM5: specialist teachers, a reverse-KL teacher signal used as a per-token advantage, and distillation prompt reuse. That pipeline has no notion of cross-behaviour interference during consolidation. This program measured exactly that interference; DA-OPD is the synthesis.
The sampled token advantage, unbiased at the behaviour policy:
The consolidation loss:
where the importance ratio is
The first term is the unmodified on-policy distillation objective. The second
term is the addition: while a teacher's domain is being consolidated, every
protected category pays an anchor-KL penalty to a frozen reference, weighted by
how expensive that category is to damage. The reference is the common SFT
initialisation, with β = 0.25 as the prior. Setting
β = 0 recovers vanilla OPD and is the control arm.
The measured comparison: β = 0.25 anchored beat
β = 0 control in 3 of 3 student seeds on overall and every
protected axis at equal transfer (0.816 vs 0.791). Premise preservation failed in
all six arm-seeds, and the β = 0.5 screen fired STOP per its frozen
rule because the β response is non-monotone.
Teacher routing is solved before any run rather than tuned after it:
where g_c is own-category installation gain per epoch,
D_{c→c'} is the expected sub-score change in c' per
epoch of category-c exposure, Ŵ_harm,c is the measured
harm threshold, and K is the total update budget. Unmeasured rows
default to the mean measured damage and are flagged. The output — per-teacher
budgets and predicted per-category deltas — is falsifiable against the run's
actual deltas.
Three measured points on the same recipe family: at 0.6B the recipe trains behaviours up and MMLU rose 3.7 points; at 1.2B the same recipe preserves the cost but damages MMLU monotonically with dose (42.34 → 40.72); at Qwen3.5-2B the recipe regressed against a strong base across every variant measured (−0.08 overall, −5.22 MMLU). The dilution law is measured, not inferred. Recipe choice is an experimental variable per substrate, not a transferred constant.
Two 2-bit builds of one model, identical in file size and bit width, scored 0.014 and 0.708. The variable was not how far the model was compressed but what the compressor was told to protect.
The quantisation gap of a lineage at a rung is
measured at identical evaluation, protocol, seed, and decode settings.
Matched-quant deltas only: candidate_rung − base_rung is reportable;
candidate_rung − base_bf16 is never reported as a delta. Each rung
is a frozen estimand with its own lock entry.
| rung | size | engram v3 | control v3 | uplift |
|---|---|---|---|---|
| Q4_K_M | 2.40 GB | 0.722 | 0.056 | 13× |
| Q2_K, imatrix-calibrated | 1.76 GB | 0.708 | 0.111 | 6.4× |
| Q2_K, naive | 1.76 GB | 0.014 | 0.000 | collapsed |
| IQ1_M, imatrix-calibrated | 1.27 GB | 0.069 | 0.014 | 5× on a collapsed base |
The mechanism is standard tooling, not an invention here.
llama-imatrix runs a calibration corpus through the model and
accumulates, per weight, an importance estimate derived from the expected squared
activation flowing through it — the diagonal of an empirical Hessian/Fisher-type
quantity. llama-quantize --imatrix then allocates bit precision by
that importance: weights the calibration data exercises heavily keep resolution,
weights it rarely touches are squeezed harder.
The contribution is the calibration corpus choice. The imatrix was built from this program's own corpora. A generic calibration set does not know which weights a given domain depends on. The consequence is practical: if a rung collapses, the first variable to fix is what you calibrated against, not the rung.
The deployable artifact is a matched pair (rung, table), and the
product claim is a curve: composite score stays flat as weights shrink, with the
table growing slightly to compensate. Pre-registered gates: at every rung there
exists a table size scoring within 0.05 of the top-rung reference, and a larger
table never hurts.
IQ1_M, calibrated from our own corpora, was measured on both
architecture classes: dense 0.069, MoE 0.097. Both collapsed. The
compression destroys the polarity capability itself, and the division of
labour in §4 says the table cannot replace the model's reject half. IQ1 + engram
parity with an uncompressed base fails on both classes; the ladder floor is
calibrated 2-bit.
A 7.9B/A1.3B mixture-of-experts held 92–93% of its Q4 score at 2 bits, where the dense looped substrate measured 0.014–0.069. Expert redundancy functions as quantisation armour. This is one measured cell on one instrument, not a general claim about MoE architectures.
On one trained 1.2B substrate the same fact table scored 0.767 at F16 and 0.767 at Q4_K_M — bit-for-bit identical — and retained 70% of its uplift at Q2_K (0.533, against a 0.383 floor) at one fifth the file size. Facts in a table survive compression; facts in the weights do not.
A model advertised at ~84–85 MMLU scored 48.2 on this harness. The gap is protocol, not model: 0-shot letter-loglikelihood here against 5-shot generative on the full set, an offset of roughly 36 points that cannot be modelled away. Vendor benchmark numbers are inadmissible as base references on this harness; base-only runs of our own are the references, and a generation-based public bench is the cleaner external anchor.
An engram here is a static, runtime-editable fact table — a few hundred to a few thousand rows, each with source provenance — whose retrieved hits are injected into the prompt. No architecture change, no training. The task splits cleanly into a half only the model can supply and a half a table supplies better than training can.
Fine-grained grading separates three outcomes on reject-polarity items: corrected (rejected the premise and stated the true fact), affirmed (accepted a false premise), and rejected without content (rejected but could not state the fact). All arms share the same six retrieval misses, so the residual decomposes cleanly.
Polarity is the model's job. Accept or reject. At 0.6B, 50 of 60 reject items failed this way. Below roughly 2B the model affirms a false premise even when handed the true fact — a judgement limit, not a knowledge limit.
Content is the table's job. What the true fact is. Supply it and content failures nearly vanish; at 1.2B the model restates an injected fact 89% of the time when it rejects.
Uplift is defined as the ratio of the engram condition to its paired control. The first true paired table in the program's history: 3.7× at 0.6B, 17× at 1.2B, 7.1× at 2B, 13× on the looped substrate at Q4, 5.1× and 4.7× on the two MoEs.
Uplift is non-monotone. It peaks mid-ladder and declines at the top as the control floor rises. The declining tail is the first measured support for the prediction that a substrate carrying knowledge natively has less headroom.
A related result: a gate that decides whether to supply the fact was falsified on both scales. Always-inject beats gated-inject, because injected facts help true-premise items too, and withholding costs more than the gate's reject decision adds.
Growing a table from 310 to 3,000 facts bought +0.02 to +0.04 — a real gain, not a step change. Gate reads: within-class consistency passed (both MoEs trend identically), between-class difference passed (MoE saturates at 1,000, dense-2B still rising, 0.6B flat), and the capacity signature is supported but shallow. Anti-cheat guard: table growth uses generator-grown, provenance-recorded facts only, never bench-derived; golds stay frozen and hit@1 is re-measured at every size.
Twenty items were authored from the frozen table's corpus facts — its actual distribution, disjoint from the bench's end-user topics — and committed before any retrieval probing at all.
| table | bench | holdout |
|---|---|---|
| frozen, hand-curated (28,425 rows) | 0.878 / 0.959 | 0.550 / 0.750 |
| challenger, mechanical (2,445 rows) | 0.980 / 1.000 | 0.850 / 0.900 |
Without the holdout rows merged, the frozen table's holdout retrieval collapsed further — hit@1 0.30, v3 0.50 — so the probe above flatters it. Every residual miss on the large table traced to same-topic near-duplicates that the mechanical hygiene pass never admits. Volume was the disease.
Final gates on the challenger, single 8B-A1B, pool 50, top-k 1, no keyword tiebreak: bench 0.959 at the strict stage-1 grader (0.980 on grader v2), blind holdout 0.900 (18/20, on-hit conversion 17/17). The 0.9 gate passes on a single model, on both the original bench and the blind holdout.
Conflict-aware distillation is the named next lever: two facts, same topic, different numbers — keep the document-pinned one.
The formula was run on three verticals, reported on one basis (stage-1 / grader v2): M365 0.959 / 0.980, SQL Server 0.878 / 0.878, PowerShell 0.781 / 0.829. On PowerShell the model leg transferred unchanged — on-hit conversion 0.96–1.00, replicating the M365 result including on affirm items — while the retrieval leg was bound by table quality (0.585 hit@1 against M365's 0.898). The gap on a new vertical is a table-quality problem, not a method problem.
Giving the model a retrieval tool made it worse than retrieving before inference — at both scales measured, through three separable failure modes.
| arm | tool-call rate | on-call conversion | v3, 8B | v3, 350M |
|---|---|---|---|---|
| pre-injection | n/a | 1.000 on-hit | 0.939 | 0.327 |
| tool available, model chooses | 36.7% | 0.722 | 0.510 | 0.265 |
| tool call, model writes the query | 98.0% | 0.625 | 0.633 | 0.061 |
The three leaks are independent and they compound. Initiation: with the tool merely available, the 8B calls it on barely a third of items and answers blind the rest of the way. Query quality: with explicit framing the call rate reaches 98%, but the model's keyword queries hit the right fact only ~49% of the time. Round-trip conversion: even when the perfect fact is returned, conversion through the tool result is 0.722 against 1.000 for pre-injection.
An MCP surface does not make a small model competent. At 350M the tool is called on 96–98% of items and conversion is still 0.27 with the perfect fact handed back — the misses are affirmed. The binding constraint at that scale is conversion, not access. For a bounded product scope, pre-injection is strictly better; tool-calling remains interesting only for open-ended retrieval the engram cannot anticipate.
One-shot multi-row spreadsheet aggregation failed at every scale measured, including 1.2B with harness arithmetic supplied — asserting 102,000 against 15,834.36. Scale bought document quality, not the composition skill. The product answer is a deterministic harness that evaluates formulas against a harvested cell grid and prints exact totals regardless of model state. The measured boundary in the trained 350M: three rows 1.000, seven rows 0.167.
The deployable object is not a model. It is a triple: a table carrying the facts with per-row provenance, editable and versioned without retraining and bit-identical at every rung; a substrate carrying the judgement — accept, reject, abstain, cite — large enough for the polarity half; and a rung, a measured choice with a measured cliff, printed on the card next to the win.
Config rules are per-vertical because retrieval geometry is: topic-diverse corpora want a wide candidate pool with top-1 and no tiebreak; narrow corpora want a narrow pool with top-3, and measured worse at the wider setting. The optimisation question changes accordingly — not how small can the model be, but which half of the problem am I fixing.
The measurements above were taken at 0.3B–8B on one 6 GB GPU. This section separates what they predict at scale from what remains open, and names the experiments the hypothesis most needs. Everything here is intended as open-source knowledge: the harness is released so each item can be run, checked, or falsified by someone else.
If windows widen with scale — and P1 is confirmed at only one size pair — two consequences follow for fine-tuning at scale. First, targeted behavioural patching should become relatively cheaper as models grow: a pulse-then-re-lock curriculum sized against a damage ledger, rather than category data mixed into the base mixture. Second, multi-teacher consolidation should weight anchor replay by measured per-category damage — the DA-OPD anchor term, equation (3) — rather than uniformly. Both are extrapolations from the measured interval, not results.
Build the importance matrix from deployment-domain text, not a generic corpus, and ship quantisation rungs as measured pairs with their table. The measured rungs: calibrated 2-bit at parity with Q4 at 73% of the size on one dense substrate, and 92–93% retention at 2 bits on one MoE. If the MoE result holds across the class, 2-bit MoE serving is roughly a further 40% memory reduction at near-parity — the cheapest untested capacity lever in this paper.
Perishable knowledge belongs in tables, not weights: per-vertical tables of roughly a thousand facts with per-row provenance, refreshed without retraining. The self-populating pipeline — search, distill a cited fact, answer — measured 0.800 against a 0.000 control on post-cutoff questions, at n = 5: a demonstration, not a result. Private enterprise knowledge is the axis on which external tables compose with rather than compete against large n-gram-table architectures: those tables freeze at pretraining; runtime-editable tables with provenance are structurally absent from them.
The experiments this hypothesis most needs, in the order a small team can run them:
W_harm keeps widening.The evaluation harness, corpus builders, retrieval pipeline, and command-line tooling described here are released as open source at github.com/chadstthomas-svg/cowork-cli, together with the frozen-instrument generators. Licenses: code Apache-2.0; catalog sources CC-BY-4.0; NLI grader MIT; training corpus under the Tablinum Dataset License v1.0. Every number recomputes from a committed run record — the intent is that you can check, and falsify.