Improvement research after 0.4.0: 5 November 2026

This research combines primary-source review with **16 completed single-model Modal evaluations and four paired experiments**. The cutoff is 5 November, not the end of November. The results change the earlier brainstorming: compact checkpoints are useful controls, but neither new float model earns a general replacement, and the released int4 artifact does not preserve coverage.

The work adds repository-only research adapters, reproducible experiments, and tests. The published 0.4.0 package and public presets retain their release behavior. No training, calibrated router, streaming service, production promotion, or new package publication occurred in this research.

Follow-up implementation: protection readiness and release 0.5.0 adds output gates and a counterfactual stage audit. It explains all ten observed medical coverage losses at taxonomy projection without redefining the public types.

Evidence and experiment design

• Six-model artifact manifest: exact revisions, current license declarations, file sizes/LFS hashes, and hashes of inspected cards, configurations, notices, and license files.
• Recomputed result summary: all report hashes, matched selection digests, paired coverage, and support counts. scripts/audit_compact_research.py checks this evidence and recomputes paired summaries from saved source-free case outcomes.
• Three public prefixes: 50 legal, 50 medical, and 50 finance records; 254, 311, and 151 in-scope gold spans. CPU and paired studies reuse these inputs. These are exploratory prefixes, not a new sealed deployment test.
• All new comparisons retain native labels, original source, and union containment. They do not claim public typed F1. No rules or confusable folding are added. The Horizon control therefore differs from the typed multi-encoder system in the previous results.
• SPY retains government-id-v2, seed 42, and author-associated annotation. Its omitted identifiers and broader native vocabularies mean character masking precision cannot by itself measure unwanted masking or task utility.
• Float experiments use the pinned Torch/Transformers runtime on a T4. The portable quantized control uses ONNX Runtime 1.23.2 on four Modal CPU threads. Warm p95 measures complete locate calls, including tokenization, windows, decoding, and source-offset handling; loading and downstream task work are excluded.
• GPU memory is peak Torch allocated memory during evaluation. CPU RSS is process-lifetime peak, including imports and loading. This shared research image imports heavyweight libraries; its RSS is not a lean edge-runtime footprint.

Native categories may include types absent from a public gold policy. A zero high-severity count covers only that sample's gold and supported measurement; it does not establish that a model recognizes every credential or identifier.

Literature that changes the next engineering decisions

REDACT remains a useful multilingual/disclosure-form expansion. Its released protocol and label schema must stay separate from SPY's author policy and our broad discovery task. Its small obfuscated slice should not become a strong robustness claim.

Artifact review and admission

The Masker card's pretrained-student initialization is a useful distillation ablation for future training. Its published quantization agreement is measured on its own validation distribution; our parity experiment below tests a different question on these public prefixes.

Decoder admission matters

Masker emits B/I/E/S/O tags. The adapter explicitly performs argmax BIOES decoding, including closing E/S tokens, recording how orphan I/E tags start spans, and preserving offsets through checked windows. It executes no remote model code. Special tokens, Unicode combining marks, malformed ranges, tag changes, and native labels have independent regression examples.

Ettin and Horizon use the checked Transformers pipeline with native labels. Their labels are not silently coerced into EdgePII's taxonomy. PII-Tracer retains its previously reviewed, hash-verified native Viterbi runtime.

Actual T4 measurements

Each row uses 50 records. Containment requires every character of a gold span to be covered by the native range union. Misses below are high-severity containment failures, not total false negatives.

Completion manifests and all four reports per corpus: legal, medical, finance. Modal runs: legal, medical, finance.

Interpretation: Masker is faster but loses substantial full-span coverage under this gold policy. Its segmented address/name inventory also differs from full-address/person gold; character recall and examples must accompany containment. Ettin provides no general quality improvement over the Horizon control here. Native Horizon is a different system from the projected, folded, rules-assisted preset; these results do not erase that system's earlier misses.

CPU and int4 results

Same medical prefix, four CPU threads:

Completion manifest, Modal run. These RSS values include the research harness. GPU and CPU Ettin outputs also differ slightly; identical weights do not establish cross-runtime span parity.

The released Masker files are 70,811,538 bytes for stored float weights and 22,344,556 bytes for the int4 ONNX graph. That smaller file was about **8.5% slower** at medical warm p95 here. Torch inference uses fp32 parameters after loading the stored float artifact; file size and live memory are distinct.

Paired CPU studies use the same tokenizer, window policy, and BIOES decoder:

Reports: medical, finance. Modal: medical, finance. Exact range parity compares canonical masked interval unions; native-span parity also requires labels and boundaries to agree. These are not token-agreement metrics.

Precision metadata correction

Direct Modal probes of GPU parameters and CPU parameters verified the actual loaded precision under the same Torch 2.8.0 / Transformers 5.22.1 pins: Ettin uses bfloat16; Horizon and Masker use float32, on both devices. The original Ettin reports' native_backend string ending in fp32 was an unverified annotation and is incorrect. Historical reports retain their bytes; their measured containment/latency values are unchanged. The adapter now records the loaded dtype instead of assuming it. The tables' float references mean non-int4 inference, not a uniform precision across model families. An Ettin fp32 ablation has not been run; the observed rows compare the loader behavior of the released candidates and cannot isolate architecture from precision. Probe runs: GPU, CPU.

Decision: retain int4 as an experiment. It adds a finance high-severity miss, so smaller storage does not justify promoting it as an equivalent runtime. See ONNX Runtime's quantization guidance for its emphasis on debugging accuracy changes.

Do complementary errors justify an ensemble?

Paired experiments compute each detector on the same original document and score the actual interval union. They can credit complementary partial ranges that jointly cover a gold span even when neither alone does. Tests include that counterexample. Saved case outcomes contain coverage flags and counts, without source strings, document IDs, or extracted values.

Reports: medical, finance. Modal: medical, finance.

Decision: test a domain-aware policy only after independent validation. Horizon + Masker recovers two finance high-severity misses, but adds no medical containment. Adding Horizon to Tracer adds no finance complete-span containment. Extra masked characters can include both omitted PII and unwanted damage; the table does not adjudicate utility. No adaptive router or routed latency was measured. A lack of detected spans cannot by itself justify skipping a verifier.

A taxonomy discrepancy worth investigating first

On identical medical inputs, native Horizon has no high-severity misses while the earlier typed system has ten. The paired support audit records four single-span supports of gold GOV_ID under native MEDICAL_ID, which the current global crosswalk leaves unmapped. Counts describe supporting predictions, not necessarily unique gold entities.

That demonstrates a concrete vocabulary/policy gap, but it does not explain all ten differences. Native versus typed systems also differ in normalization, trimming, filtering, and overlap policy. Do not map every medical identifier to a government ID to improve this sample. A stage-by-stage counterfactual audit and model-specific category definitions should precede a public mapping change. Strict protection already rejects nonzero unmapped diagnostics by default.

Prioritized improvement plan

P0/P1 are the next implementation priority. P2 has exploratory supporting evidence on finance, not a universal ensemble result. P3–P5 remain conditional experiments; no deployment-domain data or human adjudication was invented. Downstream usefulness and contextual re-identification require separate outcome tests, with empirical attacks distinguished from formal privacy guarantees.

Reproduction and first-principles checks

.venv/bin/python scripts/research_snapshot.py --out /tmp/compact-models-new-snapshot.json
.venv/bin/modal run modal/models.py --corpus spy-medical --limit 50 --gpu T4 \
  --presets masker-mini-native,ettin-native,horizon-native,tracer-native
.venv/bin/modal run modal/models.py --corpus spy-medical --limit 50 --gpu cpu \
  --presets masker-mini-native,masker-int4-native,ettin-native,horizon-native
.venv/bin/modal run modal/models.py --corpus spy-medical --limit 50 --gpu T4 --paired
.venv/bin/modal run modal/models.py --corpus gretel-finance-en --limit 50 --gpu cpu --paired --quantized
.venv/bin/modal run modal/models.py --inspect-only --gpu T4 \
  --presets ettin-native,horizon-native,masker-mini-native --out /tmp/compact-runtime-probe.json
.venv/bin/python scripts/audit_compact_research.py
.venv/bin/modal run modal/validate.py

The artifact collector records a fresh snapshot, so future upstream changes must not silently replace this dated manifest. Experiment constants pin the revisions measured here. Native models remain evaluation-only SDK rejections. The single-model run manifests retain integration gates; those permissive gates do not establish a zero-leak quality pass.

Initial paired launch failures are retained separately. They failed before inference because of unsupported function options or inconsistent remote import/resource construction. The corrected paired worker runs through the established Modal Model class. Those failed starts are not counted among the completed experiments.

Validation record

The final full Modal verification passed 257 tests, with one optional weight-dependent demo test skipped. It built and exercised the installed wheel, audited all completed model reports, recomputed the paired evidence, rebuilt the site, and checked desktop/mobile browsers. JUnit, Modal run. The earlier full run is retained with its own source hashes. The new regression examples cover BIOES boundaries, Unicode/source offsets, invalid token schemas, complementary partial masks, and range-versus-label parity. The final documentation and audit check passed 17 tests with no skips on Modal, after the precision correction and publication evidence were added.