Grounded improvement plan

Latest implementation: coverage-preserving protection and new model results.

Research cutoff: 5 November 2026. Evidence: research update, model manifest, and historical assessment.

The next-stage recommendations analyze the completed runs, correct PII-Tracer availability, and prioritize remaining coverage-policy losses, calibration, and new model experiments.

This plan separates implementable engineering changes from experiments whose promotion requires evidence or unavailable deployment data. Status is updated only after the corresponding tests and reports exist. A runnable experiment is not a production-approved model.

1. Objective and non-negotiable properties

Improve detection and measurement while preserving the public contract:

source[span.start:span.end] == span.text

The first principles imply the following acceptance rules. Tests must challenge the implementation rather than mirror its code.

2. Implementation sequence and status

IMP-01 through IMP-07 are the immediate implementation scope. Later stages are genuine outstanding work, not implied completed features. They must not be marked complete merely because the library tests pass.

3. Repair the benchmark contract first — IMP-01

Change

Preserve historical SPY generation as an explicitly named legacy protocol. Introduce a new version that selects a declared generator from a seeded local random source and uses government-ID-shaped values only when the declared public label is GOV_ID. Do not pass RIPE network handles, IBANs, or cards off as government IDs. Record generator/version, Faker version, seed, source policy, and selection digest. Avoid process-global random-state interference.

Maintain a written crosswalk from native corpus labels to public types. A generic source identifier does not establish its subtype. For ambiguous original records, exclude the unsupported type from a clearly labeled common-support diagnostic, retain its unsupported count, and do not call the result full-taxonomy recall. A separate synthetic control can establish the government-ID mechanics without claiming it represents deployment data.

Tests and evidence

• Generated ID examples conform to the declared category and reproduce under the same version/seed; the generator does not mutate global Faker randomness.
• Legacy protocol reproduces legacy inputs; changing a generation protocol changes report identity. Old report bytes are retained.
• Reports state that SPY annotations are author-associated. A broad-discovery model's extra spans are not automatically adjudicated as hallucinations.
• The model ranking is rerun after protocol changes, with all models using identical source/gold selections. No prediction-driven rewriting of gold.

4. Preserve independently useful spans — IMP-02

Write ADR 0005 before changing overlap policy. Within an overlap component, select the highest-priority candidate, remove only candidates that directly conflict with that winner, then continue resolving the remaining intervals. Deterministic ranking stays based on taxonomy severity, preferred detector lane, length, score, and stable lexical tie-breakers.

The bridge example must keep GOV_ID[5:9] and EMAIL[18:30]. Test chains, nested ranges, equal ranks, equal offsets, duplicates, and shuffled detector order. Benchmark ordinary disjoint input separately from adversarial dense overlaps so the documented complexity matches the actual algorithm.

This does not prove every discarded candidate is harmless. Keep conflict diagnostics and compare pre-merge versus post-merge containment. Do not silently turn a heterogeneous interval union into an invented public PII type.

5. Prove model input coverage — IMP-03

Character count is a resource bound, not a token budget. Derive windows from the actual tokenizer and effective model context, including special tokens and schema overhead where applicable. Preserve character offsets on each original window. Verify that the final model input fits; split further or raise rather than letting a library truncate invisibly.

The optional legacy GLiNER and ModelScope adapters remain unadmitted for token-coverage claims. Their tokenizer/configuration paths still require model-specific validation before that guarantee can be extended to them. For GLiNER2/2.5, account for schema tokens and use the supported long-input path or checked chunks. A longer-context model does not remove boundary-crossing tests. Very long mentions may require an explicit unsupported input error if no valid window can contain both endpoints.

Adversarial controls put synthetic identifiers after dense Unicode prefixes, inside overlap regions, and at every window edge. The tests must use fake tokenizers with more tokens than characters, not just ASCII examples for which the old heuristic happened to work.

6. Measure the product outcome — IMP-04

Keep exact typed precision/recall/F1/F2. Add:

• Untyped complete containment: a gold range is protected only if the union of masking intervals covers all its characters, regardless of predicted type.
• Sensitive-character recall: covered gold characters divided by gold characters, computed using interval unions so overlaps are counted once.
• Masking precision: masked gold characters divided by all masked characters, under the declared annotation policy. Interpret incomplete gold as a limitation, not proof of low utility.
• PII-free document false-positive rate: fraction of audited negative documents that lose any characters, plus masked-character fraction.
• Repeated-identifier complete coverage: all mentions must be removed; a single missed recurrence fails that identifier's outcome.
• Operational completeness: failures, invalid offsets, unsupported labels, over-budget inputs, and incomplete documents are explicit outcomes. Failed inputs cannot be removed from the success denominator.

Compare exact typed and containment metrics side by side. A broad OPF account_number can contain sensitive digits while lacking enough evidence to choose GOV_ID versus PAYMENT. Do not force a false public type to make a model fit. Native-label experiments remain labeled as such until a sound public taxonomy decision exists.

7. Run a staged model tournament — IMP-05

First batch

Retain rules, current en-closed, and current multi-open controls. Add the compatible Horizon small encoder first, then GLiNER2.5 boundary extraction. Add OPF using its native decoder and a reviewed taxonomy crosswalk, or retain it as an explicit native-label experiment where the public mapping is ambiguous. Do not reuse generic token aggregation merely because it runs.

Every adapter must pin weights, tokenizer, model code if required, decoder, label schema, and runtime package versions. Review only the necessary pinned code; do not execute floating remote helper modules. Test adapters alone and in combination with rules so the rule lane cannot hide a nonfunctional model.

Liquid and Protegrity remain conditional on license/code/access review; ShinrAI weights require access review. Kiji and OpenMed require their proper sequence decoders. Their existence does not block the first compatible batch.

Data and selection

Use the audited SPY legal/medical tracks, finance, structured forms, hard negatives, and a versioned first-principles challenge set. Declare training overlap for every model and dataset. Public benchmarks screen candidates; they do not constitute a sealed deployment freeze.

The private target set remains unavailable. Proposed collection must include the deployment's actual languages, domains, and document lengths, with two annotators and adjudication of disputed policy/boundary cases. Group splits by template family, original document, and conversation. Near-duplicates and translations cannot cross splits. Do not treat absent private data as consent to send production PII to an annotation service.

Modal execution

• Use the existing private SDK model class and persistent public-weight cache. Run CPU controls and GPU candidates explicitly; record actual GPU hardware.
• Run one representative load/inference preflight before larger evaluations. Bound document count, container count, startup time, and invocation time.
• Keep single-request warm p50/p95 separate from batch throughput, cold start, model loading, and network time. Record memory and failures where available.
• Use unique report identity and a run-completion manifest so stale files from a previous run cannot be mistaken for new evidence.
• Measure model-only and rules-plus-model outcomes. If quality does not improve, retain the result rather than changing gold or suppressing failed slices.

The first batch is exploratory and bounded. Wider GPU spending and production promotion require explicit budgets and risk floors; no paper's hardware price or throughput becomes our service SLO.

8. First-principles testing on Modal — IMP-06

Run the full repository test suite in a clean Modal CPU environment, including the actual wheel/source layout and site tools. Run actual model inference in the GPU runtime. Keep offline deterministic tests separate from statistical model quality: both must pass their own criteria.

Instruction-like text and Unicode evasion are adversarial controls. No test may assert that a detector is safe merely because its confidence is high.

9. Documentation and site — IMP-07

Publish this plan, the research update, the model manifest, and the overlap ADR through the static site's existing documentation routes. Update the docs index, research/catalog pointers, evaluation guide, and historical assessment erratum. Explain the SPY qualification near the older scores. Preserve the historical reports and source contract. Site claims must identify an experiment as an experiment, and a model-card number as external evidence.

The local build and the Modal build must pass link checks. Browser checks must confirm the new pages are reachable and readable, including a narrow viewport. Updating repository site content is separate from publishing the production GitHub Pages deployment; record which actually occurred.

10. Promotion criteria and later experiments

Keep the existing requirement of zero observed high-severity containment failures on a human-audited sealed freeze. Precision, per-type recall, latency, negative damage, and supported-language floors remain UNCONFIRMED until the deployment owner chooses them on development data and commits them before the sealed run. The earlier CLI example of 0.90 recall is not a universal product requirement.

Use paired document-level bootstrap intervals for candidate differences on larger runs, clustering by conversation/template where required. Do not claim significance from 50 prefix-selected records. Zero observed errors on a finite freeze is not zero future risk.

Only after those steps should IMP-08 test domain adaptation with a fixed train/dev/test split, repeated seeds, category-balanced sampling, hard negatives, and replay data to detect forgetting. Human-reviewed corrections stay separate from model-generated labels. Learning-curve sizes are experimental proposals, not guaranteed crossover points.

IMP-09 evaluates quantization against the identical float decoder. IMP-10 requires an explicit tool-necessity policy, repeated-mention conversation gold, and downstream task checks for surrogates. Do not invent cryptography, replace secret irreversibility, or equate retained task utility with anonymity.

11. Completion record

Implemented on 5 November 2026. IMP-01 through IMP-07 now have code and validation evidence. IMP-08 through IMP-10 remain conditional, and no model has been promoted to a production default.

First-principles implementation and verification

• SPY materialization uses local random state, seeded generator selection, government-ID-shaped synthetic values, and an explicit protocol. Legacy reports remain unchanged. Both SPY domains were rerun on identical input/gold selections for every candidate.
• Direct-conflict merge retains independent winners, with deterministic tie-breaking and O(n log n) interval indexing. The bridge, dense-overlap, secret-vault, source-offset, and permutation controls passed.
• HF and GLiNER2 check actual encoded windows, including special tokens and schema. Tests cover more tokens than characters, dense Unicode tails, and impossible schema/character budgets. All six model lanes also exercised actual Modal inference on fixed Unicode, repeated-value, long-prefix, mask/restore, and rule-disabled controls.
• Reports retain exact typed scores, complete and character containment, unmapped counts, within-document repeated-value coverage, and pre-policy candidate coverage. The final gated confirmation also records source and masked-character fractions. Hand-calculated controls prove that a wrong label can still mask a value, that a missed recurrence fails all-mentions coverage, and that audited-negative damage has an explicit denominator.
• New experiments pin Horizon Small, GLiNER2.5 boundary extraction, and OPF's official native Viterbi runtime. OPF retains its native labels and rejects changed decoded text or invalid claimed surfaces. Public typed SDK presets do not silently acquire OPF's coarse label meanings.

The full Modal verification completed with 204 passed, 1 skipped, 0 failed in the test suite. The one skip is the optional HF demo-browser test in a CPU environment without model weights; real T4 inference is recorded separately below. The run also built and installed the wheel, proved the installed-package source/mask/restore and no-torch rules contract, regenerated the site data/icons/social images, built and checked all 30 HTML pages, audited completed model reports, and exercised Chromium at 1440 px and 390 px.

Evidence: JUnit, desktop screenshot, mobile screenshot, and Modal run. The JSON retains source-file hashes, the wheel hash, dependency versions, and step exit codes. After this completion record is rendered, a separate --site-only Modal run rechecks the final pages, report links, and both browser viewports. Its narrower scope is labeled in its saved evidence.

Matched public experiments

Six lanes completed all four corpus runs: 24 corpus/preset reports, plus one explicit-gate confirmation. Rules use four CPU cores; models use an NVIDIA T4. Every main run has one matched selection hash across its six reports. Every learned typed lane also has a full **model-only ablation on the same inputs** inside its JSON. The fixed inference smoke requires model detections with the rule detector disabled.

These are bounded public prefix samples, not a sealed deployment freeze. SPY uses government-id-v2 and author-associated gold. Nemotron is an in-domain control for ModernBERT. Training overlap is not excluded for the other public checkpoints. No significance or whole-language guarantee follows from these samples. Tables below use the report's all slice; the Nemotron minority slices are shown separately. F1 is exact and typed; containment is untyped complete-span coverage. A dash for OPF means no typed projection was made, not a score of zero. p95 covers warm locate calls, including tokenization and merging, and excludes loading, transport, and scoring.

Two main experiments failed the invalid-offset gate: GLiNER2.5 on legal and GLiNER2-PII on medical each returned one rejected span. Their run manifests are quality-gates-failed, and their CLI commands exited nonzero. Other rows passing the permissive integration policy are not production approvals.

SPY legal

50 documents and 254 gold spans in all; 50 unique inputs evaluated across all requested slices. Completion manifest; Modal run.

SPY medical

50 documents and 311 gold spans in all; 50 unique inputs evaluated across all requested slices. Completion manifest; Modal run.

Gretel finance

50 documents and 151 gold spans in all; 50 unique inputs evaluated across all requested slices. Completion manifest; Modal run.

Nemotron

25 documents and 53 gold spans in all; 75 unique inputs evaluated across all requested slices. Completion manifest; Modal run.

--limit 25 means a prefix per slice, not only 25 model calls. Each cell below is exact recall / high-severity misses on 25 documents. OPF cells show untyped containment / misses. Slice memberships overlap and must not be summed.

Decisions from these results

Horizon Small is the strongest exact-recall candidate on these legal and medical prefixes, but it still misses ten high-severity spans on medical and four on finance. Its legal exact precision is only 0.319 under SPY's annotation policy. GLiNER2-PII has higher recall than Horizon on finance and zero measured high-severity misses there. ModernBERT leads exact recall on its in-domain Nemotron control. There is no single production winner.

The explicit-gate Horizon confirmation passed 0.90 exact recall and zero high-severity misses on this legal slice: recall 0.933, complete-span containment 0.972, character recall 0.990, and masked-character fraction 0.0753. That run also passed the image's Parquet import preflight and pip check. Modal run. These are scoped experimental gates. Negative damage floors, deployment precision, language coverage, private data, and the sealed freeze remain unconfirmed. Zero-gold document counts in public corpora do not establish an audited negative set.

Rules do not improve every exact-F1 result. The model-only ablations preserve that evidence: for example, GLiNER2.5 finance F1 is higher alone than with rules. Rule preference and overlap decisions can trade typed correctness for masking coverage. Before/after diagnostics also show remaining policy losses, so the bridge fix must not be described as a proof that merging cannot leak.

OPF's native containment and high-severity outcomes are useful independent measurements, but its roughly 1.3–1.5 second p95 in these runs is much slower than the small encoders. No batch-throughput number or model-card partial F1 is substituted for those measured request latencies. Peak memory and target edge-device performance were not measured.

Reproduction, failures, and publication scope

# Repeat each of the four matched runs; limit is per slice.
.venv/bin/modal run modal/models.py --corpus spy-legal --limit 50 --gpu T4 \
  --presets rules,en-closed,multi-open,multi-encoder,multi-boundary,opf-native
.venv/bin/modal run modal/models.py --corpus spy-medical --limit 50 --gpu T4 \
  --presets rules,en-closed,multi-open,multi-encoder,multi-boundary,opf-native
.venv/bin/modal run modal/models.py --corpus gretel-finance-en --limit 50 --gpu T4 \
  --presets rules,en-closed,multi-open,multi-encoder,multi-boundary,opf-native
.venv/bin/modal run modal/models.py --corpus nemotron-pii --limit 25 --gpu T4 \
  --presets rules,en-closed,multi-open,multi-encoder,multi-boundary,opf-native
.venv/bin/modal run modal/models.py --corpus spy-legal --limit 50 --gpu T4 \
  --presets multi-encoder --min-recall 0.90 --max-leaks 0
.venv/bin/modal run modal/validate.py
.venv/bin/modal run modal/validate.py --site-only
.venv/bin/python modal/check_evidence.py

The initial OPF attempt used the Transformers root conversion with the native runtime and failed. The corrected adapter explicitly loads original/ at the same pinned model revision. The initial finance image also failed because Arrow 26 required NumPy 2 at import while the runtime used NumPy 1.26.4; the image now pins compatible Arrow 20 and checks imports and dependencies before allocating a model. Failed manifests remain here and here. They are not treated as completed benchmark results. Historical legacy reports are unchanged.

The optional legacy GLiNER and ModelScope weights, gated/restricted models, custom CRF/ONNX candidates, domain adaptation, quantization, and surrogate/tool utility studies remain unvalidated or conditional as stated above. First- principles tests establish the challenged invariants, not complete PII recall or a mathematical proof of system safety.

Repository documentation and all new site routes/evidence assets are updated. The static site is built and browser-tested. The existing Pages workflow publishes changes after merge to main; its deployment runs record the actual publication status. These initial implementation runs preceded the subsequent 0.4.0 package release. The research runs do not imply a production default change, fine-tuning run, or public inference endpoint.