Next-stage AI engineering: coverage, calibration, and model selection

Implementation follow-up: protection API and measured model results. Core protection and the three reference adapters now have Modal tests and matched public measurements. The private viewer, deployment annotations, calibration, training, and optimized-runtime stages remain outstanding.

Reviewed 5 November 2026, after the implemented first-stage plan. This is a prioritized proposal. New candidate metadata and source files were inspected; no new weights were downloaded, custom model code executed, or inference results measured for this review. Existing Modal results remain the measurement baseline. The candidate manifest pins the inspected artifacts, and the baseline reanalysis records report hashes and the arithmetic behind the recommendations.

The next priority is **preserving coverage through policy, then evaluating additional models against adjudicated errors**. The current evidence does not support choosing a universal replacement or promoting a new default.

1. What the completed experiments tell us

These observations use the all prefix slice of each existing report. They are descriptive, not independent trials or significance tests. SPY's author-associated policy also differs from broad discovery: an extra predicted name is not automatically an unwanted redaction. No gold is changed to improve a candidate's score.

2. First change: distinguish entity description from protection

The existing Span contract remains useful: a typed, disjoint result on the original source. Selecting one description for an overlap does not establish that the discarded characters may safely leave the boundary.

Propose an opt-in ProtectionPlan, separate from locate() and the existing typed merger. It consumes validated candidates after explicit type and score eligibility. It records protection intervals and provenance without inventing a public PII type for a heterogeneous union.

Required invariants for this experiment:

1. Every character covered by an eligible candidate is covered by the plan. Overlap arbitration cannot reduce that coverage.
2. Intervals index the original string. Construction never searches for a repeated value to repair an invalid offset.
3. Every character covered by an eligible SECRET candidate is irreversible. No overlap with another type may put any credential fragment in the vault.
4. Protection segments are disjoint. Partitioning at candidate boundaries can preserve active-type provenance and secret handling; restoration must be tested across those segments.
5. Invalid offsets, incomplete inference, and unsupported requested categories are explicit diagnostic outcomes. They cannot be silently reported as a fully processed clean document.

Begin with counterfactual evaluation and an opt-in API. Union masking can remove more non-PII text, especially with broad false-positive candidates. It must meet a measured negative-damage budget before becoming a default. The invariant concerns supplied eligible candidates; it cannot protect an identifier missed by every detector. Keep the typed result and exact scorer unchanged so the trade is visible.

Acceptance includes nested SECRET/GOV_ID/PERSON ranges, losing bridges, adjacent intervals, duplicate and permuted inputs, repeated placeholders, Unicode, and structured mappings. Measure actual masked output, vault content, and restoration, not just interval geometry.

Also test country-specific identifier recognizers behind explicit application profiles. The current rule detector has no GOV_ID recognizer, while the corrected SPY protocol deliberately generates US SSN-shaped IDs. A constrained US profile is a useful low-cost control before fine-tuning a larger model for those misses. Use contextual cues and published structural rules, and test collisions with order IDs and unrelated numeric strings. SSA's structural exclusions provide negative controls; passing a format check does not establish that a number was issued or belongs to anyone. Keep the country scope explicit, measure recall lost by requiring cues, and do not turn this into an automatic worldwide GOV_ID rule or assign a probability to a grammar match.

3. Model candidates worth testing next

Availability correction: the earlier review missed the public Perplexity PII-Tracer checkpoint. Its official model repository and the PII-TRACE dataset are now verified in the manifest. The paper's release-promise wording did not establish artifact availability. This correction does not add a measured EdgePII result for that model.

The verified Hub release is a 500-conversation English subset, currently exposed as a train split, with 450 positive and 50 empty-control conversations according to its card. It is not the paper's entire multilingual benchmark. Its training/evaluation lineage must be checked before ranking PII-Tracer on it. Current gold spans occur in user messages; the artifact does not establish assistant-message detection quality. Preserve (turn, role, start, end) and original message text when implementing the conversation adapter.

The manifest contains full revisions; the first three are respectively ef0eff8e63c1711ec0c997c5ca9ea3b78a068684, d25c16f2e57e321f6d2527715c01df9112f956f5, and e5f6a01d465c9a2489a74a95ff1151359b759213.

PII-Tracer's sensitivity output is advisory and must not suppress span detection. Its implementation derives span scores from transformed logits; those are not established probabilities of complete protection. Its official vLLM/GGUF/MLX repositories are alternate packagings, not additional independent models. The MLX card explicitly requires a loader implementation that is not included. Start with the reference runtime before testing alternate engines.

Models to keep conditional

• Dataiku Kiji remains a useful CRF control with a custom head/decoder. Liquid's PII encoder also remains conditional on custom-code and license review. Neither is a generic pipeline drop-in merely because it carries a token-classification tag.
• Stirling-Minimo v0.2 is an interesting 10M ONNX efficiency reference, but the published CC-BY-NC-4.0 terms do not make it an unrestricted commercial default. Its card separates checkpoint behavior from the served system and uses masking F2. Do not substitute those claims for our exact typed scores or latency.
• ScienceSoft's encoder has proprietary terms that do not grant production use of the released weights. Its disclosure of seed variance and benchmark reuse is useful evaluation practice, not a deployment license.
• Deerpark's edge checkpoint is gated. Protegrity and ShinrAI retain the access/license qualifications in the first review. A public card or permissive metadata field is not enough to admit a model or all its dependencies.

4. Apply AI engineering principles to this closed task

Hamel Husain's error-analysis guidance supports reading concrete failures before choosing tools. The first artifact should be a private or synthetic/public case viewer: original text, gold, raw valid candidates, selected spans, protection intervals, and a reviewer reason. Real source values stay in the approved review boundary and out of docs/eval/, telemetry, and public pages. The current aggregate reports cannot support document-level disagreement analysis by themselves.

Anthropic's January 2026 evaluation guidance distinguishes outcomes, graders, regression checks, and capability tests. For EdgePII the deterministic outcome is the actual protected string and its source ranges. Human reviewers adjudicate ambiguous annotation policy; an LLM does not decide exact-span correctness. New multilingual/OCR/conversation cases are capability tests; verified failure fixes join the regression suite.

Operationalize those principles in this order:

1. Separate error causes. Label detection misses, boundary shifts, unsupported categories, normalization effects, decoder/offset failures, and protection-policy losses separately. Audit label granularity too: a city fragment, complete address, generic date, and date of birth are not interchangeable annotations.
2. Create a policy-aligned development set. Use adjudicated examples from actual target languages, domains, lengths, and structured formats. Include audited negatives. Keep SPY author-associated and broad-discovery tracks separate. Public or model-generated drafts require human verification. Permissioned deployment data and domain-owner review remain dependencies.
3. Make splits before optimization. Group by document, template family, conversation, and translation lineage. Keep a sealed test distinct from the already inspected public prefixes. Never fine-tune on these reported test examples and continue calling them untouched evaluation.
4. Optimize operating points, not one aggregate F1. Choose per-model/type thresholds on development data under a high-severity containment constraint, precision/negative-damage budgets, and latency/memory limits. Model scores require calibration checks; rule scores remain unscored. Freeze criteria before the sealed run and inspect every required minority slice.
5. Test complementary errors before routing. Persist privacy-appropriate paired coverage outcomes, then measure which misses a second detector recovers and how much text it damages. Use trusted application domain metadata for initial routing trials. Low confidence alone is not a reliable escalation trigger because a confident miss emits no span at all.
6. Adapt only after error taxonomy and clean splits exist. Test learning curves such as 100/500/1,000 adjudicated training examples, repeated seeds, rare-category balance, hard negatives, and replay against forgetting. These are proposed experiment sizes, not known crossover points. Distill only after a teacher demonstrates additional coverage on independent examples.

Use paired document-level intervals for model differences on larger samples, clustering related templates/conversations. Keep a small random audit alongside disagreement sampling: agreement does not reveal shared blind spots. Zero observed leaks on a small prefix is not a bound on all future traffic.

5. Concrete implementation order and evidence gates

For NEXT-03, reuse Modal's private SDK workers and cached public weights. Preflight one synthetic document with each adapter before a bounded matched tournament. Record model/code/tokenizer/decoder pins, input selection hashes, failures, cold load, warm p50/p95, throughput, peak GPU memory, CPU RSS, and cost on the same hardware and workload. Include model-only and full-system measurements. Existing permissive integration gates do not become promotion gates just because an experiment runs successfully.

Expand evaluation with long and structured inputs, OCR-like damage, non-Latin/combining-mark scripts, secrets in code/configuration, and repeated mentions across conversations. Check licenses and annotation policy for PII-TRACE and other added corpora before ingesting them. The detector's own training distribution is an in-domain control, not its decisive ranking set.

Modal CPU results do not establish Apple Neural Engine, browser, or mobile performance. Local-first inference remains the default boundary; any cloud second stage stays explicit and opt-in. Model-output logs and session caches must not become an accidental store of private source values.

Recommended next pull request: NEXT-01, followed by the three NEXT-03 reference adapters and a matched experiment once its evaluation policy is declared. Keep model default changes, fine-tuning, and edge-runtime promotion separate so each decision has reviewable evidence.

Documentation verification

This research update's site route, corrected availability note, and pinned manifest passed the Modal site verification: 17 tests passed, no skips, including desktop/mobile browser checks and the complete static-site build. This validates the documentation integration; it is not an inference test of the newly proposed models.