Evaluation

The current implementation plan adds untyped complete containment, interval-union character recall, masking precision, zero-gold masking rate, repeated-identifier coverage, pre-policy candidate coverage, and unmapped prediction counts beside exact typed metrics. A zero-gold row is not automatically audited PII-free text. The SPY protocol audit qualifies historical scores; new materializations are separately versioned.

The scorer exists before the adapters. See phase 1 of the implementation plan and principle 1 in AI engineering.

Primary metric

Exact-span F1. A prediction is a true positive only when type, start, and end equal a gold span. The rule is CoNLL exact entity match, on character spans (W03-0419). SeqScore's cautions on double-counting still apply (2021.eval4nlp-1.5). One gold span can credit one prediction.

Report the metric per type and as a micro average. Do not report a single bucket named PII. Macro F1 is a diagnostic. It is not the ship metric, because rare types would dominate it.

Diagnostics

Partial overlap is not a true positive for the ship metric. It is a shift or a wrong type.

F2 (beta = 2) is a second column for high-severity types. Presidio's own guidance favors recall when the cost is a leak (FAQ). F2 does not replace exact-span F1. It explains a threshold move.

Leakage

Leakage is a product metric on the masking ranges. A gold span leaks when the union of predicted intervals fails to cover every character in its original range, regardless of predicted type. A partial mask still leaks. The scorer uses original offsets, avoiding both shifted masked-string indices and global text searches that confuse repeated mentions. The union matches the ranges that redact removes; separate contract tests exercise actual mask/restore. High-severity classes are secret, gov_id, and payment.

Reports keep exact typed scores alongside containment: union character recall, masking precision, complete-span coverage, and all-mentions coverage for repeated identical values within a document. pre_policy_candidates measures mapped requested-type candidates before overlap and score filtering. It has an explicit unavailable denominator for adapters that do not expose candidates. A decline between candidate and final coverage points to policy loss; it does not establish which discarded category was correct.

zero_gold_document_mask_rate measures masking on documents with no in-scope gold. It becomes a PII-free false-positive measure only on audited negative fixtures; incomplete source annotations cannot establish that condition. zero_gold_masked_character_fraction reports the fraction of those source characters masked; masked_character_fraction gives the whole-slice fraction. Reports never store the repeated values or source strings.

A high-severity leakage rate above zero on the freeze means that the team must not ship. This spec keeps that gate for the masked string on the sealed freeze. It does not claim a model has no misses. If the model misses and the rule lane also misses, the preset does not ship. The mitigation is a better detector or a refusal to send, not a softer score.

The downstream task owns its utility floors. This spec leaves those floors open.

Gate the emitted output

Starting in 0.5.0, --gate-output protection applies --max-leaks, --max-leaking-documents, --min-containment, and --min-masking-precision to protection_plan. --gate-output selected retains the previous default. --min-recall always gates exact typed recall; containment has its own floor. Protection gates reject missing or partial output and require zero unmapped predictions unless --max-unmapped explicitly changes that budget. Invalid offsets remain limited by --max-dropped, which defaults to zero.

Reports include high_severity_documents and high_severity_leaking_documents; the latter is also recorded for protection. The document leakage rate uses documents with in-scope high-severity gold as its applicable denominator. Native-label experiments have no public protection projection and cannot pass that output's readiness gate. All gates apply to every requested slice, and failed reports are saved before the CLI exits 1. See actual stage audits and gate behavior.

Sets

The public corpus replaced the plan to write a freeze from scratch. nemotron-pii is the in-domain slice and gretel-finance-en is the out-of-domain slice, and both carry a commit revision, a byte count, and a sha256 in corpora.yaml. A run that does not match the pin stops. Two people can score the same preset and get the same number.

A public corpus is not the sealed freeze. It is public, so a checkpoint can train on it. The sealed freeze still needs a human-read slice of the real channel.

Split synthetic rows by template id. Split real rows by document id. A template in both train and test is an eval failure, even if F1 rises. The locate note and the builder note both require this split.

The dev freeze must include at least these slices:

Start the dev freeze from public synthetic spans (Nemotron-PII, OpenPII, SPY) plus rows you write. Tag every row with source_corpus and template_id. Do not put n2c2 or other DUA text in the repo.

A human must read the dev freeze before the first sealed run. Model drafts are not gold. Do not edit gold to match a model.

Floors

Until phase 9 writes numbers, every floor stays open. Phase 9 sets them from the dev freeze in a commit that does not include the sealed scores. Then the sealed run uses those numbers unchanged.

The kinds of floors, with no fake targets:

Do not copy 0.96 from a card into the recall floor. That number is in-domain for someone else's split.

Report

Each run writes a JSON report: preset, revision, taxonomy version, per-type counts, leakage, and the slice name. The report omits source strings. A local HTML or text dump for error reading can include text. That dump stays out of git when the text is not from the fake examples.

The harness is python -m edgepii.eval.cli. It takes a preset, a corpus key, an optional limit, and an optional type subset. The report lands in docs/eval/. Public corpora documents every field.

Compare presets on the same gold file. Two different gold files do not make a valid compare. The research note shows why: SPY tables already disagree with each other. A Nemotron score and a Gretel score are two different numbers about two different corpora. Neither one wins.

A run must score one document at a time. A pooled list of every gold span would let a prediction in one document match gold in another, and the report would claim a hit for text that document does not hold. tests/test_eval_corpora.py locks the case.

Contamination checks

1. Hash each template id in the sealed freeze.
2. If that id appears in a train list you used to fine-tune, fail the run.
3. If a builder opened the sealed file in the same change as a threshold edit, fail the run.
4. Record the git revision of the scorer next to the score.

V1 does not fine-tune. Check 2 is still required. A public checkpoint can train on OpenPII or Nemotron-PII.

When the preset card lists that corpus, mark the slice in_domain. Do not use an in_domain slice as the only ship gate. Keep one slice the card does not list. SPY is the public candidate for that slice. Our own ticket-shaped fake rows are the other.

What is not an eval