Public corpora

This file names the public PII corpora that the scorer reads, the license of each, and the fold from each external gold label into taxonomy 2026.11. The data lives in corpora.yaml. That file is the only list.

A score is replayable only when the bytes are pinned. Each corpus row carries a repository id, a commit revision, a file name, a byte count, and a sha256. A download that does not match the pin stops the run. It does not warn and continue, because a moved file makes an old score unreadable.

The two corpora

Both are fully synthetic. Neither holds real personal data. Both licenses allow redistribution and commercial use, so either one can sit in a public gate.

Why these two

The evaluation spec asks for one slice that the checkpoint card names and one slice it does not. A card score on a set that the checkpoint trained on is in domain. It is not evidence for a new domain.

• nemotron-pii is the in-domain slice. The antoniogr7/pii-identifier-v0.1 card and its preprocessing.json both name nvidia/Nemotron-PII as the source, so this corpus measures the ceiling of that checkpoint rather than its transfer.
• gretel-finance-en is the out-of-domain slice. A finance document with SWIFT fields and a chat transcript looks nothing like a synthetic persona note. A preset that passes nemotron-pii and fails here has learned a template, not a type.

SPY is the third name in the research note. The public release is not reachable without an access grant from this writing environment, so it is not a pin. The gaps section records that.

What the map keeps and drops

Each corpus uses its own label names. The map folds a label into one public type, or it drops the label with a written reason. A label that the file does not name at all is a third case: the report prints it under unknown_labels. A silent fold would inflate recall, so the run refuses to guess.

The drops fall into five groups. The reason is in the YAML next to the label, so the decision travels with the data.

Two of these deserve a note.

cvv and pin are dropped although they look secret. A SECRET span needs a cue word and a value. A bare four digit code has no cue, so scoring it would measure a detector that the library does not claim. The rule audit forbids a rule with no source.

city alone is a known over-mask risk that the taxonomy already records. Nemotron labels a city inside an address, so a keep would score a span the library never returns.

Nested gold spans

A corpus row can nest two labels. Gretel puts a person name inside a company name, and a street address inside a person name. The locator returns one non-overlapping list, so gold must be one list too. The harness keeps the longer range and counts the row under rows_with_overlapping_gold.

Nemotron holds one such row in 100 000. Gretel holds 1 866 in 2 891. The Gretel number is a property of that generator, and it is why a Gretel score is not comparable to a Nemotron score by type.

Run a score

pip install -e ".[eval]"                                  # pyarrow
python -m edgepii.eval.cli --preset rules --corpus nemotron-pii --limit 500 --print
python -m edgepii.eval.cli --preset rules --corpus gretel-finance-en --print

--limit cuts a prefix per slice, not per corpus. A corpus-wide prefix would read only the us block of Nemotron-PII, because the locales arrive in two blocks of 50 000, and it would report an empty intl slice with a perfect precision and recall of 1.000. A prefix per slice keeps every slice populated.

The report

The report is JSON. It holds counts, the two pins, the taxonomy version, and the environment. It holds no source text. A span score is a number; the text that produced it stays out of the file.

corpus_diagnostics is the audit of the map. It reports the count of in-scope spans, out-of-scope spans, dropped labels, unknown labels, bad offsets, and rows with nested gold. A change to corpora.yaml shows up here before it shows up in a ratio.

The score is per document

The scorer runs locate, then compares the spans of that one document with the gold of that one document, then adds the counts. It never pools every gold span into one list.

A pooled list would score two documents as one string. A span in document B could match a gold span in document A when the two texts share a prefix, and the report would claim a hit for a document that holds no such text. tests/test_eval_corpora.py locks that case with a fixed detector.

The metrics are the ones in the evaluation spec: exact span, per type and micro, with miss, wrong_type, shift, and extra as exclusive counts that add up to the false negatives.

Gaps