Research landscape

The 5 November 2026 update supersedes this snapshot for model selection and research versions. See its model manifest and the implementation plan before selecting a preset.

As of 4 November 2026. This file says which research lane fits the span contract. It does not rank models by a number we measured. This file marks paper figures and card figures as reported. They stay UNCONFIRMED as ship gates. See AI engineering and evaluation.

The model rows and links live in the catalog. This file is the map.

The task in the literature

The public name is span NER, or token classification when the labels are BIO or BILOU. CoNLL-2003 fixed exact-entity match as the score (W03-0419). SeqScore later set rules for fair sequence scores (2021.eval4nlp-1.5).

CoNLL types are person, organization, location, and miscellaneous. Those types are not a PII taxonomy. A strong CoNLL model can still miss an email, a card, or a national id. The locate note records that limit. dslim/bert-base-NER is a CoNLL model, not a candidate default (card).

Lanes

flowchart TD
  T["Source string"] --> R["Lane R. Rules and checksums"]
  T --> E["Lane E. Closed encoder"]
  T --> G["Lane G. Open schema"]
  T --> Z["Lane Z. Chinese schema, optional"]
  R --> M["Merger"]
  E --> M
  G --> M
  Z --> M
  M --> S["Span list on the source string"]

Lane R is necessary and not sufficient. Names, messy addresses, and context-heavy ids need Lane E or Lane G.

Lane E. Encoders

BERT showed that a bidirectional encoder can feed a token-classification head (arXiv:1810.04805). The head emits BIO or BILOU. The adapter turns those tags into character spans. That path meets the contract when labeled spans exist.

Later encoders keep the same head and change the backbone.

Clinical de-identification is a neighbor task, not the default taxonomy. Stanford published a transformer de-identifier (JAMIA, doi:10.1093/jamia/ocac219, card). Use it only when the text is clinical and the data-use terms allow it. n2c2 text stays under its data-use agreement. Do not copy it into this repo.

When the encoder is enough

Use a closed encoder when all of the lines below are true.

1. The team freezes the type list. Callers do not add a type at runtime.
2. The language and the genre match the train mix. For the ModernBERT candidate, that mix is English synthetic PII.
3. Labeled spans exist, or a public checkpoint already covers the types.
4. CPU cost and p95 latency matter more than a new type next week.
5. The context fits the window, or the window splitter in the architecture covers the tail.

Leave the encoder when any line below is true.

BERT is a solution for the contract. BERT is not a claim of the best paper F1 on every set. The locate note says the same thing, and this spec keeps that caution. Soft-kill any "BERT" wrapper that has no offset path on the card.

The 2026 closed default is ModernBERT, not 2018 BERT-base. ModernBERT is still Lane E. It is the current form of "a BERT-family token classifier is enough". Preset en-bert keeps BERT-base as the control so the team can see what the extra backbone buys on the golden set.

Lane G. Open schema

GLiNER matches label text to span vectors and returns start and end (arXiv:2311.08526, NAACL 2024). The type list is an input. That property is why Lane G exists. A closed head cannot add TAX_ID on one call without a new label set and a new train run.

GLiNER2-PII adapts that idea to a PII inventory and a multilingual synthetic train set (arXiv:2605.09973, 2026-05). The public weights are fastino/gliner2-privacy-filter-PII-multi. The card says 205M parameters. The paper abstract says about 0.3B. This spec does not resolve that gap. Read the card you pin.

Other span checkpoints in this lane: nvidia/gliner-PII, urchade/gliner_multi_pii-v1, gretelai/gretel-gliner-bi-large-v1.0, knowledgator/gliner-pii-edge-v1.0.

LOGICAL studies local obfuscation with a GLiNER PII stack (arXiv:2510.19346). It is a system paper, not a default checkpoint.

Reported scores, and why they are not the gate

SPY is a synthetic PII set with legal and medical subsets (NAACL 2025 SRW). Authors score exact span match. Two write-ups of the same models do not match.

Paper table in arXiv:2605.09973, average exact-span F1: fastino 0.471, nvidia 0.391, urchade 0.384, OpenAI Privacy Filter 0.373, knowledgator base 0.368.

The fastino card table reports a different average: fastino 0.477, nvidia 0.400, urchade 0.398, OpenAI Privacy Filter 0.380 (card).

A separate study of OpenAI Privacy Filter reports other SPY figures (arXiv:2608.02616). That paper also reports a training result. A fine-tuned encoder can pass a zero-shot filter after about 500 labeled rows, on that paper's protocol. The result supports Lane E when labels exist. Those figures are not our gate.

antoniogr7/pii-identifier-v0.1 reports F1 0.9628 and F2 0.9699 on the Nemotron-PII test inside the PIIMB harness (card). That set is the train distribution's test split. It is not SPY. A high in-domain F1 does not imply a high SPY F1.

Do not average these tables. Do not pick a winner from them. Run the golden set.

Lane Z. ModelScope and Chinese

Hugging Face is the default hub. ModelScope is the hub when the host is in a network that reaches ModelScope cleanly, or when the text is Chinese.

The ModelScope pages returned HTTP 500 from this writing environment on 4 November 2026. Search indexes still list the model ids. Re-check the card before you pin a hash.

Siamese-UIE is a general extractor. It is not a PII-tuned checkpoint. The adapter must prove source[start:end] equals the surface before a span is public. A Chinese generic NER such as RA-NER returns {type, start, end, span} (modelscope issue 274) but the types are news types, not the taxonomy. Do not use it as a PII default.

Lane R. Rules

Use rules for forms that a published grammar or checksum defines.

Every rule reports 1.0 only when a validator confirms the slice. A rule with no validator reports null. The library never invents a middle number, because a fixed value would order spans by nothing.

Microsoft Presidio already wires patterns and a transformers engine (engine doc, GitHub). Recognizers-Text covers several structured forms (GitHub). V1 does not depend on Presidio. A later bridge can convert our spans into Presidio results. One merger stays inside edgepii.

Data the golden set can cite

Synthetic rows need a template id. The sealed freeze must also hold a small human slice of the real channel. The locate note forbids a train-test bleed.

Escalation ladder

Use this order. Skip a step only when the trigger in "when the encoder is enough" is already true on the dev freeze.

1. Lane R for structured types.
2. BERT-base control (en-bert) so the team has a cheap baseline.
3. ModernBERT closed preset (en-closed) for English and a frozen type list.
4. DeBERTa plus CRF only when the residual errors are illegal BIO paths the linear head emits. Kiji is the candidate, after a license re-read.
5. Lane G (multi-open, then phi-open if health types are in scope).
6. Lane Z for a Chinese slice.
7. A generative model only if it emits offsets and the scorer checks them. Not in v1.

Systems that are adjacent

These systems mask and restore. They are consumers of spans, or rivals to the product around the library. They are not the locator contract.

Cloud detect APIs (Comprehend, Azure Language PII, Sensitive Data Protection) return spans and copy the text to the vendor. ADR 0003 keeps them out of v1.