edgepiityped character spans
taxonomy 2026.11 GitHub

First principles

A first principle here is a fact the team cannot remove without losing the job. Models, vendors, and card scores sit above these facts. When a design fights a fact, the design loses.

The chain that produces the facts is the five whys. The contract that encodes them is the span contract.

Facts

IdFactConsequence
F1The editable object is the source stringPublic offsets are character indexes on that object
F2A document flag does not name charactersThe library has no document-label API
F3The next action depends on the typeThe public type comes from one taxonomy
F4A miss can leave the trust boundaryHigh-severity recall is a gate. Extra spans are a budget
F5Formats and genres moveA frozen regex set cannot be the only detector
F6A remote detect call is a copyV1 loads weights on the host. See ADR 0003
F7Card F1 is someone else's testThe golden set is the only ranker. See evaluation
F8Token labels are not charactersAdapters convert BIO to offsets inside one module
F9Two people of one type are differentPlaceholders use a counter per type per call
F10A password is not a stand-inThe locator finds secrets. redact then destroys them. See ADR 0004

F1 to F3 come from Chain B. F4 comes from the parent note on leak cost. A miss is a leak. An extra span is a utility cost. F6 follows because a remote detector would copy the source string before the mask.

Axioms for the build

These axioms are the engineering form of the facts. They are short so a review can check a pull request against them.

  1. Spans, not a document class.
  2. Same source string. No silent normalize.
  3. Taxonomy version on every result.
  4. One span type in code. Detectors do not invent a second result shape.
  5. Code scores spans. A language model does not judge exact-span F1.
  6. Per-type metrics. No single PII bucket in the score report.
  7. Rules and models both emit spans. The merger is the only place that resolves overlap.
  8. The locator drops invalid offsets. It does not guess a repair.

Kill list

Kill for v1Fact it breaks
Regex-only hero detectorF5
Chat model as the locatorF1, F7
Whole-file delete as the privacy featureF2's useful remainder. The file still has work
Copied leaderboard F1 as a go or no-goF7
Document classifier as the productF2
Cloud PII API as a default detectorF6
Free-text types from the modelF3
Fuzzy match to repair a bad offsetF1. The repair can bind the wrong copy
Logging span.text at INFOThe log becomes a second copy

Decisions that follow

DecisionChoiceRecord
Public unitHalf-open character spanADR 0001
Default modelsOne closed encoder, one open-schema modelADR 0002
Where weights runOn the host in v1ADR 0003
SecretsLocate, then irreversible maskADR 0004
When to leave the encoderOnly after the golden set shows a miss the encoder cannot closeResearch

Trust boundary

Detection runs inside the trust boundary. A later product step can send text out only after it masks or drops the spans. edgepii does not open that socket.

diagram
flowchart LR
  subgraph Host["Trusted host"]
    IN["Source string"]
    LOC["edgepii locator"]
    SP["Span list"]
    RED["redact, optional"]
  end
  subgraph Out["Outside the host"]
    LLM["Any remote model"]
  end
  IN --> LOC --> SP --> RED
  RED -->|"masked string only"| LLM

The map from placeholder to original stays on the host. The library returns the map to the caller and does not store it. If restore fails, the caller shows the masked string. The library does not guess a person.

Counsel uses these texts to decide which types exist. The library does not embed a legal conclusion.

InputRoleLink
NIST SP 800-122US guide on PII confidentialityCSRC
GDPR Article 4EU definition of personal dataEUR-Lex 2016/679
EDPB Guidelines 01/2025Pseudonymisation is not anonymisation when the map existsPDF
PDPOHong Kong ordinance. Counsel reads it. This spec does not interpret itPCPD

A placeholder plus a host-side map is pseudonymisation for the party that holds the map. It is still personal data for that party. The remote model does not receive the original if the mask is complete. Completeness is an eval result, not a slogan.

Domain shift

A pattern that is true on synthetic email text can fail on a chat, an OCR line, or another language. F5 says the library must allow a new detector without a new result type. F7 says the team measures the new detector on a slice of the golden set before the team changes the default preset. The AI engineering note states how that measurement works.