We test our own detection. Here are the results.
Most masking vendors describe their accuracy in adjectives. We publish ours in numbers — including what PDM misses — because deterministic detection makes accuracy measurable and repeatable: the same file produces the same result every time, so the numbers below mean something. When detection improves, we rerun the studies and update this page.
91.88% F1
Representative test — synthetic financial documents
Structured financial documents (auditor reports, tax filings, transaction confirmations). This is expected performance on the file types PDM is built for.
91.26% F1
U.S.-locale test — structured + unstructured documents
A mixed-format benchmark isolated to U.S.-locale conventions. Consistent with the representative test above.
57.65% F1
Adversarial stress test
Deliberately hostile prose and markup, published so the limits are known. This is the worst case, not the typical one.
Full breakdown — all three tests
| Test | Precision | Recall | F1 | What it represents |
|---|---|---|---|---|
| Test 1 — Synthetic Financial Documents (Mendeley) | 90.21% | 93.61% | 91.88% | Representative case: structured financial documents |
| Test 2 — PII-Masking-300k (ai4privacy) | 91.98% | 41.98% | 57.65% | Stress test: adversarial general prose, markup-heavy |
| Test 3 — Nemotron-PII (NVIDIA, US-locale) | 89.80% | 92.77% | 91.26% | U.S.-locale isolation: structured + unstructured documents |
Precision: of everything PDM flagged as sensitive, how much actually was. Recall: of everything actually sensitive, how much PDM caught. F1: a single score combining both, so a weakness in one can't hide behind strength in the other.
Known limitation, stated plainly
PDM's detection is strong on emails, credit cards, names, and addresses, and weaker on non-English names and single first names without surname context.
The Full Detail
Methodology and full results
We tested PDM's live detection pipeline against three independent, public benchmark datasets.
Top-Line Results
| Test | Precision | Recall | F1 | What it represents |
|---|---|---|---|---|
| Test 1 — Synthetic Financial Documents (Mendeley) | 90.21% | 93.61% | 91.88% | Representative case: structured financial documents |
| Test 2 — PII-Masking-300k (ai4privacy) | 91.98% | 41.98% | 57.65% | Stress test: adversarial general prose, markup-heavy |
| Test 3 — Nemotron-PII (NVIDIA, US-locale) | 89.80% | 92.77% | 91.26% | U.S.-locale isolation: structured + unstructured documents |
Precision: of everything PDM flagged as sensitive, how much actually was. Recall: of everything actually sensitive, how much PDM caught. F1: a single score combining both, so a weakness in one can't hide behind strength in the other.
Synthetic Financial Documents
Mendeley Data, DOI 10.17632/tzrjx692jy.1
Built for financial-document context — auditor reports, tax filings, compliance notices, transaction confirmations. The closest match to how PDM is actually used. Sample: 750 rows, randomly seeded from the dataset's full 45,000 (seed 42, reproducible).
Scope discipline: 76.8% of this sample's labels fell within PDM's detection scope.
Precision 90.21% · Recall 93.61% · F1 91.88%
Some apparent "wrong" detections are PDM correctly finding real names the dataset never labeled — e.g. an auditor's name mentioned in a document. We don't count matches against a company name or the unlabeled cardholder name in the Credit Card field's own template, since neither is in scope. A smaller number of similar untagged names remain uncorrected and still count against precision below — so true precision is likely a bit higher than shown.
| Category | Recall | Why |
|---|---|---|
| Credit card number n=689 | 100.0% | Clean, highly structured format — no real ambiguity. |
| SSN n=756 | 100.0% | Standard US formatting throughout. |
| Email address n=531 | 100.0% | Consistent, unambiguous format. |
| Person name n=1,208 | 99.9% | 1 miss out of 1,208 labeled names. |
| Street address n=635 | 84.4% | About half the misses are military APO/DPO/FPO addresses. The rest: a synthetic-data artifact — a real state abbreviation paired with a mismatched zip code (e.g. an NC address, CO zip), which appears to confuse DLP's address validation. |
| Phone number n=689 | 72.7% | Misses include both fictional and real non-US country codes (e.g. +91 India) — the real limitation is non-US phone formats generally, not just synthetic ones. |
PII-Masking-300k
Public English-language benchmark (ai4privacy, Hugging Face) covering a broad PII taxonomy across general prose, HTML fragments, and structured text. Included because it's harder and less representative than Test 1, and because a competing detection model's recent announcement cited this same dataset — we wanted our own independently-run number, not someone else's self-reported one.
Scope discipline: graded only on entity types PDM targets — 47.3% of the dataset's labeled instances. The rest (cities, states, generic dates, etc.) were excluded, not scored against.
Precision 91.98% · Recall 41.98% · F1 57.65%
| Category | Recall | Why |
|---|---|---|
| Email address | 94.0% | Strong — consistent, unambiguous format. |
| IP address | 61.6% | No major issue identified. |
| SSN | 48.6% | US-format-specific by design — most examples here use non-US formats. |
| Street address | 42.0% | Over half the misses are inside HTML/JSON/Markdown fragments — harder, less natural than real business prose. |
| Phone number | 42.4% | Same markup-context pattern as address, plus international format diversity. |
| Date of birth | 40.8% | Markup-context artifact — dates embedded in HTML/JSON fragments (e.g. inside a rendered <strong> tag), not ordinary prose. |
| Passport | 29.0% | Markup-context artifact — passport numbers embedded in JSON-like key/value structures rather than natural sentences. |
| Person name | 28.9% | Two causes: markup-context artifacts (as above), and a real, separately-measured weakness on non-English/non-Western names even in plain prose — see "Key Takeaways" below. |
| Driver's license | 5.6% | US-format-specific detector by design — this dataset's examples used non-US license formats (e.g. "CHAVA.358139.CA.411"). |
Highlighted rows are the ones most likely to be misread in isolation — the explanation is in the same row.
What we found when we investigated the weak spots
- SSN and driver's license detection are US-format-specific, by design — this dataset's examples are largely non-US formats. A deliberate scoping choice, not a defect.
- Over half of name and address misses are inside HTML, JSON, or Markdown fragments — an artificially harder context than PDM's real target use case (ordinary business prose).
- A real, measured weakness remains after accounting for the above: name detection performs meaningfully worse on non-English and non-Western names, even in plain prose. Not documented in the underlying engine's own materials — found by testing, not a spec sheet. An open, active limitation, not a footnote.
Nemotron-PII (U.S. Locale)
NVIDIA, Hugging Face — nvidia/Nemotron-PII — CC BY 4.0
A third, complementary benchmark — not a replacement for Tests 1–2. Isolates U.S.-locale performance using a locale split ("us"/"intl") built into the dataset itself, not defined by us after the fact — across structured and unstructured documents, 50+ industries, 54 PII/PHI categories. Sample: 750 rows, randomly seeded from the full 100,000-row US-locale pool (seed 42, reproducible). One disclosed limitation: NVIDIA's own dataset card notes the "international" split retains roughly 12% U.S.-style overlap — a minor imprecision in the locale boundary, uncorrected here since we only use the "us" split.
Scope discipline: 1,828 of 6,052 labeled spans (30.2%) fell within PDM's target types — this dataset's 54-category taxonomy is far broader than PDM's scope, so the lower percentage reflects breadth, not a stricter bar. 77 of 750 sampled rows (10.3%) were mechanically excluded first, using the same ground-truth offset check as Test 1.
Precision 89.80% · Recall 92.77% · F1 91.26%
Precision uses the same company-name netting as Test 1 — a name-shaped detection on a labeled company name isn't counted against precision, since company names are out of scope. This barely moved the number here (171 → 169 false positives), since most company names in this dataset are invented brands, not surname-based firms like Test 1's.
| Category | Recall | Why |
|---|---|---|
| Email address n=273 | 100.0% | Strong — consistent, unambiguous format. |
| Phone number n=156 | 100.0% | Strong across both structured and unstructured documents in this corpus. |
| SSN n=45 | 97.8% | Strong on standard US formatting. |
| Street address n=124 | 97.6% | Strong on standard US street-address formatting. |
| Credit card number n=71 | 94.4% | A real detection gap was found here and fixed mid-benchmark — see "What we found — and fixed" below. |
| Date of birth n=73 | 91.8% | Solid overall, with a modest dip specifically in structured-document contexts (87.8% vs. 96.9% in unstructured documents). |
| Person name n=778 | 91.6% | Misses are almost all bare single first names without a nearby surname (e.g. "Annelly," "Robert") — different from Test 2's non-English-name finding; this looks like missing surname context, not name origin. |
| IP address n=84 | 56.0% | Weakest category in this test. Misses are a mix of IPv6 and private/RFC1918 addresses (e.g. 192.168.1.1) — the latter arguably aren't meaningfully identifying PII, so part of this gap may be a scope question, not a pure detection miss. |
What we found — and fixed — mid-benchmark
Credit card recall started at 11.3%. First hypothesis — space/dash-grouped formatting was the cause — tested directly against live DLP and disproven: even a plain, unspaced number was missed. Actual cause: DLP wasn't confidently flagging numbers that fail the Luhn checksum, despite these being real, structurally valid numbers that show up in practice — typos, placeholders, test data, partial entries. We built and shipped a supplementary check validating card-issuer prefix and length instead of the checksum, without double-masking anything DLP already found. Recall after the fix: 94.4%. Four residual misses remain in non-standard BIN ranges, not yet investigated.
This dataset's 54-category taxonomy has no passport label, and its closest-sounding label ("certificate_license_number") covers professional/certification licenses — engineering, nursing, CPA — not driver's licenses. Passport and driver's license recall aren't measurable here; Test 1 and Test 2's figures for those categories still stand.
Key Takeaways
No PII masking tool is perfect, regardless of vendor. We tested across three different scenarios specifically to find where PDM is strong and where it isn't. Strong: emails, credit cards, names, addresses, and standard US identifiers — whether they sit in a clean column or embedded in free text. Noticeably weaker: non-English names, and single first names without surname context. We're actively improving detection to close both gaps.
The honest summary
Tested against outside data, not assumed. We'll keep publishing what we find.
Methodology, sample sizes, seeds, and full category breakdowns available on request.