A local detector stack found more PHI in clinical notes
A local language model layered with rule-based and neural detectors for clinical text de-identification: an open-source toolkit and evaluation · arXiv:2611.12709
In plain words
Clinical notes can contain details that identify patients. DeidSharp is an open-source toolkit that runs rules, a clinical text model, and a language model locally, then removes any text span that one of them flags.
In tested ICU nursing notes, its recall-focused setup found a larger share of reference PHI spans than the Stanford/Penn tool. It also marked more non-PHI text, so the result is a tradeoff on one note set, not a guarantee for other records.
The research
For clinical de-identification, the choice is not just whether to catch identifiers: hosted language models may be barred, and broader detection can remove more non-PHI text.
These results compare local detector combinations with an open clinical model and show how recall and precision shifted on two particular note sets.
DeidSharp combines local language-model, rule-based, and clinical-model detectors to find PHI, and its tested stacks reveal different recall and precision tradeoffs.
On reporting halves of ICU nursing notes and semi-synthetic discharge summaries, the study scored fixed DeidSharp stacks against five open tools. The scorer found that recall-first detected a larger share of reference PHI spans than Stanford/Penn in both sets, while precision-first offered a different tradeoff on nursing notes.
The discharge-summary lead was concentrated in generated untyped names; all compared systems found more than 99% of typed-label spans. That pattern supports a dataset-specific explanation for the larger gap.
Both clinical note sets came from one hospital and used surrogate PHI; the discharge summaries were semi-synthetic. The study measured detection only: it did not test the known-identifier detector or the leak guard.
What the evidence shows
- A local layered toolkit
DeidSharp is an open-source toolkit that combines rule-based detectors, a clinical text model, and a locally run language model to find identifying spans in clinical text. The problem is that tools can miss PHI across note collections, while institutional rules may prevent using hosted language models.
Its local, layered design addresses both constraints; the evaluation asks how its detection compares with open tools.
- Recall came with a cost
The comparison used fixed detector stacks and five open tools on reporting halves of ICU nursing notes and semi-synthetic discharge summaries. Span recall means the share of reference PHI spans overlapped by a prediction.
On the nursing notes, the recall-first stack reached 98.88% recall against 96.06% for Stanford/Penn, but its precision was lower. This makes the gain a tradeoff, not an across-the-board improvement.
- A different operating point
Removing the clinical text model changed the tradeoff on this nursing-note reporting half. Precision, the share of predicted spans that overlapped reference PHI, was higher for the precision-first stack than for Stanford/Penn, while their recall difference was small enough that the paired interval included no difference. This comparison describes these notes, not other collections.
- Names drove the larger gap
The discharge-summary recall lead was concentrated in generated untyped names: Stanford/Penn found these spans less often than either DeidSharp setup, while all three systems found nearly all typed-label spans. This pattern means the larger overall gap reflects the generated names in this semi-synthetic corpus, rather than a comparable advantage across label types. The subgroup labels were assigned using placeholder rules and contextual review of a sample.
- Self-check lost spans
A development test also checked whether the language model should review spans it had found itself. On the nursing-note tuning half, this second opinion reduced recall by 24.6 percentage points.
That result cautions against treating this check as a general precision adjustment: it was a specific procedure tested on tuning data, not a final comparison across note sets.
Deep
- Original paper authors
- Valentin Burkin
- Submitted to arXiv
- License
- CC BY 4.0