Title: Using the UMLS MetaMap as a Cause of Death Analyzer
1Using the UMLS MetaMap as a Cause of Death
Analyzer
- Michael Hogarth, MDMichael Resendez, MSUniv. of
California, Davis
2Overview
- Causes of Death A Historical Perspective
- Overview of the California EDRS
- Cause of Death Analysis tool (BECA)
- NLM MetaMap and the UMLS
- BECA-MetaMap experiment
- Discussion
2
3Historical Perspectives on causes of death
- Bills of Mortality (1532)
- Arose from the need to better understand death
rates in medieval England -- plague
epidemics(1361,1368,1375,1390,1406, ) - John Graunt (1620-74)
- Used the Bills of Mortality and found an infant
death rate of 36 in England -- not previously
known or understood - London Bills of Mortality classification
- Used by Dr. John Snow to characterize a cholera
outbreak traced to a water source in London - Evolved to become the Intl. Classification of
Disease (1850s) - International Classification of Disease(ICD) --
used for the last 150 years
4CA-EDRS Causes of Death
4
5Causes of Death
- Importance
- key epidemiological information is contained in
the cause of death - Issues and Challenges
- absolutely correct versus close to correct
- absolute correctness requires significant
time/effort and manual effort - is close to correct in an automated fashion
still useful? - Typical process in California
- COD --gt SuperMICAR --gt Stat Master File
- turnaround for entire process can be lengthy (2
years) - could have a trend in causes of death and it
would not be known by local jurisdictions for 2
years. - Today in California
- a significant number of jurisdictions today dont
wait for the final statistical files from the
State office to look at trends --- they
manually code (if they have the staff) --
takes time and funding
5
6Preliminary COD classification
- Possible uses of a preliminary COD classification
using automated methods that are close to
correct - early identification of trends in a local
jurisdiction - disease vs. injury/poisoning -- coroner referral
cross-checking - identify specific infectious causes
(encephalitis, cholera, etc..) - What it is not
- not for absolutely correct cause of death
classification - will not replace the nosologists expertise in
understanding the sequence of events leading to
death nor their understanding of ICD-10, with its
includes/excludes
7How to analyze causes of death?
- Challenges
- text is verbatim and thus arbitrary (free text)
- need to go beyond simple keyword matching
- biomedical knowledge and content is vast -- and
constantly changing! - A possible approach - text mining and
computational linguistic techniques
8BECA
- We built BECA, a generic concept analyzer
framework that can incorporate any concept
identifier engine such as NLM MetaMap and other
text processing tools - BECA BECA Enables Concept Analysis
- Supports a plug-in design for the concept
matcher and other components (ie, spell checker) - Designed to support multiple transformations of
the text in step-by-step fashion - transformations -- strip special characters,
lower case, run it through the concept matcher
engine (MetaMap or other), run it through an
available spell checker (jazzy spell, etc..) - example transformations
- convert to lowercase, remove all punctuation, map
string using concept mapper, etc.. - First version of BECA uses the NLM MetaMap as a
concept mapper
9BECA system design
10Example transformations
10
11What is NLM MetaMap?
- The National Library of Medicines MetaMap
- a free, open source software component built by
the NLM Lister Hill Laboratory - uses computational linguistic techniques to map
biomedical text to a large corpus of biomedical
content (the NLM Unified Medical Language System) - Provides a number of text processing functions
- Includes a concept mapper that attempts to
match phrases with concepts in the UMLS
Metathesaurus - Includes a UMLS concept-to-code mapping for
multiple coding systems (ICD, SNOMED, etc..)
11
12How does MetaMap work?
- Takes text as input and attempts to identify
concepts in the text and match them to concepts
in a large corpus of phrases and concepts in
biomedicine (UMLS Metathesaurus) - The retrieved candidate matches include a score
that reflects how sure it believes the match is
correct - The candidates retrieved include their semantic
type - Disease or Syndrome, Injury or Poisoning,
etc...
12
13The UMLS
- Developed by the National Library of Medicine
- Derived from over 100 sources (ICD, SNOMED,)
- The Unified Medical Language System
- A system built to support information retrieval
in biomedicine - Used in PubMed, ClinicalTrials.gov, etc..
- Consists of
- (1) UMLS Metathesaurus
- (2) UMLS Semantic Network
- (3) UMLS SPECIALIST Lexicon
14UMLS in detail
- UMLS Metathesaurus -- the worlds largest
repository of biomedical phrases - 1.3 million concepts, 6.4 million unique phrases
(concept names) - over 100 source vocabularies (ICD,SNOMED,CPT,
etc..) - UMLS SPECIALIST LEXICON
- a file that provides individual words found in
the UMLS metathesaurus and their linguistic
information including grammatical type (noun,
verb, adjective, adverb, etc..) - UMLS Sematic Network
- a set of files that classify the metathesaurus
concept into a particular type - Examples -- Disease, Injury/Poisoning,
Neoplasm, ..
15MetaMap Algorithm
- MetaMaps algorithm consists of four steps
- (1) Parsing
- using a part-of-speech tagger text is decomposed
into one or more noun phrases - ocular complications of myasthenia gravis gt
ocular complications and myasthenia gravis. - noun phrases are processed independently by
decomposing them into their grammatical origins - ocular complications gt modifier ocular and
head of the phrase complications - (2) Variant Generation -- variants for each
phrase are generated using SPECIALIST - variants -- all synonyms of the term, acronyms
containing the term, abbreviations,
plural/singular variants - each variants has a distance score obtained
from SPECIALIST - ocular - eye, eyes, optic, opthalmic,
opthalmia, oculus, oculi
15
16MetaMap Algorithm
- MetaMap Algorithm continued
- (3) Candidate Retrieval from Metathesaurus
- all metathesaurus strings that have at least one
of the variants is retrieved - can exclude those where the variant is present in
a large number of strings (ie, very common
string) - (4) Candidate evaluation -- the MMTX score
- each metathesaurus candidate is evaluated by
calculating the strength of the similarity
between the original input phrase and the
candidate phrase from metathesaurus - the calculation involves a weighted average of
four metrics including distance scores for
variants from input noun phrase(variation),
whether the phrase is part of the head
(centrality), , coverage and cohesiveness
16
17Example
- BECA MetaMap output
- Input phrase ocular complications
17
18The question
- ?Can BECA using the NLM MetaMap be useful in
- Identifying biomedical concepts in a cause of
death literal, which is narrative text. - auto-coding literals into ICD-10 codes
18
19Cause of Death Literals in CA-EDRS
- CA-EDRS data is a combination of records
initiated in EDRS (EDRS counties) and those
submitted on paper (non EDRS counties) - Causes of death are verbatim from the certifier
and typically entered into EDRS or the typed on a
paper certificate by funeral home staff or
hospital staff - Overall COD statistics for CA-EDRS
- 462,564 registered death certificates
- 985,330 unique literals (phrases) in all COD
fields - 88,719 unique literals (phrases) in the Immediate
Cause of Death field
19
20Experiment
- We randomly selected 1,000 literals from the
88,719 unique literals in the Immediate Cause of
Death field - We submitted these as is to BECA (MetaMap, no
spell checking component) - BECA returned 7.9 candidate matches per literal
(7,791 candidates for 1,000 strings) - Candidate scores ranged from 517 - 1000
- Match score distribution for the 7,791 candidates
20
21Example Output
21
22Literals with high score matches gt800
22
23High Score Candidate Matches
- 3,017 (38.7) of the 7,791 candidates had a score
gt800 - 95.3 of the original literals (953/1000) had at
least one candidate with a match scoregt800 - 54.5 of the original literals (545/1000) had at
least one candidate with a match scoregt900 - 30.7 of the original literals (307/1000) had at
least one candidate with a match score1000 - Note only 7.5 were the exact string as found
the UMLS Metathesaurus - Match score distribution for the 3,017 candidates
24Semantic Type correct matches
- BECA with MetaMap correctly categorized 720 (72)
of the literals by semantic type - Of these, Neoplastic Process had the highest
reliability
24
25Wrong matches
- Semantic types most frequently in error
25
26ICD-10 Coding
- 252 of the 1,000 (25.2) literals had an ICD-10
matched by BECA-MetaMap - Categories
- 1 good match
- 2 approximate match (within ICD category)
- 0 incorrect code
- Results - 97 were good or approximate
- 82.5 good match
- 14.3 approximate match
- 3.2 incorrect match
26
27ICD-10 Autocoding data
27
28Some interesting challenges
- CSTFIOTRDPIRATORY FAILURE
- CHRONIC ALCOHOLISHM
- ESOPHAGELA VARICES
- END STAGE RENAL DOSEASE
- HEAR FAILURE
- OVARION CANCER WITH METASTASES
- LUNF CARCINOMA, METASTATIC
- PENDING TOX MICRO
- SEPTIC SHOCK
28
29Discussion
- MetaMap may be useful for preliminary
categorization of causes of death by semantic
type - Excluding certain semantic types would improve
match precision (at the cost of lower of
matches) - BECA-MetaMap only assigned an ICD-10 code 25.2
of the time - If BECA-MetaMap assigned an ICD-10 code, it was
correct over in 83 of cases, and near correct in
97 of cases - We found that MetaMap was confused if
- there are multiple concepts (noun phrases) in a
single string - the phrase has a compound statement (metastasis
to brain and bone or gunshot wounds of the head
and right arm - the phrases begin with certain words (ie,
complications, etc...)
29
30Future Directions for BECA
- Build a new concept mapper to replace MetaMap,
and specifically design it to analyze causes of
death phrases - include a spell checker
- disambiguation for phrases that have compound
statements - match SNOMED first, then match to ICD-10
(increases the hit rate for ICD-10 autocoding) - improve performance
- implement for ICD-10 includes/excludes using an
open source rules engine (jBoss Rules Engine)
30
31Credits
- National Library of Medicine, Lister Hill Lab
- University of California
- Michael Resendez, MS
- Cecil Lynch, MD, MS
- California Department of Health (California
Department of Public Health) - Terry Trinidad
- David Fisher
- Debbie McDowell
31
32California EDRS
- Developed by the University of California and
California DHS (2004-2005) - Implementation (2005 - 2008)
- all death certificates entered into EDRS since
Jan 1, 2005 - full EDRS (implemented counties)-- DC originates
in EDRS and electronically completed locally - KDE EDRS (non-EDRS counties) -- DC completed in
standard paper fashion, eventually entered by
State office into EDRS - June 2007 - where are we?
- today --gt 510,000 certificates (2005 - present)
- Originate locally (EDRS records) or are entered
later into EDRS (non-EDRS records) - Today, June 2007, 65 originate locally as EDRS
electronic - By Nov 2007 over 90 of all CA records will
originate in EDRS
33Cause of Death Workflow with CA-EDRS
- CA-EDRS does not provide electronic support for
gathering of the COD today
certifier and funeral home exchange
(fax) worksheet
Once COD is finalized by certifier, funeral home
staff create EDRS record and enters them