Title: Extracting Personal Names from Email: Applying Named Entity Recognition to Informal Text
1Extracting Personal Names from Email Applying
Named Entity Recognition to Informal Text
- Einat Minkov Richard C. Wang
- Language Technologies Institute
William W. Cohen Center for Automated Learning
and Discovery
School of Computer Science Carnegie Mellon
University
2What is an informal text?
- A text that is
- Written for a narrow audience
- Group/task-specific abbreviations often used
- Not self-contained (context shared by a related
group of people) - Not carefully prepared
- Contains grammatical and spelling errors
- Does not follow capitalization conventions
- Some examples are
- Instant messages
- Newsgroup postings
- Email messages
3Objective / Outline
- Investigate named entity recognition (NER) for
informal text - Conduct experiments on recognizing personal names
in email - Examine indicative features in email and newswire
- Suggest specialized features for email
- Evaluate performance of a state-of-the-art
extractor (CRF) - Analyze repetition of names in email and newswire
- Suggest and evaluate a recall-enhancing method
that is effective for email
4Corpora
- Mgmt corpora Emails from a management course at
CMU in which students form teams to run simulated
companies - Teams Each set (train/tune/test) formed by
different simulation teams - Game Each set formed by different days during
the simulation period - Enron corpora Emails from Enron Corporation
- Meetings Each set formed by randomly selected
meeting-related emails - Random Each set formed by repeatedly sampling a
user then sampling an email from that user, both
at random
Note The number of words and names refer to the
whole annotated corpora
5Extraction Method
- Train Conditional Random Fields (CRF) to label
and extract personal names - A machine-learning based probabilistic approach
to labeling sequences of examples - Learning reduces NER to the task of tagging, or
classifying, each word using a set of five tags - Unique A one-token entity
- Begin The first token of a multi-token entity
- End The last token of a multi-token entity
- Inside Any other token of a multi-token entity
- Outside A token that is not part of an entity
- Example
Einat and Richard Wang met William W.
Cohen today
Unique Outside Begin End
Outside Begin Inside End
Outside
6Top Learned Features
- Features most indicative of a token being part
of a name in a Conditional Random Fields (CRF)
extractor
Newswire (MUC-6)
Email (Mgmt-Game)
2
Results show that Email and newswire text have
very different characteristics
Note A feature is denoted by its direction
(left/right) comparing to the focus word, offset,
and lexical value
7Our Proposed Features
Note All features are instantiated for the focus
word t, and 3 tokens to the left and right of t
8Feature Evaluation
- Entity-level F1 of learned extractor (CRF) using
- Basic features (B)
- Basic and Email features (BE)
- Basic and Dictionary features (BD)
- All features (BDE)
Results show that 1) Dictionary and Email
features are useful (best when combined) 2)
Generally high precision but low recall
9Whats Next?
- Previous experiments show high precision but low
recall - Next goal Improve recall
- One recall-enhancing method
- Look for multiple occurrences of names in a
corpus - We conduct experimental studies
- Examine repetition patterns of names in email and
newswire text - Examine occurrences of names within a single
document and across multiple documents
10Doc. Frequency of Names
- Percentage of person-name tokens that appear in
at most K distinct documents as a function of K
Results show that Repetition of names across
multiple documents is more common in email corpora
Only 1.3 of names in MUC-6 appear in 10
documents
Percentage
About 20 of names in Mgmt-Game appear in 10
documents
Nearly 80 of names in MUC-6 appear only in one
document
30 of names in Mgmt-Game appear only in one
document
1
Document Frequency
11Single vs. Multiple Documents
- We define the following extractors
- CRF baseline trained with all features
- SDR (Single Document Repetition)
- Rules that extract person-name tokens that
appear more than once within a single document
hence an upper bound on recall using only names
repetition within a single document - MDR (Multiple Document Repetition)
- Rules that extract person-name tokens that
appear in more than one document hence an upper
bound on recall using only names repetition
across multiple documents - SDRCRF
- Union of extractions by SDR and CRF hence an
upper bound on recall using CRF and names
repetition within a single document - MDRCRF
- Union of extractions by MDR and CRF hence an
upper bound on recall using CRF and names
repetition across multiple documents
12Single vs. Multiple Documents
- Token-level upper bounds on recall and potential
recall-gains associated with methods that look
for name tokens that re-occur within a single
document or across multiple documents
Results show that Higher recall and potential
recall-gains can be obtained for email corpora
using MDR method
13Whats Next?
- Our studies show the potential of exploiting
repetition of names over multiple documents for
improving recall in email corpora - We suggest a recall-enhancing method
- Auto-construct a dictionary of predicted names
and their variants from test set - Statistically filter out noisy names from the
dictionary - Match names globally from the inferred dictionary
onto test set, exploiting repetition of names
Note A dictionary is simply a list of one or
more tokens
14Name Dictionary Construction
- Every name in the test set predicted by the
learned extractor (CRF), trained with all
features, is transformed into a set of name
variants and inserted into a dictionary
Transformation Example Name variants of Benjamin
Brown Smith
Original name is included by default
15Name Dictionary Filtering
- Previously constructed dictionary contains noisy
names - i.e. brown can also refer to a color
- Next goal Filter out noisy names
- We suggest a filtering scheme to remove every
single-token name w from the dictionary when
PF.IDF(w) lt T
Predicted Frequency Inverse Document Frequency
Words that get low PF.IDF scores are either
highly ambiguous names or very common words in
corpus
cpf(w) of times w is predicted as a name-token
in corpus ctf(w) of occurrences of w in
corpus df(w) document frequency of w in
corpus N of documents in corpus
T 0.16 optimizes entity-level F1 in tune sets
thus, we apply the same threshold onto our test
sets
Note Corpus mentioned here refers to the test
set in our experiments
16Name Matching
Filtered Dictionary
- A window slides through every token in the test
set - A match occurs when tokens in a window starts
with the longest possible name variant in the
dictionary - All matched names are marked for evaluation
benjamin brown smith benjamin-brown
smith benjamin brown-smith benjamin-brown-smith be
njamin brown s. benjamin-b. smith benjamin b.
smith benjamin brown-s. benjamin-brown
s. benjamin-brown-s benjamin-b.
s. benjamin-smith benjamin smith b. brown
smith benjamin b. s. b. brown-smith benjamin-s. be
njamin s. b. brown s. b. b. smith b.
brown-s. benjamin b. smith b. b. s. smith b. s.
Names Matching Example E-Mail
17Experimental Results
- Entity-level relative improvements (and final
scores) after applying our recall-enhancing
method on test sets - Baseline learned extractor (CRF) trained with
all features
Results show that 1) Recall improved
significantly with small sacrifice in
precision 2) F1 scores improved in all cases
18Conclusion
- Email and newswire text have different
characteristics - We suggested a set of specialized features for
names extraction on email exploiting structural
regularities in email - Exploiting names repetition over multiple
documents is important for improving recall in
email corpora - We presented the PF.IDF recall-enhancing method
that improves recall significantly with small
sacrifice in precision
19Thank You!
20References