Role Prediction Using Electronic Medical Record System Audits

1
Role Prediction Using Electronic Medical Record
System Audits

• Wen Zhang1, Carl Gunter3, David Liebovitz4, Jian
  Tian1 , Bradley Malin1,2
• 1Dept. of Electrical Engineering Computer
  Science, Vanderbilt University
• 2Dept. of Biomedical Informatics, Vanderbilt
  University
• 3Dept. of Computer Science, University of
  Illinois at Urbana Champaign
• 4Dept. of Medicine, Northwestern University

1
2
Misuse of EMR Systems is Real

• Medical center employees misuse medical record
  systems to breach privacy

When Where Who
2007 Palisades Medical Center George Clooney
2011 UCLA Various Celebrities

• The problem is not limited to celebrity snooping

• HIPAA Security Rule ? Access to EMRs should be
  limited

• But how?

2
3
Challenges to Security in EMRs

• Basic security principle
• Least privilege
• Separation of duty
• Access control technologies have been around
  since the 1970s
• Information systems often provide role-based
  access control (RBAC) capability1
• Privileges mapped roles
• Users mapped to privileges
• Roles are hard to define, so EMR systems often
  provide broad access rights

3
1 R.Sandhu, E.Coyne, H.Feinstein and C.Youman.
IEEE computer. 1996.
4
In Rare Cases Break the Glass

• A user may not sufficient access rights to
  perform job
• This model allows users to temporarily escalate
  privilege
• Access is logged and reviewed by administrator
• May require user to specify reason for access

5
Rare Cases?

• Central Norway Health Region enabled break the
  glass
• 53,000 of 99,000 patients (54.5) ? broken glass
• 5,000 of 12,000 users (42.7) ? broke the glass
• Over 295,000 logged breakage events in one month

Role Users Invoked Glass Breaks in Past Month
Nurse 5633 36
Doctor 2927 52
Health Secretary 1876 52
Physiotherapist 382 56
Psychologist 194 58
3 L. Røstad and N. Øystein. Proceedings of the
2nd International Conference on Availability,
Reliability and Security (ARES)
6
Idea! Refine Access ControlBased on Behavior

• Experience-based Access Management (EBAM)
• Combine static knowledge (RBAC)
• with actual actions (access logs) and
  organizational knowledge for feedback control

EMR Access Logs
Experience-Based Access Management 2
RBAC
Medical Center Knowledge
2 C.Gunter, D.Liebovitz, B.Malin. IEEE Security
and Privacy Magazine. 2011.
6
7
The Role Prediction Problem for EBAM

• Use audit logs to predict if a user is associated
  with a role
• Goals
• Determine if expert-defined job titles are
  reasonable
• Provide administrators with a better idea of how
  to refine roles

Doctor
Role Classifier
Nurse
.
Access Reason
Medical Service
Biller
Location of Patient
7
8
Evaluation with Cerner EMR of Northwestern
Memorial Hospital

• Example audit logs

User Patient Time Service User Position (Role) Reason Location
u1 p1 8/4/10 OBSTETRICS NMH Physician Office - CPOE Attending Phys/Prov Ward A
u2 p2 12/14/10 OBSTETRICS NMH Physician - CPOE Patient Care Ward A
u23 p3 12/14/10 PEDIATRICS Unit Secretary 2 Unit Secretary Orders Ward B

• Represent users as ltService, Reason, Locationgt
  vectors
• Statistics

Users Roles Reasons Services Locations
8095 140 143 43 58
8
9
Leveraging Role Hierarchies

• To assist in role management, we worked with
  organization experts to build a hierarchy
  (specialized to Northwestern)
• Optimization Tradeoff
• Goal 1 Accuracy (should increase as we step up
  in hierarchy)
• Goal 2 Separation of Duty (will increase as we
  step down)

Employee

Specific Clinician
Doctor
Conceptual (5 roles)


Dietitian
Physician
Nurse
General (62 roles)



Specific (140 roles)
Junior Dietitian
Senior Dietitian
9
10
Basis of a Role-Up Algorithm

• General idea Audit roles at different levels of
  the hierarchy
• Score each role in conceptual position general
  position
• Select role with the highest score generalize
  its children
• Repeat 1 2 until a threshold score is reached

• Allow administrators to balance between the
  prediction accuracy and separation of duties
  (number of roles)

10
11
Balanced Scoring Function

• R measures the extent to which specificity could
  be kept by the node
• A measures the extent to which predictablity
  could be achieved by the node

11
12
Employee
0.453
0.0441
Doctor
Specific Clinician
0.410
0.476
0.224
Dietary
Physician
Nurse
Junior Dietician
Senior Dietician
Nurse 1
Nurse 2
Physician 2
Physician 1
a 0.5, Threshold 0.4
12
13
Employee
0.0441
0.453
Doctor
Specific Clinician
0.224
0.410
Dietary
Physician
Nurse
Junior Dietician
Senior Dietician
Nurse 1
Nurse 2
a 0.5, Threshold 0.4
13
14
After one iteration, the role set is Doctor,
Nurse 1, Nurse 2, Dietary
Employee
Doctor
Specific Clinician
Dietary
Nurse
Nurse 1
Nurse 2
a 0.5, Threshold 0.4
14
15
Training Testing at the Same Level of the Role
Hierarchy
Level
Accuracy
Employee
82.38
Specific Clinician
Conceptual
52.45
General
Nurse
Specific
51.34
Nurse 1
15
16
Distribution of Accuracy Over the Role Hierarchy
16
17
Most Predictable Roles
Rank Role Accuracy Users
1 (tie) AP-Technologist 100 54
1 (tie) ED Assistant 100 26
1 (tie) ED NMH Physician-CPOE 100 43
1 (tie) NMH Resident/Fellow ID Clinic-CPOE 100 10
1 (tie) Patient Care Staff Nurse Lactation 100 14
17
18
Least Predictable Roles
Rank Role Accuracy Users
140 Patient Care Staff Nurse 7.6 1554
139 Rehab OT 14.3 28
138 Transfer 20.0 20
137 View Only PC 3 21.4 14
136 Patient Care Staff Nurse (Pilot) 22.1 217
18
19
Number of Users in the Role Can Influence Accuracy
19
20
Case Study Most Likely Mispredictions for
Patient Care Staff Nurse
Predicted Role Prediction
Patient Care Staff Nurse - Lactation 19.6
View Only PC 1 14.3
Radiology Nurse 14.0
Patient Care Staff Nurse (Pilot) 10.4
SN-RN/Customer Service 5.8
20
21
Most Likely Mispredictions
Original Role Predicted Role Probability
Rehab OT Rehab PT 85.7
Patient Care Staff Nurse - Agency Patient Care Staff Nurse - Lactation 75.0
Rehab PT Rehab OT 60.0
View Only PC 3 Patient Care Staff Nurse - Lactation 50.0
Medical Records - Scanner Medical Records 47.4
21
22
Parameter Bias Trades Between Accuracy and
Separation of Duty

• Biased toward Accuracy
• number of roles is small (27)
• accuracy is highest (63)

• Biased toward Specificity
• number of roles is high (60)
• accuracy is lower (52)

? 0.1 0.8 0.9
Number of Roles Recommended 27 60 64
Accuracy of Role Predictions 63.3 51.8 51.3
22
23
Conclusion and Future Plans

• EHR audit logs can be analyzed to determine if
  the users behaviors are consistent with their
  designated job titles
• Role hierarchies enable automatic discovery of
  appropriate levels of role management
• Plan to expand Role-up to allow for Role-down
  and Role-over
• Need to evaluate Role-up with real hospital
  administrators, to assess its usability and
  acceptance of results

23
24
Acknowledgements

• National Science Foundation
• CCF-024422
• CNS-0964063
• National Library of Medicine
• R01-LM010207
• Office of the National Coordinator for HIT
• SHARPS (sharps.org)

24
25

• Questions?
• wen.zhang.1_at_vanderbilt.edu

25