Test Planning

1
EHR RoadmapWebEx
Stratis Health, the Minnesota Quality
Improvement Organization in partnership with
other QIOs, presents . .

• Test Planning

2
Presenter

• Margret Amatayakul
• RHIA, CHPS, CPHIT, CPEHR, FHIMSS
• President, Margret\A Consulting, LLC,
  Schaumburg, IL
• Consultant to Stratis Health DOQ-IT Project
• Independent information management and
  systems consultant,
  focusing on EHRs and
  their value proposition
• Adjunct faculty College of St. Scholastica,
  Duluth, MN, masters program in health informatics
• Founder and former executive director of
  Computer-based Patient Record Institute,
  associate executive director AHIMA, associate
  professor Univ. of Ill., information services
  IEEI
• Active participant in standards development,
  HIMSS BOD, and co-founder of and faculty for
  Health IT Certification

3
EHR Roadmap
Implement
4
Objectives

• Appreciate the importance of testing, and test
  planning
• Describe the types of EHR system testing
• Plan for testing the EHR
• Ensure documentation of testing

5
Test Planning

• Importance

6
Business Reasons

• Trigger points for a payment
• Payments should be based on achievement of
  milestones, not on dates
• To ensure that milestones have been achieved,
  some form of testing must have occurred
• Testing will vary by stage in implementation
• Realize benefits
• EHRs impact clinicians as no other information
  system has in the past, so ensuring that
  clinicians can use the system is a critical
• EHRs with clinical decision support are close to
  being medical devices, even though they are not
  substitute for professional judgment. As a
  result, they need thorough testing before use

7
Clinical Reasons

• EHRs have a direct impact on the well-being of
  patients
• EHRs are designed to reduce/eliminate the need
  for paper records
• Although care has been rendered in the past with
  very little if any previous information about a
  patient, in the absence of an emergency medical
  situation, the expectation is that records will
  be retained and available for direct care
• Without a paper back up process, the EHR system
  becomes mission critical and must work as
  intended
• External requirements for EHR testing could
  increase over time, as product certification,
  pay-for-performance, and other incentives are
  being implemented

8
Legal Reasons

• Testing serves to satisfy the terms and
  conditions of the contract
• Any flaws, problems, or issues need to be
  identified prior to being put into productive use
  both to satisfy the contract and to avoid risk
  to patients
• Testing demonstrates what is actually occurring
  the event any flaws are contested

9
Test Planning

• Types of Tests

10
Types of Tests

• Unit and function testing
• Tests data elements and functions
• System test (a.k.a. end user/simulation test)
• Tests application capability and workflow
• Integrated test (a.k.a. interface test)
• Tests exchange of data between systems
• Performance and stress test
• Technical capability to handle volume
• There may also be tests for certifying
• Hardware configuration
• Super user training
• Adoption rate for acceptance testing
• Satisfaction of users/patients

11
Unit and Function Test

• All major functions
• All design changes
• Every screen
• Every data table
• Every data element
• Clinical alerts and reminders
• For example,
• Users are engaged to check that every drop down
  menu is complete, data elements are spelled
  correctly, acceptable values are accurate, etc.

A
B
?
C
12
System Test

• Checks
• Data availability
• Data flow
• Data integrity
• Within each application
• Frequently uses scripts to test common and
  unusual scenarios. Each application owner
  performs this type of test, literally trying to
  break the system

13
Integrated System Test

• Data exchange
• Data flows
• User interactions
• Policies
• Procedures
• Processes
• Output accuracy
• Views
• Logical rules
• Reports
• Multi-day testing
• Impact of other processes

FEEDBACK
14
Performance and Stress Test

• Performance Does system meet demands of live
  use?
• Measures response time for key transactions
• Literally uses a stop watch
• Should simulate varying conditions, including
  remote users, different kinds of input devices
• Contract should specify acceptable limits
• Stress Can the system meet the
  demands of peak use times?
• Simulate many more transactions
  than normal

15
Test Planning

• Planning

16
Testing Environment

• Timing
• Sequencing
• Test systems vs. production systems
• De-identification or not
• Simulators
• Participants
• Accountability
• Documentation
• Scripts
• Results

17
Timing and Sequencing

• Timing and sequencing are critical to ensure that
  testing reflects a complete module or application
  and does not need to be redone because only part
  of the module was built

Unit and Function Test
System Test
Integrated System Test
Performance and Stress Test
18
Test Environment

• A separate system should be devoted to building
  and testing.
• Day-to-day operations should not be jeopardized
  because a module or component of the new system
  is being tested
• A test environment should be maintained for
  testing all upgrades and changes
• In some cases, the ability to run technical
  simulations may be needed
• If using live data that have been moved to the
  test environment, many organizations prefer to
  de-identify it by changing names, etc.
• Some organizations, however, believe that
  introduces risk into the system that may mean
  they cant test that data coming from one system
  truly belongs to the given patients case being
  tested.
• The Office of Civil Rights (OCR) which is
  responsible for overseeing compliance of the
  HIPAA privacy rule has indicated in frequently
  asked questions that de-identification is NOT
  required for testing.

19
Accountability

• Participants in testing should be drawn from both
  the project team and users.
• Any person performing testing must understand the
  critical nature of the process and importance of
  accuracy
• Testers must be willing to say something does not
  work, because if does not work in the test
  environment, it will not work in real life.
• Many organizations ask testers to assume
  accountability through signing a form
  acknowledging that they have conducted a thorough
  test and certify that the system has met the
  specific criteria they tested against, and noting
  any concerns or failures
• While this may seem like a stringent process, it
  emphasizes the serious nature of the testing and
  that it cannot be performed carelessly or in a
  punitive manner

20
Acceptance Testing

• Sometimes, the term used to refer to the final
  test that triggers
• Final payment to vendor
• Preferably,
• Agreed upon in contract specifications for
  acceptance
• Usually some period after go-live
• Reflects issue resolution

21
Test Planning

• Documentation

22
Test Scripts

• Use scripts developed to
• Describe performance requirements in RFP
• Evaluate product in due diligence
• Describe process improvement
• Describe
• Most typical processes and work flows
• Complex administrative logic
• Complex clinical logic
• Known pain points with current processes
  anticipated pain points with new system learning
  curve

23
Evidence of Testing

• Written records of testing
• Describes how well system performed during tests
• Identifies any detected errors
• Describes how issues were resolved and retested
• These records must be recoverable in case they
  are needed as documentary evidence of your
  efforts
• In a payment dispute
• In a performance dispute

24
Reference

• Internal uses of documentation help deal with
  issues arising after testing, such as
• Problems identified, but not satisfactorily fixed
• Problems that occurred later that someone could
  claim should have been discovered but were not
• Problems that appear to have arisen due to
  subsequent changes or updates, but were not
  problems at the time the system was accepted
• Builds or protects your case for fixing such
  problems

25
Stress of Testing

• It should be obvious that where system build was
  highly detailed, testing must be as detailed or
  more so as a result, testing of mission critical
  systems is stressful
• Reduce stress through
• Spreading out the workload so that all
  participate
• Ensuring appropriate conditions in which to
  conduct tests
• Real scripts
• Adequate time to run test, retest if necessary,
  and document
• Appropriate location to concentrate on process
• Rewarding testers for finding problems
• Demonstrate tester credentials to others
• Celebrate successes

26
Stratis Health is a non-profit independent
quality improvement organization that
collaborates with providers and consumers to
improve health care.
This presentation was created by Stratis Health
under a contract with the Centers for Medicare
Medicaid Services (CMS). The contents do not
necessarily reflect CMS policy.