Test Automation Success: Choosing the Right People

1
Test Automation Success Choosing the Right
People Process
Kiran Pyneni, Automation ManagerAetna, Inc.
2
Agenda

Company Background

Automation Team
Automation Objectives

Our Journey

Strategy Team

Delivery Team

Benefits of Teams Strategy


Automation Lifecycle
Benefits of Process Integration

Metrics Of Our Success
Automation Coverage, Efforts Automated,
Utilization Cost Avoidance

Questions
3
Aetna Background

• Aetna is one of the nation's leading diversified
  health care benefits companies, serving members
  with information and resources to help them make
  better informed decisions about their health
  care.
• Products and management Programs
• Aetna offers a broad range of insurance and
  employee benefits products.
• The first national, full-service health insurer
  to offer a consumer-directed health plan, Aetna
  continues to lead the way with its Aetna
  HealthFund2 line of products, including HSA, HRA
  and RRA options.
• Aetna offers a wide array of programs and
  services that help control rising employee
  benefits costs while striving to improve the
  quality of health care, such as case management
  disease and patient safety programs integrated
  medical, dental, pharmaceutical, behavioral
  health and disability information.
• Aetna provides members with access to convenient
  tools and easy-to-understand information that can
  help them make better-informed decisions about
  their health and financial wellbeing.

3
4
Automation Team
Improves efficiencies and quality at our
organization by automating consistent and
repeatable manual tasks using industry standard
technologies / tools / frameworks.
Standards Best Practices
Automation Team
Assessments Designs
Reports Metrics
Automation Development Maintenance
5
Automation Objectives
Automation Objectives
Automating Right Tasks
Utilization of existing automation
Sustaining and Growing Automation Suite
6
Our Journey
Decentralized Approach
Centralized Approach

• Benefits
• Easy identification of automation opportunities
• Higher collaboration between application
  automation teams
• Challenges
• Automation development has become expensive
• Usage of different tools solutions
• Standards Best practices are not shared across
  applications

• Benefits
• Standardization of automation tool set at
  enterprise level
• Sharing of coding best practices, templates and
  standards
• Cost of automation development is lowered
• Challenges
• Slower automation coverage progression
• Lower automated scripts utilization
• Funding Challenges

7
Our Journey Contd. Challenges Solutions

• Challenges
• Identification of automation opportunities
• Prioritization of automation projects based on
  ROI
• Cost of automation
• Implementation of optimized designs

• Challenges
• Early completion of maintenance for automation
  scripts
• Funding for automation projects
• Collaboration between automation business teams

• Solution
• Structuring automation as strategy delivery
  team

• Solution
• Create automation life cycle process and
  integrate with company's overall software
  development life cycle

8
Automation Team Structure
Automation Team
Automation Strategy Team
Automation Delivery Team
9
Automation Strategy Team

• Definition
• To provide automation strategic direction and
  identify potential new automation opportunities
• Characteristics
• Project management experience
• Various automation frameworks/tools knowledge
• Ability to manage multiple projects in parallel
• Very strong written and verbal communication
  skills
• Very strong leadership and analytical skills
• Activities
• Process Best Practices
• Assessment Strategies
• Design Solutions Techniques
• Estimations Planning
• ROI Cost/Benefit Analysis
• Reports Metrics
• Tool Technical Support Analysis

10
Automation Delivery Team

• Definition
• To develop, deliver, and maintain automation
  solutions
• Characteristics
• Understanding of programming principles and best
  practices
• Knowledge of various automation tools
• Knowledge of multiple programming languages
• Strong communication skills
• Activities
• Scripts Development
• Scripts Maintenance
• Coding Standards
• Scripts Planning Execution
• Day to Day interactions w/End User

11
Benefits of Teams Strategy
Benefits

• Easy identification of automation opportunities
• Prioritization of automation projects based on
  ROI
• Improved automation design
• Optimized the cost of automation
• Clearly defined roles responsibilities
• Better communication process
• Optimized automation project planning

12
Automation Lifecycle
13
Team Collaboration
Automation Strategy Team
Automation Delivery Team
Automation Evaluation
Automation Preparation
Automation Generation
Execution Acceptance
14
Automation ROI Calculator
15
Automation ROI Calculator
16
Integration of Automation Lifecycle Software
Development Lifecycle
Automation Evaluation
Feasibility/Analysis
Automation Preparation
Design
Automation Generation
Implementation
Execution Acceptance
Testing Maintenance
17
Benefits of Process Integration
Benefits

• Improved understanding on automation process
  across the entire organization
• Better collaboration between business and
  automation teams
• Funding challenges are removed
• Improved on-time delivery of automation scripts
• Helped to promote the automation across entire
  organization
• Formal escalation process developed
• Initialized the creation of automation reports
  metrics

18
Automation Coverage Regression Testing
35
Year-2008
42
Q1-2009
45
Q2-2009
47
Q3-2009
52
Q4-2009
19
Manual Testing Efforts Automated
13,803 hrs
11,750 hrs
10,200 hrs
8,570 hrs
7,047 hrs
Year-2008
Q1-2009
Q2-2009
Q3-2009
Q4-2009
Year As of 2008 As of 2009 2009 only efforts
Total Efforts Automated 7047 hrs 13803 hrs 6756 hrs
20
Utilization Cost Avoidance
Utilization
150
134
130
87
45
Q1-2009
Q2-2009
Q3-2009
Q4-2009
Year-2008
Cost Avoidance
10,810 hrs
9,867 hrs
9,618 hrs
7,200 hrs
Q1-2009
Q2-2009
Q3-2009
Q4-2009
21
Savings Vs. Cost Avoidance

• Cost Savings
• Efforts difference between manual and automation
  testing
• Manual (Not automation) execution frequency
• Produced by automation strategy team after
  assessment
• Automation ROI Calculator
• Cost Avoidance
• Test coverage difference between manual
  automation testing
• Efforts difference between manual and automation
  testing
• Automation execution frequency
• Generated from Quality Center after release
  implementation
• Helped to improve the credibility of automation
  metrics significantly

22
Automation Metrics Best Practices

• Automation is a way of Testing and process
  followed for manual testing can be applied
• Best Practices followed in Test Management Tool
  Quality Center
• Define automation work flow
• Regression Testing identifier
• Automation Assessment status
• Automation Development status
• Store automation inventory
• Manage automated test scripts execution

23
Questions