Bob Kaiser Michael Baker Jr., Inc.

1
Act 44 Performance Standards Regulation
Development April 24, 2009

• Bob Kaiser - Michael Baker Jr., Inc.
• Walt Cherwony - Gannett Fleming, Inc.
• Anna Lynn Smith, Parsons Brinckerhoff

In association with
Draft - Discussion Purposes Only
2
Discussion Outline

• Overview of Development Process
• Progress to Date
• Research
• Data and Definitions
• Peer Selection Process
• Performance Measures Selection
• Overall Process
• Continuous Improvement
• Estimated Timeline of Events (preliminary)
• Next Steps
• Questions / Discussion

3
Overview

• Research Development Tasks
• Background
• Legal Req., Research Materials, Gap I.D.
• Peer Selection Process
• Performance Measure Development Process
• Data and Definitions
• Performance Review Process
• Potential Impacts
• Consultation

4
Overview (continued)

• Schedule

July
May
Jan
March
1 Background
2 Research
3 Peer Selection
4 Performance Measures
PA Bulletin
5 Data, Definitions
6 Devel. Overall Process
7 Impacts
8 Consultation
5
Overview (continued)
Draft Overview Pennsylvania Public Transportation
Agency Performance Review Process
Review Cycle Initiated
System Notification and Kick-Off
Field Work/Data Acquisition Analysis
DOT Planning and Preparation.
DOT-Award Recipient Consultation.
Peer selection, Data Acquisition Analysis,
Draft Initial Performance Standards
Draft Performance Report
Review Report with Transit Agency/Finalize
Follow-up, Corrective Action, and Monitoring
DOT-Award Recipient Consultation. Review
Results, Discuss Action Plan, Identify Action
Plan Elements. DOT-Grant Recipient Meeting.
Document Exemplary Performance. Track Corrective
Action Plan Implementation. Report Results.
Draft Report of Findings v. Prior-Established
Performance Standards. Identify Poor Exemplary
Performance.
3-18-09
6
PPTA Committee Recommendations
7
Overview (continued)

• Systems Encompassed
• Urban and Rural Public Transit Systems
• Modes Encompassed
• Fixed Route Public Transportation Modes
• Evaluation by Mode
• Not being considered at this time
• ADA paratransit
• Shared ride
• Other, non-public transit

8
Overview (continued)

• Trial Applications
• Peer Selection Process
• Performance Review Metrics Application
• Peer and Trend Analyses
• Potential Impacts
• Process, Schedule, Funding
• Systems/Modes
• SEPTA (MB, HR, CR, LR)
• PAAC (MB, LR)
• AMTRAN (MB)
• Endless Mountains (MB)

9
Progress To-Date

• Materials Identification and Review
• Legislation Act 44, Act 3
• PPTA Performance Review Committee Notes
• Prior Audits Processes (Act 3, TFRC)
• STAC White Paper
• Other States Activities
• Databases (NTD, dotGrant, Census)
• TCRP Synthesis 56

10
Progress To-Date (continued)

• Data and Definitions
• NTD and dotGrant Data, Definitions
• Applied By Mode
• Urban NTD Rural dotGrant
• Definitions Act 44 as data permits
• Act 44 Special Considerations
• Circumstances Beyond Award Recipients Control
  
• Events beyond grantees control which
  negatively affect performance, i.e., strikes,
  infrastructure failures, natural disasters.
• Added Consideration of
• New, restructured, revised services
• Discontinued services
• Man-made disasters
• Other (i.e., Special Events)

11
Progress To-Date (continued)

• Perf. Review Major Analytical Components
• Peer Review
• Trend Analysis (intra-agency)
• Functional Analysis (as warranted)
• Annual Risk Assessment by DOT
• Regular Cycle
• 3-6 years average 5 years
• More frequent per performance data
• Data Timeliness
• 1-2 year NTD data lag may be filled with local or
  dotGrant data, where feasible.

12
Progress To-Date (continued)

• Graduated Approach
• Basic Review
• In compliance per base metrics and no best
  practices to document. Close out.
• Step 2 Review
• In compliance minor issues or Best Practice from
  Basic Review. Possible Group 2, 3 metrics.
  Resolve via agency-developed improvement plan
  or document Best Practice.
• Step 3 Review
• Potential or demonstrated non-compliance. Group
  2, 3 metrics Functional review. May result in
  Action Plan to rectify non-compliance, potential
  funding implications per Act or Improvement
  Plan.

13
Progress To-Date (continued)

• Peer Identification
• Modal
• Number of Peers
• 10-12 Initial Selection
• 8-10 Final Selection (desired)
• 5 Minimum
• System Types
• Urban compared to Urban (NTD)
• Rural compared to Rural (dotGrant)
• Small urban if nec. to secure minimum number of
  peers.

14
Progress To-Date (continued)

• Peer Review (continued)
• Identification Process
• Process to identify and select peers based on
  attributes of system under evaluation.
• Includes consultation with subject system.
• Approaches Researched Trialed
• Modal Characteristics Primary, Sort by Descending
  Criteria
• Service Area Characteristics Primary, Sort by
  Descending Criteria
• Modal Characteristics Match

15
Progress To-Date (continued)

• Peer Review (continued)
• Key Criteria from NTD / dotGrant
• Revenue Vehicle Hours
• Revenue Vehicle Miles
• Peak Vehicles
• Service Area Population
• Other Considerations
• Modal
• Fixed Guideway Stations, Route Miles
• Bus System Design Type, Service Type
• Professional Expertise
• Major Generators
• Special Circumstances / Considerations
• Climate Car miles/hours for FG

16
Progress To-Date (continued)

• Peer Initial Identification Method Results
• 1 Modal Characteristics Primary, Sort by
  Descending Criteria
• 2 Service Area Characteristics Primary, Sort by
  Descending Criteria
• Both Inadequate
• Insufficient peers for many modes / systems
• Peers disparate from Act 3, TFRC studies
• Spread inadequate for smaller systems
• Over-reliance on professional judgment
• Add or delete prospective peers

17
Progress To-Date (continued)

• Peer Identification Results
• Modal Characteristics Match Selected
• Developed when 1 and 2 proved inadequate.
• Adequate peers for all modes
• Good match-up to Act 3, TFRC peers.
• Good spread for smaller systems
• Many prospective peers
• Minimizes need for professional judgment to
  arrive at initial set of candidates
• Final peers to be selected via DOT/Agency
  consultation

18
Progress To-Date (continued)
19
Summary Initial Peer Identification by Method

• Peers initially identified by selection
  alternative.
• Final peers to be determined after consultation
  with transit agency.

20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
Performance Measures

• Metrics for the Performance Review

29
Information Sources

• Act 44 Section 1513
• Act 3
• STAC Study
• PPTA / PennDOT Cmte.
• Other States Practices
• TFRC Transit System Audits
• TRB / TCRP

30
Cycle

• Regular
• 3 6 yr intervals (5 year avg.)
• As Needed
• If warranted based on data
• Annual Risk Assessment by PennDOT
• Annual dotGrant data
• Possibly supplement with NTD submittal
• Performed by PennDOT from agency submittals

31
Measures (continued)

• 3 Groups of Measures
• Group 1 Act 44 Metrics
• Performance Standard associated w/ these
• Group 2 Best Practices Metrics
• No performance standard associated w/ these
• Supplement and Explain Group 1 Results
• Others as appropriate to situation
• Calculated by PennDOT during performance review
• Group 3 Customer Service Metrics
• PennDOT encourages regular use by agency

32
Measures

• Group 1 Act 44 Measures
• Performance Standards to be Implemented

33
Measures (continued)

• Group 2 Supporting Meas. No Perf. Std.

34

Group 3 Customer Service Satisfaction Meas.
35
Types of Performance Review Cycles
Regular Cycle
As needed Cycle

• Not more than every three years
• Not less than every six years

• As DOT may determine appropriate

Years 1, 6
Years 2, 3, 4, 5
36
(No Transcript)
37
Performance Review Standards Process

• Progress Thoughts to Date

38
Establishing Performance Standard

• Peer Analysis to Establish Standard
• Metric
• Peer Average, Median, Mode, Other?
• Accounting for current performance (good,
  otherwise)
• Allow for Future Events, Progress
• Project at current review, or at next review?
• Next review in 3-6 years
• Trend Analysis
• Direction Improving, Declining, Flat
• 5 years in analysis Most recent 2 years

39
Sample Results Peer Review Analysis
Mode MOTOR BUS
Operating Cost / Revenue Vehicle Hour
Passengers / Revenue Vehicle Hour
Operating Revenue / Revenue Vehicle Hour
Operating Cost / Passenger
40
Sample Results Trend Analysis
Mode MOTOR BUS
Operating Cost / Revenue Vehicle Hour
Passengers / Revenue Vehicle Hour
Operating Revenue / Revenue Vehicle Hour
Operating Cost / Passenger
41
Establishing Performance Standard (cont.)

• Measuring Achievement of the Standard
• Approaches Under Study
• Standard Deviation Approach
• X of Peer Mean
• X of Peer Median
• Quartile or Quintile
• Other Suggestions?
• Incorporate Trend Results ?
• Step(s)
• One
• gt One

42
Sample Peer Mean, Median, Standard Deviation,
Quintile
MOTOR BUS PEER ANAYLSIS
Passengers / Revenue Vehicle Hour
Operating Cost / Revenue Vehicle Hour
Illustrative Only
Operating Revenue / Revenue Vehicle Hour
Operating Cost / Passenger
Peer Average
Peer Median
1 Std Dev.
5th Quintile
Direction to Pass
43
Passengers per Revenue Vehicle Hour
Illustrative Only
Peer Average
5th Quintile
Direction to Pass
Peer Median
1 Std Dev.
44
DRAFT Performance Review Outcomes
Performance Improving
Continuous improvement and potential
transferability to other agencies.
May warrant further research and possible
Improvement Plan.
Standard Achieved or Exceeded
Standard Achieved or Exceeded
Standard Not Met
Standard Not Met
Action Plan required by Act 44.
Action Plan required by Act 44.
Performance Deteriorating
04-20-09
45
Sample Results Trend Analysis
Mode MOTOR BUS
Operating Cost / Revenue Vehicle Hour
Passengers / Revenue Vehicle Hour
4.54
.57
Illustrative Only
Operating Revenue / Revenue Vehicle Hour
Operating Cost / Passenger
.12
.07
.57
5 year trend / Calculated slope
46
Progress To-Date (continued)

• Continuous Improvement Approach to Establish
  Performance Standards
• Peer Rev. Trend Analysis Functional Rev.
• Establish Minimum Perf. Std. by Mode
• Possible Performance Targets by Mode
• PennDOT Agency Discussions
• Technical Assistance Measure Progress
• Achieve Standard
• Achieve Goal

Goal
Standard
Current Performance
47
Overview (continued)
Conceptual Process Pennsylvania Transit
Performance Review
Report to Governor General Assembly
Off Cycle Review
Results
PennDOT Annual Review Risk Assessment
Fail
On Cycle Review
Pass
PennDOT Technical Assistance
PennDOT Technical Assistance
Waiver
Improvement Plan
Approved
Pass
Not Approved
Compliance Measurement
Action Plan
Financial Penalty
Fail
Draft 3/20/09
48
(No Transcript)
49
Estimated Timeline of Major Events - A
50
Estimated Timeline of Major Events B
51
Next Steps

• Finalize Perf. Review Process
• Method to Develop Performance Standards
• Peer Analyses
• Trend Analyses
• Functional Reviews
• Overall Process Map and Description
• Impacts of Inadequate Performance Process
• Process
• Impacts
• Test vs. 4 systems / 8 modes

52

• Questions and Discussion