Getting Your HMIS Data to the Next Level: Preparing for Program Reporting, Evaluation and Monitoring

1
Getting Your HMIS Data to the Next Level
Preparing for Program Reporting, Evaluation and
Monitoring

• Matt White,
• Abt Associates Inc.

2
Learning Objectives

• To give HMIS project managers and program staff
  practical tips on improving data quality
• To learn how to monitor data quality through
  utilization rates, custom reports and regular
  user meetings.
• To learn of management techniques to prepare HMIS
  data for evaluation and monitoring questions.

3
Overview

• Utilization Rates the canary in the coal mine
• Data Quality Checks Big brother is watching
• Preparing Data for Program Analysis do you have
  a dog in this fight?

4
What are Bed Utilization Rates?

• The bed utilization rate is the percentage of
  beds that are used on a particular day or the
  percentage of beds that are used on an average
  day during a particular time period.
• Utilization is calculated as follows
• of clients served on an average day
• of beds available
• Bed Utilization Rate

5
Bed Utilization Rates An Example

• For example, if a community serves 180
  individuals in emergency shelters on an average
  day during a particular month and has 200 beds
  available for emergency shelters serving
  individuals, the bed utilization rate is
• 180 200 90
• Bed utilization rates can be calculated for a
  single provider, an entire community, or any
  other level of aggregation.
• For non-residential providers, a similar
  utilization rate can be calculated in terms of
  service slots.

6
Inaccurate Bed Utilization Rates Means
Inaccurate Information for Making Policy Decisions

• Inaccurate bed utilization rates means your
  program does not know
• The number of homeless people using shelters
• The number of days people are using shelters
• The characteristics of sheltered homeless people
  served on a particular day or period of time
• System wide what types of providers need more
  capacity and what types have extra capacity.

7
Reasons for Excessively High Utilization Rates
(over 100)

• Missing exit dates
• Missing exit dates for people who leave the
  shelter leads to an over-count of homeless people
  served on a particular day or time period.
• Inaccurate, low bed counts
• Inadequate deduplication (affects over time
  counts and average daily utilization based on
  over time counts)
• Accounting for people who use overflow beds, but
  not the beds

8
Reasons for Excessively Low Utilization Rate
(below 50)

• Not entering information on everyone served in
  HMIS leading to undercount of the number of
  people served.
• Inaccurate, high bed counts
• For family programs, not all beds in occupied
  unit may be used

9
Suggestions for Improving Accuracy of Counts of
Number of People Served

• Regularly review utilization rates by program and
  ask for confirmation of number of people served
  for unusual rates (see sample worksheet)
• Compare utilization rates across similar
  providers to identify possibly erroneous rates.
• Use the counts and utilization rates for funding,
  policy decisions, and evaluations. When people
  see the numbers are important to decisions, they
  have and incentive to ensure they are accurate.
• Compare the number of people served from HMIS to
  point-in-time count reported for CoC Application
  (if from a different source)

10
More Suggestions for Improving Accuracy of Counts
of Number of People Served

• Match people listed in HMIS to bed logs if
  separate logs are kept.
• Automatically enter exit dates for overnight
  shelters.
• Identify specific records for confirmation if
  length of stay seems excessive.
• If long-term problem, exit everyone and have
  provider re-enter only those persons still being
  served.
• Design system so persons cannot be listed as
  staying at two residential providers at the same
  time.
• Increase training of front-line staff who enter
  the information.

11
Suggestions for Improving Accuracy of Number of
Beds Available

• Regularly review utilization rates by program and
  ask for confirmation of bed inventory for unusual
  rates (see sample worksheet)
• Calculate bed utilization rate with overflow beds
  if appropriate
• Make sure CoC Housing Inventory information is
  internally consistent.
• Some apps have family units listed, but not beds
• Others have of beds participating in HMIS, but
  nothing in column for total of beds available
• Some list providers that do not serve homeless
  people
• Some have wrong geocode for location of program
• Aim for 100 accuracy of information needed for
  deduplication

12
Worksheet to Monitor Bed Utilization Rates Monthly

• Include seasonal beds in the denominator if
  they are open during the month.

13
Utilization Wrap Up

• Inaccurate bed utilization rates point to
  problems in the count of the number of sheltered
  homeless people or to inaccurate counts of beds
  available to serve homeless people
• Accurate utilization rates are important they
  can help you determine whether you have too few
  or too many resources for serving certain types
  of residential providers
• They also point to problems in all the
  information you have on sheltered homeless people
• You can address the issue the main emphasis has
  to be on entering information on everyone who is
  served and exiting people no longer being served

14
Data Quality Checks
Timeliness Completeness Accuracy Consistency
15
Ensuring Quality HMIS Data

• Monitor program activity
• Clients served, services provided, needs, etc.
• Monitor client-level completion rates
• Clients without service transactions, DOBs,
  disability status,program entry dates or other
  Universal Data Elements
• Compare data from different reports
• APR vs. Clients Served Report

16
Identifying Data Inconsistencies

• Run reports that are clear and concise
• Be familiar with data being entered and reported
• Entries should be done in real time
• Use data quality reports to track user entries
• Disability incongruities (yes, but no type)
• Chronic homeless (flag doesnt match LOS)
• Zip codes (no quality indicator)

17
Potential User Entry Problems

• Overload on users from data entry backlogs
• Users who are not computer literate
• Users who do not understand HMIS at the
  beginning/training level
• Users who rush through data entry
• Lack of communication between users, System
  Administrators and HMIS providers
• Users who dont have a dog in the fight

18
Solving User Entry Challenges

• Be selective in staff who will be entering data
  at the beginning
• Integrate program functions with HMIS data entry
  functions
• Hire additional data entry support (or use
  existing personnel)
• Use specific staff for data entry to reduce
  errors
• Use/hire additional staff during peak times

19
Solving User Entry Challenges

• Constant communication done at training level
  will ensure a comfort level with HMIS
• Re-train individuals until comfortable with HMIS
• Run reports on individual users to ensure quality
  data entry
• Run random reports that match up with paper
  trails
• Follow up with client entries with users that are
  new to system

20
Preparing Data for Program Analysis

• Fitness for Use Tests

21
Fitness for Use

• Completeness
• Information is entered on all consumers
• Information on the consumer is complete
• Accuracy
• Data reflects reality
• Data is entered correctly
• Data has face validity reflects what we know
• Consistency
• Performance information is consistent over time

22
Fitness for Use

• Program staff reviews reports monthly for
  completeness, accuracy and consistency.
• Clear protocols for correcting data.
• Agency signs off on reports monthly
• Errors systematically result in corrective action
• Procedures for correcting are defined
• Software has error checking functions (out of
  range, missing values, incongruous data).
• Staff look at data reliability and validity
  issues prior to data analysis and publishing
  reports.

23
The Next Step

• Preparing for Reporting, Evaluation, and
  Monitoring

24
Process

• Define research/monitoring question
• Identify necessary data variables required for
  analysis
• Determine exceptions, limits, exclusions
• Define query and run report
• Review/test results

25
Reporting, Evaluation, Monitoring

• Prepare data for the following analysis
• Is the male Veteran LOS longer than the average
  LOS for my program?
• What of families have a positive exit outcome
  within 90 days?

26
Is the male Veteran LOS longer than the average
LOS for my program?

• Universal Data Elements
• Program entry/exit
• Gender
• Race/Ethnicity
• Date of birth to calculate age
• Disability status
• Program Level Data Elements
• Employment/Income/Non-Cash Benefits
• Education
• Health/Physical and/or Developmental
  Disability/HIV/AIDS

• Veteran status
• Residence prior to
• program entry
• ZIP of last permanent
• address

• Mental Health/
• Substance Abuse/
• Domestic Violence
• Services Received
• Reason for Leaving and
• Destination

27
Is the male Veteran LOS longer than the average
LOS for my program?

• Universal Data Elements
• Program entry/exit
• Gender
• Race/Ethnicity
• Date of birth to calculate age
• Disability status
• Program Level Data Elements
• Employment/Income/Non-Cash Benefits
• Education
• Health/Physical and/or Developmental
  Disability/HIV/AIDS

• Veteran status
• Residence prior to
• program entry
• ZIP of last permanent
• address

• Mental Health/
• Substance Abuse/
• Domestic Violence
• Services Received
• Reason for Leaving and
• Destination

28
Is the male Veteran LOS longer than the average
LOS for my program?
29
Is the male Veteran LOS longer than the average
LOS for my program?
30
Is the male Veteran LOS longer than the average
LOS for my program?

• Male Veteran LOS 16.5 (round up to 17)
• Ave LOS 25
• Answer No, the male Veteran LOS is 8 days less
  than the average of all other program
  participants.

31
What of families have a positive exit outcome
within 90 days?
32
What of families have a positive exit outcome
within 90 days?
33
What of families have a positive exit outcome
within 90 days?

• Answer 22 (2 of 9) families had a positive
  outcome within 90 days.