University of Florida Survey Research Center - PowerPoint PPT Presentation

Loading...

PPT – University of Florida Survey Research Center PowerPoint presentation | free to view - id: 715bee-YjE1N



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

University of Florida Survey Research Center

Description:

University of Florida Survey Research Center – PowerPoint PPT presentation

Number of Views:78
Avg rating:3.0/5.0
Slides: 78
Provided by: mark1229
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: University of Florida Survey Research Center


1
University of Florida Survey Research Center
2
Bureau of Economic and Business Research (BEBR)
  • Established in 1929
  • Population Program (State estimates and
    Projections)
  • Publications (Florida Statistical Abstract)
  • Economic Analysis (Ad hoc studies, such as
    Florida Price of Living Index)
  • University of Florida Survey Research Center
    (Established 1979)

3
Some of Our Clients
4
(No Transcript)
5
(No Transcript)
6
Selected Health Related Surveys
Survey Population Sample Type Completed Interviews Number of Attempts Total Sample Released Total Number of Calls Response Rate Refusal Rate of Refusals Converted
BRFSS Cell Phone Survey FL Cell Phone users RDD Cell Phone 300 12 7,200 24,569 0.32 0.22 3.00
College of PHHP--CAHPS Eval. Medicaid for Reform Implementation Plan Children enrolled in Medicaid from Broward, Duval, Baker, Clay and Nassau counties Listed 3091 21 14,695 68,006 0.38 0.07 14.43
FL DCF --Eval. substance abuse programs Adults who received substance abuse treatment from state Listed 422 13 3,357 13,188 0.62 0.08 6.45
FL AHCA--Report Card Children enrolled in a Medicaid HMO Listed 3714 20 18,397 72,553 0.72 0.06 16.85
FL DOH--Eval. anti-tobacco program FL children (11-18 yrs old) Listed 2205 15 20,000 94,521 0.37 0.07 2.52
FL ICHP--FL KidCare eval. Children enrolled in FL KidCare Listed 1714 31 7,859 70,600 0.57 0.15 9.64
FL ICHP--Eval. health care in Partners in Care Children enrolled in PIC(terminally ill children) Listed 140 33 352 2,904 0.76 0.12 23.75
TX STARPLUS--Eval. elderly Medicaid enrollees with disabilities Adults enrolled in STARPLUS Listed 955 34 9,128 47,618 0.26 0.14 13.80
FL DCF --Eval. substance abuse programs Adults who received substance abuse treatment from state Listed 422 11 3,953 14,507 0.66 0.07 7.69
FL AHCA--Report Card Children enrolled in a Medicaid HMO Listed 4004 20 22,209 90,889 0.66 0.12 15.08
TX STAR--Medicaid HMO health care eval. Adults in a Medicaid HMO under TX STAR Listed 2255 50 20,133 173,000 0.77 0.05 11.65
7
Organizational Chart
8
Breakdown of OPS Staff
  • 11 Supervisors (13.00/hr)
  • 10 Assistant Supervisors/Interviewers (9.00)
  • 15 Refusal Convertors (9.50/hr)
  • 25 Spanish Speaking Interviewers (9.00/hr)
  • 100-150 General Interviewers (8.50/hr)
  • Bonus based on productivity for interviewers and
    listening to recordings for supervisors

9
Refusal convertors
  • Our most skilled interviewers
  • Higher pay, higher productivity standard
  • Allowed to call our most difficult cases
  • Promotion process
  • Minimum of 208 hours of interviewing
  • Maintained higher productivity standards
    consistently for last 40 hours
  • No penalty boxes in last two months

10
Shift Staffing
  • 1 Shift Supervisor
  • 0-4 Assistant Supervisors
  • 0-5 Mentors

11
Hours of Operation
  • Mon-Friday 9am-12am
  • Saturday 12pm-9pm
  • Sunday 3pm-12am

12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
FACILITIES
20
Recently expanded location
21
Dedicated network backbone
22
Security
  • Building security guards and alarm system
  • Dedicated on-site servers kept in a locked room
  • Off-site backups to prevent data loss
  • Virus and malware detection
  • All network infrastructure protected with
    filtered power and battery backup
  • Unauthorized access detected 24 hours a day,
    seven days a week by IT staff

23
Telephone system
  • 93 telephone interviewing stations
  • 69 outgoing lines with VOIP overflow option
  • Extra server in case first server fails
  • Asterisk VOIP software
  • Reducing background noise
  • Headsets
  • Noise absorbent dividers and ceiling

24
OTHER SERVICES
  • Focus Groups
  • Data entry
  • Transcription

25
Software
  • Wincati Sample management software (Sawtooth)
  • Ci3 Questionnaire authoring software (Sawtooth)
  • Scheduler Lab management software

26
WINCATI Sample Management Software
27
Survey Specifications
28
Example Record
29
Disposition Report
30
Response Rate
  • Proportion of completed interviews out of all
    eligible respondents
  • What is eligible?
  • Listed sample -- All cases may be eligible
  • RDD sample There are rules to determine which
    cases to count

31
AAPOR RR1
  • I Complete interview
  • P Partial interview
  • R Refusal and break-off
  • NC Non-contact
  • O Other
  • UH Unknown if household/occupied housing unit
  • UO Unknown, other

32
Monthly survey response rates by county
33
Ci3 Questionnaire Authoring Software
34
Lab Management and Quality Control
35
Scheduler
36
Interviewers receive feedback
37
Reviewing recorded calls
38
Reviewing call recordings
  • Automatically select random calls targeted at
    mistakes and policy violations
  • Supervisors listen to and evaluate calls
  • Determine whether interviewer correctly described
    the outcome of the call
  • Check for policy violations

39
Real-time call monitoring
  • Assistant supervisors listen to calls during a
    shift
  • Evaluate six skills verbatim, probing,
    clarifying, feedback, pace, and voice
  • Meet with interviewer during same shift to
    discuss their performance

40
Real-time call monitoring
41
Miscoding Penalty Box
  • Receive penalty box for one call with low rating
    on call skills
  • Can also receive penalty boxes for frequently
    choosing the wrong description of a calls
    outcome
  • May be terminated for third penalty box

42
Productivity
43
Top performer
44
This interviewer needs help
45
Measuring employee productivity
  • Compare completed interviews per hour to average
    for survey
  • Average across time spent on each survey
  • Interviewer score calculated over large 45-day
    window to smooth out bad days
  • Survey average calculated over smaller 15-day
    window for higher accuracy

46
Rewarding top performers
  • Productivity incentive for keeping score above
    100
  • Incentive proportional to hours worked and how
    far above 100
  • Some top performers earn over 600 in
    productivity incentives for a 2-month incentive
    period

47
Penalty box
  • Required to keep productivity score above
  • 84 for general interviewers
  • 91 for spanish-speaking interviewers
  • 107 for refusal convertors
  • 117 for spanish-speaking refusal convertors
  • Hold most skilled to higher standard
  • Meet with supervisor after second box
  • Terminated or demoted after third box

48
How Much Does A Survey Cost?
49
Negotiating a survey
  • Estimate productivity per interviewer hour
  • We try to get 26 per interviewer hour
  • A survey where we complete one interview per hour
    costs 26/complete
  • A survey where we complete one interview every
    two hours costs 52/complete

50
Relation between hours worked and cost per hour
51
Survey Cost Structure
  • Major factors that influence Survey Costs
    include
  • Population
  • Mode
  • Length
  • Sample
  • Subject/Salience

52
Survey Mode Characteristics
53
2010 Surveys by Mode
54
Results from AASRO survey
55
Effect of Population on Cost
  • How easy is it to identify, locate and/or select
    the group being studied?
  • Survey of the Elderly (Elder) Statewide study of
    Florida Seniors Selection screening
    simple 38/complete
  • Oral Cancer Awareness (Port9) Oversamples
    African Americans, age 25 in selected
    counties Selection screening challenging
  • 55/complete

56
Effect of Length on Cost
  • Generally, a shorter survey lower costs
  • Survey of Star Plus program members (Star)
  • 36 min 53/complete
  • Consumer Confidence Index (CCI)
  • 11 min 26/complete

57
Effect of Mode on cost
  • Type of survey telephone, mailout, etc.--
    affects survey cost.
  • Dept. of Fine Arts Evaluation Web survey
    10/complete
  • FL Dept of Transportation Survey of Drivers
  • Telephone 28/complete

58
Effect of Subject/Salience on Cost
  • What is the survey about? How important is it to
    the respondent? Surveys with threatening
    questions (disability, alcohol abuse, etc.) may
    increase costs. Florida BRFSS Subject Health
    practices care Inc. threatening
    questions 75/complete CAHPS Subject
    Health care No threatening questions 23.50/compl
    ete

59
Effect of Sample on Cost
  • Sample type (Random Digit Dial vs. Listed
    sample) and quality can substantially affect
    costs.
  • HMO Report Card Survey (HMO)
  • Listed sample 3.2 recs/comp.
  • Consumer Confidence Index (CCI)
  • Random Digit Dial 10 recs/comp.

60
Sampling
61
Telephone survey sampling schemes
  • RDD random digit dialing
  • Numbers are created using information about
    existing area codes (GENESYS)
  • Best for obtaining a representative sample
  • Less productive than listed sample

62
SAMPLING SCHEMES
  • Targeted RDD sample
  • Overlay U.S. census tracts with working telephone
    banks
  • Targets specific subgroups
  • Regions
  • Ethnic groups
  • Levels of income

63
SAMPLING SCHEMES
  • Predetermined lists
  • Efficient
  • Cost effective
  • Quality depends on the frequency of updates

64
Sample size estimate
65
Do Not Call Lists
  • National Do Not Call Registry (www.donotcall.gov)
  • Only calls that solicit the sales of goods or
    services are covered
  • Surveys and charities are exempt
  • Respondents usually do not know that

66
Common sources of lists
  • Telephone numbers and households listed in
    telephone directory
  • Can pull national sample
  • Unlisted numbers vary a lot by geographic area
    and respondent characteristics
  • Drivers licenses from state Department of Motor
    Vehicles
  • Must select samples by state, and states vary in
    laws regarding drivers licenses
  • Data may be old as people move without informing
    Department of Motor Vehicles
  • Nor every one drives and there are biases (old
    and young, people in urban settings with public
    transportation and high insurance costs)

67
Common sources of lists (continued)
  • Voter Registration
  • Potentially more updated than drivers license
    database
  • Not everyone votes potentially very biased
    unless survey concerns potential voters
  • Lists from behavioral surveys and credit card
    evaluation
  • Usually expensive
  • Can often select people with particular
    characteristics (e.g. smokers)
  • Potentially biased based on source
  • Member and User Lists such as patient records,
    HMO membership, recipients of Temporary
    Assistance for Needy Families (TANF)
  • Source is often variable in maintaining records
    (e.g. HMOs do not have common database practices
    for recording membership data)

68
Telephone sample options and costs
  • 300 minimum on all orders
  • Random Digit Dialing (RDD) uncleaned
    .04/record
  • RDD with business purging from yellow pages
    .05/record
  • RDD with business purge and attended dialing
    using automated detection .09/record
  • Claritas Modeled data
  • Experian Behavior Bank .35/record

69
Example Consumer Confidence Survey
70
What is Consumer Confidence?
  • Telephone survey based measure calculated each
    month from the responses of approximately 500
    Florida households
  • Designed to predict consumer spending which
    accounts for 70 of Gross Domestic Product (GDP)
  • Barometer of public opinion about economy

71
How is consumer confidence measured?
  • Consumer Sentiment from the University of
    Michigan
  • Consumer Confidence from the Conference Board
  • 500 Random Digit Dial telephone interviews a
    month across the U.S.
  • Preliminary release second Friday, final release
    last Friday
  • Approximately 3,500 mail out surveys each month
    across the U.S.
  • Released last Tuesday

The University of Florida Survey Research Center
uses the University of Michigan method for
Florida, but releases the results the last
Tuesday of each month
72
Comparison of Index Questions
University of Michigan Index of Consumer Sentiment Conference Board Consumer Confidence Index
Current Conditions Current Conditions
Overall Buying Conditions Business Conditions in Area
Current Personal Financial Condition Current Job Availability in Area
Future Expectations Future Expectations
Business Conditions Over Next 12 Months Business Conditions in 6 Months
Business Conditions Over Next 5 Years Job Availability in Area in 6 Months
Personal Financial Condition in 12 Months Total Family Income in 6 Months
73
Comparison of U.S. Consumer Sentiment and
Consumer Confidence
74
Comparison of Florida and U.S. Consumer Sentiment
75
What are some drivers of Consumer Confidence?
  • Economic Shocks
  • Prices
  • Stocks
  • Employment

76
Our Survey Design
  • Approximately 5,000 Random Digit Dial phone
    numbers released each month
  • Phone numbers are released proportionate to
    households by county
  • A preliminary consumer confidence release is made
    the last Tuesday of each month with at least 400
    completed interviews
  • By the end of the month we collect at least 500
    interviews
  • Faculty can add questions to our monthly survey

77
News sources that have used our data
About PowerShow.com