Title: Objective Evaluation: A Search for Parity in an Online Geography Certificate Program
1Objective Evaluation A Search for Parity in an
Online Geography Certificate Program
- Scott L. Walker, ScEdD
- Texas State University-San Marcos
- walks_at_txstate.edu
2Distance Education Program Evaluation
- Implies measurement and judgment
- Poor measurement leads to reliance upon decision
makers beliefs or beliefs of others that are
imposed upon the decision maker
3Gen 5 Distance Teaching/Gen 1 Program Evaluation
- Distance education program evaluation remains
less than rigorous - Measurements typically fall into three
categories - Student outcomes
- Attitudes
- Satisfaction
- Mostly captured via anecdotal observation or
low-rigor instruments
4This presentation is about
- A rapid, multi-modal evaluation of an online
geographic information systems (GIS) certificate
program
5Program evaluation One way to look at it
- Overall purpose is to answer questions and
address issues raised by stakeholders - Evaluation combines techniques, proficiencies,
and sensitivities (politics) required to
establish - If a service is needed
- If it is conducted as planned
- If it actually works
6Program evaluation One way to look at it
- Embedded within the overall purposes from the
previous slide are more specific purposes - Evaluation is used to
- Determine accountability and effectiveness
- ID weaknesses so improvements can be made
- Gather effectiveness evidence for doubters
- Provide info for renewal
7Approaches to Evaluation
- Objectives-oriented
- Consumer-oriented
- Management-oriented
- Expertise-oriented
- Adversary-oriented
- Participant-oriented
8Objectives-Oriented
- Are program objectives being met?
- Poorly-defined objectives
- Unintended outcomes
- Hidden agendas
- Mis-measurement
9Consumer-Oriented
- Considers students as consumers
- Typically summative in nature
- Does not take into consideration student
differences (i.e. one size program fits all)
10Management-Oriented
- Used to allocate funding
- Evaluation conducted by those involved in the
program promote the status quo
11Expertise-Oriented
- Program accreditation evaluation
- Content experts look at program curriculum and
administration - Content expert ? distance education expert
12Adversary-Oriented
- Tries to get adversary buy-in
- Attempts to address adversarys issues
13Participant-Oriented
- Qualitative and involves all stakeholders
- Squeaky wheel may get all the grease
14Approaches Summary
- Seldom are these 6 approaches used exclusively
- Little consensus on which are best
- Often mixed with course evaluation
15Measures Actually Used
- Student grades are the most prominent measure
second only to technology used. - Other measures typically include
- Class size
- Target population
- Student characteristics
- Admin issues
- Distance ed vs. face-to-face
- Requirements of students
- Requirements of faculty
16This Evaluation-Background
- Four-year, doctoral granting, regional university
- Program consisting of four, 4-semester hour
online courses - Undergraduate level
- Offers a certificate (not a degree, but courses
are part of a degree and can be used for
attaining one)
17This Evaluation-Background
- Instructors Two doctoral students with strong
content background - Evaluator Strong content background 10-yrs.
experience developing, evaluating, teaching, and
participating in distance ed (eliminates the
expert-oriented approach problems)
18This Evaluation-Approaches, Purposes
- Approaches Combined Management-, Expertise-,
Participant-Oriented - Purposes
- ID weaknesses to target improvement areas,
- Ensure parity with equivalent face-to-face
classes - Provide info for renewal
19This Evaluation-Strategies
- Determine psychosocial learning environment
- Content analysis
- Cost comparison
- Academic outcomes
- Retention comparison
- Benchmarking
20Psychosocial Learning Environment
- a.k.a. class climate
- Used DELES (validated online instrument)
21Psychosocial Learning Environment
22Content Analysis
- Investigates parity between online and
face-to-face (f2f) sections - Considered only one classes content
23Content Analysis
- Classroom Content
- Data input
- Evaluation quality data
- Modeling
- Remote sensing
- Digital elevation models (DEM)
- Visualizations
- Skill Content
- Personal Web page development
- Data acquisition
- Geocoding
- Spatial analyst
- Network analyst
24Cost Comparison
- Done simply, Considered only personnel costs of
instructors/lab assistants
25Academic Outcomes(Grade Distribution Trend)
- Used only because it is one of the few
commonalities - Aggregated for all students over 3 years
26Academic Outcomes
27Academic Outcomes
28Retention
29Benchmarking
- Considers this program in relation to similar
programs regarding required semester hours and
delivery mode (online, hybrid, f2f)
30Benchmarking
- Mean required semester hours of 25 other GIS
certificate programs (distance, f2f, hybrid) 19 - This program 16
- Mean online GIS certificate programs 16
- 20 of other university GIS certificate programs
are online - 16 of other programs are offered in some hybrid
of online and face-to-face - 36 not online
31Summary of Evaluation
- Presented as SWOT (includes additional
observation notations) - Provides opportunity for decision makers to make
data-informed decisions from multiple, non-biased
perspectives
32Contact Info
- Scott L. Walker, ScEdD
- Texas State University-San Marcos
- http//uweb.txstate.edu/sw36/
- walks_at_txstate.edu