Title: Improving academic achievement: Methodologies to find and track valid educational indicators at Scho
1Improving academic achievement Methodologies to
find and track valid educational indicators at
School and Board levels
Dany Laveault, Ph.D. Faculty of Education,
University of Ottawa
AEA-- Europe, Dublin November 3rd, 2005
2Issues with indicators of change
- What kind of changes should we pay attention to?
Some changes are more relevant than others. - How can we appreciate change? Stable points of
reference are required to appreciate change. - How can we explain change? The capacity to effect
change is related to our capacity to link it to
relevant processes. - How can we observe change without disturbing it?
3A system of indicators should
- Be relevant and convivial for the development of
policies and / or improvement plans - Be supported by a conceptual framework
- Be valid and reliable
- Be cost efficient
- Have no undesirable consequences
Blais, J.-G., Laurier, M. Pelletier, G. (1999).
Regards sur la problématique de la production des
indicateurs en éducation. Mesure et évaluation en
éducation, , 22(2-3), 47-69.
4Alignment
- Alignment refers to the degree of match between
test content and the subject area content
identified through state academic standards.
La Marca, P. (2001). Alignment of Standards and
Asseessments as an Accountability Criterion. ERIC
Digest/AE Series EDO-TM-01-07.
5Evidence of curriculum alignment
6Indicators and curriculum alignment Which
curriculum?
- The formal curriculum and its many possible
interpretations. - The effective curriculum and its many possible
implementations of each single interpretation. - The curriculum of learning achievements with all
individual student differences associated with
the same implementation.
Perrenoud, P. (1993). Curriculum le formel, le
réel, le caché. In J. Houssaye (ÉD.). La
pédagogie une encyclopédie pour aujourdhui. 2e
édition. Paris ESF, pp. 61-76.
7Factors affecting the objective evaluation of
alignment
- Test / items tasks often provide measurement of
multiple content standards/objectives and this
may introduce error into expert judgments. - Standards that reflect only general expectations
tend to include limited information for defining
the breadth of content and determining cognitive
demand. - Standards that contain excessive detail also
impede the development of assessments, making an
acceptable degree of alignment difficult to
achieve.
La Marca, P. (2001). Alignment of Standards and
Asseessments as an Accountability Criterion. ERIC
Digest/AE Series EDO-TM-01-07.
8Zoom in on trends
1
2
achievement
achievement
3
6
5
4
3
Grade level
3
6
5
4
achievement
Grade level
3
6
5
4
Grade level
Seltzer, M., Choi, K., Thum, Y.M. (2002).
Examining relationships between where students
start and how rapidly they progress. Los Angeles
Center for the study of evaluation, University of
California.
9Percentage of improvement
2/20 10
6/60 10
Province
School
10Zoom in on trends
1
2
achievement
achievement
3
6
5
4
3
Grade level
3
6
5
4
achievement
Grade level
3
6
5
4
Grade level
Seltzer, M., Choi, K., Thum, Y.M. (2002).
Examining relationships between where students
start and how rapidly they progress. Los Angeles
Center for the study of evaluation, University of
California.
11AYP 37 indicators(Adequate Yearly Progress)
Scott M. White, C. (2002). Making valid and
reliable decisions in Determining Adequate Yearly
Progress. Waxhington, DC Council of Chief State
School Officers.
12Value-added indicators
Expected (local)
Expected (France)
Observed
Peretti, C. (2002). Indicateurs de résultats des
lycées. Paris Ministère de la jeunesse, de
léducation nationale et de la recherche
13Indicators of progress
- Cross-sectional study progress of different
cohorts at fixed periods. - Longitudinal study progress of the same cohort
at different periods. - Median change median change for different
cohorts (such as schools or classes) - Added-value discrepancy between the obtained
and the expected results for a specific cohort.
14One mark The median change
Median 0
15Two examples of median change
Median 2.5
Median 0
16Stability across results Grade 3, Reading
80
16
33
27
33
70
Improvement
60
50
Stability
students performing at standards, Grade 3,
2000-2001
27
40
30
Deterioration
Écriture 9e, 2002-2003
20
lecture 3e, 2000-2001
12
22
20
10
28
0
80
60
40
20
0
Écriture 9e, 2001-2002
students performing at standards, Grade 3,
1999-2000
17Stability across time
18Integration reading-writing
19Local diagnosis at the classroom level
Case 1
Different impacts of Reading?
Case 2
20Relative position of single-level and multiple-
level schools. Reading Writing (Gr. 6)
at levels 3 or 4 in Reading
Multiple levels
Single level
at levels 3 or 4 in Writing
21Differentiating information.
Percentages of boys and girls who agreed with the
statement, I like to read.
22Relationship between attitude and instructional
method
23Establishing relationships between
self-perceptions and achievement (Grade 6)
24I review my piece of writing and correct my
mistakes (99-00)
25Grade 3 underestimation values for writing
26Grade 6 underestimation values for writing
27Indicators as factors that influence and reflect
student learning
- Context
- Inputs
- Processes
- Results
28Indicators and change
Before
After
Initial condition
Final condition
Change processes
- Prior conditions
- Starting situation
-
Intervention
Indicators of inputs
Indicators of processes
Indicators of results
Adapté de Gaudreau, L. (2001). Évaluer pour
évoluer. Montréal Éditions Logiques.
29Examples of indicators in Reading
- Inputs
- Processes
- Results
- Efficiency
- Performance
- Impact
- Success
Prior results in Reading. Context.
Motivational strategies / Instructional methods /
Etc.
Results in Reading have increased.
Results have been obtained by liberating one
lead-teacher.
Results are superior to the last three-year
average.
Improvement in Reading results have had a
positive impact in Mathematics problem solving.
The target has been reached.
30Possible sources of bias
- Availability sources of data are selected
because they are easy to use and highly visible. - Classifiability data are selected on the
basis that they correspond to already known
categories. Consequently, new or different
characteristics are neglected. - Mesurability dependence towards characteristics
that may be easily measured and quantified.
Singer, B.D. (1996). Towards a Sociology of
Standards Problems of a Criterial Society.
Canadian Journal of Sociology, 21(2), 203-221.
31Different improvement trends
32Selection of indicators
- Indicators make little sense if taken in
isolation, outside of context and if they do not
have a basis for comparison. - Indicators allow to compare one characteristic at
a time. It is preferable to use them as a group. - Indicators may be summarized by one statistic or
a statistical composite index that represent a
basic construct. - Indicators should evolve to be sensitive to
changing contexts. They may require some
reconstruction from time to time.
Blais, J.-G., Laurier, M. Pelletier, G. (1999).
Regards sur la problématique de la production des
indicateurs en éducation. Mesure et évaluation en
éducation, , 22(2-3), 47-69.
33Validity of a system of indicators
Conceptual
Consequential
- Does the system focus on real problems?
- Is the system successful in identifying schools
which are experiencing difficulties? - Is the system linked theoretically and logically
to student learning improvement?
- Does the system produce the expected impact?
- Does the system leads to more equity towards
students? - Are there unexpected or perverse effects
occurring as a result of the system? (e.g.
teachers leaving the profession, curriculum
defined in narrower terms )
Scott M. White, C. (2002). Making valid and
reliable decisions in Determining Adequate Yearly
Progress. Washington, DC Council of Chief State
School Officers.
34In all fairness, the test is the same for
everyone. Please climb up that tree.