Loading...

PPT – EPSY 546: LECTURE 1 SUMMARY PowerPoint presentation | free to download - id: 4167cc-M2NkY

The Adobe Flash plugin is needed to view this content

EPSY 546 LECTURE 1 SUMMARY

- George Karabatsos

REVIEW

REVIEW

- Test ( types of tests)

REVIEW

- Test ( types of tests)
- Item response scoring paradigms

REVIEW

- Test ( types of tests)
- Item response scoring paradigms
- Data paradigm of test theory (typical)

DATA PARADIGM

REVIEW Latent Trait

- Latent Trait ? ? Re (unidimensional)

REVIEW Latent Trait

- Latent Trait ? ? Re (unidimensional)
- Real Examples of Latent Traits

REVIEW IRF

- Item Response Function (IRF)

REVIEW IRF

- Item Response Function (IRF)
- Represents different theories

- about latent traits.

REVIEW IRF

- Item Response Function (IRF)
- Dichotomous response
- Pj(?) PrXj 1
- PrCorrect Response to item j ?

REVIEW IRF

- Item Response Function (IRF)
- Polychotomous response
- Pjk(?) PrXj gt k ?
- PrExceed category k of item j

?

REVIEW IRF

- Item Response Function (IRF)
- Dichotomous or Polychotomous response
- Ej(?) Expected Rating for item j ?
- 0 lt

Ej(?) lt K

IRF Dichotomous items

IRF Polychotomous items

REVIEW SCALES

- The unweighted total score Xn stochastically

orders the latent trait ? - (Hyunh, 1994 Grayson, 1988)

REVIEW SCALES

- 4 Scales of Measurement
- Conjoint Measurement

REVIEW

- Conjoint Measurement
- Row Independence Axiom

REVIEW

- Conjoint Measurement
- Row Independence Axiom
- Property Ordinal Scaling and unidimensionality

of ? (test score)

INDEPENDENCE AXIOM (row)

REVIEW

- Conjoint Measurement
- Row Independence Axiom
- Property Ordinal Scaling and unidimensionality

of ? (test score) - IRF Non-decreasing over ?

REVIEW

- Conjoint Measurement
- Row Independence Axiom
- Property Ordinal Scaling and unidimensionality

of ? (test score) - IRF Non-decreasing over ?
- Models MH, 2PL, 3PL, 4PL, True Score, Factor

Analysis

2PL

3PL

4PL

Monotone Homogeneity (MH)

REVIEW

- Conjoint Measurement
- Column Independence Axiom (adding)

REVIEW

- Conjoint Measurement
- Column Independence Axiom (adding)
- Property Ordinal Scaling and unidimensionality

of both ? (test score) and item difficulty (item

score)

INDEPENDENCE AXIOM (column)

REVIEW

- Conjoint Measurement
- Column Independence Axiom (adding)
- Property Ordinal Scaling and unidimensionality

of both ? (test score) and item difficulty (item

score) - IRF Non-decreasing and non-intersecting over ?

REVIEW

- Conjoint Measurement
- Column Independence Axiom (adding)
- Property Ordinal Scaling and unidimensionality

of both ? (test score) and item difficulty (item

score) - IRF Non-decreasing and non-intersecting over ?
- Models DM, ISOP

DM/ISOP (Scheiblechner 1995)

REVIEW

- Conjoint Measurement
- Thomsen Condition (adding)

REVIEW

- Conjoint Measurement
- Thomsen Condition (adding)
- Property Interval Scaling and unidimensionality

of both ? (test score) and item difficulty (item

score)

Thomsen condition (e.g.,double cancellation)

REVIEW

- Conjoint Measurement
- Thomsen Condition (adding)
- Property Interval Scaling and unidimensionality

of both ? (test score) and item difficulty (item

score) - IRF Non-decreasing and parallel

(non-intersecting) over ?

REVIEW

- Conjoint Measurement
- Thomsen Condition (adding)
- Property Interval Scaling and unidimensionality

of both ? (test score) and item difficulty (item

score) - IRF Non-decreasing and parallel

(non-intersecting) over ? - Models Rasch Model, ADISOP

RASCH-1PL

REVIEW

- 5 Challenges of Latent Trait Measurement

REVIEW

- 5 Challenges of Latent Trait Measurement
- Test Theory attempts to address these challenges

REVIEW

- Test Construction (10 Steps)

REVIEW

- Test Construction (10 Steps)
- Basic Statistics of Test Theory

REVIEW

- Total Test Score (X) variance
- SumItem Variances
- SumItem Covariances

EPSY 546 LECTURE 2 TRUE SCORE TEST THEORY AND

RELIABILITY

- George Karabatsos

TRUE SCORE MODEL

- Theory Test score is a random variable.
- Xn Observed Test Score of person n,
- Tn True Test Score (unknown)
- en Random Error (unknown)

TRUE SCORE MODEL

- The Observed person test score Xn is a random
- variable (according to some distribution) with

mean Tn E(Xn) and variance ?2(Xn) ?2(en).

TRUE SCORE MODEL

- The Observed person test score Xn is a random
- variable (according to some distribution) with

mean Tn E(Xn) and variance ?2(Xn) ?2(en). - Random Error en Xn Tn
- is distributed with
- mean E(en) E(XnTn) 0,
- and variance ?2(en) ?2(Xn) .

TRUE SCORE MODEL

- True Score
- Tn true score of person n
- E (Xn) expected score of person n
- s Possible score s ? 0,1,,s,,S
- pns PrPerson n has test score s

TRUE SCORE MODEL

- 3 Assumptions
- Over the population of examinees, error has a

mean of 0. Ee 0 - Over the population of examinees, true scores and

error scores have 0 correlation. - ?T, e 0

TRUE SCORE MODEL

- 3 Assumptions
- For a set of persons, the correlations of the

error scores between two testings is zero.

?e1, e2 0 - Two testings when a set of persons take two

separate tests, or complete two testing occasions

with the same form. - The two sets of person scores are assumed to be

randomly chosen from two independent

distributions of possible observed scores.

TRUE SCORE ESTIMATION

TRUE SCORE ESTIMATION

TRUE SCORE ESTIMATION

TRUE SCORE ESTIMATION

is test reliability. The proportion of

variance of observed scores that is explained by

the variance of the true scores.

TEST RELIABILITY

is the error of

measurement.

TEST RELIABILITY

is the standard error of

measurement. (random error)

TEST RELIABILITY

is the standard error of

measurement. (random

error) Estimated ((1?)100) confidence

interval around the test score

TEST RELIABILITY

- It is desirable for a test to be Reliable.

TEST RELIABILITY

- Reliability the degree to which the

respondents test scores are consistent over

repeated administrations of the same test.

TEST RELIABILITY

- Reliability the degree to which the

respondents test scores are consistent over

repeated administrations of the same test. - Indicates the precision of a set of test scores

in the sample.

TEST RELIABILITY

- Reliability the degree to which the

respondents test scores are consistent over

repeated administrations of the same test. - Indicates the precision of a set of test scores

in the sample. - Random and systematic error can affect the

reliability of a test.

TEST RELIABILITY

- Reliability the degree to which the

respondents test scores are consistent over

repeated administrations of the same test. - Test developers have a responsibility to

demonstrate the reliability of scores obtained

from their tests.

ESTIMATING RELIABILITY

Estimated item variance

Estimated total test score variance

ESTIMATING RELIABILITY

Estimated covariance between

items i and j Estimated total

test score variance

OTHER FORMS OF RELIABILITY

- Test-Retest Reliability
- The correlation between persons test scores

over two administrations of the same test.

OTHER FORMS OF RELIABILITY

- Split-Half Reliability
- (using Spearman-Brown correction for test

length) - ?AB Correlation between scores of Test A
- and Test B

TEST VALIDITY

- VALIDITY A test is valid if it measures what it

claims to measure. - Types Face, Content, Concurrent, Predictive,

Construct.

TEST VALIDITY

- Face validity When the test items appear to

measure what the test claims to measure. - Content Validity When the content of the test

items, according to experts, adequately represent

the latent trait that the test intends to

measure.

TEST VALIDITY

- Concurrent validity When the test, measuring a

particular latent trait, correlates highly with

another test that measures the same trait. - Predictive validity When the scores of the test

predict some meaningful criterion.

TEST VALIDITY

- Construct validity A test has construct

validity when the results of using the test fit

hypotheses concerning the nature of the latent

trait. The higher the fit, the higher the

construct validity.

RELIABILITY VALIDITY

- Up to a point, reliability and validity increase

together, but then any further increase in

reliability (over .96) decreases validity. - For e.g., when there is perfect reliability

(perfect correlations between items), the test

items are essentially paraphrases of each other.

RELIABILITY VALIDITY

If the reliability of the items were increased

to unity, all correlations between items would

also become unity, and a person passing one item

would pass all items and and another failing one

item would fail all the other items. Thus all the

possible scores would be a perfect score of one

or zeroIs the dichotomy of scores the best that

would be expected for items with equal

difficulty? (Tucker, 1946, on the

attenuation paradox)

(see also Loevinger, 1954)