Evaluating Research Claims - PowerPoint PPT Presentation

Loading...

PPT – Evaluating Research Claims PowerPoint presentation | free to download - id: 77283f-ZDg3N



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Evaluating Research Claims

Description:

Evaluating Research Claims – PowerPoint PPT presentation

Number of Views:0
Avg rating:3.0/5.0
Date added: 10 August 2018
Slides: 54
Provided by: Martin940
Learn more at: http://people.uncw.edu
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Evaluating Research Claims


1
Evaluating Research Claims
2
Evaluating TV Ad Claims
  • An ad on TV says,
  • USE Cocoa Creme. Youll have youthful skin in
    10 minutes. All natural ingredients.
  • You say, Yeah, sure! Youthful skin.
  • I bet your face falls off in 10 minutes.

3
Evaluating TV Ad Claims
  • Another ad tells you to,
  • Use Motor Magician. Your engine will purr like
    a kitten. And youll get 80 miles per gallon!
  • You say, I dont want a motor that sounds like
    a cat. 80 miles per gallon? HA! Baloney.

4
Evaluating TV Ad Claims
  • In other words.
  • You arent fooled by ads promising what is very
    unlikely, and that give no EVIDENCE to support
    the inflated promises.

5
Evaluating Medical Claims
  • The same goes with medicine.
  • Lets say your child is ill with an infection.
  • The physician says, Heres a prescription for
    Megabiotic.
  • You ask, Will it work, Doctor?
  • Physician. Oh, yes. It will fix her right
    up.
  • You. Does it have bad side effects?
  • Physician. Oh, no. No side effects.
  • You. How do you know, Doctor?

6
Evaluating Medical Claims
  • How do you know, Doctor?
  • That is THE question.
  • How do you know?
  • What makes you so sure?
  • Do you have trustworthy evidence to back up your
    claim?

7
Evaluating Medical Claims
  • Will you let your child take the prescribed
    medication if these (below) are the physicians
    answers?....
  • Well, I and my doctor friends think it works
    very well.
  • In a study of five persons, everyone got well.
  • In a study of 1000 persons, many said they felt
    much better.
  • You wouldnt be satisfied by these answers!
  • In fact, youd tell the physician, See you
    later!

8
Evaluating Medical Claims
  • What DO you want the physician to tell you?
  • How about these?...
  • Megabiotic was tested in 10 experimental studies.
  • The research was not conducted by the drug
    company, but by independent researchers who had
    nothing to gain whether Megabiotic was effective
    or not effective.
  • Each study lasted from one to five years.
  • Each study had at least 1000 persons in it.

  • More

9
Evaluating Medical Claims
  • Each study sample had a mix of different kinds of
    persons in it younger/older, more ill/less ill,
    male/female.
  • Each study compared the effects of Megabiotic
    with other medications (alternative treatments)
    and with no treatments (control group).
  • Each study measured the exact amount of infection
    before treatment, each week during treatment, and
    at the end of the course of treatment. The
    measurement was objective---the amount of
    infection in the blood.
  • Side effects were also measured, such as blood
    pressure, skin rash, and kidney function.
  • More

10
Evaluating Medical Claims
  • All of the studies found that persons who
    received Megabiotic had NO serious side effects.
  • Persons who received Megabiotic showed directly
    observable (in the blood) reductions in infection
    each week until, by the end of one year, 95 of
    the persons were cured.
  • Now would you let your child take the medication?

11
Evaluating Medical Claims
  • Probably.
  • Why?
  • Why is the second set of answers from the doctor
    convincing?
  • Think about it.

12
Evaluating Medical Claims
  • The second answers are more convincing because
  • The second answers are more believable.
  • The second answers give you solid REASONS to
    believe that the medication is likely to cure
    your childs infection and not cause harm.
  • The physician gave you evidence that is hard to
    dismiss as just baloney.

  • More

13
Evaluating Medical Claims
  • You cant say, It wont work with my child,
    because the research on Megabiotic was tested
    with thousands of persons---many of whom were
    just like your child.
  • You cant say, Maybe persons got well by
    themselves. It wasnt Megabiotic, because
    persons in the control group (who got no
    Megabiotic) did NOT get well.
  • You cant say, They didnt measure improvement,
    because in fact they measured what can be SEEN
    (the amount of infection in the blood)---an
    objective measure of improvement, not an opinion.

  • More

14
Evaluating Medical Claims
  • You cant say, The effects are probably
    temporary persons will get sick again, because
    some of the studies lasted five years, and no one
    got sick again with the infection.
  • You cant say, The research was rigged to get
    good results. For example, researchers dropped
    from the study all persons who were NOT getting
    well. You cant say this because the drug
    company (that wanted the drug to work) did not DO
    the research.

15
Summary So Far
  • Lets summarize.
  • What makes research believable?
  • What makes the conclusions of research
  • The (drug, treatment, or method) is
    effective. Therefore, its a good idea to use
    it.
  • so trustworthy that you can be confident in
    your decision to use it?

16
Summary So Far
  • Here is what makes research believable and the
    conclusions trustworthy.
  • The effect of the thing being tested (program,
    materials, treatment, or method) is defined and
    measured objectively.
  • The thing being tested is tested against
    alternatives (to see which works better) and
    against no treatment (to see if the thing works
    at all---that is, improvement is much greater
    than when there is NO treatment).
  • The research is done many times (replicated), to
    ensure that results are not chance, not a fluke.
  • The research is done for a long period of time
    (longitudinal), to see if the effects are
    long-lasting.
  • More

17
Summary So Far
  • The research is done with samples of persons who
    have the same features as persons in the larger
    population who might later receive the program,
    treatment, or method. representative sample
  • The researchers do not have a stake in the
    outcomes. independent research

18
Lets Apply This to Education
  • Just as you feel responsible for the care of
    persons you love, you also feel responsible for
    the children you teach.
  • You dont want students to be taught with
    curriculum materials or with teaching methods
    that dont work and that cause harm---for
    example, teaching students the wrong way to read
    or to solve math problems.
  • Therefore, you want to ask tough questions about
    research done on curriculum materials and
    teaching methods that are advertised, or that a
    school district wants you to use, or that your
    colleagues say are terrific.
  • What tough questions should you ask about
    research?

19
Questions About Purpose. Selling?
  • What tough questions should you ask about
    research?
  • Lets say you are reading a research report, or
    a description of a new program on a website, or
    an advertisement for curriculum materials.
  • Does the ad, report, or program description sound
    as if the authors are selling an idea or a
    productto persuade you that they are right that
    they have the answer that they can fix schools
    and raise achievement?
  • Just (believe us, join us, buy our
    materials).
  • If selling is their purpose, this is NOT good.
    Why?
  • You can be almost certain that the only data
    (evidence) the writers will show are data that
    support their claims. They will not show data
    that do NOT support their claims. This is
    called dishonest.

20
Purpose Selling
  • In other words, you should be highly suspicious
    of their claims about effectiveness.
  • Our method works.
  • Our ideas are sound.
  • Use our ideas and program.
  • These claims do not come from objective
    scientific research. They come from persons and
    groups who are making up a story (a sales pitch)
    that will get you to buy.
  • If they had solid research evidence that their
    ideas, materials, or procedures were effective,
    their writing would not sound like a sales pitch.

21
Purpose Reporting
  • However, the purpose of a journal article or a
    program description may simply be to REPORT the
    findings of research. For example, the authors
    are telling you what they learned---not what they
    want you to believe or to do.
  • For instance, authors may report which one of
    three methods of peer tutoring produces the
    highest achievement.
  • Or what percentage of fourth graders are
    proficient in reading on grade level after a new
    reading program was used.
  • Simple reporting is good. If authors are not
    trying to sell you an idea or materials or
    programs, then they are probably going to report
    honestly what they found they have nothing to
    gain simply by passing on information.

22
Questions About the Research Strategy
  • There are different KINDS of research questions
    for instance, there are questions.
  • About persons opinions. How do teachers feel
    about.?
  • About what persons do. How do students respond
    when teachers.?
  • About factors that cause or predict changes.
    Does timely correction of errors reduce those
    same errors happening in the future?
  • About what is effective. Which program (A, B,
    or C) is more effective in raising the reading
    achievement of diverse learners?

23
Questions About the Research Strategy
  • There are different kinds of research questions,
    and there are different kinds of research
    strategies---or WAYS of trying to answer the
    questions. For instance, there are
  • Surveys.
  • Observation.
  • Experimentation.

24
Questions About the Research Strategy
  • Heres a big idea
  • Certain kinds of research questions require
    certain research strategies,
  • or else the research will NOT give evidence
    (data, findings) that you can trust.
  • Do you have high blood pressure?
  • Which is the better research strategy to get an
    answer you will trust?
  • Do a survey of the opinions of 20 of your
    friends.
  • Use a blood pressure cuff (every day for a week,
    at different times of day) to get exact
    measurement.
  • Blood pressure is NOT an opinion-kind-of-thing!
  • It is an objective thing. PRESSURE. Therefore,
    it requires an OBJECTIVE measurement---with the
    cuff.

25
Questions About the Research Strategy
  • Here are different kinds of research questions
    and the proper research strategies.
  • What are teachers opinions about a new math
    program that they are using?
  • How well do students think they read three months
    after theyve been taught with a new reading
    program?
  • How satisfied are teachers with the supervision
    and professional development they receive?
  • What are the evaluations of parents on the
    helpfulness of their childrens teachers?
  • These are opinion questions.

26
Is the Research Strategy Appropriate for the
Research Questions?
  • These are opinion questions.
  • The researcher wants information on how
    teachers, students, and parents SEE things.
  • Therefore, a useful strategy would be SURVEY
    research.
  • Example of survey research ?

27
Is the Research Strategy Appropriate for the
Research Questions? Survey Research
  • Consider the question on parents satisfaction.
  • The researcher selects a sample of families.
  • This sample is representative of the whole
    population of families of children at schools
    for example, small and large families two-parent
    and single-parent families families with one
    child and families with more children poor
    families and affluent families.
  • Then the researcher either uses a questionnaire
    (that parents fill out) or uses interviews
    (face-to-face) in which the researcher asks the
    same kinds of questions as on the questionnaire
    or gives questionnaires to all families and also
    interviews a smaller sample to obtain more
    detailed information.

28
Is the Research Strategy Appropriate for the
Research Questions? Survey Research
  • The researcher can examine (analyze) the survey
  • information (data) from the questionnaires or
  • interviews, and can then determine
  • What kinds of help parents feel is adequate.
  • What kinds of help parents feel is inadequate.
  • What percentage of parents are satisfied vs. not
    satisfied with teachers helpfulness.
  • These findings could assist school administrators
    to
  • improve family-school relations.

29
Is the Research Strategy Appropriate for the
Research Questions?
  • Here is a different kind of research question.
  • What are the different ways that teachers respond
    or try to handle students disruptive behavior?
    What are the effects of each way?
  • When students are doing peer tutoring, do
    students learn to play the role of tutor and the
    role of tutee? If so, what are the behaviors in
    these roles?
  • When a class is organized into learning centers
    (e.g., reading, math, science), and students are
    allowed to come and go as they wish, what do
    students actually DO in the centers?

30
Is the Research Strategy Appropriate for the
Research Questions?
  • These are questions about what persons
    DO---about observable activity---NOT about how
    persons see things---their opinions.
  • Therefore, a useful research strategy is
    OBSERVATIONAL research.
  • Example of observation research?

31
Is the Research Strategy Appropriate for the
Research Questions? Observational Research
  • Consider the question of how teachers respond to
    disruptive
  • behavior.
  • The researcher selects a sample of classrooms.
  • Standing out of the way, the researcher takes
    running notes (like a sportscaster) on
    teacher-student interaction.
  • After several weeks, the researcher examines the
    notes and
  • a. Identifies different kinds of disruptive
    behavior e.g., making noises, talking out of
    turn, noncompliance with teacher requests.
  • b. Identifies how the teacher typically
    responds to each kind of disruptive behavior
    e.g., telling a student to stop, threatening to
    send the student to the office, ignoring the
    disruptive behavior but praising students when
    they are engaging in desirable behavior.
  • c. Determining what generally happens when
    teachers respond in different ways e.g.,
    students do the disruptive behavior more often
    vs. less often.

32
Is the Research Strategy Appropriate for the
Research Questions?
  • Here is yet another kind of research question.
  • Is peer tutoring more effective (does it yield
    higher achievement) for some subjects than other
    subjects?
  • Are teachers opinions about their new math
    programs affected by whether a math program
    provides written procedures for teachers to
    follow vs. does not provide written procedures
    for teachers to follow?
  • More

33
Is the Research Strategy Appropriate for the
Research Questions?
  • Under what conditions (e.g., size of school,
    recent levels of student achievement, quality of
    teaching staff) is it better (there is faster
    change, less opposition) when school reform is
    rapid and extensive vs. slow and in small steps?
    In other words, what factors predict which kind
    of school reform works better.
  • Which math program yields the most math
    proficiency in students overall and students by
    subgroups, such as ethnicity and social class?
  • These are questions about causation.

34
Is the Research Strategy Appropriate for the
Research Questions?
  • These are questions about causation. The idea
  • is that
  • Certain factors ? Predict or produce
  • or variables changes in other
    factors
    or variables

35
Is the Research Strategy Appropriate for the
Research Questions?
  • Here are examples of questions about which
    factors or
  • variables predict or produce (cause) effects in
    other variables.
  • Which features of a math program (Teaching
    procedures? Practice? How tasks are sequenced?)
    have the largest effects on (cause the largest
    changes in) learning?
  • Which features of a school affect (cause or
    inhibit) the success of efforts to reform the
    school curriculum?
  • How does subject matter affect (cause or inhibit)
    learning through peer tutoring?
  • Do written procedures or the absence of written
    procedures in a math program affect (cause)
    teachers opinions of the program?

36
Is the Research Strategy Appropriate for the
Research Questions?
  • This kind of question.
  • Which factors cause what changes?
  • Which curriculum, program, or procedure is more
    effective?....
  • .requires the strategy of experimental
  • research.

37
Is the Research Strategy Appropriate for the
Research Questions? Experiment
  • The essential feature of the strategy of
    experimental
  • research is that you
  • Compare two or more situations (e.g., schools)
    that are as similar as you can make them (e.g.,
    of students who are poor, size of school, quality
    of teaching staff), but these schools are
    different in the few factors (e.g., reading
    materials) whose effects you are trying to find
    out, or test.
  • If the only main way that the schools differ is
    the
  • reading materials used, and if one school has
    consistently
  • higher reading achievement than the others, then
    it is
  • very likely that the reading materials make the
    difference.
  • What ELSE could it be? After
    all. ?

38
Is the Research Strategy Appropriate for the
Research Questions? Experiment
  • The factor that makes the difference in reading
    achievement cant be class size---because this is
    the same across all the schools.
  • The factor that makes the difference in reading
    achievement cant be teacher quality---because
    this is the same across all the schools.
  • The factor that makes the difference in reading
    achievement cant be the percentage of students
    who are poor---because this is the same across
    all the schools.
  • Therefore, the factor that makes the difference
    in reading achievement must be the ONE major way
    the schools are different----reading curriculum.

39
Is the Research Strategy Appropriate for the
Research Questions? Experiment
  • However, not all experiments give information
    that you can
  • trust. It depends on how the experiment is
    carried out. For
  • example
  • 1. Maybe students in schools with the highest
    reading achievement were already reading at a
    higher level. Therefore, these students dont
    have to learn very much in order to come out
    first. In other words, maybe students
    pre-skills (and not the curriculum materials)
    account for higher achievement.
  • To rule out this possibility, an experiment
    should have pre-
  • test measures of the variables whose change we
    are interested
  • in. For instance, we should measure reading
    skills in all of the
  • schools BEFORE new reading materials are
    introduced, to
  • see if some classes are ahead of other classes.

40
Is the Research Strategy Appropriate for the
Research Questions? Experiment
  • So, if an article says that a program or
  • method produced more change or is more
  • effective than another program or method, ask
  • if the researchers did pre-tests to show that
  • students all started at the same level. If they
  • did not do pre-tests, then you cannot trust the
  • claim about effectiveness. The group with the
  • highest achievement may have started at a
  • higher level.

41
Is the Research Strategy Appropriate for the
Research Questions? Experiment
  • 2. How do you know how much students learned?
    The only way to find out is to do post-tests
    (e.g., at the end of the semester and year) to
    see where students reading achievement ended up.
    And you have to see how much students changed
    (learned, achieved) between the pre-test (reading
    proficiency at the start) and the post-test
    (reading achievement at the end). And you need
    to report how much difference there is between
    the pre-test and post-test scores.
  • If researchers do NOT do post-tests, or do not
    REPORT the
  • size of differences between the pre-test and
    post-test, then you
  • dont know HOW MUCH students learned as a result
    of the
  • curriculum that was used between the pre-test and
    the post-
  • test. Therefore, you cant trust claims that one
    curriculum
  • produced more achievement than the others.

42
Is the Research Strategy Appropriate for the
Research Questions? Experiment
  • 3. Were measures objective?
  • You dont rely on a physicians subjective
    opinion about your
  • health.
  • You look fine to me.
  • Maybe the physician is right.
  • But what if the physicians opinion is wrong?
  • Just so, you want objective evidence of (in this
    experiment)
  • reading achievement. It is nice to know how much
    teachers
  • think their students learned, but you cant judge
  • effectiveness based on opinion. Reading is a
    real thing.
  • It can be measured objectively---for example, how
    many
  • words students read correctly per minute.

43
Is the Research Strategy Appropriate for the
Research Questions? Experiment
  • So, if researchers do NOT use objective measures
    of (for example) reading and math achievement
    (e.g., counting correct words students read or
    problems students solve per minute), but instead
    use the subjective opinions of teachers
    (Students improved a great deal.) then you
    cannot trust claims about effectiveness----because
    the researchers did not give evidence of
    effectiveness. They only gave evidence of how
    teachers see things. But that was not the
    question.

44
Is the Research Strategy Appropriate for the
Research Questions? Experiment
  • 4. Were instruments and measurement methods
    tested for validity (they measure what they are
    supposed to measure) and reliability (different
    persons measuring the same thing would come up
    with about the same scores)?
  • Validity. Do you think a good measure of
    reading proficiency is whether students can name
    the parts of a book and can turn the pages?
  • If your school principal is selecting a new
    reading program, is that the information you want
    the principal to use?
  • Students taught with Weird Phonics learned to
    turn the pages. So, Ill use that program with
    200 students in my school.
  • I dont think so.

45
Is the Research Strategy Appropriate for the
Research Questions? Experiment
  • Turning pages is not what READING means.
  • You want to know if a program teaches students
    to (1) say the sounds that go with the letters
    (2) accurately and quickly read single words (3)
    accurately and quickly read passages (4) learn
    many new vocabulary words each week (5) answer
    both simple and hard comprehension questions
    about what they read.

46
Is the Research Strategy Appropriate for the
Research Questions? Experiment
  • So, if researchers use instruments and measures
    that do not measure what they OUGHT to measure,
    then the findings have nothing to do with the
    research question.
  • The question is whether a curriculum teaches
    students to read. But the findings have to do
    with turning pages. Invalid measure.
  • The question is whether a certain teaching
    procedure increases math achievement. But the
    findings have to do with how much students
    enjoyed being taught with the method. Invalid
    measure.
  • The question is whether peer tutoring is
    effective at helping students to spell better.
    But the findings have to do with changes in
    self-esteem. Invalid measure.

47
Is the Research Strategy Appropriate for the
Research Questions? Experiment
  • 5. Is the sample size, composition (kinds of
    participants), and selection/allocation of
    participants to groups appropriate for the type
    of study and for how the findings might be used?
  • Poor.
  • The researcher claims that a teaching method is
    effective. However, the method was used in only
    one class, or in one school. There is no way to
    tell if it would be effective anywhere else.
  • Poor.
  • The researcher claims that a method is not
    effective. However, it was used in only one
    class, or in one school. There is no way to tell
    if it would have been effective elsewhere.
  • More

48
Is the Research Strategy Appropriate for the
Research Questions? Experiment
  • Poor.
  • The researcher claims that an instructional
    procedure or curriculum is more effective than
    another. However, the comparison groups (one
    class used one curriculum the other class used a
    different curriculum) were not created by random
    assignment of children to the groups, or by
    matching the groups to make sure they had, for
    example, the same percentage of boys and girls.
  • Therefore, the groups may have been different in
    other ways besides the curriculum e.g., more
    children in one group got reading instruction at
    home, and this (not the curriculum) caused
    differences in achievement.
    More

49
Is the Research Strategy Appropriate for the
Research Questions? Experiment
  • So, if research involves comparing (for example)
  • achievement from one group/sample (class,
  • school) to another, but the researchers did NOT
  • make sure (through random assignment of
  • students to the samples, or through matching)
    that
  • the groups have the same characteristics, then
    you
  • cant trust that differences in achievement are
  • caused by the curriculum or teaching method.

50
Is the Research Strategy Appropriate for the
Research Questions? Experiment
  • Here is one kind of experimental research that
    you
  • DO want to see. Level-1 research.
  • The researcher conducts a pilot study to see if a
    method
  • seems to work well enough that it ought to be
    given a more
  • valid test. The researcher uses a small sample
    (one class or
  • school). It is not a representative sample of
    the school
  • population, but that is alright. Do you want to
    use a method
  • that may not work on a lot of kids? The
    researcher finds that
  • the new method is followed by significant student
  • achievement. But since the sample is small and
    not
  • representative of the larger school population,
    and since
  • the research was done only once, the researcher
    cautions that
  • the findings are very tentative and cannot be
    generalized
  • anywhere else. Further research is needed.
  • This is called honesty.

51
Is the Research Strategy Appropriate for the
Research Questions? Experiment
  • Here is another kind of experimental research
    that you DO
  • want to see. Level-2 research.
  • The researcher repeats research on the same
    method in
  • different schools. Each new study is called a
    replication. The
  • sample of schools is representative of the
    population of
  • schools---small/large, wealthy/poor, rural/urban.
  • The researcher claims that data from most of the
    studies
  • show that the method is effective in most of the
    schools, but
  • the researcher cannot figure out which factors
    hinder its
  • effectiveness or what additional factors may be
    needed to
  • make it work better. The researcher advises
    caution in using
  • the method. This is honest and morally
    responsible.

52
Is the Research Strategy Appropriate for the
Research Questions? Experiment
  • Here is yet another kind of experimental research
  • that you DO want to see. Level-3 research.
  • The researcher conducts a study in several school
  • districts at once. The sample of districts is
  • representative of the larger population of
    schools and
  • school districts. Three new reading curricula are
  • tested against each other. The three curricula
    are
  • randomly assigned to the districts. Objective and
  • validated measures of essential reading skills
    are
  • made before the new curricula are introduced
    (pre-
  • test), every two weeks during the use of the
    curricula
  • (progress assessment). and at the end of the year
  • (post test, or outcome assessment.
    More

53
Is the Research Strategy Appropriate for the
Research Questions? Experiment
  • At the end of the year, researchers examine the
    amount of change in students skills from
    pre-test to post-test. They find that curriculum
    A produced significantly more achievement (and
    objectively high achievement) in all of the
    reading skills (compared to achievement produced
    by curricula B and C) in 8 out of the 10 schools
    in which it was used. The researchers therefore
    claim that curriculum A is more likely than the
    other curricula to produce high reading
    achievement, and that it can be used with at
    least modest confidence. However, they say that
    more research is needed to determine the school
    conditions in which curriculum A works best.
  • Would you use curriculum A?
About PowerShow.com