Science students' critical examination of scientific information related to socioscientific issues

1
Science students' critical examination of
scientific information related to socioscientific
issues

• Stein Dankert Kolstø1, Berit Bungum2, Terje
  Kristensen2, Erik Arnesen1, Anders Isnes2, Ketil
  Mathiassen2,
• Idar Mestad1, Andreas Quale2,
• Anne Sissel Vedvik Tonning1, Marit Ulvik1
• 1University of Bergen, 2University of Oslo

2
Two agendas

• 1. Present a study
• Theoretical framework
• Method
• Findings
• Discussion
• 2. Discuss and exemplify qualitative analysis of
  qualitative data

3
Focus for the study

• Science for all (scientific literacy)
• Political dimensions of science (STS)
• Intellectual independence of experts and
  authorities
• Socioscientific issues (SSI)
• Critical assessment of scientific claims
• (Science for further education - not in focus
  here!)

4
The study

• 89 science teacher education students
• Article on the Internet of their own choice
• Related to a socioscientific issue
• To assess the reliability of scientific claims
• Present their assessment in a short text.

5
Theoretical perspectives

• Socioscientific issues
• Include two main questions
• 1 Political / ethical
• Should irradiation of food be legalised or not?
• 2 Scientific
• Do irradiated food have lower nutritional value?

6
Theoretical perspectives

• Constructivist epistemology
• From disputed claims from the frontier of
  research
• To reliable consensual core science
• Thus degrees of reliability
• Not necessarily correct or wrong

7
Framework for the analysis

• Criteria defined
• Values, norms or ideas appealed to when making a
  judgment
• Examples of criteria
• Consistency
• Relating findings to existing theories

8
Framework for the analysis

• Criteria for judging scientific claims
• Scientific criteria?
• Contextual criteria?
• Possible to differentiate?
• Underderermination of theories by empirical
  evidence
• Focused on identifying the kind of criteria used
  by the students
• all kinds focused on content and source!

9
From the literature Criteria for judging
scientific arguments

• Criteria referring to scientific values
• Empirical adequacy
• Internal and external theoretical consistency
• Contextual factors
• Competence, reputation, interests, personal
  qualities,
• Other?
• Completeness (publication), level of consensus

10
Research questions

• What criteria do the science teacher students use
  in their examinations?
• What knowledge do they draw upon in their
  evaluations?
• Which criteria might be relevant for inclusion in
  science education?
• Discussed in the paper

11
Method

• Data collection
• 89 science teacher education students
• Groups of 2 and 3
• Search the Internet for articles
• Socioscientific issue of interest

12
Method

• Task
• Write an evaluation
• Focus in particular on the trustworthiness of
  science-related claims
• Twenty-eight articles
• Between a half and a full written page

13
Method

• Analysis
• Analyzed using qualitative methods
• Constant comparative method
• Code and retrieve
• Inter-coder reliability of 69 percent

14
(Back stage Codes in three levels)

• 3 Main categories (4 groups of criteria)
• 2 Categories (the 13 criteria identified)
• Based on codes constantly compared for
  similarities and differences
• Judged to express similar judgments
• Categories and retrieved codes constantly
  compared to quotations
• 1 Codes
• Condensations of quotations which also included
  tentative indication of category (label for easy
  retrieve and inspection)

15
(Back stage Memos)

• Possible useful observations
• Tentative hypothesis
• Judgments about strategies in the analysis
• Keep track of process, like new categories
• Impression from the different texts
• Definitions of the evolving categories

16
Findings part I

• Criteria focusing on
• empirical and theoretical adequacy

17
Quality of references

• "relevant references
• "prominent scientific journals
• "trustworthy and respected sources",
• Gr.14. Because the article ... makes references
  to a great many concrete scientific articles, we
  consider, as a starting point, the article to be
  trustworthy.
• Requires knowledge enabling differentiation
  between different kinds of sources

18
Consistency of argumentation

• Logical correctness and consistency of an
  argument
• Gr.21 The article says nothing about the
  possibility that increasing number of instances
  of cancer might have other causes, like for
  instance more frequent sunbathing.
• Requires the idea that evidence and logical
  consistency is needed

19
Face validity of argumentation

• Focuses on the quality of arguments presented,
  but does not involve any focus on consistency
• Gr.18 The factuality of the text seems to be
  great, due to the numerical statements and
  evaluations, ....
• "a scientific method, "sounds reasonable"
• "detailed explanations
• "professional argumentation"

20
Compatibility with own scientific knowledge

• Many have put forward this claim as one of the
  most risky. How stable new genes are in the new
  organism is difficult to predict. It is proved
  that naturally occurring bacteria can transfer
  genetic material from one plant to another. This
  way you can not make a guarantee saying that a
  gene installed in one plant to increase its
  resistance against disease, herbicides or
  something of the sort can not be transferred to
  weed. In this way we might attain a super weed
  that will be difficult to conquer. Moreover, ...
• Presupposes relevant knowledge at a relatively
  sophisticated level

21
Findings part II

• Criteria focusing on
• completeness of information

22
Completeness of references

• Complained that an article wanted references for
  claims
• Gr.22 We can not find any direct references to
  these claimed research findings, and this is a
  weakness of this page. ... We want more links
  that could make it easier for us to check facts
  and claims.
• Relevant for differentiating between documented
  claims and mere guesses

23
Completeness of an argument

• Wanting explanations, details, counter-arguments
  or argumentation in general
• Gr.19 A discharge reduction of dioxin at 99,7
  with new cleaning technology is unquestionably
  very good, but the environmental organization
  does not state any reasons for why 0.027g/MT
  gram per megaton incinerated waste is an
  acceptable value.
• Emphasis the importance of requiring disclosure
  of arguments for further inspection

24
One-sidedness in the presentation

• Commented explicitly on the lack or presence of
  counter-arguments
• Gr.9 The article has a one-sided focus. The
  author writes a great deal about advantages
  related to the use of nutritional supplement, but
  mentions little about advantages of the
  alternative action of changing eating habits.

25
Findings part III

• Criteria focusing on social aspects

26
Possible underlying interests

• Make inferences from what they read to what might
  have caused a possible bias
• Gr.23 It should not be concealed that the
  specialized competence that the gene
  technologists possess can be used as a "smoke
  screen" in order to secure their own work.
  Especially in relation to the fact that gene
  technologists within the industry in general need
  to get their products and ideas sold, in order to
  make a living of their expertise.

27
Possible underlying interests

• economy and workplace
• professional prestige and position for experts
• loyalty to friends
• Requires knowledge about institutional
  characteristics

28
Personal value-related qualities

• Critical attitude towards "both sides"
• Gr.13 The critical attitude towards both
  homeopathy and scholastic medicine is one of the
  things which contribute to our impression of
  seriousness.
• Rare use of criteria like integrity,
  conscientiousness, honesty

29
Authors or cited experts competence

• Relevant education and current occupation
• Gr.24 The article emerges as trustworthy as it
  is medical practitioner name that states this.
  He is director at the name medical clinic and
  therefore he has some credibility.
• Place of publication to judge competence

30
Level of professional recognition

• Gr.25 It is made a reference to Dr. name as an
  acknowledged biochemist, but he has only
  published works about noni. It therefore emerges
  as improbable that he really is an acknowledged
  biochemist, and not a biochemist working on a
  contract for noni juice.
• Presupposes knowledge of different experts'
  prestige in science

31
Level of expert agreement

• About the standing of the science-related claim
• Gr.8 According to the researcher, this whale
  Keiko is not suited for a life outside
  captivity, and costs very much money. This
  corresponds to other researchers' utterances and
  experiences.
• Requires knowledge about the role of critique,
  argumentation and consensus in science

32
Findings part IV

• Criteria focusing on
• manipulative strategies

33
Appeals to emotions

• Whether deliberately manipulating readers
• Gr.25 The history of the noni fruit is used to
  show that it is a "natural" and therefore healthy
  product. They are again appealing to popular
  feelings and to the popular opinion that
  everything that is old is good.
• Used surprisingly infrequent

34
Additional findings

• Most student-groups did not conclude on
  reliability
• Yang (2004) attribute their uncertainty to
  insufficient information
• Echoed in this study
• Rare use of strategies for cross-checking claims
• Even if they worked online!

35
Summary

• Ma Manipulative str.
• Em Empirical and th.

• So Social aspects
• Co Completeness of inf.

36
Discussion

• Context-dependent findings
• Additional criteria in other contexts
• Certain criteria in the literature not used
  (internal consistency, simplicity, novelty, )
• Epistemic independence?
• Critical examination Yes!
• Cross-checking No!
• Relevance of criteria used?
• Both relevant and legitimate

37
Consequences for science education

• Critical examination of science dimension of
  issues
• Scientific content knowledge is not sufficient?
• The students draw upon their knowledge of
• Methodological norms in science
• Science content
• Social processes in science
• Institutional aspects of science
• Need to be included in the science curricula?

38
Implications for science teacher education

• Science teacher students weaknesses
• Rare use of strategies to cope with
  incompleteness of information
• Students evaluations varied considerably in
  depth and number of criteria used
• Facione (1990) critical thinking should be
  taught explicitly
• Including criteria!
• Need to bring more science teachers up to a
  proper level for teaching critical examination

39

• Thank you for your attention!
• The project was supported by a grant from the
  Norwegian Ministry of Education