Three Evaluative Tools to Empower Local Communities in the Environmental Cleanup of Sediment Contami

1
Three Evaluative Tools to Empower Local
Communities in the Environmental Clean-up of
Sediment Contaminated Sites A Comparative
Analysis EPA ORD 11 May, 2005
2
The Project Team

• Dr. Seth Tuler, Ph.D., SERI
• Dr. Thomas Webler, Ph.D., SERI
• Dr. Susan Santos, Ph.D., University of Medicine
  Dentistry of New Jersey, School of Public Health,
  New Brunswick, NJ
• Dr. Caron Chess, Ph.D., Department of Human
  Ecology, Rutgers University
• Stentor Danielson, Department of Geography, Clark
  University

3
Presentation Outline

• Evaluation in Superfund
• The research question
• Our approach
• Expected results from project
• Case Study
• Ciba-Geigy Chemical Corporation site in Toms
  River, NJ
• Preliminary findings
• Next steps

4
Why Evaluate?

• To improve
• the community involvement effort
• clean-up decisions
• To learn from past mistakes
• To give voice to community members
• To justify and account for past efforts

5
EPA Already Evaluates at Superfund Sites

• Informal evaluation
• Interviews
• Meeting feedback forms
• Formal evaluation
• Surveys sponsored by the EPA Community
  Involvement and Outreach Center (CIOC)
• EPA evaluation is not systematic, and may not be
  adequate.

6
Connection of Evaluation to Empowerment

• Evaluation can focus attention on community needs
  and concerns as they emerge (formative
  evaluation)
• Community participation can inform evaluation
  (participatory evaluation)
• Evaluation can lead to community empowerment

7
The Basic Research Question

• What are the strengths and weaknesses of three
  different methods for evaluation and their role
  in community empowerment?
• Surveys
• Focus Groups
• Q method

8
Our Approach

• Apply three evaluation methods in two case
  studies
• Case studies
• Contaminated sediment sites
• Region 1 and Region 2
• Completed RI and FS
• Three methods
• Surveys
• Focus groups
• Q method
• Assess their relative strengths and weakness?
• From participants
• From systematic comparison on key dimensions

9
Expected Results

• Identify the strengths and limitations of the
  three methods for getting feedback on
• effectiveness of community involvement
• preferences for outcomes related to clean-up
• NOT an evaluation of the agency or PRP
• NOT an evaluation of community involvement at the
  site

10
Case Study 1 Overview

• Ciba-Geigy Chemical Corporation site in Toms
  River, New Jersey
• Site to manufacture dyes, pigments, resins and
  epoxy additives (1952 to 1990)
• On-site disposal of drums
• Contamination leaching into groundwater, which is
  tapped by municipal, industrial, and private
  wells
• Groundwater treated on-site (ROD in 1989, ESD in
  1993)
• Drum removal and soil bioremediation on-site
  began in 2004 (ROD in 2000)
• Cancer cluster assessments, lots of initial
  controversy and distrust
• Highly engaged community groups, technically
  informed
• High level EPA Region 2 administrative presence,
  stable staff

11
Case Study 1 Steps

• Background (scoping) interviews
• site history
• identify key participants
• Create evaluation protocols
• Implement each evaluation method
• Analyze data
• Prepare draft reports of findings
• Gather feedback from participants about
  usefulness, relevance, etc. of findings.

12
Toms River Mail Survey

• CIOC Protocol
• CIOC OMB-approved survey
• Potential respondents created from two lists
• EPA Mailing List
• Abutters List
• Up to four contacts
• Initial letter
• Survey
• Postcard reminder
• Replacement survey

13
Mail Survey Responses

• Total Sample
• 604 potential respondents
• 191 usable responses
• 34 response rate
• EPA mailing list responses
• 280 individuals
• 40.2 105 responses
• Abutters list responses
• 324 individuals
• 28.6 86 responses

14
Focus Groups

• A qualitative research method that is essentially
  a group interview technique.
• A guided group discussion to generate an
  understanding of participants experiences and
  beliefs
• Focus groups allow
• exploration and discovery among different groups
• exploration of complex behavior or topics and
  motivations
• enable discussion to generate new ideas/topics
  while providing structure
• provide greater insight into why certain opinions
  are held

15
Toms River Focus Groups

• Highly Involved Activists and community group
  leaders (8)
• Officials EPA, DEP, County and Township
  government (7)
• General Public Names selected from EPA mailing
  list had some involvement (e.g. went to a
  meeting) (8)

16
Content Areas for Discussion

• General awareness about the site and its
  remediation
• Views about the community involvement process
• Opportunities for involvement
• Purpose of community involvement
• Measures of success and satisfaction with
  outcomes
• Suggestions for improvement

17
Q Method

• Provides holistic perspective of a person's
  subjectivity regarding a research question.
• Maintains an individuals responses as a whole
  rather than disaggregating responses according to
  various traits.
• Identifies underlying collective perspectives and
  the extent to which individuals subscribe to
  them.

18
Toms River Q Sorts

• Selection of participants (19)
• sort 54 statements in 11 categories about the
  community involvement process
• sort 42 statements in 11 categories about
  remediation outcomes
• interview about sort
• identify social perspectives about good process
  (factor analysis)
• validate narrative descriptions with participants

19
Q sort about community involvement

• We asked people for their views of what should be
  done next
• When you think about where the process is now,
  what should happen next? Sort the statements
  according to most like I think the process needs
  to be to least like I think the process needs to
  be.

20
Q sort about outcome preferences

• We asked people for their views about the
  outcomes and remediation
• When you think about the remediation of the
  Toms River Ciba Geigy site, what do you think
  about what has been done in the past and is being
  done currently? Sort the statements according to
  most like I think to least like I think.

21
Preliminary Findings

• Evaluation Methods Feedback Group
• Project Team Observations/Reflections

22
What We Learned from the Feedback Group (5 people)

• Each of the methods provided some information
  that people found interesting and useful.
• Any report must be free of numbers or statistics.
  Graphs are good.
• Be aware of the susceptibility of the method to
  intentional manipulation. Surveys and Q sorts
  are vulnerable, focus groups are more robust.
• Four people liked the focus group report the
  most. One person liked the Q report the most.

23
What We Learned from the Feedback Group (cont.)

• Survey was useful to EPA to tell them what the
  community at large was thinking.
• Survey might be useful as an outreach tool it
  might get people interested so they would want to
  find more out.
• The Q report gave perspectives of the community
  and those were seen as somewhat useful to people,
  although they were seen as a snap shot in time
  and some noted that periodic snap shots should be
  taken.
• Q process was seen as illegitimate because
  statements are open to interpretation.

24
What We Learned from the Feedback Group (cont.)

• The FG report was the most legible and
  comprehendible. Most felt that it provided the
  best information.
• Focus groups are very useful for having a
  dialogue among stakeholders.
• But, focus groups would be hard to do at a time
  when people did not trust each other or when
  emotions were running high. But, focus groups
  can be used early on to bring up and discuss
  controversies.
• Some people may not feel comfortable talking on
  record during a focus group.
• Elected officials will not come to a focus group.

25
Whose Experiences Tapped
26
What We Learn and Time Requirements
27
Our ObservationsStrengths and Limitations

• Survey
• Low response rates may mean results are less
  meaningful about a population
• Good at revealing perspectives that may not stand
  up to the scrutiny of peers in an open dialogue
• No opportunity to learn from others during the
  process
• Basic statistical tests used, easy to learn
• Focus Groups
• Good at getting rich history, basis for
  justification
• Can descend into a complaint session or argument
• could be a disaster if there is not enough trust,
  high conflict among participants
• How much heterogeneity is appropriate and what
  kinds?
• If people don not show up to a focus group, you
  loose their input
• Q Method
• Good at revealing perspectives that may not stand
  up to the scrutiny of peers in an open dialogue
• Prior knowledge needed
• No opportunity to learn from others during the
  process
• Non-representative sample could bias findings
• Unfamiliarity with method can be confusing,
  although our experience shows that people find
  interesting
• Interpretation of results of unfamiliar method
  can be confusing
• Analysis can be complex, researcher skills needed

28
Next Steps

• Second case study
• Modify methods (esp. Q) and further consider
  issues of empowerment
• Comparative analysis of case study findings
• Recommendations and guidance