Have been studied throughout a number of contexts (e.g., clinical, secondary and post-secondary education; Gould, 2006; Karabenick, 2004

1
Engaging in Rigorous Instrument Development to
Increase Validity of Assessment Results Civic
Learning Program in Judicial Affairs
Jilliam N. Joe Wendy M. Young Mary M.
Johnston
Snapshot of Civic Learning Program

1. Identify Construct-of-Interest

• General Help-Seeking

2. Establish Student Development Objective

• As a function of the Civic Learning Program,
  students will be more likely to increase
  attitudes toward help-seeking and increase
  help-seeking behavior.

3. Research Available Instruments
And when an off-the-shelf instrument cant
measure up
4. Select Instrument Development Team

• Judicial Affairs Assessment Committee
• Consultant from the Center for Assessment
  Research Studies

DeVellis, R.F. (2003). Scale development Theory
and applications, 2nd ed. Thousands Oaks,CA Sage
Publications.
Example Help-Seeking Items 32 item-instrument
Defining the Construct
Help-Seeking Threat1 To an individual with a
high level of help-seeking threat, the cost to
seeking help is depreciation in self-esteem.
Because seeking help implies inadequacy, it is
more inconsistent, and therefore more
threatening, to individuals with high self-esteem
than to individuals with low self-esteem
(Karabenick Knapp, 1991, p.222). Help-Seeking
Avoidance1 An individual with a high level of
help-seeking avoidance will choose to not seek
help even in the face of failure. Instrumental
Help-Seeking1 An individual who engages in
instrumental help-seeking does so when necessary
and in order to gain the least amount of
assistance possible in order to problem-solve
(e.g., asks for a hint). Executive
Help-Seeking1 An individual who engages in
executive help-seeking, on the other hand, does
so in order to minimize the cost (effort) of
completing the task on his own (e.g., asks for
direct answers).
Informal vs. Formal Help-Seeking1 Help-seeking
tendencies are directed toward either informal or
formal sources. That is, an individual will seek
help from different sources sources that are
formal (e.g. instructor) or informal (e.g. peer).
Confidence (Perception of Help-Source)2 Help
source perception is the degree to which an
individual has confidence in the help source.
Recognition of Need for Help2 Help-seeking
recognition is the acknowledgement of the need
for professional help in the face of a problem
(personal or emotional).
1 2 3 4 5 6 7 Not at all
Completely true true of me of me
HST When I need help, I dont care that others know.
HSA If I cant fully resolve a problem on my own, I would rather make a best guess than ask someone for assistance.
HSI I seek out help in order to more fully understand a problem.
HSE When I ask for help my primary reason is to reduce the amount of effort I have to put forth on my own.
HSSinf I typically seek out help from my friends rather than professionals on campus.
HSSfor If I seek help, I prefer to ask professionals on campus rather than another student.
HSP I believe that professionals on campus are too busy to help when I have a problem.
1 Karabenick, S.A. (2003). Seeking help in large
college classes A person-centered approach.
Contemporary Educational Psychology, 28, 37-58.
Karabenick, S.A. (2004). Perceived goal
achievement structure and college student help
seeking. Journal of Educational Psychology, 96,
569-581.
HSR I generally dont know I need help for a problem until someone points it out to me.
2 Fischer, E.H. Turner, J. L. (1970).
Orientations to seeking professional help
Development and research utility of an attitude
scale. Journal of Consulting and Clinical
Psychology, 35(1), 79-90.
Piloting Items Using Concurrent Think-Aloud
Method
Help-seeking tendencies
Results
Discussion As shown in Figure 1, participants
had a wide range of conceptualizations about
campus resource (e.g., student assistant,
roommate). In many ways, students interpreted
these items in the way the developers intended.
That is, the test developers wanted students to
consider campus resource in a broad sense and not
in a narrow sense (e.g., faculty members).
Participants interpretation of problem, on
the other hand, presented a challenge. How a
respondent might answer items that include the
word problem depends on the context. More often
than not, that context was academic (i.e., within
the classroom), which we believe was a result of
the language of some of the items that may have
implied a classroom context (e.g., task).
Moreover, whenever an item was vague, that is,
whenever the student said it depends on the
context, he or she would use the middle category.
Had we not conducted the think-aloud, we might
have interpreted middle category responses as a
neutral response and not as an I dont
understand or it depends proxy.
Method Participants Participants (N 3) used in
this exercise where volunteers from a residence
life program. The sample consisted of two (2)
females and one (1) male. Procedures Three (3)
think-aloud moderators were used for the exercise
(two female and one male). All three were members
of the assessment committee. Members were trained
on the think-aloud process and materials. A mock
think-aloud was conducted between the lead
moderator and another moderator to make sure
everyone had the same conceptualization of the
process. Data collection A concurrent verbal
report (think-aloud) method (Ericsson Simon,
1980 1993) was chosen for the purposes of this
exercise. Moderators met with students
one-on-one. Interviews lasted, on average, for
about one half hour. Afterwards, moderators
reconvened to debrief and provide general
impressions of students responses.

• Have been studied throughout a number of contexts
  (e.g., clinical, secondary and post-secondary
  education Gould, 2006 Karabenick, 2004 2003
  Deane, Wilson, Ciarrochi, 2001 Karabenick
  Knapp, 1991 Fischer Turner, 1970).
• Currently, however, there is no general measure
  of help-seeking in a college context.
• The objective of the item development process was
  to modify current measures used in academic and
  clinical settings to a more generalized setting.
• Throughout the item development process
• Members on the development panel noticed how some
  items may elicit different responses depending on
  the context to which students think the items
  refer (e.g., academic, personal/psychological).
• As such, it was deemed necessary to pilot items
  on students in a way that will help us to
  understand how items might be interpreted.

Analysis Each statement made by respondent were
analyzed. Initial codes were developed using
impressions drawn from a review of the data, as
well as observers field notes. Coding is meant
to be mutually exclusive each statement can
receive only one code. An analytic coding
approach was used (Richards Morse, 2006). Codes
served as keywords. Each time a statement with
depends for, example, was encountered, that
statement was highlighted.
HSA I would rather do worse on a task I couldnt finish, than ask for help.
Figure 1. Conceptual map of think-aloud analysis
for the Getting Help from Others Questionnaire