A Longitudinal Study of Student Skill Development

1
A Longitudinal Study of Student Skill Development

• Kathy Ray, University of the Pacific
• RUSA/MOUSS 10th Annual Reference Research Forum,
  June 2004

2
Why a Qualitative Longitudinal Study?

• Libraries are expected to show evidence of their
  impact on student learning as a quality indicator
  within their institutions
• Many libraries assessment studies measure the
  direct impact of instructional efforts many are
  quantitative approaches
• Few studies have tracked individual students over
  time to examine the development of research
  behaviors and skills few are in-depth,
  qualitative approaches

3
Why a Qualitative Longitudinal Study? contd.

• Explore student perspectives on research see
  the process, the library through student eyes
• Gain insight into student learning styles and
  other aspects of development
• Explore implications for design of library
  instruction and other library services

4
Use of Cohort and Longitudinal Studies

• Extensive use of cohort studies in human
  development and sociological research
• Few cohort studies in library literature
• Library-related longitudinal studies compare data
  sets but dont follow individuals over time
• Cohort studies are time consuming and require
  commitment over a multi-year period

5
Cohort Study at University of the Pacific

• The study originated as part of the Librarys
  assessment plan, in an effort to answer this
  question
• Do Pacifics students have the desired level of
  information competency skills at graduation?
• Information competency skill development is
  distributed across general education courses and
  is not centrally defined or assessed
• Based on the outcomes of the study, should the
  universitys General Education program be revised
  to include specific information literacy goals?

6
Research Questions

• Do students have the desired information
  competency skills at graduation?
• How do they develop their skills?
• What factors and experiences shape this
  development?
• How can student behaviors and development be
  explained (using learning, development theories)?
• What conclusions can be reached about the design
  of library instruction?

7
Research Design / Methodology

• Qualitative researcher as participant
• Phenomenological approach how do students
  experience the phenomenon of research, library
  use?
• Population and sample 19 student volunteers from
  a core GE course enrolling 700 freshmen
• Longitudinal data gathering
• structured interviews, recorded and transcribed
  field notes student papers (portfolio analysis)
  final competency quiz
• Confidentiality of participants maintained
• Coding of data and analysis
• Pay!

8
Some Observations on the Research Design

• Issues of validity how students want to be
  perceived by librarians (saying the right
  thing)
• End of semester fatigue level apparent in some
  interviews
• We have a limited view of the great variety of
  factors that influence students
• We must avoid generalization
• Enables librarian to conceptualize instruction at
  the level of the individual

9
Coding and Analysis

• An initial reading of the transcripts suggests
  categories, themes around which to organize data
• In a second reading, student responses and
  statements are coded according to the agreed upon
  set of themes
• Each student has his or her own unique experience
• Data cannot be generalized to describe the
  average student
• Themes help the researcher find both commonality
  and diversity of experience and begin to suggest
  meaning and interpretations of the experience

10
Interview Questions/Themes

• Why and how students pick research topics
• Where and how they look for information
• Research techniques Innate vs.taught
• Factors affecting skill development
• Inputs Student background characteristics
• Environment Teachers the curriculum other
  students, friends Library use, library
  instruction
• Learning styles
• Maturation (cognitive, ego, social, moral)

11
Cohort Demographics

• 19 students
• 11 female
• 8 male
• 18 first year students, 1 soph.
• Residence halls or Greek houses
• GPAs 2.0 to 3.8

12
Majors

• 2001
• Business 3
• Education 1
• Engineering 4
• Exploratory 5
• Music 1
• Pre-Pharmacy 2
• Pre-Med 1
• Psychology 1
• Religious Studies 1
• Sports Sciences
• English
• Sociology
• Graphic Design
• Communication

2002 2003 5 4 1 1 2
2 1 0 1 1 1
1 1 1
1 1 1 1 1
1 1 1
13
Where and how they were taught to look for
information
Other databases
Specialized encyclopedias
E.A.I.
Books
Web
Assignment
No FT
print journals
FT
read
read
copy
Expanded Academic Index
print/email
14
Where and how they look for information
Books
WEB
E.AE.A.I..I.
Other databases
Assignment
print out
No FT
copy paste
FT
read
print journal
e-mail
Expanded Academic Index
15
Why do you go straight to the Internet when
starting a research project?

• I definitely didnt have time to go look for
  books or anythingI just needed to be right there
  on my computer, get it printed out, read it,
  write the paper.

Its just most convenient for me and usually I
can find a lot of information. Ive done papers
where Ive used only Internet sites and Ive
gotten decent grades
16
Why do you go straight to the Internet when
starting a research project?

• On the Internet, it pretty much gets straight
  to the point. If I were to go to the library and
  try to find a book on Roman marriages, then I
  have to read a few chaptersits so much easier
  just to read a few lines, jump to the next
  paragraph, get an idea of what theyre talking
  about

17
Where is the first place you look for information?

• Well, actually, I like to check out books
  just because I know its the easiest thing go in
  here, look at the library catalog and find your
  books. Once you go through them you can go online
  and look for keywords that were in the books to
  help you specify what youre looking for.
• I find books helpful because you actually
  have something in your hand at 200 am. Sometimes
  Internet searches go bad.

18
Preliminary Findings 1st 2nd year Where and
How They Look for Information

• Web search - Yahoo, MSN, AOL, whatever
• Continue habits from high school (use of the
  Internet)
• Easy, convenient, can always find something
• Expanded Academic Index
• Taught how to use in library workshop teachers
  encouraged use
• Credible, scholarly articles most full-text
• Other databases frustrating and useless
• Books avoided by most
• Past papers things my friends or teacher gives
  me
• Barnes Noble innovative research strategy

19
Preliminary Findings 1st 2nd year Research
Techniques

• Keywords in basic search mode avoidance of
  advanced search features
• Some sophistication in narrowing a search
• Start at top of list and work down
• Follow links from one website to another
• Ask Jeeves (natural language)
• Ask for help friends, professor, librarian
• Browsing shelves (journals or books)

20
Preliminary Findings Varying Methods of Making
Judgments Regarding Sources

• High preference for credible full-text articles
• Just Right articles not too brief, not too
  long, not too difficult to understand
• Look at titles, abstracts, some skim article
  itself
• Find enough to fulfill assignment requirements
• No need to be comprehensive or look for the
  best articles
• Use papers written by other students
• Models for approximate length and complexity (and
  content?)
• Look for credible, reliable sources
• But judgment made on simple and usually
  superficial basis

21
Scholarly Journals

• I had a hard time sometimes. The articles
  wouldnt mention if they were scholarly or not so
  I basically had to ask my professor. Most of the
  articles used bigger words than I do so they all
  seemed pretty scholarly to me
• I figured if I had heard of the magazine, it
  wasnt scholarly.

22
Conflicting Information

• A One article said that emissions from hybrid
  cars were really low and another article said
  that the emissions were really high.
• Q So what did you do?
• A We used the one that supported our side.
• Q Did you feel it was a better argument? Did you
  look at the other one and worry that it might be
  correct?
• A Yeah, well, not really...

23
Preliminary Findings for 1st 2nd year students
Influential Factors Affecting Skill Development

• Habits developed in high school
• Experience using the Web on his/her own
• Use of high school or public libraries
• Strategies techniques learned by associating
  with other students
• Library workshops where a database is
  demonstrated (associated with certain types of
  projects or topics)
• Assignments directions from teachers (although
  greatly varying and sometimes unsound)
• Major course of study (sometimes)

24
Preliminary Findings Not-Very-Influential
Factors Affecting Skill Development

• Curriculum
• Few substantive research papers or assignments
  requiring use of scholarly resources
• Use of the library home page
• Library workshops
• Reference librarians

25
Preliminary Conclusions Implications for
Library Instruction

• 1st 2nd year students...
• need to be introduced to a greater variety of
  sources more than they need to be taught search
  techniques
• Some need instruction in specialized databases
• Most use limit features to find scholarly or
  refereed sources and full-text articles
• want a well-integrated research environment (SFX,
  etc)
• need a basic orientation to where things are
  located
• are not heavily invested in their research in the
  2 years
• But when a topic piques their interest and they
  master a body of material, intellectual growth is
  clearly evident.

26
Preliminary Conclusions Implications for
Library Instruction

• 1st 2nd year students...
• have a casual understanding of the ethics of use
  of information

What I usually do is just minimize them
(articles) so I can keep them right there. Ill
find a paragraph or a phrase that I want to
incorporate into my paper and Ill cut and paste
Ill go to Word and then Ill just have a whole
page of different quotes that Ill use. When I
start writing my paperIll cut and paste where I
want them to go and then rewrite it.
Ill take parts of it and paste it into another
document I think it can be confusing because
youre just cutting and pasting and then you have
to really decide what is plagiarism and what is
not.
27
3rd year students research strategies

• Think!
• Books
• Library databases
• Internet
• Talk to others
• Original research

28
Preliminary Findings in 3rd year Influential
Factors Affecting Skill Development

• Major course of study
• Closer relationships with professors higher
  expectations
• More engagement in assignments/projects more
  printing highlighting, less cut-n-paste
• More sophisticated understanding of various types
  of information
• Books for overview, articles for specialized
  info, web for images
• But sources needed to provide evidence and
  stuff
• Research paper not a universal experience
• Primary research, presentations, group projects

29
Preliminary Findings in 3rd year Influential
Factors Affecting Skill Development

• Internet sufficient for many assignments
• Often an opinion piece or analysis outside info
  not required
• Confident about ability to discern credible web
  sites
• I look for a logo or something professional
• Ethics well understood
• But time pressures cause them to rationalize
  short-cuts
• Appeal of Barnes and Noble
• All students rank higher than library

30
Barnes Noble Strategy 2nd year student
I got two of my best references from Barnes
Noble. I wasnt able to get any books here
because they were all checked out. So I borrowed
money from my Dad and went and bought these
books, and they were really, really helpful. The
majority of this paper is from those books. I
bought them and then I took them back. But
dont tell Barnes Noble!
31
Barnes Noble 3rd year student
I usually go to bookstores and they have
their own online catalog and at customer serviced
theyll say, we can order a book for you and if
you like it you can buy it but you dont have to,
and usually I wont but I can look at it and sit
there and drink coffee and stuff Ill ask
for a list of authors on nano technology and they
print out a whole list of all these different
authors. Once I have the names, I go to the
library and then I have these authors to type in.
32
Case Studies3rd year students

• Nora
• English major, Film studies minor
• Moki
• Computer Engineering/Engineering Mngmt, double
  major Physics minor
• Shura
• Sociology major, criminal justice emphasis

33
Final Goals

• Conduct portfolio analysis based on works
  collected over four years
• Compare student skill set in 4th year to desired
  competency level
• Relate observations on student development to
  development and library literature
• Disseminate results to campus faculty and
  profession
• Reconsider approaches to group and individualized
  instruction