Methodology and Design in Heritage Language Research

1
Methodology and Design in Heritage Language
Research

• Tania Ionin, UIUC
• tionin_at_illinois.edu
• Third Heritage Language Summer Institute
• June 22, 2009

2
Conducting research with heritage language (HL)
speakers some goals

• To describe and explain the state of HL learner
  knowledge, across different domains of the
  grammar
• To compare HL learners to other learner groups
  (children, or second language learners)
• To determine how the state of HL learner
  knowledge is affected by such factors as
• amount and type of exposure
• oral and/or written proficiency
• literacy
• years of instruction
• age of exposure to the majority language
• To examine how the HL is affected by
  cross-linguistic influence from the dominant
  language (English)
• To develop pedagogical materials that are
  appropriate for HL learners of different levels

3
Typical HL speaker in the U.S.

• Exposure to the HL begins at birth
• HL speakers are born in the U.S. or move here as
  children
• Exposure to the majority language (English)
  begins at birth or in early to middle childhood
• Input in the HL is interrupted or decreases as
  input in English increases
• Input in the HL is primarily in the home, from
  family members
• Schooling is (typically/mostly) in English
• HL speakers may or may not be literate in the HL
• HL speakers may be exposed to a non-standard
  variety of the HL, and/or to informal /
  colloquial registers
• For more discussion, see Montrul 2002, 2008
  Polinsky, 1997 Polinsky and Kagan 2007
  Silva-Corvalan 2003 among others.

4
What does this mean for research with HL learners?

• Many research methodologies have been developed
  for use with
• adult native speakers (cf. Cowart 1997)
• adult second language (L2) learners (cf. Mackey
  and Gass 2005)
• young children (cf. Crain and Thornton 1998)
• Where do adult HL speakers fit in?
• HL speakers are less likely to be literate in the
  language under investigation than adult
  monolinguals or L2-learners
• HL speakers may be more familiar with a
  non-standard / dialectal variety of the language
• the variation in proficiency levels and literacy
  among HL speakers is very wide
• Relevance for research methodology
• formal, written tasks may not be appropriate for
  some groups of HL speakers
• the researcher needs to consider what dialectal
  variety HL speakers use
• the researcher needs to decide what level(s) of
  proficiency / literacy to target

5
Goals of this presentation

• To give an overview of types and components of
  research design for research with language
  learners
• To discuss specific components of a language
  study, including control groups, proficiency
  measures, and language background questionnaires
• To discuss data collection methods commonly used
  with first and second language learners
• To discuss the advantages and disadvantages of
  each data collection method
• To discuss the appropriateness of different data
  collection methods for use with heritage language
  learners
• To briefly discuss components of classroom
  intervention research

6
Outline of the presentation

• Research design
• Types of research design
• Research questions, hypotheses, and predictions
• Independent and dependent variables
• Components of a language study
• Control group
• Proficiency test
• Language background questionnaire
• Data collection methods
• Naturalistic production tasks
• Elicited production tasks
• Comprehension tasks explicit measures
• Comprehension tasks implicit measures
• Classroom intervention research
• Suggested readings

7
Outline of the presentation

• Research design
• Types of research design
• Research questions, hypotheses, and predictions
• Independent and dependent variables
• Components of a language study
• Control group
• Proficiency test
• Language background questionnaire
• Data collection methods
• Naturalistic production tasks
• Elicited production tasks
• Comprehension tasks explicit measures
• Comprehension tasks implicit measures
• Classroom intervention research
• Suggested readings

8
Types of research design

• Longitudinal
• follow one subject or a small group of subjects
  over time
• Cross-sectional
• test a group of subjects at a single point
• Mixed
• test a group of subjects several points,
  comparing results both across subjects and over
  time

9
Types of research design

• Naturalistic
• observe subjects in their natural setting
• Experimental
• set up particular tasks that subjects participate
  in
• Classroom intervention
• include pre-tests and post-tests
• measure the effect that an intervention has on
  the students performance

10
Outline of the presentation

• Research design
• Types of research design
• Research questions, hypotheses, and predictions
• Independent and dependent variables
• Components of a language study
• Control group
• Proficiency test
• Language background questionnaire
• Data collection methods
• Naturalistic production tasks
• Elicited production tasks
• Comprehension tasks explicit measures
• Comprehension tasks implicit measures
• Classroom intervention research
• Suggested readings

11
Research questions, hypotheses and predictions

• Any research project starts with a research
  question
• A research question may turn into a hypothesis
  an expectation, based on theoretical
  considerations and/or prior findings, on what
  learners knowledge and/or performance will be
• A hypothesis generates one or more predictions
  for a particular group of learners and a
  particular experimental design and data
  collection method

12
A hypothetical example acquisition of Case
morphology in heritage Russian

• Research questions
• general RQ How well do Russian HL speakers
  command the different Russian case forms (e.g.,
  Nominative vs. Accusative vs. Dative)?
• specific RQ 1 do errors consist primarily of
  overuse of one default case forms (e.g.,
  Nominative), or do Russian HL speakers use
  different case forms interchangeably?
• specific RQ 2 are Russian HL speakers more
  accurate on case forms in production or in
  comprehension?
• specific RQ 3 is knowledge of Russian case
  forms related to literacy?

13
A hypothetical example acquisition of Case
morphology in heritage Russian

• Hypotheses
• H1. Errors with case forms largely consist of
  overuse of Nominative case in place of other case
  forms.
• H2. HL speakers command of Russian case forms is
  better in comprehension than in production.
• H3. There is a positive relationship between
  literacy and knowledge of case forms

14
A hypothetical example acquisition of Case
morphology in heritage Russian

• Prediction based on H1
• naturalistic design in naturalistic production,
  Nominative case forms will be used in place of
  other case forms, but not vice-versa
• Predictions based on H2
• experimental design Russian HL speakers will be
  more accurate at judging the (un)grammaticality
  of case forms in a grammatlicality judgment task
  than at producing appropriate case forms in an
  elicited production task
• Prediction based on H3
• naturalistic design high-literacy Russian HL
  speakers will produce a greater variety of case
  forms than low-literacy Russian HL speakers

15
Outline of the presentation

• Research design
• Types of research design
• Research questions, hypotheses, and predictions
• Independent and dependent variables
• Components of a language study
• Control group
• Proficiency test
• Language background questionnaire
• Data collection methods
• Naturalistic production tasks
• Elicited production tasks
• Comprehension tasks explicit measures
• Comprehension tasks implicit measures
• Classroom intervention research
• Suggested readings

16
Hypotheses and variables

• Variables
• features that change in the experiment
• Null hypothesis there is no relationship between
  the variables
• Research hypothesis there is a relationship
  between the variables
• The goal of statistical analysis to show that
  there is support for the research hypothesis

17
Types of variables

• Independent variable
• is manipulated by the experimenter (by assigning
  subjects to different groups, by constructing
  sentences of different types, etc.)
• may have an effect on the dependent variable
• Dependent variable
• is not manipulated by the experimenter
• but may change under the influence of the
  independent variable
• There can be more than one of each!

18
A hypothetical example production

• The study HL learners of Spanish are tested on
  their production of gender marking on Spanish
  articles
• RQs
• Are Spanish HL learners more accurate at
  producing the right article with masculine or
  feminine nouns?
• Is accuracy greater for Spanish HL learners who
  have had at least one year of schooling in
  Spanish, compared to those who have had no
  schooling in Spanish?
• Independent variables
• between-subjects independent variable schooling
  in Spanish (yes vs. no)
• within-subjects independent variable type of
  noun (masculine vs. feminine)
• Dependent variable the rate of suppliance of the
  target article

19
A hypothetical example comprehension

• The study HL learners of Korean and monolingual
  Korean speakers are asked to judge the
  (un)grammaticality of Korean sentences with
  correct vs. incorrect word order, using a scale
  from 1 (incorrect) to 4 (correct)
• Research questions
• Do Korean HL learners appropriately distinguish
  between grammatical and ungrammatical word
  orders?
• How do Korean HL learners compare to monolingual
  Korean speakers in their sensitivity to word
  order (un)grammaticality?
• Independent variables
• between-subjects independent variable language
  group (heritage speaker, or monolingual speaker)
• within-subjects independent variable sentence
  type (grammatical vs. ungrammatical)
• Dependent variable mean sentence rating on a
  scale from 1 to 4

20
Outline of the presentation

• Research design
• Types of research design
• Research questions, hypotheses, and predictions
• Independent and dependent variables
• Components of a language study
• Control group
• Proficiency test
• Language background questionnaire
• Data collection methods
• Naturalistic production tasks
• Elicited production tasks
• Comprehension tasks explicit measures
• Comprehension tasks implicit measures
• Classroom intervention research
• Suggested readings

21
Comparison groups

• Adult HL learners may in principle be compared to
  any of the following groups
• age-matched fully fluent native/monolingual
  speakers
• age-matched and proficiency-matched second
  language (L2) learners (e.g., HL Spanish speakers
  vs. L2-Spanish learners)
• another group of HL learners in a different
  learning situation (e.g., uninstructed vs.
  instructed HL speakers)
• What would we learn from each of these
  comparisons?

22
Comparison groups

• Comparing HL speakers to fully fluent native /
  monolingual speakers
• important to ensure that the test instrument
  works as expected that fluent native speakers
  perform at or near ceiling
• fluent native speakers performance provides
  information about the target
• Comparing HL speakers to proficiency-matched
  L2-learners
• teases apart the effects of proficiency from the
  effect of type of exposure (naturalistic vs.
  instructed) and/or age of exposure (in childhood
  vs. adulthood)
• cf. work by Silvina Montrul
• Comparing two groups of HL speakers
• can compare groups on such factors as amount of
  input, proficiency level, degree of instructed
  learning in the HL, etc.

23
Outline of the presentation

• Research design
• Types of research design
• Research questions, hypotheses, and predictions
• Independent and dependent variables
• Components of a language study
• Control group
• Proficiency test
• Language background questionnaire
• Data collection methods
• Naturalistic production tasks
• Elicited production tasks
• Comprehension tasks explicit measures
• Comprehension tasks implicit measures
• Classroom intervention research
• Suggested readings

24
Proficiency measures

• HL learners range widely in proficiency (cf.
  Montrul 2008)
• from learners who do not produce the language but
  can comprehend it, to learners to sound nearly
  fluent
• from learners who have no literacy in the
  language to those who can read and write quite
  well
• The choice of data collection measure greatly
  depends on proficiency
• written tasks can be used only with literate
  learners
• oral production tasks are not appropriate for
  learners who cannot speak the HL
• And it is important to use proficiency measures
• to check whether a given group of learners is
  homogeneous in proficiency
• to divide learners into proficiency levels and
  examine performance at each level

25
Oral proficiency measures

• Participants tell an oral narrative, often based
  on pictures
• The oral narrative is transcribed, and various
  measures can be taken to estimate proficiency
• Speech rate
• Mean length of utterance (MLU)
• Lexical complexity (e.g., type-token ratio)
• Syntactic complexity (e.g., proportion of
  embedded clauses)
• See Polinsky (1997, 2008), Kagan and Friedman
  (2004), for use of these measures with Russian HL
  learners
• See Unsworth (2008) on a measure that combines
  lexical and syntactic complexity
• developed for child-adult comparisons in second
  language acquisition, but also potentially
  applicable for use with HL learners

26
Oral proficiency measures

• Frog story books (Mayer 1969, 1973, etc.)
• well-established methodology in the field of
  L1-acquisition (Berman 1988, Berman Slobin
  1994)
• also used with heritage language learners (e.g.,
  Kagan and Friedman 2004)
• participants tell a story based on the pictures,
  the story is transcribed
• advantage comparable length and content across
  participants
• Example from English two L1-Russian L2-English
  adult learners describing the same pictures in a
  frog story book (pilot study by Ionin and
  Zubizarreta)
• SUB7 Once upon a time it was a littles boy
  birthday. His parents got him a big box as a
  gift. He couldnt wait to open and see whats in
  there. Once he open it he was very surprised to
  see a little frog on the bottom of the box. A
  little boy was so happy with his present. But a
  big frog was not happy with this present. She was
  jealous of the new present the new little frog.
• SUB3 So the boy on a picture got a present and
  he want to open the box. He open the box and saw
  a frog in the box. He really happy and he got a
  frog out of the box. He put the frog on the
  ground. And hes waiting what happen next. So
  there was another frog there who probably was his
  pet before. Hes not happy about another frog
  there.

27
Written proficiency measures

• Vocabulary test
• Cloze test
• a passage of text, with every nth word replaced
  with a blank
• researchers examine learners ability to fill in
  the blank with an appropriate word
• Example (English cloze test created by William
  Rutherford, used by Ionin et al. 2008 for a study
  of L2-English)
• excerpt When President Lincoln was shot, the
  word____ communicated by telegraph to most parts
  of the United ____, but because we had no links
  to England, it ____ five days before London heard
  of the event.
• For use of vocabulary and cloze tests with
  Spanish HL learners, see Montrul et al. 2008

28
Which proficiency test to use?

• Oral narration tasks
• do not require learner literacy
• measure production rather than comprehension
• allow comparisons of children and adults
• Written cloze tests
• require learner literacy
• measure both production and comprehension
• allow comparisons of adult HL learners with adult
  L2 learners

29
Outline of the presentation

• Research design
• Types of research design
• Research questions, hypotheses, and predictions
• Independent and dependent variables
• Components of a language study
• Control group
• Proficiency test
• Language background questionnaire
• Data collection methods
• Naturalistic production tasks
• Elicited production tasks
• Comprehension tasks explicit measures
• Comprehension tasks implicit measures
• Classroom intevention research
• Suggested readings

30
Language background questionnaires

• Background questionnaires are crucial to any
  language study
• Some questions that should be asked in any study
  with native and/or non-native speakers
• age
• gender
• level of schooling
• languages known, and degree of fluency in each
  one
• native / family language
• Some questions specific to heritage language
  speakers
• self-ratings in both the HL and English
• self-report on which language is dominant
• family linguistic history (languages spoken by
  parents)
• language(s) spoken during childhood
• language(s) of schooling
• country of birth and if applicable, age of U.S.
  arrival
• age of first exposure to English

31
Uses of language background questionnaires

• Determining which participants meet the criteria
  for inclusion
• Determining how to classify participants into
  groups based on type of exposure, dialect, etc.
• Assessing HL learner proficiency based on
  self-ratings and self-reports (should be used in
  conjunction with proficiency tests)

32
Outline of the presentation

• Research design
• Types of research design
• Research questions, hypotheses, and predictions
• Independent and dependent variables
• Components of a language study
• Control group
• Proficiency test
• Language background questionnaire
• Data collection methods
• Naturalistic production tasks
• Elicited production tasks
• Comprehension tasks explicit measures
• Comprehension tasks implicit measures
• Classroom intervention research
• Suggested readings

33
Some methodological issues to keep in mind

• Does the task measure oral or written language?
• Does the task measure production or
  comprehension, or a mixture of both?
• Does the task test primarily explicit knowledge
  or implicit knowledge?
• Which aspects of linguistic knowledge is the task
  most appropriate for?

34
Measuring explicit vs. implicit knowledge (cf.
Ellis 2005)

• More explicit tasks are likely to
• Give the learners time to think about their
  responses
• and/or Require metalinguistic awareness (e.g.,
  judging a sentence as (un)grammatical,
  conjugating a verb, etc.)
• and/or Make it fairly easy for the learner to
  guess what aspect of the grammar is being studied
• and/or Use written format
• More implicit tasks are likely to
• Put time pressure on the learners
• and/or Ask learners to perform activities that do
  not appear related to grammar (e.g., picture
  description, judging the truth or falsity of a
  sentence, etc.)
• and/or Use oral format

35
Measuring explicit vs. implicit knowledge

• Tasks measuring explicit knowledge are generally
  more straightforward and easier to construct.
• But can we be sure that such a task measures
  what the learner truly knows about language on an
  underlying level?
• On the other hand, a task measuring implicit
  knowledge may not always tap into the precise
  area of linguistic knowledge that we are
  interested in.

36
Making tests less explicit use of fillers

• The target items test what the experimenter is
  interested in
• The filler items test something else, in order to
  prevent the learner from focusing on a particular
  grammatical structure
• The target items and filler items should be
  similar in structure so that the subject cant
  tell which are which

37
Areas of linguistic knowledge

• Phonetics/phonology the sound system
• The lexicon word knowledge
• Morphology word construction
• Syntax grammar
• Semantics meaning
• Pragmatics language use
• Focus today primarily on the middle four

38
Todays focus

• Task type
• focus on offline tasks, which measure accuracy in
  production or comprehension
• will not discuss online tasks, which measure
  reaction times as well as accuracy (see, e.g.,
  Sekerina et al. 2008)
• Research design
• focus primarily on cross-sectional experimental
  design
• brief discussion of naturalistic and classroom
  research designs

39
Outline of the presentation

• Research design
• Types of research design
• Research questions, hypotheses, and predictions
• Independent and dependent variables
• Components of a language study
• Control group
• Proficiency test
• Language background questionnaire
• Data collection methods
• Naturalistic production tasks
• Elicited production tasks
• Comprehension tasks explicit measures
• Comprehension tasks implicit measures
• Classroom intervention research
• Suggested readings

40
Naturalistic production tasks

• Spontaneous speech production
• in natural conversation (the learner chooses the
  topic)
• in interactions with friends/family members
• Controlled naturalistic production (oral or
  written)
• describing pictures (e.g., in frog story books)
• answering specific questions
• constructing narratives on a given topic
• etc.

41
Naturalistic production tasks can be used to
study

• overall language proficiency (e.g., through
  speech rate, cf. Kagan and Friedman 2004)
• lexical knowledge
• inflectional morphology
• e.g., use of past-tense morphology can be studied
  through narratives about past events
• basic syntax (e.g., word order)
• but not complex sentences, which are unlikely to
  be produced spontaneously very often

42
Naturalistic production tasks

• Advantages
• very easy to conduct the study
• the closest we can come to measuring language as
  its naturally used
• measure implicit knowledge subjects do not know
  what they are being tested on
• Disadvantages
• very time-consuming to create and analyze
  transcripts
• the experimenter does not have much control over
  the structures produced
• low-proficiency learners may be very reluctant to
  produce anything
• absence of evidence is not evidence of absence
  learners might know a form but fail to produce it
  
• Appropriateness of use with HL learners
• generally appropriate
• but difficult to use with low-proficiency HL
  speakers (see above)

43
Outline of the presentation

• Research design
• Types of research design
• Research questions, hypotheses, and predictions
• Independent and dependent variables
• Components of a language study
• Control group
• Proficiency test
• Language background questionnaire
• Data collection methods
• Naturalistic production tasks
• Elicited production tasks
• Comprehension tasks explicit measures
• Comprehension tasks implicit measures
• Classroom intervention research
• Suggested readings

44
Elicited production tasks

• Oral format
• provide a context, ask a question, and prompt the
  beginning of the answer
• provide a picture, ask for a description
• Written format
• provide the context, replace individual words or
  phrases with blanks (cloze test format)
• some variations give a choice of lexical items
  provide an uninflected form of verb or noun and
  instruct learner to use it appropriately

45
Example elicited oral production in Spanish
(Montrul et al. 2008)

• Target Veo una banana madura

46
Example elicited written production in English
(Ionin et al. 2008)

• Barbara Did Betsy get anything at the bookstore
  yesterday?Rick Yes she bought a novel and a
  magazine. She read ___the__ magazine first.
• Fillers elicit other lexical items pronouns,
  negation, prepositions

47
Elicited production tasks can be used to study

• overall language proficiency (cloze tests)
• lexical knowledge
• inflectional morphology
• morphology for gender, case, tense, etc.
• semantics of closed-class words
• use of the vs. a
• use of her vs. herself

48
Elicited production tasks

• Advantages
• more control over type of production than with
  naturalistic production
• much easier to analyze the data
• can target specific words/phrases
• Disadvantages
• may be testing explicit rather than implicit
  knowledge (especially with written tasks)
• the most controlled type of elicited production
  is written ? not appropriate for use with
  children or low-literacy subjects
• it may not be possible to elicit particularly
  complex syntactic structures
• Appropriateness of use with heritage speakers
• written format not appropriate for speakers with
  low literacy
• otherwise appropriate

49
Outline of the presentation

• Research design
• Types of research design
• Research questions, hypotheses, and predictions
• Independent and dependent variables
• Components of a language study
• Control group
• Proficiency test
• Language background questionnaire
• Data collection methods
• Naturalistic production tasks
• Elicited production tasks
• Comprehension tasks explicit measures
• Comprehension tasks implicit measures
• Classroom intervention research
• Suggested readings

50
Explicit measures of comprehension

• Grammaticality judgments
• Acceptability judgments
• Preference judgments
• Learners make a conscious decision about the
  grammaticality and/or appropriateness of a
  sentence
• Fillers are used to ensure that the learner does
  not become too focused on just one construction
  type
• A dont know option should be provided to avoid
  random guessing
• The task may be timed or untimed

51
Hypothetical example use of negation

• Grammaticality judgment rate each sentence as
  grammatical or ungrammatical
• Mary did not finish her homework.YES NO DONT
  KNOW
• John not wash the dishes.YES NO DONT KNOW

52
Hypothetical example use of negation

• Acceptability judgment rate each sentence as
  acceptable or not, on a scale from 1 to 4, where
  1 is least acceptable and 4 is most acceptable
• Mary did not finish her homework.1 2 3 4 DONT
  KNOW
• John not wash the dishes. 1 2 3 4 DONT KNOW
• NB an alternative is to have a
  negative/positive scale -2, -1, 0, 1, 2? but
  then the zero response is likely to be used when
  the learner is uncertain

53
Hypothetical example use of negation

• Preference task for each pair of sentences
  below, state which one is more acceptable
• A. Mary did not finish her homework.B. Mary not
  finished her homework
• Choose one
• A is better
• B is better
• A and B are equally good
• Dont know

54
Explicit measures of comprehension can be used to
study

• complex sentence structures (that are not often
  produced spontaneously)
• embedding, relative clauses, complex
  wh-questions
• inflectional morphology
• semantic appropriateness
• whether one grammatical sentence is acceptable in
  the context of another

55
Explicit measures of comprehension

• Advantages
• can test just about any kind of grammatical
  construction
• very straightforward testing exactly what we
  want to know
• easy to analyze the data (especially if test is
  computerized)
• Disadvantages
• tests highly explicit knowledge ? may not be
  getting at what the learner knows at a more
  underlying level (but timing the task makes it
  less explicit)
• learners are often uncertain about their
  judgments
• it can be hard to distinguish uncertainty from a
  true judgment of neither good nor bad
• not appropriate for use with young children or
  low-literacy subjects
• Appropriateness of use with heritage speakers
• very formal task, may be inappropriate for
  learners exposed primarily to an informal variety
  of the language
• not appropriate for learners with low literacy
  and/or lack of schooling in the language
• but, appropriate for literate, schooled HL
  learners

56
Outline of the presentation

• Research design
• Types of research design
• Research questions, hypotheses, and predictions
• Independent and dependent variables
• Components of a language study
• Control group
• Proficiency test
• Language background questionnaire
• Data collection methods
• Naturalistic production tasks
• Elicited production tasks
• Comprehension tasks explicit measures
• Comprehension tasks implicit measures
• Classroom intervention research
• Suggested readings

57
Implicit measures of comprehension

• Truth-value judgments
• Picture-matching
• Learners perform tasks which do not appear to
  have anything to do with grammar
• But the results provide information about how
  learners interpret individual sentences
• Appropriate for testing interpretation rather
  than grammaticality

58
Truth-value judgment tasks (TVJTs)

• Learners see or hear a target sentence in the
  context of a story (either written or narrated by
  the experimenter) and/or a picture
• Learners have to state whether the target
  sentence is true or false in the context of the
  story
• The set-up must be such that in order to give the
  appropriate response, the learner needs to assign
  a particular grammatical structure to the
  sentence (i.e., the correct response cannot be
  deduced from the context alone)

59
An example interpretation of definite plurals
(Ionin and Montrul 2009, under review)

• English The tigers like meat
• YES specific interpretation these tigers like
  meat
• NO generic interpretation tigers (as a kind)
  like meat
• Spanish A los tigres les gusta la carne. 
• YES specific interpretation these tigers like
  meat
• YES generic interpretation tigers (as a kind)
  like meat
• L1-transfer would cause L1-Spanish L2-English
  learners to accept the generic interpretation for
  the tigers

60
Why use a TVJT here?

• A grammatical acceptability test would not work
• The tigers like meat is perfectly grammatical
• need to know what interpretation it has
• TVJT is appropriate
• can set up a context where what these tigers do
  is different from what tigers as a kind do

61
Sample story-picture pair
In our zoo, we have two very unusual tigers. Most
tigers eat meat all the time. But our two tigers
are vegetarian they love to eat carrots, and
they hate meat. The tigers eat
meat TRUE FALSE (Ionin and Montrul 2009 based
on a similar format from Perez-Leroux et al.
2004, with children)
62
Picture-matching

• The learner sees a single sentence in the context
  of two or three pictures
• The learner has to choose which picture matches
  the sentence
• As with the TVJT, the appropriate response
  requires the learner to assign a particular
  grammatical structure to the sentence
• But unlike the TVJT, which present
  picture-sentence pairs individually,
  picture-matching tests the preference of one
  picture over another

63
An example inalienable possession (Ionin and
Montrul, under review)

• The boys closed the eyes (Choose A, B, or both)
• A B

64
Why use picture-matching here?

• The boys closed the eyes
• In the context of picture A TRUE
• In the context of picture B still true (??) but
  not as felicitous
• better variant The boys closed their eyes
• With picture-matching, subjects can express a
  preference for picture A
• L1-Spanish L2-English learners are expected to
  express a preference for picture B, or to select
  both
• in Spanish, the definite article is used for
  inalienable possession

65
Implicit measures of comprehension can be used to
study

• interpretation of pronouns and reflexives
• aspectual interpretation
• reference to completed vs. incomplete events
• reference to habitual vs. ongoing events(e.g.,
  Mary is playing the piano can be false even
  though Mary plays the piano is true)
• generic vs. specific interpretation
• interpretation of quantifier scope
• is A cat climbed every tree a statement about one
  cat or many cats?

66
Implicit measures of comprehension

• Advantages
• more implicit than grammaticality judgment tasks
  focus on meaning rather than form
• use of pictures and oral stories makes the task
  appropriate for children and low-literacy
  subjects
• Disadvantages
• constrained application can only be used for
  linguistic knowledge at the syntax/semantics
  interface
• tests are time-consuming and difficult to
  construct (especially with pictures)
• have to control for many factors be sure that
  the response results from the grammatical
  interpretation and not from some other factor (a
  yes-bias, an aesthetic preference for some
  pictures, etc.)
• Appropriateness of use with heritage speakers
• appropriate, when well-controlled and carefully
  constructed
• oral format appropriate for low-literacy learners

67
Summary

• Which tasks are most appropriate for research
  with HL learners?
• It depends on the linguistic phenomenon and the
  population
• For low-literacy / unschooled HL learners, tasks
  that dont rely on literacy or explicit
  (textbook-based) knowledge are best
• naturalistic or elicited production (oral format)
• truth-value judgments and picture matching (oral
  format)
• But for literate / schooled HL learners, written
  and more formal tasks are also appropriate
• Different task types are needed to supplement
  each other

68
Outline of the presentation

• Research design
• Types of research design
• Research questions, hypotheses, and predictions
• Independent and dependent variables
• Components of a language study
• Control group
• Proficiency test
• Language background questionnaire
• Data collection methods
• Naturalistic production tasks
• Elicited production tasks
• Comprehension tasks explicit measures
• Comprehension tasks implicit measures
• Classroom intervention research
• Suggested readings

69
Intervention research

• Research on acquisition of grammar can address
  two different questions
• What is learners linguistic performance like?
  What do they already know?
• How can we influence learners linguistic
  performance? How can we cause them to learn
  something new?
• The data collection methods discussed so far
  address the first question
• Intervention studies address the second question

70
Typical intervention study design

• Pre-test
• check what the learners already know
• Intervention/treatment
• attempt to influence the learners knowledge
• Post-test
• compare to pre-test, to find out whether the
  intervention has made any difference
• Delayed post-test (not all studies)
• check whether intervention benefits continue over
  time

71
The need for multiple groups

• Subjects may perform better on the post-test than
  the pre-test
• But is this the result of the intervention?
• or just general learning over time?
• or familiarity with the test format?
• Need control/comparison groups
• The groups must be maximally balanced in terms of
  native/dominant languages, proficiency, exposure
  situation, pre-test scores

72
Hypothetical example

• Research question how effective is the use of
  feedback in the classroom?
• Control group design
• class 1 gets implicit feedback (recasts)
• class 2 gets no recasts, and no other grammar
  instruction
• ? can learn whether recasts have any effect
• Comparison group design
• class 1 gets explicit feedback (correction)
• class 2 gets implicit feedback (recasts)
• class 3 gets both types of feedback
• ? can learn which type of feedback is most
  effective

73
Applications of intervention research in the
L2-classroom or Heritage Language classroom

• To examine the benefits of different
  instructional methods within the same population
• To examine whether L2-learners and heritage
  language learners benefit from same or different
  instructional methods
• To see whether theoretical findings about L2/HS
  knowledge can be applied in the classroom

74
Conclusion designing a research study with HL
learners

• Formulate a research question or hypothesis
• Define the learner population
• Decide on the necessary controls
• Choose a data collection methodology

75
Outline of the presentation

• Research design
• Types of research design
• Research questions, hypotheses, and predictions
• Independent and dependent variables
• Components of a language study
• Control group
• Proficiency test
• Language background questionnaire
• Data collection methods
• Naturalistic production tasks
• Elicited production tasks
• Comprehension tasks explicit measures
• Comprehension tasks implicit measures
• Classroom intervention research
• Suggested readings

76
Suggested readings on research methods and design

• Guides to research methods
• Cowart, W. 1997. Experimental syntax. Sage
  Publications.
• Crain, S. and Thornton, R. 1998. Investigations
  in Universal Grammar A guide to experiments on
  the acquisition of syntax and semantics. MIT
  Press.
• Mackey, A. and Gass, S.. 2005. Second language
  research Methodology and design. Lawrence
  Erlbaum.
• Rasinger, S. 2008. Quantitative research in
  linguistics. Continuum International Publishing.
• Proficiency measures and cross-task comparisons
• Ellis, R. 2005. Measuring implicit and explicit
  knowledge of a second language A psychometric
  study. Studies in Second Language Acquisition,
  27, 141-172.
• Montrul, S., Foote, R. and Perpiñán, S. 2008.
  Gender agreement in adult second language
  learners and Spanish heritage speakers The
  effects of age and context of acquisition.
  Language Learning 58, 3, 503553.
• Polinsky, M. 2008. Heritage language narratives.
  In D. Brinton, O. Kagan and S. Bauckus (eds.),
  Heritage Language Education A New Field
  Emerging, Routledge, p.149-164
• Unsworth, S. 2008. Comparing child L2 development
  with adult L2 development How to measure L2
  proficiency. In Gavruseva, E. and Haznedar, B.
  (eds.) Current Trends in Child Second Language
  Acquisition. Amsterdam John Benjamins, p.
  301-336.
• Issues in heritage language research
• Montrul, S. 2008. Incomplete Acquisition in
  Bilingualism. Re-examining the Age Factor.
  Amsterdam John Benjamins.
• Polinsky, M. and Kagan, O. 2007. Heritage
  languages in the wild and in the classroom.
  Language and Linguistics Compass, 1/5, 368-395.
• Online data collection measures
• Marinis, T. 2003. Psycholinguistic techniques in
  second language acquisition research. Second
  Language Research, 19, 144-161.
• Papadopoulou, D. 2005. Reading time studies of
  second language ambiguity resolution. Second
  Language Research, 21, 98-120.
• Sekerina, I. Fernández, E. and Clahsen, H.
  (eds.). 2008. Developmental Psycholinguistics
  On-Line Methods in Childrens Language
  Processing. Amsterdam John Benjamins.
• A very useful statistics guide
• Field, A. 2006. Discovering statistics using
  SPSS. Sage Publications.