Lecture 7: Further Student Modelling:

1
Lecture 7 Further Student Modelling
2
Contents

• 1. Analytic Methods
• 2. e.g. ArtCheck
• 3. Other Machine Learning egs
• 4. Current trends

3
1. Analytic Models
4
Analytic Models

• Analyse students input and try to account for
  the structure
• represents misconceptions
• not restricted to pre-defined errors
• often use of machine learning techniques
• - doesnt explain cause of error or bug
• - needs significant samples of data

5
e.g. Spelling

• Work back from the misspelling to the correction
• k w i c quick
• r i d red, rid, ride, write, rite, read
• f a t fate
• Letter as its name? i for i_e a for a_e
• Slides of Phoncode

6
e.g. Phoncode -gt Phonological Information

• Based on phoneme-grapheme grammar consider what
  phonemes error was intended to represent, then
  see if any word matches

7
WEST (Brown and Burton, 1979)

• Development of mathematical skills through a
  game
• students move compared to systems
• if differ, moves are analysed
• identifies issues student is weak on
• - coaches on weakness, in context of move

8
Wenger, figure 7.1 How the West was won (Burton
and Brown, 1979)

• Insert figure

9
Excerpt from a dialogue with WEST (Burton and
Brown, 1976)

• Bob is at 54 -- the COMPUTER is at 40.
• West Its MY turn Spinners are 2 4 2
• My expression is 2 (4/2) which gives ME 4
• I took a SHORTCUT
• I bumped YOU
• the COMPUTER is at 54 -- Bob is at 40
• West Its YOUR turn Spinners are 1 2 2
• What arithmetic expression did you form?
• Student 1 2 2
• West What number does that give you?
• Student 5
• West You dont seem to be bumping very much.
  Bumps are hard to get but they are usually a good
  idea. One good example would be the expression
  (1 2) 2, which would give you a SHORTCUT and
  a BUMP!! So you could have been at 54 while I
  would have ended up at 40. Bob is at 45 -- the
  COMPUTER is at 54.

10
WEST

• Issues
• Mathematical skills use of addition, division,
  parentheses
• Game-specific skills
• use of shortcuts and landing on your opponent
• development of strategy such as always aiming for
  towns if possible
• knowing that order of numbers in expression does
  not have to be same as on the spinners
• General game skills
• learning from your opponent
• exploring the space of possible strategies
  permitted by the game.

11
Recognising and Evaluating Issues

• For each issue, there is a RECOGNISER
• - checks to see whether issue features in the
  student's move,
• - whether it is necessary in that expression
• - whether it is necessary in the optimal move
• For each issue there an EVALUATOR
• - determines whether student weak in issue, by
  failing to use it to good effect.
• - blame shared equally among all issues that
  might be implicated in failure to find best move.

12
Dealing with different strategies

• What if the user employs different strategy?
• WEST watches for what is called a tear
• the appearance of a fairly high number of
  issues not used when they should have been, and
  others used when they should not have been.
• WEST considers whether another strategy by the
  expert would give a better fit, (according to
  each of pre-wired strategies)

13
2. Analytic Methodse.g. ArtCheck
14
Detecting and Analysing Errors

• Using Articles
• - how do you know whether you are talking about
  a specific object, or any old object?
• - how do we choose the correct article to
  indicate the indefinite or definite property of
  the noun in an utterance?
• Detecting Errors
• how do we detect errors when some uses the
  incorrect article?
• how could we generate an appropriate explanation
  to help them learn to do this correctly?

15
Artcheck (Sentence, 1993)

• Indefinite
• a/an, eg John is a teacher
• zero, eg Do you take milk in coffee?
• Definite
• the, eg He is the only teacher in the school
• Some native languages do not include an article
  category, e.g. Finnish - definiteness and
  indefiniteness expressed in quite different ways
• For such EFL learners, the correct use of English
  articles can be a major problem.

16
e.g in Finnish.

• Otan kirjat
• I take the books
• Otan kirjoja
• I take some books
• Huoneessa on poyta
• There is a table in the room
• Poyta on huoneessa
• The table is in the room
• Similar problems in Basque and Russian

17
Examples of errors

• I have visited Tower of London
• I have visited the Tower of London
• Aeroplane has revolutionised travel
• The aeroplane has revolutionised travel
• We discussed our plans for the day over the
  breakfast
• We discussed our plans for the day over breakfast

18
ArtCheck ITS

• Aims to help such non-native speakers of English
  use articles appropriately.
• Domain knowledge rules which determine correct
  article usage
• Applying knowledge user input, in order to detect
  any article usage errors
• Having detected errors, rules can be used as the
  basis for generating explanations, customised to
  the individual learner.

19
Determining correct article usage

• Rules indicate whether article before noun should
  be
• the definite article the
• the indefinite article a/an
• no article at all, (the zero article)
• Some are fixed rules the definite article should
  be used when the noun is modified by a
  superlative adjective,
• eg the largest dog
• Other depend on context of use the indefinite
  article should be used to introduce new
  information.
• The sources of information used by the system
• the lexicon, the parser, the morphological
  analyser, and a discourse history module.

20
Examples of rules
21
Easily implemented rules, e.g.
22
Rules requiring more information

• e.g. occurrence of the zero article in certain
  prepositional phrases with nouns of some semantic
  categories, as in in spring and by car.
• In this case, lexicon augmented with information
  about the semantic categories of nouns
• So with certain singular count nouns to do with
  seasons, meals, time, transport, and
  institutions, and with certain prepositions,
  information required is
• List of types of nouns and Prepositional phrase
  modifier

23
Detecting article usage errors

• 1. Student answer compared with expert model
• Incorrect article usage identified
• 3. Uses rule induction to learn rules from
  training instances
• 4. Artcheck produce new rules, based on the
  expert ones, describes what students doing
• i.e. identify the (incorrect) rules (
  mal-rules)
• 5. If the system can determine a mal-rule which
  represents the students error, then this is
  explained to the student

24
Form of the mal-rules

• When conditions X apply, use article Y.
• The conditions which can apply are
• Whether a noun is a proper or common noun
• Whether a noun is a singular or plural noun
• Whether a noun is a count (e.g. three eggs) or
  mass bread, milk noun
• Whether a noun is modified and with what
• What immediately precedes the noun (for
  example, the verb to be)
• How the noun is categorised in the lexicon
  (semantic category)
• Whether a noun is new or given information

25
Generating mal-rules

• Mal-rule cannot be generated from one error
• - system does not have enough information about
  the consistency of the error to be able to decide
  that a student is applying an incorrect rule.
• A number of errors must be accumulated, before
  the system can generalise about the errors and
  find a common rule to explain them.
• Errors in ArtCheck are the positive training
  instances from which the system can learn.

26
Positive and negative instances

• Positive instances of the error (incorrect noun
  phrases)
• a. John is teacher.
• b. Sandy is pig.
• c. I am doctor.
• Negative instance of the error (correct noun
  phrases)
• d. John is a good man.
• (a) - (c) include incorrect noun phrases all
  using the zero article where the following rule
  should have applied
• Rule 11 Use the article a/an where a singular
  count noun is used as the complement of the verb
  to be.
• Mal-rule proposed to account for data
• Where there is a singular, unmodified, common,
  count noun preceded by a singular form of the
  verb be, use the zero article.

27
Learning from errors

• One goal is for user of the system to learn from
  any errors made. To do this it must
• be able to understand the observed errors,
• be able to communicate effectively with the user
• provide a good explanation for that error.
• The explanation is tailored to the learner in
  three ways
• relating to the learners level of ability,
• learning style, and
• the type of error observed.
• In addition, the learner is given some control
  over the information received.

28
Explanations

• Relevant if it answers the users questions.
• Convincing justification is one which is sound,
  logical, and based on facts which the user
  believes.
• Whether a user can understand the explanation
  depends on
• being a well-organised explanation,
• not unnecessarily long-winded, and
• using terms with which the user is familiar.
• A tutoring explanation should vary its
  explanation according to the individual student.
• The student should both understand and be helped
  by the explanation.

29
Example dialogue

• AC Enter sentence
• Student I am doctor
• AC identify error doctor in I am doctor is
  incorrect.
• AC correct error It should be a doctor .
• AC ask student for feedback
• Select m more q quit explanation
• Student m
• AC state rule The rule is RULE 11
• Use a or an before singular count nouns which
  come after the verb to be .
• AC ask student for feedback
• Select m more q quit explanation
• Student m
• AC explain mal-rule
• I have noticed that you seem to use no article
  instead of a or an before a singular count
  and after the verb to be in the singular

30
Dialogue, continued

• AC ask student for feedback
• Select m more q quit explanation
• Student m
• AC exemplify mal-rule
• You also said Sandy is pig
• John is teacher
• which are similar errors.
• Try one of these again
• Sandy is pig
• Choose the correct article
• 1 a
• 2 an
• 3 the
• 4 no article
• Student 1
• AC Well done. That is the correct answer.
• Continue? (y/n) n

31
Evaluation of ArtCheck

• Can understand many types of sentence structures.
  
• Cannot understand questions and imperatives.
• - grammar could easily be extended
• Sometimes wrongly predict appropriate article
  usage
• - less common idiomatic usages.
• distant or complex referring expressions, where
  semantic information required
• Lot of data required for mal-rule to be
  generated.
• Feedback during external evaluation generally
  positive
• students confirmed this was an area of difficulty
• were enthusiastic about experimenting with
  system.
• found the system easy and helpful to use.
• Most showed some improvement after using ArtCheck
  for a short period of time.
• Verbal and written feedback generally very
  positive.

32
References

• Kass, Robert and Finin, Tim, (1988).
• The Need for User Models in Generating Expert
  System Explanations,
• Technical Report MS-CIS-88-37, University of
  Pennsylvania.
• Sentence, S.(1993).
• Recognising and responding to English article
  usage errors an ICALL based approach,
• Unpublished PhD Thesis, 1993, University of
  Edinburgh.

33
3. Other Machine Learning egs
34
Machine Learning

• Machine Learning
• developing computational theories of the
  learning process and building machines which
  learn
• Where used in Student Modelling
• - PIXIE (Sleeman)
• - ACM Langley, Ohlsson and Sage
• - Self - co-operative learning

35
PIXIE (Sleeman, 1983)

• Solving algebraic equations
• Has set of mal-rules heuristics
• Takes student answer, works back to question,
  infers missing steps, generates new mal-rule
• Generates large search tree - not clear how
  decides which possible mal-rule
• only generates one mal-rule per problem
• no cause suggested for mal-rule domain-dependent
  heuristics.

36
Automated Cognitive Modelling(Langley, Ohlsson
and Sage 1984)

• Production rule representation of skill
• search through problem space
• operators for chosen domain plus distinguishing
  properties of domain
• e.g. subtraction greater than, above, is zero
• Obtains student response to problem
• Searches problem space - get 'solution path'
  (describes operations performed by student)
• Applies inductive-learning techniques to
  distinguish those operators applied successfully
  and those not
• e.g learning by example ve -ve instances
• 4. Constructs discrimination network, providing
  model of constraints student placed on operators
• Depends a lot on properties supplied initially
• No explanation of where errors come from, how to
  correct

37
Collaborative Learning System For Concepts
(Gilmore and Self, 1988)

• Envisaged situation
• - access to database of elements
• - has details of properties of elements
• - elements assigned to 3 classes
• - explore why assignments made
• - student and system work together
• system has learning strategies and tactics (
  learning algorithm)
• Focusing generalises from ve examples and
  discriminates from -ve examples
• (uses a relation tree)
• System's knowledge in learning components

38
4. Current Trends
39
Student Modelling - Current Trends

• Continued Emphasis on Diagnosis and Error
  Modelling
• Modelling Affective and Motivational Aspects of
  Learner
• Open and Participative Models
• Use of Statistical and Machine Learning
  Techniques
• Adaptive to Wider Range of Users (Including
  Disabled)

40
User Modelling Motivation and Affect(de Vicente
and Pain, 2000)

• Dialogue Planning Rules
• What type of feedback to give?
• - depends on student's performance, effort and
  confidence
• Should we allow the student to give up?
• - depends on effort, challenge
• Is the interest so low that we should interupt
  the student?
• - depends on sensory-interest, cognitive-interest