CrossLanguage Information Retrieval: Layout Strategies for Gloss Translation

1
Cross-Language Information Retrieval Layout
Strategies for Gloss Translation
Eiman M. Elnahrawy Nagia M. Ghanem Moustafa A.
Youssef
2
Goal

• In University of Maryland Translingual
  Information Retrieval System
• The three most common translations for each word
  are displayed
• Using this gloss translation, users can assess
  whether the document is relevant or not
• In our experiment
• We studied the effect of changing the number of
  translations on the users assessment

3
Variables

• Independent Variables
• The number of displayed translations per word.We
  have 3 treatments 2,3 or 4 translations. (1x3
  experiment)
• Dependent Variables
• The percentage of documents that the user could
  take a decision within a fixed period of time
• Precision measured using non-relevant documents
  the fraction of the selected documents that are
  non-relevant (fallout)
• Recall measured using non-relevant documents the
  fraction of the non-relevant documents that are
  selected
• User Satisfaction

4
Hypotheses

• Hypothesis 1 Increasing the number of displayed
  translations per word will decrease the
  percentage of documents that the user can take a
  decision within a fixed period of time
• Hypothesis 2 Increasing the number of displayed
  translations per word will improve the users
  assessment
• We expected that users will prefer fewer
  translations per word

5
Subjects

• Eighteen subjects (14 males, 4 females)
• All subjects were students at the UMD (15 grad.,
  3 undergrad.)
• The majority were Computer Science students, the
  others were Electrical, Mechanical, and Computer
  Engineering students
• The experiment was conducted within subjects
• To minimize the effect of getting used to the
  system on the user performance
• We had 3 treatments that can be permuted in 6
  different ways and 18 subjects
• We assigned 3 subjects to each one of the 6
  permutations

6
Results

• The number of documents that each subject was
  able to judge their relevance within the allowed
  five minutes were recorded.
• We measured the Fallout and the non-relevance
  Recall.
• The single factor analysis of variance (ANOVA)
  showed that there were no statistically
  significant differences at Alpha 0.05.

7
Results

• The percentage of documents the user could make a
  decision

8
Results

• Fallout

9
Results

• Non-relevance Recall

10
Results

• Subjective Satisfaction

11
Results

• We were interested in the subjects responses for
  the following five questions
• Overall Reaction to the System (1Difficult --
  9Easy)
• Overall Reaction to the System (1Frustrating --
  9Satisfying)
• Overall System Efficiency (1Efficient --
  9Inefficient)
• Screen Layout were Helpful (1Never -- 9Always)
• Learning to Operate the System (1Difficult --
  9Easy)

12
Results
13
Results
14
Results
15
Discussion

• Number of documents in which the user took a
  decision
• Large Variance? Decision on the title or the
  entire document
• Fallout and non-Relevance Recall
• Decision based on bold and red words
• Better selection of the document collection
• Satisfaction
• 83 of the users preferred one or two
  translations per word

16
Observations

• The order in which the subject takes the
  treatments is important
• Most users performed better on second and third
  treatments than on the first one
• Long loading time of the whole document web
  page
• Some subjects used only the title or judged the
  relevance as do not know

17
Conclusions

• Impact for practitioners
• Users prefer two translations per word
• The title should be formatted in a way that
  facilitates users decision
• The time it takes the system to format the
  document should be reduced

18
Conclusions (continued)

• Suggestions for future researchers
• Working on the entire document collection or
  selecting a better subset
• Other measures of user performance can be used
• Adding a fourth option, not seen
• Other layout strategies such as vertical vs.
  horizontal layout

19
Acknowledgement

• Thanks to
• Dr. Ben Shneiderman
• Dr. Doug Oard
• Clara Cabezas
• Gina Levow
• Pilot subjects
• Subjects

20
Thank You