Information Elicitation in Scheduling Problems

1
Information Elicitation in Scheduling
Problems

• Ulas Bardak
• Ph.D. Thesis Defense

Committee Jaime Carbonell (chair), Eugene Fink,
Stephen Smith, and Sven Koenig (University of
Southern California)
2
Outline
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• Introduction
• Related work
• Domain
• Optimization
• Elicitation
• Evaluation
• Conclusions

3
What is information elicitation?
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• For example

4
Why elicitation?
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• Scheduling problems include information about
  resources, constraints, and preferences
• Uncertain information can lower the quality of
  schedules
• We need to select and ask questions that help to
  reduce uncertainty

5
Example problem
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• We are organizing a small conference,using three
  available rooms
• We have incomplete information about speaker needs

6
Initial schedule
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion
Available rooms
1
2
Roomnum. Area Projector
123 LargeMed.Small YesNoYes
3

• Missing info
• Invited talk Projector need
• Poster session Room size Projector
  need

• Requests (by importance)
• Invited talk, 910am Needs a large room
• Poster session, 9-11am Needs a room

• Assumptions
• Invited talk Needs a projector
• Poster session Smaller room is OK
  Needs no projector

7
Choice of questions
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion
Initial schedule
1
2
Posters
3
Talk

• Candidate questions
• Invited talk Needs a projector?
• Poster session How big a room? Needs a
  projector?

• Requests
• Invited talk, 910am Needs a large room
• Poster session, 911am Needs a room

8
Improved schedule
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• Requests
• Invited talk, 910am Needs a large room
• Poster session, 911am Needs a room

Initial schedule
1
2
Posters
3
Talk
Info elicitation
System Does the poster sessionneed a projector?
How big a room does it need?
Posters
UserA projector may be useful.A small room is
OK.
9
Motivation
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• Improve optimization results by
• reducing uncertainty of the
• available knowledge.

10
Related work
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• Example critiquing (Burke)
• Have users tweak result set
• Collaborative filtering (Resnick and Hill)
• Have the user rank related items
• Similarity-based heuristics (Burke)
• Look at past similar user ratings
• Focusing on targeted use (Stolze)

11
Related work
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• Clustering utility functions (Chajewska)
• Decision tree (Stolze and Ströbel)
• Min-max regret (Boutilier)
• Choose question that reduces max regret
• Auctions (Smith, Boutilier, and Sandholm)

12
What is different?
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• No bootstrapping
• Both continuous and discrete variables
• Large number of uncertain variables
• Tight integration with the optimizer
• Synergy of multiple approaches

13
Explored domains
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• An academic conference
• Assigning rooms to sessions
• Placing vendor orders
• Assigning orders to sessions
• Social networking
• Matching users to other users

14
Selected domain
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• Scheduling a conference
• Rooms are our resources
• We need to assign rooms to sessions

15
Collaborative scheduling
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion
Automatic operations
16
Collaborative scheduling
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion
Automatic operations
Automatic operations
Process new dataand advice
return the control to the user
invoke theauto scheduling
17
Architecture
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion
Top-level control and learning
Processnew info
18
Rooms
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion
2
1
3

• Rooms have a set of properties
• Size, seating capacity,...
• Microphones, projectors,...
• We also know distances between rooms

Room 1 is 2000 square feet and has one projector.
Room 1 is 400 feet away from Room 3.
19
Sessions
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• Session description includes
• Importance
• Hard constraints, such as the minimal acceptable
  room size
• Soft preferences, such as the desired room size

The invited talk is more important than the
poster session. The assigned room has to be at
least 500 square feet, and preferably 1000 square
feet.
20
Sessions
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion
We represent preferences by piecewise-linear
functions.
1.0
Quality
0.5
0
Room size
250
500
750
1000
Unacceptable
The invited talk is more important than the
poster session. The assigned room has to be at
least 500 square feet, and preferably 1000 square
feet.
21
Uncertainty
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• We usually have incomplete knowledge of room
  properties, session importances, and constraints
  and preferences.

22
Uncertain properties
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• We represent an uncertain value as either
• a completely unknown value, or
• a probability density function, approximated by a
  set of uniform distributions.

23
Uncertain properties
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion
Example An auditorium has about 600 seats.
0.2 chance 450..549 0.6 chance 550..650 0.2
chance 651..750
Probability
0.006
.6
0.004
0.002
.2
.2
0
0
200
400
600
800
Capacity
24
Uncertain preferences
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• We represent an uncertain preference as
• completely unknown function,
• piecewise-linear function with uncertain
  y-coordinates of endpoints, or
• set of possible piecewise-linear functions with
  related probabilities.

25
Uncertain preferences
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion
The description of a demo session does not
include a room-size preference.
1.0
.95 chance
Quality
.05 chance
0.5
0
Room size
250
500
750
1000
Unacceptable
Demo sessions usually require at least 250 square
feet, and preferably 750 square feet however,
there is a 5 chance that a big sponsorshows up
unexpectedly and asks for additional 250 square
feet.
26
Optimization
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• The optimizer assigns rooms to sessions.
• Input Rooms and sessions
• Output Room and time for each session

27
Session quality
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• Quality value of a session is based on how much
  each preference is satisfied
• Uncertainty is taken into account when
  calculating quality

28
Schedule quality
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• Overall schedule quality value is a weighted sum
  of session quality values
• If any session violates hard constraints, the
  whole schedule is unacceptable

29
Optimizer
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• Simple version is based on hill-climbing
• Advanced version uses randomized hill-climbing,
  similar to simulated annealing

30
Elicitation
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• We use elicitation to reduce uncertainty
• User can selectively answer any questions

31
Elicitation
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion
Synergetic Elicitor
Heuristic Elicitor
Rule-based Elicitor
Search Elicitor
All Potential Questions
Merged List
Re-ranked List
32
Heuristic elicitor
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion
Synergetic Elicitor
Heuristic Elicitor
Rule-based Elicitor
Search Elicitor

• Selection of questions based on the standard
  deviation of schedule quality
• Fast calculation, once per variable
• Domain-independent

33
Heuristic elicitor
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion
Each uncertain variable is a potential question
Get list of questions
For each question, determine impact on schedule
quality of possible answers
Plug in possible answers to the quality function
to get change in schedule quality

For each question, calc. question score
Return top questions
34
Rule-based elicitor
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion
Synergetic Elicitor
Heuristic Elicitor
Rule-based Elicitor
Search Elicitor

• Selection of additional questions, based on
  domain-specific heuristics, such as Room
  capacity is more important than ceiling height.

35
Search elicitor
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion
Synergetic Elicitor
Heuristic Elicitor
Rule-based Elicitor
Search Elicitor

• Ranks selected questions using B search
• Relies on the optimizer for evaluating nodes in
  the search space
• Domain-independent and optimizer-independent

36
Example
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion
Uncertain room size versus uncertain projector
number
The minimal possible utility of asking about the
room size is greater than the maximal possible
utility of asking about the number of projectors.
37
Evaluation
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• The synergetic elicitor is far more effective
  than each of its individual components, simple
  heuristics, and random selection of questions.

38
Evaluation
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• Four scenarios with 88 sessions
• 10 rooms, 100 uncertain values
• 20 rooms, 500 uncertain values
• 50 rooms, 1000 uncertain values
• 84 rooms, 3300 uncertain values

39
Evaluation
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• For each setting, we use fivedifferent
  elicitation systems
• Synergetic Elicitor
• Heuristic rule-based
• Search rule-based
• Rule-based
• Random

Synergetic Elicitor
Heuristic Elicitor
Rule-based Elicitor
Search Elicitor
40
Evaluation
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• We plot
• Change in the schedule quality
• Change in the quality loss dueto uncertainty
  (100 ? 0)

41
Evaluation
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• 100 variables

y
42
Evaluation
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• 3300 variables

y
43
Evaluation
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion
1000 q.
500 q.
100 q.

• Problem size

3400 q.
44
Summary
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• We have applied the elicitor to conference
  scheduling
• Synergetic elicitor outperforms its components
  and simple heuristics
• Improvement is more prominent for larger problems

45
Contributions
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion
We have investigated a novel approach to
information elicitation, which has led to three
main contributions.

• Fast heuristic computation of the expected
  utility of potential questions
• Use of B search for determining more accurate
  question utilities
• Synergy of domain-independent and domain-specific
  elicitation techniques

46
Future work
Outline Introduction Related Work Domain
Optimization Elicitation - Evaluation -
Conclusion

• Learning question costs
• Learning elicitation strategies

47
(No Transcript)
48
Additional Slides
49
Vendor elicitation

• Domain
• Sessions can require services that external
  vendors provide
• e.g. mobile equipment, food deliveries
• Each item can satisfy multiple services
• e.g. Laptop ? Computer, Portable computer
• Penalty for spending money
• A vendor optimizer finds a near optimal placement
  of vendor orders
• Uncertainty can exist in prices, availability of
  items

50
Vendor elicitation

• Elicitation Algorithm
• Enumerate all of the services
• Order based on affecting the overall cost penalty

51
Vendor elicitation

• Evaluation