A Theoretical Scheduling Toolbox

1
OPEN VERSUS CLOSEDA CAUTIONARY TALE Bianca
Schroeder Adam Wierman Mor
Harchol-BalterComputer Science
DepartmentCarnegie Mellon UniversityTo
appear at NSDI 2006
2
THE RESEARCH PROCESS
standard system
3
MANY WAYS TO GENERATE REALISTIC WORKLOADS
User driven
Model the behavior of a typical user then fork
one process for each user.
User requests web page, receives page, reads
page, clicks on new link
CLOSED SYSTEM MODEL
4
MANY WAYS TO GENERATE REALISTIC WORKLOADS
file sizes from trace
next arrival time from trace
OPEN SYSTEM MODEL
5
MANY WAYS TO GENERATE REALISTIC WORKLOADS
Distribution driven
Use distributions of interarrival times and
service demands (typically using trace info)
interarrival time dist.
service demand dist.
sample dist.
sample dist.
OPEN SYSTEM MODEL
6
Arrivals are independent of completions
Arrivals are completely dependent on departures
There is a fixed population of users, called the
Multi-Programming-Level (MPL)
There is no max number of simultaneous users
7
Do you use an open or closed model?
Surge

1. Workload generators for thesame purpose use
   differentsystem models!
2. Its often not clear which model workload
   generatorsuse!

SPECWeb
TPC-W
Sclient
RUBiS
WebBench
Webjamma
8
OUR GOAL TODAY
What is the impact of the choice of an open or
closed model?
9
HOW DO WE COMPARE OPEN AND CLOSED SYSTEMS?

1. Fix the service distribution acrossthe systems
2. Fix the load across the systems

load depends only on mean arrival rate and mean
service demands
adjust load using the arrival rate
load depends on MPL, think times, mean of service
demands, variability of service demands,
scheduling policy,
adjust load using the think time
10
How do open and closed response times compare?
FCFS scheduling open ? Poisson arrival
process closed ? Exponential think times
11
FCFS scheduling open ? Poisson arrival
process closed ? Exponential think times
Open
CLOSED ltlt OPEN
Closed (MPL10)
12
FCFS scheduling open ? Poisson arrival
process closed ? Exponential think times
Open
CLOSED ? OPEN
Closed (MPL1000)
Closed (MPL100)
Closed (MPL10)
13
CLOSED ? OPEN AS MPL GROWS
As MPL grows arrival rate becomes independent of
completion rate
14
How quickly does Closed ? Open?
Open
Web Workloads
Closed (MPL1000)
Closed (MPL100)
Closed (MPL10)
15
SUMMARY SO FAR
high variability
Open gtgt Closed
It depends
low variability
Open Closed
Open gt Closed
Small MPL
Large MPL
16
OUR GOAL TODAY
What is the impact of the choice of an open or
closed model?

1. What is the impacton the effectivenessof
   scheduling?
2. What is the impactin practice?

It matters a lot!
17
SCHEDULING IS A KEY COMPONENT OF SYSTEM DESIGN
WEB SERVERS
Does the effectiveness of scheduling depend on
the system model (open vs. closed)?
18
SCHEDULING IN OPEN SYSTEMS
OPEN
1000 600 300 0
How do the closed results compare?
mean response time
0 .25 .5 .75 1
load
19
CONTRASTING THE IMPACT OF SCHEDULING
OPEN
CLOSED
1000 600 300 0
mean response time
0 .25 .5 .75 1
0 .25 .5 .75 1
load
load
20
OUR GOAL TODAY
What is the impact of the choice of an open or
closed model?
It matters a lot! Especially when
evaluating scheduling policies
What is the impact in practice?
21
OPEN VS CLOSEDIN PRACTICE
4 CASE STUDIES

1. Serving static web content
2. Database backend ofan e-commerce site
3. Auctioning web site
4. Supercomputing center

testbed implementation
trace-based simulation
22
Case study Open generator Closed generator Scheduling policies
Static web (LAN) Sclient on World Cup trace Modified Sclient on World Cup trace PS, SRPT
E-commerce Modified TPC-W TPC-W PS, PESJF
Auctioning Trace-based simulation(top 10 auction site trace) Trace-based simulation(top 10 auction site trace) PS, SRPT
23
OPEN VS CLOSEDIN PRACTICE
STATIC WEB SERVER
OPEN
CLOSED
300 200 100
MPL50
mean response time
0 .25 .5 .75 1
0 .25 .5 .75 1
load
load
Different models give different conclusion about
benefits of SRPT
24
OPEN
CLOSED
10 8 4 0
MPL50
PS
E-COMMERCE SITE
PS
PESJF
PESJF
mean response time
20 14 7 0
load
load
MPL50
PS
AUCTION SITE
PS
SRPT
SRPT
0 .25 .5 .75 1
0 .25 .5 .75 1
load
load
25
OUR GOAL TODAY
What is the impact of the choice of an open or
closed model?
It matters a lot in practice! Especially when
evaluating scheduling policies
26
CHOOSING A SYSTEM MODEL
Web workloads

1. A site being Slashdotted
2. Online gaming site
3. Science Institute - USGS
4. Online dept. store
5. Financial service provider
6. Kasparov vs Deep Blue
7. CMU web server
8. World cup site

Open or Closed?
27
NEITHER THE OPEN OR CLOSEDMODEL IS COMPLETELY
REALISTIC
28
A MORE REALISTIC ALTERNATIVE
What parameters affect the load?
Does think time affect the load?
How do think times affect response times?
29
THE EFFECT OFTHINK TIME
STATIC WEB SERVER
300 200 100 0
mean response time
1 10 100 1000
mean think time
30
A MORE REALISTIC ALTERNATIVE
Workload generators are only Open/Closed!
number of requests per visit ?
number of requests per visit ?
31
THE TRANSITION FROM OPEN ? CLOSED
STATIC WEB SERVER
CLOSED
300 200 100 0
OPEN
mean response time
0 5 10 15
20
mean number of requests per visit
32
THE PARTLY-OPEN SYSTEM IN PRACTICE
mean response time
mean number of requests per visit
33
CHOOSING A SYSTEM MODEL
Web workloads
A site being Slashdotted (1.2) Financial
service provider (1.4) CMU web server
(1.8) Kasparov vs Deep Blue (2.4) Science
Institute USGS (3.6) Online dept. store
(5.4) World cup site (11.6) Online gaming site
(12.9)
Use partly-open system to decide
34
OUR GOAL
35
WRAPUP
36
OPEN AND CLOSED CAN BEHAVE VERY DIFFERENTLY
THESE DIFFERENCES ARE IMPORTANT IN PRACTICE
OPEN
CLOSED
VS
37
A CAUTIONARY TALE
Be careful of the underlying model in workload
generators open vs. closed has huge practical
impact
Web workload generators need to be flexible
38
OPEN VERSUS CLOSEDA CAUTIONARY TALE Bianca
Schroeder Adam Wierman Mor
Harchol-BalterComputer Science
DepartmentCarnegie Mellon UniversityTo
appear at NSDI 2006
39
CHOOSING A SYSTEM MODEL
Web workloads

1. A site being Slashdotted
2. Online gaming site
3. Science Institute - USGS
4. Online dept. store
5. Financial service provider
6. Kasparov vs Deep Blue
7. CMU web server
8. World cup site

Open or closed?
Use a partly-open model...
40
FITTING A PARTLY-OPEN MODEL
file sizes from trace
41
FITTING A PARTLY-OPEN MODEL

• Fitting the interarrival times
• Distinguish userse.g. use ip address in a web
  trace
• Identify user session boundaries ? Use periods
  of inactivity of length gt timeout

Cant use trace directly because
dependencies between completions and follow-up
requests would be lost!
42
CHOOSING A TIMEOUT VALUE
43
CHOOSING A SYSTEM MODEL
Web workloads

1. A site being Slashdotted
2. Online gaming site
3. Science Institute - USGS
4. Online dept. store
5. Financial service provider
6. Kasparov vs Deep Blue
7. CMU web server
8. World cup site

Open or closed?
Use a partly-open model... ...to decide which is
more accurate
44
HOW TO CHOOSE A SYSTEM MODEL
How many simult. users are there?
Gather a trace
45
CHOOSING A SYSTEM MODEL
Web workloads

1. A site being Slashdotted
2. Online gaming site
3. Science Institute - USGS
4. Online dept. store
5. Financial service provider
6. Kasparov vs Deep Blue
7. CMU web server
8. World cup site

Open or closed?
Use a partly-open model...
46
FITTING A PARTLY-OPEN MODEL
file sizes from trace
47
FITTING A PARTLY-OPEN MODEL

• Fitting the interarrival times
• Distinguish userse.g. use ip address in a web
  trace
• Identify user session boundaries ? Use periods
  of inactivity of length gt timeout

Cant use trace directly because
dependencies between completions and follow-up
requests would be lost!
48
CHOOSING A TIMEOUT VALUE
49
CHOOSING A SYSTEM MODEL
Web workloads

1. A site being Slashdotted
2. Online gaming site
3. Science Institute - USGS
4. Online dept. store
5. Financial service provider
6. Kasparov vs Deep Blue
7. CMU web server
8. World cup site

Open or closed?
Use a partly-open model... ...to decide which is
more accurate
50
HOW TO CHOOSE A SYSTEM MODEL
How many simult. users are there?
Gather a trace
51
CHOOSING A SYSTEM MODEL
lt5 expected visits
Web Workloads
OPEN

1. A site being Slashdotted
2. Online gaming site
3. Science Institute - USGS
4. Online dept. store
5. Financial service provider
6. Kasparov vs Deep Blue
7. CMU web server
8. World cup site

5-10 expected visits
PARTLY OPEN
gt10 expected visits
CLOSED
52
CHOOSING A SYSTEM MODEL
lt5 expected visits
Web Workloads
OPEN

1. A site being Slashdotted
2. Online gaming site
3. Science Institute - USGS
4. Online dept. store
5. Financial service provider
6. Kasparov vs Deep Blue
7. CMU web server
8. World cup site

5-10 expected visits
PARTLY OPEN
gt10 expected visits
CLOSED
53
CHOOSING A SYSTEM MODEL
lt5 expected visits A site being Slashdotted
(1.2) Science Institute USGS (3.6) Financial
service provider (1.4) Kasparov vs Deep Blue
(2.4) CMU web server (1.8)
Web Workloads
OPEN
5-10 expected visits Online dept. store (5.4)
PARTLY OPEN
gt10 expected visits Online gaming site
(12.9) World cup site (11.6)
CLOSED
54
CHOOSING A SYSTEM MODEL
Web Workloads
A site being Slashdotted (1.2) Financial
service provider (1.4) CMU web server
(1.8) Kasparov vs Deep Blue (2.4) Science
Institute USGS (3.6) Online dept. store
(5.4) World cup site (11.6) Online gaming site
(12.9)
55
CHOOSING A SYSTEM MODEL
lt5 expected visits A site being Slashdotted
(1.2) Science Institute USGS (3.6) Financial
service provider (1.4) Kasparov vs Deep Blue
(2.4) CMU web server (1.8)
Web Workloads
OPEN
5-10 expected visits Online dept. store (5.4)
PARTLY OPEN
gt10 expected visits Online gaming site
(12.9) World cup site (11.6)
CLOSED
56
CHOOSING A SYSTEM MODEL
Web Workloads
A site being Slashdotted (1.2) Financial
service provider (1.4) CMU web server
(1.8) Kasparov vs Deep Blue (2.4) Science
Institute USGS (3.6) Online dept. store
(5.4) World cup site (11.6) Online gaming site
(12.9)