Experiment Tutorial

1
Experiment Tutorial

• Network Computing Lab.
• 2005. 5. 19

2
Table of contents

• What is experiment?
• What is experiment design?
• Case study - CIGMA
• Three keys for experiment
• Step by step for experiment

3
What is experiment?

• Proof of your proposal
• Convince people that your goal is well achieved
• Make your contribution to be clear and true
• Support that your technical problems are well
  solved

4
What is experiment design?

• Experiment design
• ?? ??? ?? (Pre-report)
• ?? ??
• 1. Decide key aspects you want to show
• 2. Choose the proper verification method
• Demo, implementation and measurement, simulation,
  and analysis
• 3. Identify metrics (dependent variables)
• 4. Classify parameters (independent variables)
• ?? ?? ??
• 1. Decide key system components to be implemented
• Plan to test your implemented component
• 2. Prepare workload (benchmark, trace, synthetic
  workload)
• 3. Plan to prepare test-bed (machine, OS,
  software, network link, simulation tools)
• ?? ??
• 1. Draw the expected graphs which you want to
  show (imaginary experiment w/ paper and pencil)
• Assignment
• Presentation slides of experiment design for your
  own experiment

5
Verification methods

• 1. Analysis
• ?) Quick sort O(N log N))
• Deterministic and Statistical Analysis
• 2. Simulation
• ?) Network protocol simulation (topology,
  bandwidth, link delay)
• ????? ???? ?? ??? ?? ???? ??
• Simulation? ?? ?????
• ??? ?? ????? ???? ??? ??? ? ?
• ?? ? ????? ???? ??? ? ?
• ?? ? ???? ??? ?? ???? ?? ???? ?? ?
• ?? Easy to control, ?? Unrealistic
• 3. Implementation and measurement
• ?) ? ?? ?? ?, ?? ?? ? ??

6
Case study CIGMA Motivation

• CIGMA aCtive Inventory service in Global
  e-MArket based on efficient catalog management

• Highly dynamic e-commerce environment
• Constantly changing sales item information
• Flooding information a lot of shopping sites
• I want to know how I can purchase goods with the
  best condition!!

7
Case study CIGMA Goal

• CIGMA aCtive Inventory service in Global
  e-MArket based on efficient catalog management

• CIGMA service and system
• Provide aggregated catalog information from a lot
  of sites
• Assure up-to-date information
• Serve large number of customers with fast response

8
Case study CIGMA Design and Implementation
9
Case study CIGMA Experiment

• Questions!!
• What kinds of experiments do you want to do?
• Why do you want to do those experiments?

10
Case study CIGMA Experiment

• What and Why - Remind your goal and
  contribution
• CIGMA should deal with a lot of shopping sites.
  So it should process a large number of update
  requests from multiple sites
• Catalog update throughput
• CIGMA should assure up-to-date information. So
  update delay should be small
• Catalog update delay from shopping site to CIGMA
• CIGMA should handle a lot of customers requests.
  So it should process a number of customer
  requests at the same time
• Catalog browse throughput
• CIGMA receives update requests and customer
  requests concurrently. So it should process both
  of requests
• Mixed throughput

11
Case study CIGMA Experiment

• One more Question!!
• How do you do those experiments?

12
Case study CIGMA Experiment

• What to do after you decide metrics such as
• Catalog update throughput
• Catalog update delay
• Catalog browse throughput
• Mixed throughput
• What do you need in order to measure those
  metrics??
• Do you have machines and software needed for
  implementation?
• Do you have your implementation which is working
  well?
• Do you have something to generate the workload?
• What kinds of parameters affect the experimental
  results??
• What is your expectation about your experimental
  results??

13
Case study CIGMA Experiment

• What do you need in order to measure those
  metrics??
• Do you have machines and software needed for
  implementation?
• ? Prepare your machines and software.
• Pentium III 1GHz, 2GB memory machine, 100M LAN
  Link, Linux OS, Tomcat server, FastDB Main Memory
  DB
• Do you have your implementation which is working
  well?
• ? Implement your systems core functionality to
  be verified and test your system.

14
Case study CIGMA Experiment

• What do you need in order to measure those
  metrics??
• Do you have something to generate the workload?
• ? Use existing benchmark tools. If not available,
  implement your own workload generator.
• Browse throughput Httperf benchmark used (c.f.
  TPC-W, SPEC-Web)
• Update throughput Update request generator
  implemented
• Update delay WAN emulation tool used

15
Case study CIGMA Experiment

• What kinds of parameters affect the experimental
  results??
• ? Find possible parameters
• ? Plan to choose and vary just one parameter at a
  time
• What is your expectation about your experimental
  results??
• ? Draw your expected graphs which show not only a
  pattern of result (linearly increasing,
  logarithmically increasing, decreasing, or
  keeping constant)
• but also specific numbers
• Help you to analyze your measured results
• ? Calculate your expected results mathematically

16
Case study CIGMA Experiment

• And then??
• Experiment!! Experiment!! Experiment!!
• Analyze your results
• Verify your expectations
• Modify your parameters, implementations, even
  design

17
Case study CIGMA Experiment

• Example result graphs

18
Three keys for experiment

• What
• Why
• How

19
Three keys for experiment

• What
• What do you want to show to people?
• Strongly connected to what you propose as your
  system goals and your contributions
• Quantitative
• Q. What kinds of metrics can you use?
• E.g. Throughput, latency, execution time, storage
  usage, of Disk I/O, of lookups, of
  failed lookups, BW consumption, etc.
• vs.
• Qualitative
• Q. What are core functionalities to show?
• E.g. Demo, screenshots

20
Three keys for experiment

• Why
• Why do you want to show it to people?
• Also strongly connected to your system goals and
  your contributions
• Q. Does it really prove your contribution?
• Q. Does it show your achievement more
  effectively than others?

21
Three keys for experiment

• How
• How can you show it to people?
• Experiments using real implementation and
  benchmark tool
• Simulations
• Demos

22
Step by step for experiment

• Determining What and Why
• Setting experimental environment
• Implementation
• Benchmark tools
• While (hypothesis verified to be true your
  approach turn to be wrong give-up)
• Making hypothesis
• Expect results and draw your expected graphs
• Verifying hypothesis
• Do experiment and get results
• Analyze results
• Giving feedback from your analysis
• To experimental parameters, system tuning
• To your implementation
• To your system design
• Explaining your results and lessons if you succeed