Cache Benchmarking with Web Polygraph

1
Cache Benchmarking with Web Polygraph

• Alex Rousskov
• Duane Wessels
• NLANR/IRCache
• www.ircache.net

2
Our Goals

• To develop high-quality, freely available
  benchmarking tools for proxy caches and related
  devices.
• To design reproducible and meaningful benchmarks
  modeled on real Web traffic.
• To develop standardized workloads in
  collaboration with customers and vendors.

3
About Web Polygraph

• It simulates web traffic based on user-provided
  parameters.
• versus replaying trace files
• versus measuring live devices
• It is highly configurable
• this has a downside
• It is free
• It is portable -- runs on many modern Unix
  systems
• Does not require special hardware

4
Interesting Metrics

• Throughput (requests per second)
• we prefer this to be an input parameter
• Response time (mean, histogram)
• Hit ratios
• Price
• HTTP features (persistent connections)
• Correctness (failed connections, false hits,
  stale responses, content validation)

5
Web Traffic Characterization

• File sizes
• Popularity, or recurrance
• how this changes with time
• Cachability
• Life cycle (last-modified, expires)
• Containers and embedded objects
• Content types (text, image, compressed blob)
• Real content (parsable HTML)
• Packet delays, packet loss, long-lived
  connections
• HTTP uglies (reload, Vary, etc)

6
Why Benchmarking Cachesis Hard

• Test duration needs to be very long
• need to reach steady-state conditions
• compressing time has strange side effects
• Many components involved (switches, NICs, etc)
• Cheating can be difficult to detect or prevent.
• Simulation of certain traffic parameters may not
  be possible.
• Some traffic characteristics are not known.

7
Mapping to the Real World

• Actual performance depends significantly on
  customer environment.
• There is no typical environment
• Benchmark stresses a product at its peak capacity
• A production system running at peak capacity
  leaves no room for growth or spikes.
• Vendors sometimes test unrealistic
  configurations.
• No solid data yet perhaps as workloads mature
  and stabilize.

8
Future Work

• Second IRCache bake-off in January
• Constantly improving Polygraph to support complex
  Web workloads.
• Always a need for better traffic characterization