SEDA:%20An%20Architecture%20for%20Well-Conditioned,%20Scalable%20Internet%20Services

1
SEDA An Architecture for Well-Conditioned,
Scalable Internet Services

• Matt Welsh, David Culler, and Eric Brewer
• Computer Science Division
• University of California, Berkley

2
An Application for
SEDA

• servicing requests for Internet services
• servicing a load (demand) that is never constant
• providing a service that scales to load
• keep in mind these issues of scalability
• load fluctuates, sometimes wildly
• you only want to allot the resources servicing
  the load requires
• every system has a limit! Scale responsibly!
• dont over commit the system

3
Design Goals
SEDA

• service requests for BOTH static content AND
  dynamic content
• demand for dynamic content is increasing
• work to deliver static content is predictable
• retrieve the static content from disk or cache
• send it back on the network

4
Design Goals
SEDA

• work to deliver dynamic content is unknown
• build content on the fly
• how many I/Os to retrieve the content?
• layout, insert, and format content
• may require substantial computation
• posing queries of a database and incorporating
  results
• all the more reason to scale to load!

5
Design Goals
SEDA

• adaptive service logic
• different levels of load require different
  service strategies for optimum response
• load is determined by BOTH the number of requests
  for service and the work the server must do to
  answer them
• platform independence by adapting to resources
  available
• load is load, no matter what your system is

6
Consider (just) Using Threads
SEDA

• the model of concurrency
• two ways to just use threads for servicing
  internet requests
• unbounded thread allocation
• bounded thread pool (Apache)
• hint remember !!
• (somewhat different reasons though!)

Threads!
7
Consider (just) Using Threads
SEDA

• unbounded thread allocation
• issues thread per request
• too many use up all the memory
• scheduler thrashes between them, CPU spends all
  its time in context switches
• bounded thread pool
• works great until every thread is busy
• requests that come in after suffer unpredictable
  waits --gt unfairness

8
Consider (just) Using Threads
SEDA

• transparent resource virtualization (!)
• fancy term for the virtual environment
  threads/processes run in
• threads believe they have everything to
  themselves, unaware of constraint to share
  resources
• t. r. v. delusions of grandeur!
• no participation in system resource management
  decisions, no indication of resource availability
  to adapt its own service logic

9
DONT Consider Using Threads!
SEDA

• threaded server throughput degradation
• as the number of threads spawned by the system
  rises, the ability of the system to do work
  declines
• throughput goes down, latency goes up

10
Consistent Throughput is KEY
SEDA

• well-conditioned service
• sustain throughput by acting like a pipeline
• break down servicing of requests into stages
• each stage knows its limits and does NOT over
  provision
• so maximum throughput is a constant no matter the
  varying degree of load
• if load exceeds capacity of the pipeline,
  requests get queued and wait their turn. So
  latency goes up, but throughput remains the same.

11
Opaque Resource Allocation
SEDA

• incorporate Event-Driven Design in the pipeline
• remember -- only one thread!
• no context switch overhead
• paradigm allows for adaptive scheduling decisions
• adaptive scheduling and responsible resource
  management are the keys to maintaining control
  and not over-committing resources to account for
  unpredictable spikes in load

12
Back to the argument
SEDA

• Event-driven Programming vs. Using Threads
• the paper recommends event-driven design as the
  key to effective scalability, yet it certainly
  doesnt make it sound as easy as Ousterhout
• event handlers as finite state machines
• scheduling and ordering of events all in the
  hands of the application developer
• certainly looks at the more complex side of
  event-driven programming

13
Anatomy of SEDA
SEDA

• Staged Event Driven Architecture
• a network of stages
• one big event becomes series of smaller events
• improves modularity and design
• event queues
• managed separately from event handler
• dynamic resource controllers (its alive!)
• allow apps to adjust dynamically to load
• culmination is a managed pipeline

14
Anatomy of SEDA
SEDA

• a queue before each stage
• stages queue events for other stages
• note the modularity
• biggest advantage is that each queue can be
  managed individually

15
Anatomy of a Stage
SEDA

• event handler
• code is provided by application developer
• incoming event queue
• for portions of requests handled by each stage
• thread pool
• threads dynamically allocated to meet load on the
  stage
• controller
• oversees stage operation, responds to changes in
  load, locally and globally

16
Anatomy of a Stage
SEDA

• thread pool controller manages the allocation of
  threads. Number is determined by length of queue
• batching controller determines the number of
  events to process in each invocation of the
  handler

17
Asynchronous I/O
SEDA

• SEDA provides two asynchronous I/O primitives
• asynchronous non-blocking
• asynchronous socket I/O
• intervenes between sockets and application
• asynchronous file I/O
• intervenes to handle file I/O for application
• different in implementation, but design of each
  uses event-driven semantics

18
Performance
SEDA

• througput is sustained!

19
Summary
SEDA

• load is unpredictable
• most efficient use of resources is to scale to
  load without over committing
• system resources must be managed dynamically and
  responsibly
• staged network design culminates in a
  well-conditioned pipeline that manages itself
• event-driven design for appropriate scalability
• threads are used in stages when possible