StarTrace: e2e PathBased Tracing - PowerPoint PPT Presentation

Loading...

PPT – StarTrace: e2e PathBased Tracing PowerPoint presentation | free to view - id: 267ea6-NDJmO



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

StarTrace: e2e PathBased Tracing

Description:

Path information can be used as input to machine learning algorithms to optimize ... Path-based tracing is a good idea. Design Considerations ... – PowerPoint PPT presentation

Number of Views:37
Avg rating:3.0/5.0
Slides: 21
Provided by: james396
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: StarTrace: e2e PathBased Tracing


1
(Star)Trace e2e Path-Based Tracing
  • James Zhang, Jesse Trutna, Armando Fox, Dave
    Patterson

2
RAD Datacenter / Node
(datacenter cluster or RAMP)
Policy Maker
Per node SW stack
Load- Balancer (IDLB)
Intrusion- Detection (IDID)
Service (IDS)
Web 2.0 Applications
Firewall (IDF)
Ruby on Rails Interpreter
Web Svc APIs
Trace, X-trace
Local OS functions
trace, X-trace, Lib log, D-trigger,
Identity-based Routing Layer, …
Virtual Machine Monitor
Actuator Commands
Sensor Data
3
Overview
  • (Star)Trace is a framework for end-to-end
    path-based request tracing in distributed web
    applications
  • Trace leverages existing tracing tools
  • X-Trace (Berkeley in development)
  • D-Trace (Sun Microsystems production code)
  • Trace aims for low overhead (12) to make
    tracing feasible in production
  • We have build a preliminary implementation in
    Ruby-on-Rails and we are in the process of
    improving and evaluating the system

4
Motivation
  • Past work demonstrated that path information can
    be extremely useful for
  • Detecting and diagnosing failures
  • Identifying problematic interactions and
    bottlenecks
  • Obtaining a global view of distributed systems
  • Most current production web apps cannot track the
    path of a request
  • Path information can be used as input to machine
    learning algorithms to optimize applications

5
A Simple Example
XML-RPC
FCGI Request
HTTP POST …
lthtmlgt ltbodygt ….
SQL Insert …
Frag. Cache Static Cont.
6
A Simple Example
HTTP GET …
Hits page cache…
lthtmlgt ltbodygt ….
FCGI request
7
A Simple Example
FCGI Request
HTTP POST …
SQL Select …
lthtmlgt ltbodygt ….
8
Problem
  • Of course, all requests are interspersed and
    affect each other.
  • As applications evolve and expand, things get
    ugly quickly.
  • Bottlenecks and errors becomes very hard to
    diagnose.
  • Path-based tracing is a good idea.

9
Design Considerations
  • Instrumentation in the middleware to avoid source
    changes in business logic
  • J2EE Containers
  • Ruby interpreter Rails library/framework
  • Minimum overhead (12) to allow always on
    tracing in production
  • Architecture should be easily implemented in
    different middleware frameworks

10
Trace Overview
11
X-Trace Trace
  • Trace uses X-Trace identifiers (id/xid)
  • Requests are assigned probabilistically unique
    ids at each component (web server, app server,
    database, etc.)
  • Ids parent-child information used for path
    reconstruction

12
Logging Trace
  • Most web application components already performs
    logging
  • Apache, Ruby on Rails, Tomcat, MySQL
  • Modify logging framework to insert id in the
    logs
  • ids are delimited differently for fast pattern
    matching
  • Production logging overhead is low and carrying
    Trace information adds no measurable overhead

13
Logging Overhead
  • Different logging levels exists
  • Example RoR has production or development level
    logging, in addition to webserver logs and/or
    database logs.
  • On a Linux VM
  • No logging --gt production logging 2.5
  • Production logging --gt development logging 5.5
  • On a Solaris VM
  • No measurable effect

14
Prototype Implementation
  • Preliminary implementation of Trace exists for
    Ruby-on-Rails
  • Apache and Lighttpd webserver
  • Propagate and log Trace id
  • Rails framework
  • Stores Trace in thread-local variable
  • Propagate and log Trace id
  • Database
  • SQL queries are tagged with Trace id as comments
  • Deployed as a plug-in, require no modifications
    to the application code
  • Path-Analysis can be done online or offline

15
Dtrace Add-on
  • Dtrace is a ubiquitous tracing tool
  • Currently in Solaris, will be in FreeBSD and OS X
  • Trace anything as long as theres a provider
  • Created a custom Trace provider
  • Pass id into dtrace with a dummy function call
  • Loupe Coordinate dtrace with RoR dispatchers
  • Runs dtrace scripts from a central location
  • Can use arbitrary dtrace scripts
  • More detailed tracing on demand

16
Dtrace Trace
17
Path Reconstruction
  • Implemented a Version 0 tool for path
    reconstruction/visualization
  • Lumberjack extract data from logs
  • Apache Log, Rails Log, MySQL Query Log
  • SQLite save extracted data
  • id, timestamp, host, and other data (url, sql,
    errors)
  • Display paths using an AJAX website

18
Trace Demo Version 0
19
Experience So Far
  • Even in this small example trace was useful
  • Found some requests generated over 1000 SQL
    requests
  • Some SQL requests repeated over and over
  • Example show fields from XX

20
Future Work
  • Instrumentation BackPlane (IBP)
  • Need a better infrastructure to gather path
    information and reconstruct paths.
  • See Poster…
  • Implement Trace for other web frameworks
  • Java Servlets
  • Python (Django), Perl, PHP?
  • Evaluation with real websites with real users
About PowerShow.com