HTTP-Tester- A web appliance testing tool

1
HTTP-Tester- A web appliance testing tool

• Kiran Komaravolu
• Dept of Computer Science and Engg
• Lehigh University

2
What is the need to test Web Appliances

• Need to use best components.
• Compare performance of different web devices.
• Verify if a device is performs as per
  specifications.
• Choosing between different service providers.

3
Related Work

• The Ethereal network analyser (www.ethereal.com).
• TCPDUMP/PCAP (www.tcpdump.org).
• Httpflow (Grevnin and Davison)
• Simultaneous proxy evaluation. (Davison).
• Medusa Proxy. (Koletsou and Voelker).

4
Http-Tester

• Replays captured requests to an existing server
  to one that is being evaluated.
• No active deployment needed.
• Does not affect existing services.
• Can be used to evaluate any Web appliance running
  HTTP.

5
HTTP-Tester

• Topology setup

6
HTTP-Tester

• Components
• The Sniffer.
• Logs all packets onto a packet database.
• Request Replicator.
• Replays all requests to the tested device
• The Post Processor.
• Reads the packet logs, extracts HTTP requests and
  responses and correlates them.

7
HTTP

• HTTP Message Structure
• A command line. (GET, POST, OK etc.)
• GET http//www.google.com / HTTP 1.0
• Zero or more header lines.
• Content-Type text/html charsetiso-8859-1
• Control Return Line Feed.
• Optional Message Body
• Response message

8
HTTP Transactions.

• HTTP 1.0
• No persistent connections.
• A new connection is opened up for every request
  made, and is closed down after response sent.
• HTTP 1.1
• Supports Persistent Connections.
• Client may make a series of requests on a
  connection. Server returns responses in same
  order.
• Supports chunk encoding.

9
The Sniffer

• Blindly captures all relevant network packets and
  logs them.
• The pcap packet capture library has been used to
  read packets off the wire.
• Berkeley DB package has been used to log the
  packets into a db database.
• The job of efficient memory and disk usage has
  been delegated to DB.

10
Request Replicator

• Replays all requests flowing into original device
  onto the tested device.
• Works in parallel with the sniffer module.
• Looks for TCP SYN packets and determines the
  beginning of a request flow. And opens up a new
  connection to the second device.
• The payload every next packet sent to on this
  connection is sent to the second device.

11
Request Replicator.

• This way the second server sees the same workload
  as the first one.
• The replies sent by the server are ignored. They
  are flushed out as the socket buffers overflow.
  The sniffer module though logs the TCP packets.

12
Post Processing.

• Extracts HTTP requests/responses from the packet
  logs.
• Every request is mapped onto its response. The
  same request response pair on the second
  connection is also identified.
• Httpflow routines used to extract HTTP requests
  from packet stream.

13
Identifying Requests.

• We know which way requests flow.
• Assumption First TCP packet of every request has
  one of GET, HEAD,OPTIONS, DELETE, POST, PUT.
• End of headers is determined by a CRLF.
• content-length field is specified in case of
  PUT/POST requests.

14
Identifying responses

• Response headers extrated by looking for a CRLF.
• End of Response message determined in three ways
• Server sends a FIN or RST packet.
• Response headers contain a content-length
  field.
• Response is chunk encoded. Each chunk has its own
  length. Zero length chunk indicates end of stream.

15
Correlating requests and responses

• HTTP 1.0