Designing a Workload Scenario for Benchmarking MessageOriented Middleware

1
Designing a Workload Scenario for Benchmarking
Message-Oriented Middleware
Kai Sachs, Samuel Kounev, Marc Carter
, Alejandro Buchmann Databases and
Distributed Systems Group, TU Darmstadt /
Germany Computer Laboratory, University of
Cambridge / UK IBM Hursley Labs, Hursley Park,
Winchester / UK
2
Overview

• Introduction
• Workload Requirements and goals of the SPECjms
  benchmark
• Application Scenario for SPECjms
• Implementation Details
• Summary

3
Message Oriented Middleware (MOM)

• Used in many business domains
• Financial services and enterprise applications
• Health care
• Supply chain
• ...
• And in many technologies
• Enterprise Service Bus (ESB)
• Service Oriented Architecture (SOA)
• Enterprise Application Integration (EAI)
• ...
• Increasing importance Need for benchmark

4
Requirements of a benchmark

• Scenario representative of real-world
  applications.
• Exercise all critical services provided by
  platforms.
• Not optimized for a specific product.
• Reproducible results.
• No inherent scalability limitations.

5
Current State of MOM Benchmarking

• Many proprietary benchmarks for MOM servers
• Used for performance testing and product
  comparisons
• However
• These benchmarks do not meet all of the defined
  requirements
• Typically they...
• concentrate on stressing individual MOM features,
  and
• do not provide a comprehensive and representative
  workload for evaluating the overall MOM
  performance
• Currently no industry-standard benchmark for MOM
  Benchmarking SPECjms 2007

6
What is SPECjms 2007?

• Worlds first industry standard benchmark for MOM
  products supporting Java Message Service (JMS)
• Developed by the SPEC OSG-Java subcommittee with
  the participation of
• IBM
• TU Darmstadt
• Sun
• Sybase
• BEA
• Apache
• Oracle
• JBoss

7
Goals of SPECjms 2007

• Provide a standard workload and metrics for
  measuring and evaluating JMS-based platforms
• Provide a flexible framework for JMS performance
  analysis

8
Overview

• Introduction
• Workload Requirements and goals of the SPECjms
  benchmark
• Application Scenario for SPECjms
• Implementation Details
• Summary

9
Categories of Workload Requirements

• Representativeness
• Comprehensiveness
• Focus
• Scalability
• Configurability

10
Categories of Workload Requirements

• Representativeness
• Comprehensiveness
• Focus
• Scalability
• Configurability

11
Representativeness

• The goal
• Allow users to relate the observed behavior to
  their own applications and environments.
• Should simulate the way platform services are
  exercised in real-life systems.
• Therefore
• It should be based on a representative workload
  scenario
• Communication style and the types of messages
  should represent a typical transaction mix.

12
Comprehensiveness I

• The goal
• Exercise all platform features typically used in
  the major classes of JMS applications.
• Point-to-point and publish/subscribe messaging
  domains should be covered.
• The features and services stressed should be
  weighted

13
Comprehensiveness II

• Therefore
• A workload transaction mix has to consider many
  different dimensions
• Transactional vs. non-transactional messages.
• Persistent vs. non-persistent messages.
• Usage of different message types, e.g.
  TextMessages, ObjectMessages, ...
• Usage of messages of different sizes (small,
  medium, large).
• Publish/subscribe vs. point-to-point messages
  (topics vs. queues).
• ...

14
Focus

• Focus of the workload / benchmark
• performance and scalability of the JMS server's
  software and hardware components.
• Therefore
• Minimize the impact of other components and
  services.
• Especially JMS servers are typically less loaded
  than e.g. database or application
  servers.Example Is a database was used to
  store business data and manage the application
  state ? it could easily become the bottleneck
• Minimize the impact of client-side operations
  (e.g. XML parsing).

15
Scalability

• Dimensions of scaling the workload
• Horizontal scaling
• De/Increase the number of destinations (queues
  and topics)
• Keep the traffic per destination constant
• Vertical scaling
• De/Increase traffic per destination
• Keep the number of destinations fixed
• Preserve real-life relationships in modeled
  scenario
• Additionally Support for freeform scaling,e.g.
  user defined traffic per destination and number
  of destinations

16
Configurability I

• Provide a flexible performance analysis tool
• Allows users to configure and customize the
  workload, e.g. for research purposes
• Produce and publish standard results for
  marketing purposes
• Therefore
• Need for a framework which supports
• tuning,
• analyzing and
• optimizing
• performance of certain features / platforms

17
Configurability II

• A benchmark framework should allow
• precise configuration of workload and transaction
  mix
• to switch off business interactions(implies that
  interactions should be decoupled)
• Providing such a configurability is a great
  challenge
• Freeform modeDesign and implement interactions
  so that they can be run in different combinations
  depending on the desired transaction mix
• Standard modeIt has to be ensured, that the
  interactions always behave like defined in the
  application scenario

18
Overview

• Introduction
• Workload Requirements and goals of the SPECjms
  benchmark
• Application Scenario for SPECjms
• Implementation Framework
• Summary

19
The Application Scenario

• Represents a supply chain of a supermarket
  company.
• Participants
• Headquarters (HQ)
• Supermarkets (SM)
• Distribution Centers (DC)
• Suppliers (SP).
• Based on the previously discussed requirements.

20
The Application Scenario

• Why again a Supply Chain Scenario?
• Excellent basis for defining different
  interactionsMany destinations, use cases, ...
• Typical real word application
• Importance of performance (RFID!)
• Allows scaling the workload in a natural way
• Horizontal e.g. scale the number of SMs
• Vertical e.g. scale amount of products sold per
  SM

21
Participants
22
Participants - Supermarkets

• Supermarket (SM)
• sells goods to end customers.
• manages its inventory.
• every supermarket offers different products.
• every supermarket is supplied by exactly one of
  the distribution centers.

23
Participants - Distribution Center

• Distribution Center (DC)
• supplies the supermarket stores which sell goods
  to end customers.
• responsible for a set of stores in a given area.
• is supplied by external suppliers.

24
Participants - Suppliers

• Supplier (SP)
• deliver goods to distribution centers (based on
  an offer of the supplier).
• not every supplier offers the same products.
• offers either all products of a given product
  family or none of them.

25
Participants - HQ

• Company HQ
• manages the accounting of the company.
• manages information about the goods and products.
• manages selling prices.
• monitors the flow of goods and money in the
  supply chain.

26
Interactions

• The following interactions are part of the
  scenario
• Order / Shipment Handling (SM / DC)
• (Purchase) Order / Shipment Handling (DC / SP)
• Price Updates
• Inventory Management
• Sales Statistics Collection
• Product Announcements
• Credit Card Hotlists

27
Interactions

• The following interactions are part of the
  scenario
• Order / Shipment Handling (SM / DC)
• (Purchase) Order / Shipment Handling (DC / SP)
• Price Updates
• Inventory Management
• Sales Statistics Collection
• Product Announcements
• Credit Card Hotlists

28
Interaction 1

• Order / Shipment Handling (SM DC)
• Includes two major steps
• SM sends an order to DC
• DC ships goods to SM
• Point-to-Point based communication /
  Intra-Company.

29
Interaction 1 - Order / Shipment Handling
Suppliers
Supermarket Company
Company HQ
Super-markets
1. A SM sends an order to its DC.
1
goods info. flow
DistributionCenters
only info. flow
30
Interaction 1 - Order / Shipment Handling
Suppliers
Supermarket Company
Company HQ
Super-markets
1
goods info. flow
DistributionCenters
2
only info. flow
2. The DC sends a confirmation to the SM and
dispatches shipment.
31
Interaction 1 - Order / Shipment Handling
Suppliers
Supermarket Company
Company HQ
Super-markets
3. The DC sends a (non-transactional,
non-persistent) message to the HQ (transaction
statistics).
3
1
goods info. flow
DistributionCenters
2
only info. flow
32
Interaction 1 - Order / Shipment Handling
Suppliers
Supermarket Company
Company HQ
Super-markets
4. The shipment arrives at the SM and
confirmation is sent to the DC.
3
goods info. flow
1
4
DistributionCenters
2
only info. flow
33
Example Interaction 2

• Purchase Order / Shipment Handling (DC SPs)
• Point-to-Point and Publish/Subscribe
  communication.
• Inter company communication.
• Includes six steps

34
Interaction 2 Purchase Order / Shipment Handling
Suppliers
Supermarket Company
Company HQ
Super-markets
1. DC sends a call for offers.
1
1
goods and info flow
DistributionCenters
only infoflow
35
Interaction 2 Purchase Order / Shipment Handling
Suppliers
Supermarket Company
Company HQ
2. All SPs offering the requested products send
an offer.
Super-markets
2
2
goods and info flow
DistributionCenters
only infoflow
36
Interaction 2 Purchase Order / Shipment Handling
Suppliers
Supermarket Company
Company HQ
Super-markets
3.Based on the offers, the DC selects a SP and
sends a purchase order to it.
3
goods and info flow
DistributionCenters
only infoflow
37
Interaction 2 Purchase Order / Shipment Handling
Suppliers
Supermarket Company
Company HQ
4.b SP sends an invoice to the HQ
Super-markets
4
4.c The SP dispatches a shipment to the DC.
4
goods and info flow
DistributionCenters
4.a SP sends an order confirmation to the DC
only infoflow
38
Interaction 2 Purchase Order / Shipment Handling
Suppliers
Supermarket Company
Company HQ
Super-markets
5
5. The shipment arrives at the DC and
confirmation is sent to the SP.
goods and info flow
DistributionCenters
only infoflow
39
Interaction 2 Purchase Order / Shipment Handling
Suppliers
Supermarket Company
Company HQ
Super-markets
6. The DC sends a message to the HQ (transaction
statistics).
6
goods and info flow
DistributionCenters
only infoflow
40
Overview

• Introduction
• Workload Requirements and goals of the SPECjms
  benchmark
• Application Scenario for SPECjms
• Implementation Details
• Summary

41
Message Types and Destinations

• Message types
• All messages types supported by the JMS
  Specification excepted ByteMessages
• 19 different messages are defined
• Three different sizes per message (small, medium,
  large) with a certain probability
• Acknowledgment mode
• Standard AUTO_ACKNOWLEDGMENT (can be changed in
  several interactions)
• (Non-)Persistent, (Non-)Transactional, Durable,
  ...

42
Message Types and Destinations

• Number of queues per location instance
• Number of topics3 one for every product family

43
Event Handler

• Every Location has a set of different Event
  Handler (Message Consumer) for different incoming
  messages.ExampleA DC has an Event Handler for
• Incoming Orders of the SMs
• Shipping Confirmations
• Incoming Offers, ... .
• Event Handlers ...
• contain the business logic of the scenario.
• are stateless from a business point of view.

44
Driver Framework

• Many locations represented by many event handlers
  (message consumers)
• Event handlers may be distributed across many
  physical machines.
• Reusable driver framework addresses this issues
  without any inherent scalability limitations.
• Plain Java
• Maximum choice in laying out workload to achieve
  maximum performance.

45
Driver Framework
46
A Flexible Framework for Performance Analysis

• Allows to configure and customize the workload /
  transaction mixes
• Provides three different topologies corresponding
  to three different modes in which the benchmark
  can be run
• Vertical
• Horizontal
• Freeform
• Many features

47
A Flexible Framework for Performance Analysis

• Some features
• Number of physical locations (HQ, SM, DC, SP)
  emulated.
• Number of agents representing a single physical
  location.
• Number of event handlers in an agent of each
  type.
• Number of driver instances for each interaction.
• Total number of invocations of each interaction
  (as an alternative to specifying a rate).
• Message size distributions for each interaction.
• The driver nodes on which agents are run.
• Number of JVMs run on each node and the way
  agents are distributed among them.
• Number of javax.jms.Connection objects shared
  amongst event handler classes within a single
  agent.
• ....

48
Overview

• Introduction
• Workload Requirements and goals of the SPECjms
  benchmark
• Application Scenario for SPECjms
• Implementation Details
• Summary

49
Summary

• The presented scenario models a set of
  interactions in the supply chain of a supermarket
  company.
• These interactions are used as a basis in SPEC's
  new SPECjms benchmark.
• SPECjms will be the world's first
  industry-standard benchmark for MOM products.
• SPECjms can be used to stress and evaluate the
  different aspects of JMS performance.
• SPECjms is more than a benchmarkOffers also a
  performance analysis tool for JMS-based
  infrastructures.

50
Summary II

• SPECjms provides a complete level playing field
  for performance comparisons of competitive MOM
  products.
• Producing and publishing standard results (e.g.
  for marketing purposes)
• to tune and optimize a platform or to analyze the
  performance of certain specific features.
• It provides a flexible performance analysis
  framework which allows the user to precisely
  configure the workload and transaction mix to be
  generated.

51
SPECjms Benchmark

• SPEC(Standard Performance Evaluation
  Corporation)
• Goal JMS Benchmark based on real world
  application.

52
SPECjms Scenario

• Supply chain of a supermarket company
• Four roles Supplier, Headquarter, Distribution
  Center, Supermarktes
• Selected 7 typical interactions
• (Purchase) Order Management
• Price Updates
• Product Announcements
• Inventory Movements
• Statistical Data
• Workload verified by multiple vendors

53

• Thanks for
• your attention