BP2110: Whats New in SonicMQ V5.0

1
BP2110Whats New in SonicMQ V5.0

• Dave Chappell
• mailtochappell_at_sonicsoftware.com
• Vice President Chief Technology Evangelist
• Technology Advancement Group

2
Agenda

• Messaging Review
• Clustering in SonicMQ CWADS CWAQS
• Dynamic Routing Architecture
• Security
• Management
• HTTP Direct
• Performance
• Flow Control

3
What is Messaging For?

• In support of distributed computing
• Communicating events
• Making requests
• Most typically for integration
• Where a business process is realised by multiple
  applications
• Creating an application out of components
• Monitoring equipment on a network
• Front ending RPC systems

4
RPC-based Solution
Point-to-Point Integration
Data A
Data B
Data C
Data D
Data E
?
Application A
Application B
Application C
Application D
Application E
?
API A
API B
API C
API D
API E

• Each application must understand all other APIs
  and how to locate/call them across the network
• n(n-1)/2 up to 10 interfaces!
• If one application changes or breaks, others will
  be affected

5
Message-Oriented Middleware (MOM)
Robust, Loosely-Coupled
Data A
Data B
Data C
Data D
Data E
Application B
Application A
Application C
Application D
Application E
Message i/f
Message i/f
Message i/f
Message i/f
Message i/f
Messaging Backbone

• Able to cope with API changes
• n 5 interfaces
• Continues to operate if one component dies

6
SonicMQ at Work
Multi-National Bank
Middle Office
Back Office
Internal External Users
Front Office
Web Browser
Derivatives Presentation / Trading
Client Application
Firewall
SonicMQ Messaging Backbone
7
What Makes SonicMQ Different?

• Manageability
• Addresses cost of ownership
• Scalability
• Single broker
• Clusters
• Communities of clusters
• Performance
• Security
• More secure
• Few CPU cycles
• Better integration with 3rd party products

8
The Programming Model

• Picture of JMS objects
• Followed by code

JMS Client
JMS Client
Connection
Connection
Message Broker
Session
Message
Message
Session
Destination eg Queue1
Consumer
Producer
9
Hello World

• TopicConnection conn conFactory.createTopicConne
  ction(username,password)
• TopicSession pubSession conn.createTopicSession(f
  alse, Session.AUTO_ACKNOWLEDGE)
• TopicSession subSession connection.createTopicSe
  ssion(false,
• 
  Session.AUTO_ACKNOWLEDGE)
• TopicPublisher publisher pubSession.createPubli
  sher(chatTopic)
• TextMessage message pubSession.createTextMessage
  ()
• message.setText(Hello world)
• publisher.publish(message)

10
Message Types
11
Queues vs Topics

• Queues
• One message goes to one consumer
• Point to point conversation between 2 parties
• Or workflow with many parties
• Topics
• One messages goes to many subscribers
• Typically publishers and subscribers are very
  decoupled
• Fire and forget
• Or, point to point plus audit

12
Qualities of Service

• Discardable
• Non-persistent async
• Non-persistent
• Persistent
• Durable
• Locally Transacted
• Distributed Transacted

13
Guaranteed Delivery
Ensures Reliable Messaging
Publisher
Subscriber
SonicMQBroker
Inventory
Retail
2-Disconnect
7-Publish() method returns
RecoveryLog
5-Message retained in persistent store
14
Anatomy of a Broker


SonicMQ Broker

Producer

TCP/IP

As necessary,

Consumer






Check permissions

send()

TCP/IP


Producer




Log

process





Deliver to clients

send()



Buffer for slow clients

process

send()



Buffer for disconnected




Recovery Log
Database
15
Agenda

• Messaging Review
• Clustering in SonicMQ CWADS CWAQS
• Dynamic Routing Architecture
• Security
• Management
• HTTP Direct
• Performance
• Flow Control

16
Clusters

• The challenge
• When I reach the scalability limits of a single
  broker how do I add in more capacity
• How do I ensure that each broker is contributing
  equally
• How do ensure that it is completely transparent
  to clients
• Destinations, durable subscriptions,
  authorization, authentication
• Can I add capacity dynamically with no impact to
  cluster or clients

Broker 1
Broker 3
Broker 2
17
Load Balancing and Failover

• Clusters have recognized entry points
• Stored in connection factory
• Multiple entry points for failover purposes
• The entry points then redirect connections to
  another broker in the cluster
• Ability to weight brokers so that the algorithm
  is not purely round robin

Broker 1
Broker 2
Broker 3
18
Cluster Wide Queues
Queue Receivers

• Clustered Queues are defined on the cluster
  rather than a broker
• Both senders and receivers can be load balanced
  across the cluster
• If a broker fails the rest of the queues
  contents are available

Broker 1
Broker 3
Broker 2
Queue Senders
19
Cluster Wide Queues

• New Option when Creating the Queue
• Management Console
• Configure Queue on the Cluster (not on the
  Broker)
• One configuration definition ? applied to each
  broker as it joins cluster
• same name/size/properties for all instances
• Fragment of each Clustered Queue is created on
  every broker in the cluster
• Total state of the queue is federated across the
  fragments
• Senders and receivers manipulate messages in
  Queue fragment on broker they are connected to
• Should a receiver empty his local queue fragment
  messages are pulled from other fragments

20
Cluster Wide Durable Subscriptions
Durable Subscribers

• Adds the ability to subscribe to messages
  regardless of which broker the client is
  connected to
• Allows use of load balancing when reconnecting to
  the cluster
• If a broker fails the rest of the subscriptions
  contents are available

Broker 1
Broker 3
Broker 2
Topic Publishers
21
Group Subscriptions
Subscriber Pool A

• One slow subscriber can slow down the performance
  for all subscribers
• One solution is to add in more clients to process
  the subscription
• Allow a single subscription to be shared by a
  group of clients
• Stream of messages for subscription is divided up
  amongst the clients
• Used to allow load sharing for a single
  subscription

Subscriber A
Subscriber A
Subscriber A
Publisher
Topic X
Subscriber B
Subscriber C
22
Syntax

• Create Topic name with Group Prefix
• group-prefixTopic-name
• Topic Object
• Can be stored in JNDI
• Old Clients can subscribe to this topic
• No non-JMS API calls
• Therefore works with App Servers
• Cannot Publish to this topic
• ACLs / QoP based only on Topic-name

23
Agenda

• Messaging Review
• Clustering in SonicMQ CWADS CWAQS
• Dynamic Routing Architecture
• Security
• Management
• HTTP Direct
• Performance
• Flow Control

24
Dynamic Routing Architecture - DRA

• The challenge how do I scale beyond a single
  cluster?
• The cluster is getting too big, or
• I need to extend outside my organization
• Departments, sites, subsidiaries, business
  partners
• Need a loosely coupled solution
• Connections
• Security domain
• Solution is the ability to link clusters together
  into more complex topologies

25
DRA and Connections
Broker 4
Broker 1
Broker63
Broker 5
Broker 3
Broker 2

• Choice of outgoing broker
• Choice of load balancing incoming broker
• Messaging pending transfer in routing queue

• Connections on demand
• Disconnected when idle
• Routing User for incoming connection

26
DRA Flavor 1 Directed Sends

• Directed sending for Queues and Topics
• Use NODEDestination syntax
• Sender directs a send to a different Node
• DRA Connection acts as a on-demand proxy
  connection
• Not delivered to clients on sending node
• Behavior
• Send / Publish (NodeTopic) puts message in
  Routing Queue
• Routing Connection triggered by message
• Delivered broker-to-broker

27
Flavor 2 - Global Subscriptions

• A very flexible extension to the existing
  clustering of topics
• Administrators specify how subscriptions are
  propagated
• Table of topic / node maps
• Can use wildcards for topic names
• Publishers are completely unaware of who is
  subscribed
• Message makes its way across routing connection
  to remote subscriber
• Multi-hop delivery can be configured

Topic A publisher
Topic A subscriber
Cluster 1
Topic A subscriber
Cluster 2
Topic A subscriber
Cluster 3
28
GSA Multi-node Diagram
29
Agenda

• Messaging Review
• Clustering in SonicMQ CWADS CWAQS
• Dynamic Routing Architecture
• Security
• Management
• HTTP Direct
• Performance
• Flow Control

30
Comprehensive Security
Flexible Deployment Options

• Encryption
• Built-in payload encryption
• Secure messaging without performance cost of SSL
• Channel encryption
• SSL with up to 256-bit keys
• Authentication
• Built-in username/password authentication
• Supports digital certificates for user
  authentication
• Authorization
• Specify rights of authenticated users
• Exploit destination hierarchies and groups of
  users to simplify administration

31
Pluggable Authentication

• At the client
• A plug-in capability to insert whatever
  authentication technology you want
• User / password, operating system login,
  thumbprint, iris scan,
• Uses an open standard - JAAS
• In authentication stacks if appropriate
• Will allow integration with Single Signon,
  operating system sign on, 3rd party products, etc
• At the server
• 5.0 will add integration with external LDAP store
  of principals
• Will require a small amount of configuration via
  PSO
• Other stores or schemas through PSO

SonicMQ Client
JAAS Plugin
Credentials
Connect
Broker
SPI
Credentials
Login
LDAP Security Repository
32
Agenda

• Messaging Review
• Clustering in SonicMQ CWADS CWAQS
• Dynamic Routing Architecture
• Security
• Management
• HTTP Direct
• Performance
• Flow Control

33
Driving down the TCO
Cluster 1
Cluster 2
100 Clustered Queues 1000 Topics across both
clusters5000 Users in 100 groups Deployed on
single- processor Intel boxes
Broker 1
Broker 2
Broker 3
Broker 4

• 8 Capacity Units WebSphere MQ _at_ 2,250
• 4 Capacity Units Event Brokers _at_ 22,500
• 4 PCs _at_ 2000
• 0.5 Administrator _at_ 50,000 / year

• 4 Enterprise Plus _at_ 7,500
• 4 PCs _at_ 2000
• No Administrator166,000 vs 38,000

34
How Do We Achieve This?

• Tools
• Boost productivity with new management tools
• 1st rate infrastructure
• Underpin new tools with a robust infrastructure
• Dynamic discovery vs static configuration
• Automate much of the configuration work
• Use of Open Standards
• JMX for management
• Underpinned by SonicMQ for transport

35
Managing Clusters
Broker 1
Directory Storage
JMS Agent
Directory Service
Mgmt Broker
Cluster
Domain Manager
36
Management Console - Configure

• Create / edit configuration of brokers
• No more broker.ini
• Create configuration templates which are reused
  many times
• All configuration data written to the Directory
  Service
• Also pushed out to broker and cached locally

37
Management Console - Manage

• Perform lifecycle operations
• Start / Stop
• Reload
• Schedule start / stop
• View Metrics / Notifications
• The health of brokers can be monitored remotely
• What are they?
• Component specific operations
• Clear messages on a queue
• Drop connections
• Browse Durable Subscriptions

38
Agenda

• Messaging Review
• Clustering in SonicMQ CWADS CWAQS
• Dynamic Routing Architecture
• Security
• Management
• HTTP Direct
• Performance
• Flow Control

39
HTTP Direct Features

• New in 5.0
• Web Services handlers
• Inbound SOAP
• Outbound SOAP
• HTTP Direct
• Polling receive
• Allows light-weight devices to receive messages
• HTTP Direct for JMS

Web Service
SonicMQ
JMS Client
40
HTTP-Direct Features
- New to MQ 5.0
41
Agenda

• Messaging Review
• Clustering in SonicMQ CWADS CWAQS
• Dynamic Routing Architecture
• Security
• Management
• HTTP Direct
• Performance
• Flow Control

42
Where have we improved?

• Durable Subscriptions
• Reconnected durables are 2.5 times faster
• Transactions
• Have re-architected the transaction sub-system
• Now 5 times faster than before
• Secure Messaging
• Strong QoP approximately 100 faster than SSL

43
Agenda

• Messaging Review
• Clustering in SonicMQ CWADS CWAQS
• Dynamic Routing Architecture
• Security
• Management
• HTTP Direct
• Performance
• Flow Control

44
Flow Control

• The challenge for all messaging providers
• What do I do when production rate exceeds
  consumption rate
• The options
• Buffer in memory
• Buffer on disk but not in memory
• Throw an exception on the producer
• Block the offending producer
• Our approach
• 1 (3 4)

45
Questions