Internet and Intranet Protocols and Applications

1
Internet and Intranet Protocols and Applications

• Lecture 14
• Introduction to MQSeries
• And Asynchronous Messaging
• May 1, 2002
• Joseph Conron
• Computer Science Department
• New York University
• jconron_at_cs.nyu.edu

2
What is MQSeries?

• A middleware product that implements a messaging
  and queuing framework.
•  
• Middleware - an intermediate software component
  that bridges dissimilar computing environments.
• Unix, MVS, OS/400 Tandem, VMS, NT, etc.
•  SNA, NetBios, TCP/IP
• Cobol, C, JAVA

3
Messaging and Queueing

• Messaging - programs communicate by sending data
  in messages rather than by calling each other
  directly.
• Queuing - messages are put on queues in storage,
  eliminating the need for programs to be logically
  connected.
• A messaging and queuing framework is inherently
  ASYNCHRONOUS!

4
Asynchronous vs. Synchronous Communications

• Synchronous App sends request, then blocks until
  request is processed.
• Requires service available at EXACTLY same time
  as client needs service.
• Asynchronous App sends request and checks at
  some future time if complete.
• Service need not be available when client sends
  request

5
Synchronous good/bad
Good
Bad

• Easy to program
• Outcome is known immediately
• Error recovery easier (usually)
• Better real-time response (usually)

• Service must be up and ready
• requestor blocks, held resources are tied up
• Usually requires connection-oriented protocol

6
Asynchronous good/bad
Good
Bad

• Requests need not be targeted to specific server.
• Service need not be available when request is
  made
• No blocking, so resources could be freed
• Could use connectionless protocol

• Response times are unpredictable
• error handling usually more complex
• Usually requires connection-oriented protocol
• Harder to design apps?

7
Messaging vs. Procedure Calls

• As programmers, many of us think procedurally, so
  using procedures is natural extension of how
  we think.
• Messages are an abstract concept harder for us
  to conceptualize relationship between actions and
  messages!
• But Wait!
• Something good happens when we use abstractions!

8
ltSoap Boxgt
We are free to concentrate on the design of the
application itself. We are no longer concerned
with details of the environment. Our application
is suddenly portable and to some degree,
extensible.
lt/Soap Boxgt
9
A Brief History of MQSeries

• 1992
• Systems Strategies (SSI) develops ezBridge, a
  messaging and queuing product for VMS, Tandem,
  and Unix
• IBM announces Networking Blueprint defining three
  standard APIs for program to program
  communication CPI-C, RPC, MQI
• 1992-3
• State Street Bank (Boston) evaluates IBM
  messaging product (code name Victory) for IBM
  CICS/ESA and SSIs ezBridge on VMS and Tandem.
• State Street Bank would like to announce the
  wedding of
• IBM and Systems Strategies!
• 1993
• IBM buys intellectual property rights for
  ezBridge from SSI

10
The Brides Wedding Preparation(What SSI had to
do)

• Implement IBM Channel Protocol over TCP/IP and
  LU6.2 (more about channels later).
• Implement the MQI interface functions (MQCONN,
  MQOPEN, MQPUT, MQGET, MQCOMMIT, MQCLOSE, MQDISC).
• Implement MQI Error Semantics (failure conditions
  should look the same on all systems)

11
MQSeries/ezBridge Integration Objectives

• Applications written in C using ezBridge on VMS
  and Tandem had to exchange messages with
  applications written in (COBOL?) Using MQI under
  CICS/ESA.
• System intercommunication using channel protocol
  over TCP/IP and LU6.2 (that is, A VMS system had
  to look to IBM machine just like another IBM
  system!).

12
MQSeries Platform Rollout

• Initially, IBMs version of MQSeries ran only on
  mainframe
• (CICS/ESA, IMS/ESA, and eventually VSE).
• SSI version (called MQSeries Version 1) initially
  released on
• VMS, Tandem, AS/400, and Unix (SCO, UnixWare).
• 1994/1995 -IBM releases the first three
  distributed platforms
• AIX, OS/2, and AS/400.
• The AIX version becomes the reference port.
• Over time, IBM replaced SSI versions with ports
  of its reference system.
• Today - MQSeries runs on over 35 platforms

13
How Messaging Queuing Works
Programs communicate by putting messages on
queues. Here, program A puts a message on
Queue1, which is read by program B.
Note A and B need not be on the same machine!
14
How Messaging Queuing Works (2)
Communication can be one-way or two-way. Here, A
sends to B on Queue1, and B responds to A on
Queue2
15
Messaging and Queuing Characteristics

• Three key facts about Messaging and Queuing
  differentiate it from other communication styles
• 1) Communicating programs can run at different
  times.
• 2) There are no constraints on application
  structure.
• 3) Programs are insulated from environmental
  differences.
• Lets examine each of these characteristics in
  more detail.

16
Applications Can Run at Different Times
Either program can be unavailable
Key Concept message queue exists independently
from programs that use them!
17
No Constraints on Application Structure
There can be a one to many relationship between
applications
Or a many to one relationship between applications
18
Applications Shielded from Environmental
Differences

• Dont care what OS is used.
• Dont care what language theyre written in
• Dont care what the underlying communication
  protocol is used.

19
Applications Shielded from Environmental
Differences
Applications
Programmatic API
Queue Manager
Queue Manager
Communications using Message Channels
Message Queue
20
MQSeries Objects Queue Manager

• Queue Manager
• Controls access to queues
• administration (create, delete, etc)
• usage (Put, Get)
• serves as transaction (syncpoint) coordinator for
  all queue operations.
• Accessed through the Message Queue Interface
  (MQI)
• Queue Managers have names (identities) that are
  UNIQUE in a network (like host names).

21
MQSeries Objects Queues

• MQSeries defines four types of queues. A queue
  instance is fully qualified by its queue manager
  and queue name.
• Local Queue - an actual queue for which storage
  is allocated.
• Remote Queue - a definition of a queue on a
  different queue manager (acts somewhat like a
  pointer)
• Alias Queue - another name for a local or remote
  queue. Typically used to switch queue
  destinations without modifying program code.
• Model Queue - a template whose properties are
  copied when creating a new dynamic local queue (
  create queue xxx like queue yyy).

22
MQSeries Queues Properties

• Maximum Message Size
• Maximum Queue Depth
• High/Low Factors
• Enable/Disable Put or Get
• Persistent/Not Persistent

23
MQSeries Queues Events and Triggering

• Local queues can generate events (messages) under
  certain conditions (like queue full).
• These event messages can be used to trigger
  the execution of a program.
• These events are called trigger messages. The
  queue on which they are put is called an
  Initiation Queue.

24
MQSeries Objects Processes

• Process defines an application to an MQSeries
  queue manager. A process definition object is
  used for defining applications to be started by a
  trigger monitor.
• A trigger monitor is a program that listens on an
  initiation queue and executes commands named in
  Process definitions.
• Triggering is useful when you dont want to
  deploy long-running programs.

25
MQSeries Objects Message Channels

• provide a communication path between two queue
  managers on the same, or different, platforms.
• A message channel can transmit messages in one
  direction only. If two-way communication is
  required between two queue managers, two message
  channels are required.
• Question why make message channels one-way?

26
MQSeries Objects Message Channels(2)

• Types of message channels
• Sender - initiates connection to Receiver
• Server - Accepts request to start from requester,
  then becomes Sender
• Receiver - Passive waits for initiation sequence
  form Sender
• Requester - Active at start, then becomes
  Receiver
• Cluster-sender (used amongst Cluster Queue
  Managers)
• Cluster-receiver (ditto)

27
MQSeries Message Channel Concepts

• The Sender side of the session is the
  transaction coordinator.
• Message channels implement a protocol that
  includes a commitment protocol.
• Channels recover from failure by agreement they
  must agree on the last committed unit of work
  would this be harder if channels were
  bi-directional??
• Message Chanels are implemented by programs
  called Message Channel Agents (MCA)

28
How messages move across channels
(1) Application puts message on transmission queue
(4) Message is available on local queue for
applications
(2) Sender MCA gets message and sends to partner
MCA
(3) Receiver MCA puts message on target queue
29
MQSeries Assured Delivery

• If queues are persistent, and message is
  persistent, then MCAs will eventually deliver the
  message to the target queue, and target
  application will get it!
• MCAs have recoverable state - theyre STATEFUL -
  and a connection-oriented protocol.
• Message is not removed from xmit queue until
  partner MCA confirms placement on target queue

30
MQSeries Objects Messages

• Messages consist of
• Header (MQMD)
• Used by Queue Manager and application to handling
  properties of the message
• User Data
• The application-to-application data (payload)
• transparent to MQSeries

31
MQSeries Messages Properties

• Destination Queue
• Reply Queue name
• Time to live (expiry)
• Format
• Correlation ID
• Persistence
• Report options

32
MQSeries Messages Properties (2)

• Messages can be individually designated
  persistent or non-persistent (persistent messages
  are logged to enable recovery)
• Correlation ID - select which message to get from
  queue
• Segmented Messages - allows ending of VERY LARGE
  messages (gt 100 MB)

33
MQSeries Messages message types

• Request - used by one program to ask another
  program for something (usually data). A request
  message needs a reply.
• Reply - used in response to a request message.
• A one-way message, as you would expect, doesnt
  need a reply, though it can carry data.
• A report message is used when something
  unexpected occurs, or to give extra information
  like
• message delivered to target queue
• message taken by application
• message not deliverable
• message exceeded time-to-live

34
MQSeries MessagesUnits of Work and Transactions

• Messages are added and removed from queues in
  Units of Work
• The smallest Unit of Work is one message.
• Units of work are atomic.
• When an app reads a message from a queue, a
  message appears to have been removed, but in
  fact, it is still in storage until the app
  commits the unit of work.

35
MQSeries Transaction Support
Unit of recovery - a piece of work that changes
data from one point of consistency to
another. Syncpoint - A point of consistency
(also called a or commit point). It is a moment
at which all the recoverable data that an
application program accesses is consistent.
36
MQSeries Transaction Support (2)

• Applications are responsible for delimiting the
  beginning and end of a transaction. How can
  messaging be coordinated with a data base update?
• MQSeries is XA compliant and can operate with
  other XA compliant systems as either a
  transaction manager (coordinator) or resource
  manager (particpant).
• Some examples Sybase, DB2, Oracle.

37
MQSeries Logging and Recovery

• All operations that affect the state of the
  Queue Manager and its objects are logged to a log
  file.
• What is state?
• Object definitions (queue manager, queues,
  processes, channels, etc)
• Queue content (messages)
• What about message channel state?
• Message channel states are logged separately by
  each channel

38
MQSeries Logging and Recovery (2)

• Two forms of Logging
• Circular log records are written sequentially
  across several files, then wrap back to the
  first file.
• Linear - log records are written sequentially
  across files. New files are allocated as current
  files fill. No automatic reuse of file space!
• Problem Length (in time) of longest running
  transaction vs amount of writes to log determines
  size of log needed.

39
MQSeries Logging and Recovery (3)

• Observations
• Circular logging is easy to manage, but is fatal
  if log is damaged (hard to backup circular
  logs!).
• Linear logging is hard to maintain but provides
  for archiving of previous logs (still a problem
  if current log is damaged).

40
MQI The MQSeries Programming Interface
MQCONN Connect to queue manager MQDISC
Disconnect from queue manager MQOPEN Open
queue MQCLOSE Close queue MQPUT Put
message MQGET Get message
41
MQI The MQSeries Programming Interface (2)
MQPUT1 Put one message (implied
open/put/close) MQBEGIN Begin unit of
work MQCMIT Commit MQBACK Back out MQINQ
Inquire about queue attributes MQSET Set queue
attributes
42
MQSeries Language Support

• C
• C
• Cobol
• JAVA (native and JMS)
• PL/I
• System 390 Assembler
• TAL (Tandem)
• Visual Basic
• For More Information
• http//www-4.ibm.com/software/ts/mqseries/