Best Practices for Implementing CA SPECTRUM Service Assurance Manager in an HA MSCS Environment

1
Best Practices for ImplementingCA SPECTRUM
Service Assurance Manager in an HA MSCS
Environment

2
Disclaimer

• Although the focus of this presentation is 2 node
  clusters, Microsoft Cluster supports more than 2
  server nodes. The concepts that apply to 2 node
  clusters, however, also apply to multiple server
  node clusters.
• The procedures specified here apply to Microsoft
  Cluster Server (MSCS) 2003. No tests have been
  performed for MSCS 2008
• Basic knowledge of cluster setup is required,
  such as how to offline resources, move group,
  etc. Basic knowledge on how to define cluster
  resources, customize SAM files is not required
  as this process has been fully automated

3
Agenda

• This presentation will cover the following
  topics
• Installing CA SPECTRUM Service Assurance Manager
  components in an MSCS environment
• Failover Considerations
• Installing SAM connectors with Silo Managers
  running in an MSCS environment

4
Objectives

• Support for Fault Tolerance is targeted for
  future release. May include scenario similar to
  SPECTRUM fault tolerance where primary and
  secondary SpectroSERVERs are required.
• This presentation reviews implementation and
  customization for CA SAM in a cluster
  environment.
• This presentation assumes you are already
  familiar with MSCS and the benefits of HA
  solutions provided by MSCS.
• Main objective of HA solution is to aim for zero
  downtime

5
Architecture Reviews
6
HA SAM Manager
SQL Virtual Node
SamStore
SILO Manager
Cluster Node A
SAM Manager
SAM UI
SILO Manager
Local Connectors
EEM
SILO Manager
SILO Manager
7
HA SAM Manager with Remote Data Store
HA SQL Server
SamStore
SILO Manager
SILO Manager
SILO Manager
8
HA SILO Managers
SAM Manager
Cluster Node A
Silo Mgr (NSM)
Local Connectors
SAM UI
Cluster Node B
HA SQL Server
Silo Mgr (NSM)
SamStore
Local Connectors
9
Preparing the Installation
10
SAM Installers

• SAM Components

11
Java Runtime

• SAM Manager requires Embedded Entitlements
  Manager (EEM) to be installed - not necessarily
  on cluster node
• If installed on cluster node, install Java
  Runtime, as well
• Java Runtime should be installed on each cluster
  node independently with following attributes
• Use real nodes
• Install on local drive
• No sharing of data

12
EEM Considerations

• SAM Manager requires EEM to be installed - not
  necessarily on cluster node
• IF EEM component must be installed on cluster
  nodes, it should be installed on each cluster
  node independently (similar to non-cluster node
  install)
• Use Real node
• No cluster resources are defined
• Configuration has to be carried out on all
  cluster nodes
• If EEM is installed on the cluster nodes, then
  Data Store replication should be implemented

13
Business Objects

• Business Objects Enterprise software provides
  reporting functionality to CA SPECTRUM Service
  Assurance Manager
• Should be installed in a non-clustered
  environment
• If High Availability is required for this, then
  review Best Practice document how to make this
  Highly Available.

14
CA SPECTRUM SAM

• If SAM manager is installed in cluster
  environment, review Installing SAM Manager in
  MSCS environment section
• If SILO Manager is running in cluster environment
  and silo manager connector will be installed,
  review Installing SAM Connectors with HA Silo
  Manager section
• For example, if NSM manager is installed in a
  cluster environment, NSM Connector must also be
  installed in the cluster environment

15
Cluster Resource Group Selection

• If installing SAM Manager, some of the data files
  will be moved to shared disk. This does not
  require large amount of space on shared disk.
• If SQL Server is installed on cluster node, you
  must select the shared disk that is associated
  with the SQL Cluster resource group
• If SQL Server is NOT installed on cluster node,
  identify Cluster Resource Group that can be used
  for SAM failover. Resource group must have
• Cluster Resource type Network Name defined
• Cluster Resource type IP Address defined
• Cluster Resource type Physical Disk defined

16
Cluster Resource Group Selection

• In this example, two cluster groups are eligible
  for SAM failover. They are
• Cluster Group
• SAM

17
Cluster Resource Group selection

• Select SAM cluster group for failover since it
  has SQL resources
• This group meets SAM failover requirements since
  it has shared disk - which is not QUORUM disk

18
Shared Disk Considerations for SAM Manager

• If installing SAM manager, some of the data files
  will be moved to shared disk. It does not require
  large amount space on the shared disk.
• This shared disk must be defined in the Cluster
  group selected for SAM Failover.
• NOT QUORUM disk
• Note drive letter for the shared disk and its
  cluster resource name. This will be required
  later for customization

19
Install Summary
20
Overview of Tasks

• Install SAM High Availability Resource kit
• Identify cluster group for SAM Failover
• Identify shared disk used for SAM data files
• Logon on active node (NodeA)
• If required, install Java Runtime and EEM
  components
• Install SAM Manager
• Run SAM Failover customization resource kit
• Defines SAM cluster resources and customizes
  required files

21
Task List Continued

• Offline SAM resources
• Move Cluster Group to another node (NodeB)
• Install Java Runtime and EEM components if
  installed on NodeA
• Install SAM manager on NodeB
• Select same SAM options as NodeA install. Do
  not change any options such as Install Path,
  Ports, Passwords
• Since SAM database will have been created by
  NodeA install, when prompted by installer, select
  Use Existing Database option

22
Task List Continued

• Execute setupMSCS.hta to customize failover
• This utility is part of the resource kit
• Customizes required files. Note cluster resources
  would have been defined by NodeA
• SAM services should be started by the installer
• Some of these cluster resources may not exist.
  The cluster resources defined depend which SAM
  components have been installed
• Offline SAM Cluster resources (to apply
  customized files)
• Online SAM Cluster resources
• Launch UI from a different server

23
Task List Continued

• Verify status of all connectors
• Connectors will be displayed with Virtual and
  Real nodes. Real node connectors will appear
  offline
• Move Group
• Verify the connector status on the new active
  node
• If status is correct on all cluster nodes,
  remove real node connectors from UI

24
Installing SAM High Availability Resource Kit
25
Resource Kit

• Available for download from CA Support Site.
• Extract contents to your shared drive

Contents of zip file
26
Resource Kit

• To launch resource kit, execute setupMscs.hta
  from SAMHA directory and select the relevant
  options

27
Installing SAM Manager in the Active Cluster Node
28
SAM Manager Install on NodeA

• This section provides information on SAM Manager
  install on NodeA (first Active Node)

29
Install Path

• Specify local path

30
Select Features to Install

• Select required components (in this example, SAM
  Manager Components and CMDB connector are
  selected)

31
3rd Party Software Folder

• No special considerations for this dialog

32
Username Configuration

• Must specify same username and password on each
  cluster node install

33
Server Configuration

• Ensure port number specified is same on each
  cluster node

34
Database Configuration

• If SQL server is also installed on the cluster
  node, specify virtual node. In this case real
  node will NOT work

35
UI Server Configuration

• Ensure same port number specified on each cluster
  node

36
EEM Settings

• Specify virtual node name for EEM server and
  change Application Name to Virtual Node. This
  should be same on cluster nodes

37
ActiveMQ Configuration

• Real node name picked by the installer. This will
  be corrected by the resource kit (see next
  section)

38
DNS Connector Settings

• Ensure the same setting is selected on each
  cluster node

39
Local Connector Configuration

• Here it shows CMDB configuration

40
Install Complete

• Install complete on Node A

41
Install SAM HA Resource Kit

• Install SAM HA resource kit if not previously
  installed. Review Installing SAM High
  Availability Resource kit section for additional
  information
• Launch SAM HA resource kit installer
• Double click on setupMSCS.hta file

42
Execute Resource Kit

• Click CA SPECTRUM SAM HA Setup option

43
Introduction Dialog

• Select Yes to continue with the setup

44
Specify Shared Disk Drive

• Provide drive letter of shared disk on which SAM
  data files will reside. This does not require a
  large amount of disk space but is required to
  share SAM data files.
• Review Preparing for Installation section on
  the prerequisites for the shared disk.

Note If the shared disk is not available you
will not be able to continue the install
45
Select Shared Disk Cluster Resource

• Resource Kit will list all available Physical
  Disk resources. Select the one associated with
  specified shared drive letter
• For example, here option 2 is entered to select
  Disk H cluster resource

46
EEM Data Store Replication

• Resource Kit will automatically detect if EEM is
  installed on the cluster node and prompt to
  confirm if data store replication is required.

47
EEM Data Store Replication

• EEM data store should be set up on all cluster
  nodes. HA Setup will list all the cluster nodes -
  excluding the real node where HA Setup was
  executed from.

48
EEM Data Store Replication

• Here it shows data store replication between two
  cluster nodes

49
Cluster Resources

• Resources added by Resource Kit.

50
Offline Resources - Move Group

• Offline SAM cluster resources, then Move Group to
  install SAM on other cluster node

Some SAM resources can take up to 5 minutes to
shutdown. Service shutdown timeout value has been
set to 5 minutes by SAM install process. The
Resource Kit will change it to 1 minute
51
Offline Resources - Move Group

• Offline SAM cluster resources, then Move Group to
  install SAM on other cluster node.

Some SAM resources can take up to 5 minutes to
shutdown. Service shutdown timeout value has been
set to 5 minutes by SAM install process. The
resource kit will change it to 1 minute
52
Install SAM on NodeB

• Now you are ready to install SAM on the second
  cluster node.

53
Installing SAM Manager in the Second Cluster Node
54
SAM Manager Install on NodeB

• This section walks through install of SAM Manager
  on NodeB
• NodeB is now active after Move Group
• SAM Cluster resources defined in NodeA are
  offlined prior to move group.

55
SAM Manager Database

• Identical to install process on NodeA (first
  node) except that database has already been
  created by NodeA therefore, select Use
  Existing Database option

56
Install SAM HA Resource Kit on NodeB

• Since Resource Kit should have been installed on
  shared disk it should be available for execution
  on NodeB.
• The SAM cluster resources should have been
  defined and set to offline status prior to moving
  to NodeB.

57
Execute Resource Kit

• Launch Resource Kit

58
Select Shared Disk Drive

• Select same shared disk drive that was selected
  for first node

59
Select Shared Disk Cluster Resource

• Select same shared disk resource name that was
  selected for first node

60
EEM Data store replication

• Similar to NodeA, Resource Kit will automatically
  detect if EEM is installed on the cluster node.
  If so, will confirm if data store replication is
  required.
• If EEM Data store replication was enabled in
  NodeA, then you must click Yes

61
HA Setup Complete

• If the HA setup successfully completed, you will
  receive the following message

62
Cluster Resources

• Cluster resources were defined by the first node
  and the Resource Kit will online them.

63
Enable HA changes

• To enable changes carried out by the Resource
  Kit, offline and re-online SAM cluster resources
• Verify connector status and failover
• Remove registration for real nodes

64
Installing Remote Connectors
65
Installing Connectors

• No special considerations for installing remote
  connectors reporting to Highly Available SAM
  manager
• Specify Virtual Node name for SAM Manager when
  requested by the SAM install process
• The following slides show the NSM connector
  install.
• SAM manager is installed in HA mode in a
  Microsoft Cluster environment.

66
NSM Connector Install

• Select required connectors

67
User Configuration

• No special considerations for this dialog

68
ActiveMQ Configuration

• Specify name of Virtual Node

69
NSM / WV Connector Configuration

• No special considerations for this dialog

70
Spectrum Connector Install

• The following slides show Spectrum Connector
  install reporting to Highly Available SAM manager
• No special considerations - except to specify
  Virtual Node name for SAM Manager

71
Destination Host
72
Destination Location
73
SPECTRUM Owner
74
Main Location Server
75
One Click Port Number
76
ActiveMQ Client Configuration

• Specify the Virtual Node for the SAM Manager

77
User Interface
78
User Interface

• If UI launched from remote servers, ensure
  Virtual Node name is used

79
Failover
80
UI Failover

• If failover occurs, active UI sessions may
  briefly lose connection
• UI console may indicate lost connection and will
  automatically reconnect when failover has been
  completed. It may prompt for user credentials.
• If the shutdown of SAM services does not take a
  long time then it is likely no interruptions will
  be observed during failover.
• The shutdown wait interval for SAM Services is
  customized by the Resource Kit to minimize the
  interruptions on the failover.

81
UI Failover

• Dashboard may indicate lost connection but it
  will not automatically reconnect. Refresh of
  Internet Explorer will be required to reconnect
  to the new active node if the connect is lost for
  long period.
• Similar to UI Console, if the shutdown of SAM
  services does not take a long time, then no
  interruptions will be observed during failover.

82
Dashboard

• Here it shows Dashboard connection was lost when
  failover occurred. IE refresh will reconnect when
  failover completes

83
UI Console

• Here it shows failover in progress. It will
  automatically reconnect to new active node when
  it is up

84
Connector Registration and Status
85
Connector Registration

• In some cases, the newly registered connectors
  may be displayed by the User Interface.
• It may take few minutes for the UI to discover
  connectors and display their status.

86
Newly Registered Connectors

• Here it shows the newly registered HA NSM
  connector is not displayed by UI

87
Newly Registered Connectors

• Here it shows the newly registered HA NSM
  connector automatically discovered by the UI

88
Connector Status

• Here it shows local connectors reported against
  Virtual node (LOD2SQLCLUSTER)

89
Remote Connectors Status

• Remote Connectors - note JMS host name set to
  Virtual Node (LOD2SQLCLUSTER)

90
Local Connectors

• SAM Manager install will register local
  connectors using names of real nodes
• Once HA mode customization completes, you need to
  manually un-register connectors that are
  registered with real node names
• The following slide shows connector status for
  the real nodes, as well. In this example, LOD1095
  and LOD1096 represent real nodes and
  LOD2SQLCLUSTER represents virtual node

91
Real Registration

• Here it shows local connectors registered with
  real node names nodes, as well. This will have to
  be removed from the UI

92
Removing Registration for Real Node

• Select connector on the real cluster node and
  click Remove Registration Tab. Repeat for local
  connectors reported as offline against the real
  cluster nodes

93
Installing Connector with HA Silo Manager
94
Installing Connector with HA Silo Managers

• Install and customization process is nearly same
  as SAM Manager install
• Main difference is that SAM manager, Active MQ
  server and User Interface server options are not
  selected during SAM install.

95
Installing Connectors

• In example, connectors are installed with HA Silo
  Manager running in an MSCS environment.
• This example uses CMDB but same principle applies
  to other SILO managers, such as NSM
• This section does NOT apply if SPECTRUM is
  installed in Fault tolerance mode (i.e., with
  Primary and Secondary SpectroServer). It only
  applies to Silo Managers running in Microsoft
  Cluster environment

96
Installing Connectors
97
Install Folder

• Select Local Disk for SAM Install path

98
Select Required Connectors

• Select the connector required for the Silo
  manager

99
Username Configuration

• Specify same username and password on all cluster
  nodes

100
ActiveMQ Client Configuration

• Specify ActiveMQ server details

101
Silo Connector Install

• Here it shows Connector install details

102
Resource Kit

• Install Resource Kit if not previously installed
• Launch Resource Kit by double clicking
  setupmscs.hta
• Like SAM manager, it will require shared disk.
• If only connectors are installed, no data files
  are moved to shared disk. However, shared disk is
  still required to allow additional components to
  be installed in future.
• Follow instructions provided by the Resource Kit.
  The procedure is same as SAM Manager
  customization.

103
Cluster Resource

• Here it shows the cluster resource CA SAM
  Integration Services defined for Connector-only
  installation

104
Install Connector on NodeB

• Offline SAM Cluster resource
• Move Group.
• Cluster NodeB is now active
• Note, SAM Cluster resource is offline
• Install SAM connectors on NodeB
• Ensure, you select the same options as NodeA.
• Once the install is complete, launch HA Resource
  Kit to customize required files

105
Post Install Customization

• Online SAM Cluster resources if in Offline status
• Note, Resource Kit should have moved cluster
  resources to online status
• Offline SAM Cluster resources
• Required to pick up customization from Resource
  Kit
• Online SAM Cluster resource again
• Verify the connectors
• Ensure Connector status for virtual node is
  online
• Move Group and verify the connector status

106
Post Install Customization

• Remove registration for real nodes
• Connector status for the real nodes stays offline
  unless manually removed
• Launch UI and remove registration

107
EEM Data Store Failover
108
User Preferences

• If EEM failover is not implemented, then SAM User
  Interface preferences, users and group
  configuration will have to be repeated on each
  cluster node.
• Note, the users and groups referenced here are
  SAM defined users and groups not external users
  and groups from external data sources, such as
  Active Directory.
• Since EEM runs on all cluster nodes, users and
  groups from external sources will be current on
  each cluster node and no replication is required
  for this.
• However, UI configuration will not be replicated
  and this has to be configured on each cluster
  node, otherwise implement EEM Data Store
  replication

109
SAM User Configuration - Groups

• Here it shows how to define SAM User Group for
  SAM UI

110
SAM User Configuration - Groups

• Here it shows how new SAM group is defined with
  specific role

111
SAM Users Configuration - Users

• Here it shows how to add external user to SAM
  Group

112
SAM Users Configuration - Users

• Here it shows how external user is added to SAM
  defined user group

113
EEM Data Store

• Here it shows EEM login using SAM application
  with virtual node name (CASAM-LOD2SQLCLUSTER).
  The application name has be consistent on all
  cluster nodes

114
EEM Data Store

• Here it show user group defined from UI

115
References

• Review EEM Getting Started Guide Chapter 7
  Configuring Failover.
• This document is available from the SAM Install
  Image.
• ltInstallBuildDrivegt\Disk1\EEM\Documentation
  directory

116
Uninstalling HA SAM Manager
117
Uninstalling HA SAM Manager

• Uninstalling HA SAM requires special
  consideration
• Like any other HA installation, SAM database
  should not be destroyed until after the last
  cluster node has been removed

118
Uninstall from First Cluster Node

• Do not check Destroy Database unless this is
  the last cluster node

119
Uninstall from last Cluster Node

• Select option Destroy Database
• Delete SAM Cluster resources
• Delete Shared drive SAM data.
• ltSharedDrivegt\Program Files\CA\SAM\
• If EEM is installed and not shared by any other
  applications, then uninstall EEM.
• Special consideration for uninstall EEM. It
  requires all the application are deleted. This
  can be done from EEM GUI.
• If EEM is installed and shared by other
  applications, then disable EEM datastore
  replication.

120
How to disable EEM data store replication

• From a command prompt, execute
• DXServer Stop all
• Ssld stop all
• Restore Knowledge directory. Copy the contents of
  
• DXHOME\config\knowledge_Habackup
• To
• DXHOME\config\knowledge
• Restore SSLD directory. Copy the contents of
• DXHOME\SSLD\SSLD_Habackup
• To
• DXHOME\SSLD\SSLD

121
How disable EEM data store replication

• Restart DXServer services, execute
• ssld start all
• DXServer Start all

122
Gotchas
123
Connector Status

• Remote connector may display Initialization
  status for very long period. If this is case,
  verify clocks are synchronized.
• If the clocks are not in-sync, connector may
  fail to change the status to Online.
• For local connectors this would not be a problem

124
Integration Service Shutdown

• During shutdown or failover, SAM integration
  service may take up to 200 seconds to shutdown.
  This will then hold Application server to
  shutdown as well.
• Timeout value of 200 seconds can be configured
  but it should done with care.
• Review the following option in IntegrationServices
  .conf file
• file resides in C\Program Files\CA\SAM\jsw\conf
  directory
• Update 200 seconds to the required value
• wrapper Shutdown failed Timed out waiting
  for the JVM to terminate.
• wrapper Java Virtual Machine did not exit on
  request, terminated
• wrapper.jvm_exit.timeout200

125
Questions and Answers