Prepared by

1
Large PI System Redundancy, Performance and
Security Strategies

• Prepared by
• Craig Taylor
• (ctaylor_at_caiso.com)
• Christopher Russo (crusso_at_alum.mit.edu)
• Presentation to
• 2002 OSISoft Users Conference

2
Agenda

• California ISO System Overview
• PI-UDS Hardware Cutover
• PI-UDS Network Monitoring Tool
• Large PI-UDS Primary Design Goals
• Tips/Potential Problems for Large Systems
• Discussion

3
California ISO Factoids

• Territories covered
• Pacific Gas and Electric
• Southern California Edison
• Comision Federal de Electricidad
• Covers 124,000 square miles
• 21,000 circuit miles of transmission
• Approximately 600 generators
• 45,000 Megawatt summer peak load
• 23 billion energy consumed annually

4
PI System Cutover

• Energy Management System Cutover December 2001
• From ABB Spider to ABB Ranger
• Ranger system improved reliability
• Included Universal Data Server hardware
  upgrades
• 150,000 points
• Currently worlds largest single system

5
PI System Hardware Changes
6
PI-UDS Issues Description and Containment

• ISO experienced client disconnects
• Potential service denial causes
• Large DataLink data queries tied up PI Archiving
  Subsystem
• Network suspected in some cases
• Disk subsystem suspected
• Question was How do we identify and fix? We
  decided to get more network use information and
  identify pi hogs
• We wrote program to organize PICONFIG data in web
  page

7
Monitoring UDS Use with PICONFIG

• Visual Basic program ran PICONFIG Command (every
  5 minutes)
• PICONFIG commands
• _at_login UDS_SERVER,piadmin,password,5450
• _at_mode list
• _at_table pinetmgrstats
• _at_ostru ID,ConStatus,ConTime,ConType,MsgRecv,MsgSen
  t,Name,PeerAddress,PeerName,PID,RecvErrors,SendEr
  rors,BytesRecv,BytesSent
• _at_select ID
• _at_ends

8
Monitoring UDS Use with PICONFIG

• PICONFIG Output
• 1948,0 Success,20-Feb-02 074624,PI-API
  connection,224.,224.,pideE,IP.IP.IP.IP,PeerName,-1
  ,0.,0.,8620.,42437
• 56,0 Success,13-Feb-02 105606,PI-API
  connection,107.,107.,pideE, IP.IP.IP.IP,
  PeerName,-1,0.,0.,4752.,19628
• 3000,0 Success,4-Mar-02 055529,PI-API
  connection,30.,30.,pideE, IP.IP.IP.IP,
  PeerName,-1,0.,0.,1600.,1081.
• 2727,0 Success,4-Mar-02 020711,PI-API
  connection,1282.,1282.,pideE, IP.IP.IP.IP,
  PeerName,-1,0.,0.,1.0027E005,2.614E005
• 2932,0 Success,26-Feb-02 104058,PI-API
  connection,1.0431E005,1.0432E005,pideE,
  IP.IP.IP.IP, PeerName,-1,0.,0.,4.2131E005,2.5783E
  006
• 3202,0 Success,4-Mar-02 082919,PI-API
  connection,15054,42290,pideE, IP.IP.IP.IP,
  PeerName,-1,0.,0.,7.0155E005,1.2982E008
• 8625,0 Success,1-Mar-02 135418,PI-API
  connection,79446,79446,PIPeE, IP.IP.IP.IP,
  PeerName,-1,0.,0.,4.1275E007,1.934E007
• The important information is
• PeerName (The persons computer)
• Connection Time
• BytesRecv and BytesSent (Data amount transferred)

9
Monitoring UDS Use with PICONFIG

• PICONFIG Output

10
Monitoring UDS Use with PICONFIG

• Web Pages

11
Monitoring UDS Use with PICONFIG

• Web Pages

12
Monitoring UDS Use with PICONFIG
IP.IP.IP.IP, Peer Name IP.IP.IP.IP, Peer
Name IP.IP.IP.IP, Peer Name IP.IP.IP.IP, Peer
Name IP.IP.IP.IP, Peer Name IP.IP.IP.IP, Peer
Name IP.IP.IP.IP, Peer Name IP.IP.IP.IP, Peer
Name IP.IP.IP.IP, Peer Name IP.IP.IP.IP, Peer
Name IP.IP.IP.IP, Peer Name IP.IP.IP.IP, Peer
Name IP.IP.IP.IP, Peer Name IP.IP.IP.IP, Peer
Name IP.IP.IP.IP, Peer Name IP.IP.IP.IP, Peer
Name IP.IP.IP.IP, Peer Name
13
Monitoring UDS Use with PICONFIG
NOTE IP Addresses And Peer Names Removed for Secu
rity
14
Monitoring Tool Benefits

• Enabled quick PI user hog identification
• Helped to identifying and optimize abusive
  queries

15
PI System Hardware Changes
16

• Christopher Russo
• Russo Associates
• www.russoandassociates.com
• Large PI-UDS Primary Design Goals
• Tips/Potential Problems for Large Systems

17
Designing Large PI SystemsPrimary Goals

• Reliability Robustness
• Avoiding single-element failure
• Redundancy
• Clustering Solutions
• Hardware (GeoCluster, Marathon, Legato, EMC)
• Software (MSCS, Unix)
• Performance
• Server tweaking
• Archive parameters, bottlenecks
• Redundant Solutions
• IP Load Balancing
• Different Servers
• PI to PI Distribution

18
Peak Performance Tips

• Decide what youre realistically going to need
• Consider dedicated systems for specific
  applications
• OS/Hardware Specific
• Network performance and latency
• Disk performance disk striping, fiber-channel,
  dedicated hardware
• Processor, network subsystem
• PI-Specific Parameters
• Archive Subsystem Tuning
• Archive cache record tuning
• Update subsystem tuning
• Tuning for real-time performance versus
  historical retrieval
• Considerations for totalizer users
• Pre-digesting of specific calculations

19
Large System Potential Problems

• Performance monitoring only manages some
  bottlenecks
• Certain requests can still nuke the server
• Loading and unloading records into memory is
  time-consuming
• Shutdown times increase linearly with archive
  ratio size
• Memory image cannot exceed 2 GB
• Microsoft IP Load-Balancing doesnt help PI
• Connections are stateful and are not all equal
• Achieving bumpless transfer is difficult
  without hardware solutions

20
Archive Subsystem Bottlenecks
21
Our PI 4 Wish List

• A true multi-threaded archive subsystem
• A connection and request logging facility
• Not just who and how much, but what
• A way to restrict expensive API queries or users
• A point-database change-log feature
• The issue of meta-data
• Some current workarounds with scripts
• Implemented in PI 3.3
• Questions Discussion