A System Administrators Guide to Preliminary Troubleshooting Techniques


1
A System Administrators Guide to Preliminary
Troubleshooting Techniques

• Presented By Darryl Wright and Volker
  Kleinschmidt

April 12 2005
2
And now a word from our lawyers

• Any statements in this presentation about future
  expectations plans and prospects for Blackboard
  and other statements containing the words
  believes anticipates plans expects
  will and similar expressions constitute
  forwardlooking statements within the meaning of
  The Private Securities Litigation Reform Act of
  1995. Actual results may differ materially from
  those indicated by such forwardlooking
  statements as a result of various important
  factors including the factors discussed in the
  Risk Factors section of our most recent 10K
  filed with the SEC. In addition the
  forwardlooking statements included in this press
  release represent the Companys views as of April
  11 2005. The Company anticipates that subsequent
  events and developments will cause the Companys
  views to change. However while the Company may
  elect to update these forwardlooking statements
  at some point in the future the Company
  specifically disclaims any obligation to do so.
  These forwardlooking statements should not be
  relied upon as representing the Companys views
  as of any date subsequent to April 11 2005.
  Blackboard in its sole discretion may delay or
  cancel the release of any product or
  functionality described in this presentation.

3
Objectives

• Provide Blackboard System Administrators with an
  overview of their roles responsibilities and
  their impact on the success of the Blackboard
  Academic Suite at their institutions
• Highlight best practices for keeping your
  Blackboard Academic Suite running smoothly and
  problem free
• Improve and standardize the troubleshooting
  process on campus. Methods for better
  troubleshooting.
• Describe ticket submission and escalation process
  within Blackboard
• Open Discussion The evolution of your Blackboard
  Networked Learning Environment

4
System Administrator Roles I

• Backend administrator
• Installations webserver admin database
  configuration and maintenance application
  monitoring snapshot and data integration and
  even OS maintenance tasks
• Frontend GUI administrator
• Course shell creation user enrollment
  export/import/copy control over feature
  availability and design set policies and
  business rules for account creation content
  preservation etc. portal administration
• System Extension Developer
• Creation of Building Blocks Application Bridges
  and Portal Modules using Java and Blackboards
  APIs

5
System Administrator Roles II

• CMS Help Desk Manager
• Troubleshooting issues reported via email
  phone or a ticket system or selfdiscovered
  escalation of issues to Blackboard Technical
  Support training and supervision of general help
  desk staff
• Trainer Documentation Author
• Oneonone training and seminars for faculty
  staff and students creating online
  documentation and tip sheets
• Instructional Designer
• In some cases the system administrator may also
  train instructors and staff on pedagogically
  sound uses of technology and participate in the
  creation of course materials and the tracking and
  evaluation of student performance

6
Show of hands

• Which roles do you fill at your institution?
• Backend administrator
• Frontend administrator
• System Extension Developer
• CMS Helpdesk manager
• Trainer / Documentation Author
• Instructional Designer

7
User community mailing lists

• Backend administrator
• Frontend administrator
• System Extension Developer
• CMS Helpdesk manager
• Trainer / Documentation Author
• Instructional Designer

• BBADMINL
• BLKBRDL
• BBOPEN_SRC
• BBFS
• BBID

8
Evolution of System Administration Roles
Practices
INSTITUTIONAL GROWTH
MISSION CRITICAL
EXPLORATORY
SUPPORTED
STRATEGIC
TRANSFORMATIVETHE NETWORKED LEARNING
ENVIRONMENT
Business Continuity
Strategic Owner
High Performance / High Availability
System Update Deployment Plans
Identified system manager
Student /Faculty Training Programs
Help Desk Management
Integrated Production Environment
Monitoring Backup Maintenance Practices
Long Range Planning
Change management practices
Multiple People
Custom Instructional Tools
Single Server Single Admin
Ticketing System
Help Desk
Phase 2
Phase 3
Phase 4
Phase 5
Phase I
9
Best Practices

• Be Prepared
• Avoiding is better than solving
• Have a wellestablished troubleshooting process
  on campus
• Train your users to be aware of pitfalls
• Document local policies and differences
• Top 10 tips for System Administrators

10
Best practices Top 10 tips for System
Administrators

• Maintain current Operations Workbook
• Document all server and application changes
• Maintain a Test server environment
• Set up and test a Disaster Recovery Plan
• Plan ahead for increased usage
• Train users well to avoid problems
• Keep it clean purge old records
• Manage and monitor your application logs
• Graph system metrics MRTG cricket
• Participate in user community

11
Top 10 Best practices for Course Management
beginning of semester operationalreadiness

• Fully documentyoursystem environment.This
  documentation should include the
  systemsapplicationsas well as the owners.
  Fill in the Operations Workbook.
• Practice change management. Make sure all
  changes to your configuration are logged and
  that a rollback procedure is in place.If
  something goes wrong the first question should be
  what changed?
• Maintain staging and test servers. Ensure that
  any change to your environment goes through an
  acceptance testing process. Use the test server
  to prototype and model changes and the staging
  server for acceptance testing.
• Create daily full backups and regularly practice
  a restore to a test server.
• Understand how many courses and userswill be /
  have beenadded between semesters.Analyze
  whetheryour hardwareis powerful enough to
  handle the increase.
• Train your users. Schedule enough workshops early
  on to highlight new features and provide actual
  use cases for better user adoption. Have
  documentation ready early.
• Purge records.Use the tools available to archive
  out courses a predetermined period after the
  course is over. Maintain the archive for long
  term storage and record keeping and then purge
  the course from your system.Keeping unused
  courses and accounts on your systemcan slow
  performance.
• Know your logs.Enterprise applicationshave a
  number of error and system logs.Monitor them for
  unusual events and size. Make suretasks such as
  log rotationand stats purging are scheduled
  appropriately. Verify regularly that they are
  working correctly.
• Actively monitor your environment using tools
  like Big Brother and graph crucial performance
  data with MRTG cricket or similar tools.
• Participate in the user community. Share
  experiences gather resources and tips hear
  about possible problems before they hit you.

12
Network diagramworkbook

• Use Visio templates or simple flowcharting tool
• Make sure to label machines with function
  hostname and IPaddress
• Indicate ports used on each connection
• Include servers firewalls routers proxies
  loadbalancers
• Keep it uptodate

13
Disaster Recovery Plan

• Ready adequate replacement hardware
• Set up full daily backups to it
• Test recovery regularly
• Train multiple staff on recovery
• Document all necessary steps in standard location
  Workbook Backup/Recovery guide local
  customizations
• Dont commit replacement system to other
  productiongrade tasks

14
Sample performance graphs
CPUbusy statsweekly graph
dual weekly graph for total number of logins per
dayand number of recent loginslast 10 mins
15
Part II

• Troubleshooting

16
Troubleshooting Process on Campus

• The SOAP Method A Basic Troubleshooting Method
• Subjective Who? What? Where? and When?
• Objective Steps to recreate Error messages
  log collection and access information
• Assessment Review steps errors logs and
  assess whether it can be resolved or has to be
  escalated
• Plan Resolution steps or Workaround
• Examples of SOAP interaction
• Used by the Helpdesk troubleshooting enduser
  issues
• Used by the System Administrator
  recognizing analyzing reporting systemissues
  focused collection of info for escalation

17
SOAP at the Helpdesk
Troubleshooting UserReported Issues

• User or coursespecific issue?
• if so need exact steps and location
• Browserspecific?
• test on different machines and browsers
• Recent changes?
• OS or software update configuration change
  malware infection
• Restore knowngood configuration
• system restore spyware removal virus scan

18
ExampleResolving a login problem

• Account exists?
• Correct password used? Caps?
• If campuswide pwd does it work elsewhere?
• Correct URL used?
• Machinespecific?
• Browser configuration SSL JS cache
• Spyware and virus check
• Create account reset password clean machine

19
SOAP for the GUI admin

• Wordofmouth vs. direct report
• Issue affects multiple users?
• Issue repeatable?
• Issue known in user community?
• Useful info in logs?
• Report results of this investigation when filing
  support case

20
ExampleDisappearing forum posts

• Group forum or course forum?
• Who has forum admin rights?
• Can authors delete own posts?
• whodunnit check modperl access logs for POST
  requests in timeframe in question
• Inform users that forums are not personal
  mailboxes and read messages should not be deleted
• Restore backup to test machine and send text of
  lost messages to users for reposting

21
SOAP for the backend admin

• Intermittent problem vs. event driven
• Issue affects server performance?
• Issue repeatable?
• Issue tracking in the system logs?
• Useful info in server OS logs?
• Report results of this investigation when filing
  Blackboard support case.

22
ExampleDatabase Connection Unavailable

• Blackboard application not working
• Users reporting error messages upon connection
• Problem is not browser specific.
• Review of Log files
• Server OS DB Logs and Blackboard log files
• Log results
• Blackboard log files revealed full transaction
  logs.
• Server OS logs reported low drive space on DB
  server.
• Database Maintenance Plans
• System Administrator scheduled Database
  maintenance routines Data integrity checks
  Backup and truncation of logs. Updated Blackboard
  maintenance routines to include nightly and
  weekly DB backups

23
The Escalation Process within Blackboard Client
Support

• Clarifying the process towards realistic user
  expectations
• sample case lifecycle
• sample case interaction between customer and
  Blackboard support
• How and where can you as the system administrator
  help to resolve cases quickly and smoothly? SOAP

24
Sample case lifecycle

• Customer case creation on Behind the Blackboard
• TSM replicates problem
• TSM researches KB Wiki tracking system for
  similar cases and known resolution
• TSM obtains access info from customer
• TSM creates bug report and escalates case
• PSE verifies problem and attempts resolution
• PSE escalates to Development
• Dev PSE Analysis of customer impact
  prioritization
• Dev Developer assigned target release version
  determined
• Dev Issue is replicated root cause is
  determined
• PSE communicates between customer and
  development
• Dev solution is created
• Dev fix goes to QA
• Customer Betarelease testing
• Dev Software updated released

25
Roadblocks in the lifecycleor why do things
take so long?

• Communication failures
• Everyone is too busy
• Incorrect or incomplete information
• Customer impact not clear
• Issue is not high priority
• Other issues hold up a release

26
Sample case interaction

• C Our system keeps going down
• S Symptoms please? Logs? Access?
• C Supplies some logs access info
• S Need recent log and perfreport
• S Access not working SSH blocked
• C now SSH works
• S escalating
• S perfreport useless no DB info in it
• you get the picture

27
Or how it should be

• C occasionally modperl crashes with CPU
  exhaustion
• C logs and perfreport from latest crash access
  info
• S this has been reported by other customers
  have you recently updated any building blocks?
• C yes block XYZ
• S any courses with course menu links to this
  BBB? If so export sample such course restore to
  test server and see whether accessing that
  course makes modperl CPU usage rise over all
  bounds
• C verified heres sample course export
• S workaround remove bad menu items in DB

28
How can YOU help?

• Clear problem description
• Precise steps for replication
• Preliminary analysis SOAP
• Supply requested data promptly
• Check case status online
• Focus on important issues
• We are all only human

29
Open Discussion

• The Evolution of theNetworked Learning
  EnvironmentJohn Fontaine Jeff Kelley John
  Knight
• Managing the growth of your installation of
  Blackboard Academic Suite
• Scaling to the Blackboard Enterprise Application
• Strategy and planning for future growth
• Blackboard Consultings role in evolution of the
  NLE
• Consider Blackboard ASP Services

30
Evolution of System Administration Roles
Practices
INSTITUTIONAL GROWTH
MISSION CRITICAL
EXPLORATORY
SUPPORTED
STRATEGIC
TRANSFORMATIVETHE NETWORKED LEARNING
ENVIRONMENT
Business Continuity
Strategic Owner
High Performance / High Availability
System Update Deployment Plans
Identified system manager
Student /Faculty Training Programs
Help Desk Management
Integrated Production Environment
Monitoring Backup Maintenance Practices
Long Range Planning
Change management practices
Multiple People
Custom Instructional Tools
Single Server Single Admin
Ticketing System
Help Desk
Phase 2
Phase 3
Phase 4
Phase 5
Phase I
31
Wrapping up

• Innovating Together in 05
• Use SOAP method for troubleshooting
• Preparation is better than problem solving
• Resources Available
• Operations Workbook
• http//behind.blackboard.com/b3/sysadminls/refcen
  ter/r61/wkbk.htm
• Training Center
• http//behind.blackboard.com/b3/sysadminls/traini
  ng/index.htm
• Web Forums
• http//behind.blackboard.com/b3/sysadminls/forums
  /index.htm
• Follow up Contacts
• Support site http//behind.blackboard.com
• Product Support Domestic 18007885264
  International 12027156019
• [e-mail]
• [e-mail]
• IF YOU ONLY REMEMBER 1 THING

32
Thank You