Title: A System Administrators Guide to Preliminary Troubleshooting Techniques
1A System Administrators Guide to Preliminary
Troubleshooting Techniques
- Presented By Darryl Wright and Volker
Kleinschmidt
April 12 2005
2And now a word from our lawyers
- Any statements in this presentation about future
expectations plans and prospects for Blackboard
and other statements containing the words
believes anticipates plans expects
will and similar expressions constitute
forwardlooking statements within the meaning of
The Private Securities Litigation Reform Act of
1995. Actual results may differ materially from
those indicated by such forwardlooking
statements as a result of various important
factors including the factors discussed in the
Risk Factors section of our most recent 10K
filed with the SEC. In addition the
forwardlooking statements included in this press
release represent the Companys views as of April
11 2005. The Company anticipates that subsequent
events and developments will cause the Companys
views to change. However while the Company may
elect to update these forwardlooking statements
at some point in the future the Company
specifically disclaims any obligation to do so.
These forwardlooking statements should not be
relied upon as representing the Companys views
as of any date subsequent to April 11 2005.
Blackboard in its sole discretion may delay or
cancel the release of any product or
functionality described in this presentation.
3Objectives
- Provide Blackboard System Administrators with an
overview of their roles responsibilities and
their impact on the success of the Blackboard
Academic Suite at their institutions - Highlight best practices for keeping your
Blackboard Academic Suite running smoothly and
problem free - Improve and standardize the troubleshooting
process on campus. Methods for better
troubleshooting. - Describe ticket submission and escalation process
within Blackboard - Open Discussion The evolution of your Blackboard
Networked Learning Environment
4System Administrator Roles I
- Backend administrator
- Installations webserver admin database
configuration and maintenance application
monitoring snapshot and data integration and
even OS maintenance tasks - Frontend GUI administrator
- Course shell creation user enrollment
export/import/copy control over feature
availability and design set policies and
business rules for account creation content
preservation etc. portal administration - System Extension Developer
- Creation of Building Blocks Application Bridges
and Portal Modules using Java and Blackboards
APIs
5System Administrator Roles II
- CMS Help Desk Manager
- Troubleshooting issues reported via email
phone or a ticket system or selfdiscovered
escalation of issues to Blackboard Technical
Support training and supervision of general help
desk staff - Trainer Documentation Author
- Oneonone training and seminars for faculty
staff and students creating online
documentation and tip sheets - Instructional Designer
- In some cases the system administrator may also
train instructors and staff on pedagogically
sound uses of technology and participate in the
creation of course materials and the tracking and
evaluation of student performance
6Show of hands
- Which roles do you fill at your institution?
- Backend administrator
- Frontend administrator
- System Extension Developer
- CMS Helpdesk manager
- Trainer / Documentation Author
- Instructional Designer
7User community mailing lists
- Backend administrator
- Frontend administrator
- System Extension Developer
- CMS Helpdesk manager
- Trainer / Documentation Author
- Instructional Designer
- BBADMINL
- BLKBRDL
- BBOPEN_SRC
- BBFS
- BBID
8Evolution of System Administration Roles
Practices
INSTITUTIONAL GROWTH
MISSION CRITICAL
EXPLORATORY
SUPPORTED
STRATEGIC
TRANSFORMATIVETHE NETWORKED LEARNING
ENVIRONMENT
Business Continuity
Strategic Owner
High Performance / High Availability
System Update Deployment Plans
Identified system manager
Student /Faculty Training Programs
Help Desk Management
Integrated Production Environment
Monitoring Backup Maintenance Practices
Long Range Planning
Change management practices
Multiple People
Custom Instructional Tools
Single Server Single Admin
Ticketing System
Help Desk
Phase 2
Phase 3
Phase 4
Phase 5
Phase I
9Best Practices
- Be Prepared
- Avoiding is better than solving
- Have a wellestablished troubleshooting process
on campus - Train your users to be aware of pitfalls
- Document local policies and differences
- Top 10 tips for System Administrators
10Best practices Top 10 tips for System
Administrators
- Maintain current Operations Workbook
- Document all server and application changes
- Maintain a Test server environment
- Set up and test a Disaster Recovery Plan
- Plan ahead for increased usage
- Train users well to avoid problems
- Keep it clean purge old records
- Manage and monitor your application logs
- Graph system metrics MRTG cricket
- Participate in user community
11Top 10 Best practices for Course Management
beginning of semester operationalreadiness
- Fully documentyoursystem environment.This
documentation should include the
systemsapplicationsas well as the owners.
Fill in the Operations Workbook. - Practice change management. Make sure all
changes to your configuration are logged and
that a rollback procedure is in place.If
something goes wrong the first question should be
what changed? - Maintain staging and test servers. Ensure that
any change to your environment goes through an
acceptance testing process. Use the test server
to prototype and model changes and the staging
server for acceptance testing. - Create daily full backups and regularly practice
a restore to a test server. - Understand how many courses and userswill be /
have beenadded between semesters.Analyze
whetheryour hardwareis powerful enough to
handle the increase. - Train your users. Schedule enough workshops early
on to highlight new features and provide actual
use cases for better user adoption. Have
documentation ready early. - Purge records.Use the tools available to archive
out courses a predetermined period after the
course is over. Maintain the archive for long
term storage and record keeping and then purge
the course from your system.Keeping unused
courses and accounts on your systemcan slow
performance. - Know your logs.Enterprise applicationshave a
number of error and system logs.Monitor them for
unusual events and size. Make suretasks such as
log rotationand stats purging are scheduled
appropriately. Verify regularly that they are
working correctly. - Actively monitor your environment using tools
like Big Brother and graph crucial performance
data with MRTG cricket or similar tools. - Participate in the user community. Share
experiences gather resources and tips hear
about possible problems before they hit you.
12Network diagramworkbook
- Use Visio templates or simple flowcharting tool
- Make sure to label machines with function
hostname and IPaddress - Indicate ports used on each connection
- Include servers firewalls routers proxies
loadbalancers - Keep it uptodate
13Disaster Recovery Plan
- Ready adequate replacement hardware
- Set up full daily backups to it
- Test recovery regularly
- Train multiple staff on recovery
- Document all necessary steps in standard location
Workbook Backup/Recovery guide local
customizations - Dont commit replacement system to other
productiongrade tasks
14Sample performance graphs
CPUbusy statsweekly graph
dual weekly graph for total number of logins per
dayand number of recent loginslast 10 mins
15Part II
16Troubleshooting Process on Campus
- The SOAP Method A Basic Troubleshooting Method
- Subjective Who? What? Where? and When?
- Objective Steps to recreate Error messages
log collection and access information - Assessment Review steps errors logs and
assess whether it can be resolved or has to be
escalated - Plan Resolution steps or Workaround
- Examples of SOAP interaction
- Used by the Helpdesk troubleshooting enduser
issues - Used by the System Administrator
recognizing analyzing reporting systemissues
focused collection of info for escalation
17SOAP at the Helpdesk
Troubleshooting UserReported Issues
- User or coursespecific issue?
- if so need exact steps and location
- Browserspecific?
- test on different machines and browsers
- Recent changes?
- OS or software update configuration change
malware infection - Restore knowngood configuration
- system restore spyware removal virus scan
18ExampleResolving a login problem
- Account exists?
- Correct password used? Caps?
- If campuswide pwd does it work elsewhere?
- Correct URL used?
- Machinespecific?
- Browser configuration SSL JS cache
- Spyware and virus check
- Create account reset password clean machine
19SOAP for the GUI admin
- Wordofmouth vs. direct report
- Issue affects multiple users?
- Issue repeatable?
- Issue known in user community?
- Useful info in logs?
- Report results of this investigation when filing
support case
20ExampleDisappearing forum posts
- Group forum or course forum?
- Who has forum admin rights?
- Can authors delete own posts?
- whodunnit check modperl access logs for POST
requests in timeframe in question - Inform users that forums are not personal
mailboxes and read messages should not be deleted - Restore backup to test machine and send text of
lost messages to users for reposting
21SOAP for the backend admin
- Intermittent problem vs. event driven
- Issue affects server performance?
- Issue repeatable?
- Issue tracking in the system logs?
- Useful info in server OS logs?
- Report results of this investigation when filing
Blackboard support case.
22ExampleDatabase Connection Unavailable
- Blackboard application not working
- Users reporting error messages upon connection
- Problem is not browser specific.
- Review of Log files
- Server OS DB Logs and Blackboard log files
- Log results
- Blackboard log files revealed full transaction
logs. - Server OS logs reported low drive space on DB
server. - Database Maintenance Plans
- System Administrator scheduled Database
maintenance routines Data integrity checks
Backup and truncation of logs. Updated Blackboard
maintenance routines to include nightly and
weekly DB backups
23The Escalation Process within Blackboard Client
Support
- Clarifying the process towards realistic user
expectations - sample case lifecycle
- sample case interaction between customer and
Blackboard support - How and where can you as the system administrator
help to resolve cases quickly and smoothly? SOAP
24Sample case lifecycle
- Customer case creation on Behind the Blackboard
- TSM replicates problem
- TSM researches KB Wiki tracking system for
similar cases and known resolution - TSM obtains access info from customer
- TSM creates bug report and escalates case
- PSE verifies problem and attempts resolution
- PSE escalates to Development
- Dev PSE Analysis of customer impact
prioritization - Dev Developer assigned target release version
determined - Dev Issue is replicated root cause is
determined - PSE communicates between customer and
development - Dev solution is created
- Dev fix goes to QA
- Customer Betarelease testing
- Dev Software updated released
25Roadblocks in the lifecycleor why do things
take so long?
- Communication failures
- Everyone is too busy
- Incorrect or incomplete information
- Customer impact not clear
- Issue is not high priority
- Other issues hold up a release
26Sample case interaction
- C Our system keeps going down
- S Symptoms please? Logs? Access?
- C Supplies some logs access info
- S Need recent log and perfreport
- S Access not working SSH blocked
- C now SSH works
- S escalating
- S perfreport useless no DB info in it
- you get the picture
27Or how it should be
- C occasionally modperl crashes with CPU
exhaustion - C logs and perfreport from latest crash access
info - S this has been reported by other customers
have you recently updated any building blocks? - C yes block XYZ
- S any courses with course menu links to this
BBB? If so export sample such course restore to
test server and see whether accessing that
course makes modperl CPU usage rise over all
bounds - C verified heres sample course export
- S workaround remove bad menu items in DB
28How can YOU help?
- Clear problem description
- Precise steps for replication
- Preliminary analysis SOAP
- Supply requested data promptly
- Check case status online
- Focus on important issues
- We are all only human
29Open Discussion
- The Evolution of theNetworked Learning
EnvironmentJohn Fontaine Jeff Kelley John
Knight - Managing the growth of your installation of
Blackboard Academic Suite - Scaling to the Blackboard Enterprise Application
- Strategy and planning for future growth
- Blackboard Consultings role in evolution of the
NLE - Consider Blackboard ASP Services
30Evolution of System Administration Roles
Practices
INSTITUTIONAL GROWTH
MISSION CRITICAL
EXPLORATORY
SUPPORTED
STRATEGIC
TRANSFORMATIVETHE NETWORKED LEARNING
ENVIRONMENT
Business Continuity
Strategic Owner
High Performance / High Availability
System Update Deployment Plans
Identified system manager
Student /Faculty Training Programs
Help Desk Management
Integrated Production Environment
Monitoring Backup Maintenance Practices
Long Range Planning
Change management practices
Multiple People
Custom Instructional Tools
Single Server Single Admin
Ticketing System
Help Desk
Phase 2
Phase 3
Phase 4
Phase 5
Phase I
31Wrapping up
- Innovating Together in 05
- Use SOAP method for troubleshooting
- Preparation is better than problem solving
- Resources Available
- Operations Workbook
- http//behind.blackboard.com/b3/sysadminls/refcen
ter/r61/wkbk.htm - Training Center
- http//behind.blackboard.com/b3/sysadminls/traini
ng/index.htm - Web Forums
- http//behind.blackboard.com/b3/sysadminls/forums
/index.htm - Follow up Contacts
- Support site http//behind.blackboard.com
- Product Support Domestic 18007885264
International 12027156019 - dwright@blackboard.com
- vkleinschmidt@blackboard.com
- IF YOU ONLY REMEMBER 1 THING
32Thank You