Chapter 3 outline

1
Chapter 3 outline

• What is network management?
• Internet-standard management framework
• Structure of Management Information SMI
• Management Information Base MIB
• SNMP Protocol Operations and Transport Mappings
• Security and Administration
• ASN.1
• Network management in the real world
• External pressures and constraints
• Time issues
• Tools of the trade

2
Tools of the Trade

• Do not fear. There is still hope for the network
  manager. This hope comes in the form of
  different kinds of tools to assist in management.
• Analytical skills
• Procedures and supports
• Software tools
• Experience
• Keep in mind that these tools tend to be
  expensive, hard to find, and do not necessarily
  do everything you need to do.
• Nevertheless, having lots of options is good.
  Remember When your only tool is a hammer, every
  problem looks like a nail!

3
Analytical Approaches

• There are many different analytical approaches
  that are useful in diagnosing problems and
  developing solutions that solve them.
• In this course, we look at the following
  different techniques
• The Delta method identifying network changes
• The Napoleon method divide and conquer
• The Sesame Street method one of these things is
  not like the other
• The SOAP method Subjective data, Objective
  data, Analysis, and Plan
• The Simple Simon method who has done this
  before
• Scientific methods formal evaluation techniques

4
The Delta Method

• We can be our own worst enemies sometimes.
• A lot of problems dealt with in network
  management arise from changes made by humans in
  the network.
• Sometimes there was a mistake in the change that
  led to the problem at hand.
• Sometimes the change was fine, but there was a
  hidden dependency that broke when the change was
  made.
• Sometimes the problem manifests itself
  immediately. Sometimes it remains hidden, only
  surfacing later when not expected (e.g. the next
  reboot).

5
The Delta Method

• The worst kinds of changes are
• Changes that were made and forgotten.
• Changes that were made and not communicated.
• Changes that were attempted, failed, and
  therewas no fallback position planned.
• What does this tell us?
• All changes must be planned and communicated.
• All changes must be documented.
• All changes must be immediately verified
  afterbeing made to uncover problems as soon as
  possible.
• All changes must have a fallback position, just
  incase something unforeseen goes horribly wrong.

6
The Delta Method Troubleshooting

• Suppose you did not follow the rules on the
  previous slide. What should you do?
• Blame yourself. Always ask yourself the
  question What have I done lately? Be prepared
  to undo those changes to find out if you are the
  culprit.
• Start asking other network managers if they have
  changed anything. Perhaps a coworker changed
  something, did not let you know, and that is now
  the source of the problem.
• Ask vendors to see if they have made changes that
  you have adopted that could cause the problem.
• Ask outsiders that connect with your network to
  see if they have made changes (e.g. your ISP).
• Check with users with enough access privileges
  that they could cause the problem.

7
The Delta Method Planning for Change

• Changes should not be implemented without a lot
  of thought and planning in advance!
• Here is what should be done
• Risk/benefit analysis. Each change can cause
  unforeseen problems. Does the benefits from the
  change outweigh the risks involved? Remember
  If it isnt broken, dont fix it!
• Do a pilot test. Test out changes in an isolated
  non-production environment where they would do
  little damage if something went wrong.
• Do incremental rollouts. After a pilot test, do
  not make the change everywhere at once. Do it in
  pieces to see if something happens along the way.
• Have a fallback position. Make sure you can
  recover if things go wrong in your changes.

8
The Napoleon Method

• This is basically a divide and conquer approach
  to diagnosing network problems.
• The premise is that the location of a problem can
  be found more easily by splitting the problem
  area into smaller, manageable pieces.
• Once you find the piece containing the problem,
  you repeat the divide and conquer process again,
  until you have completely isolated and located
  the problem at hand.
• This approach can be useful when the delta method
  is not helping you solve a problem.
• Not all changes are under your control, and many
  of them can be hidden and hard to detect.

9
The Napoleon Method Getting Started

• When you first find out about a problem in the
  network, what should you do first?
• See if it is a problem affecting only one user or
  host (a local problem) or with a large group of
  users or hosts (a systemic problem).
• If it is local, you are done with systemic divide
  and conquer and can proceed to local analysis of
  the problem (which can include local divide and
  conquer as well).
• If not, you need to collect more information and
  proceed with systemic divide and conquer.

10
The Napoleon MethodSystemic Divide and Conquer

• The first step is to determine exactly which
  group of users or hosts is affected.
• It is not likely that everyone with the problem
  has noticed or reported the situation.
• Once you have a good idea of just who is
  affected, you need to figure out where those
  affected fit into your networks organization.
• Basically, what do they have in common?
• If need be, you may need to divide your network
  organization into smaller and smaller pieces
  until you find the portion that contains the
  problem.
• You are then ready for local divide and conquer
  (and other local techniques).

11
The Napoleon MethodLocal Divide and Conquer

• Think about your first exposure to search
  algorithms in first and second year.
• You started with a linear search. Simple, but
  pretty expensive when there were a lot of items.
• You then were introduced to the much faster
  binary search. Split the items in half, check
  for what you are looking for, and repeat in the
  appropriate half until you find it. This was a
  divide and conquer search algorithm!
• This same general idea can help in further
  isolating and locating problems in networks and
  systems!

12
The Napoleon MethodLocal Divide and Conquer

• A network example of local divide and conquer
• Suppose you have a network connected by a series
  of hubs, and there is a problem on the network
  that is disrupting communications.
• Instead of checking hub-by-hub, disconnect half
  of the hubs and see if the rest can communicate.
  If they can, the problem lies in the disconnected
  half.
• Otherwise, the problem should lie in the half
  that is still connected.
• Repeat this process on the half with the problem
  until the hub with the problem is located and
  isolated.
• This approach can then be applied to individual
  ports on the hub to further localize the problem
  and to help find a solution.

13
The Napoleon MethodLocal Divide and Conquer

• A host example of divide and conquer
• Suppose you have a PC with Windows that fails to
  boot properly.
• Instead of trying to disable individual drivers
  or modules started at boot time to locate the
  problem,first try a safe boot with no extras
  loaded.
• If that works, add a group of them back to the
  system configuration, and try again. If it
  works, the problem lies in the group not added.
  If not, the problem is in this group.
• By repeating the process over and over, the
  problem module can be located without much
  difficulty.

14
The Napoleon Method Drawbacks

• This approach does not always work as well as
  one would like.
• What if the problem involves multiple network
  components, instead of just one? An unlucky
  division of the network can result in two halves
  that work on their own, but not when put
  together.
• What if the problem is intermittent? You might
  falsely assume the problem is in a location where
  it really is not, because it worked the one time
  when you tested the portion of the network it is
  in.
• What if the problem is not a broken or not broken
  situation? If the problem is somewhere in the
  middle (for example, the network is not
  performing as fast as it could, but still works),
  this approach might not help out very much.

15
The Sesame Street Method

• Think of the old Sesame Street song
• One of these things is not like the others, One
  of these things just isnt the same!
• The general principle involved is fairly simple.
• Given a group of two or more items on a network,
  and one of these items is not functioning
  correctly, a comparison to a correctly working
  item should indicate what the problem is and how
  to solve it.
• Typically, the way in which the items are
  different is also the reason the non-functioning
  item is broken.
• Changing the configuration of the non-functioning
  item to be just like the configuration of a
  working item often fixes the problem.

16
The Sesame Street Method Ruling Out

• Ruling out can be a very good problem solving
  technique for network management.
• If you can determine what a problem is not, you
  are one step closer to figuring out what the
  problem is.
• Examples
• Suppose a user is having difficulty using a
  certain application on their workstation. You
  can rule out workstation problems by having the
  user try it on a different workstation. If it
  works there, it is not a workstation problem, but
  an account problem.
• Suppose the only differences between a working
  computer and one that is not is the amount of
  memory, video card, and network adapter. If you
  give both the same memory, and the problem stays,
  you know it must be one of the other two.

17
The Sesame Street Method Good Comparisons

• Making good comparisons is not always as easy as
  it sounds.
• Are you really comparing the same thing? Make
  sure you compare apples to apples, and not to
  oranges!
• Some identical things are intentionally
  configured differently. For example, routers and
  servers will often have vastly different
  configurations, even when they perform similar
  functions.
• If different people or tools configure identical
  items, chances are good they could look very
  different but both still be functionally correct.
  The same people or tools may still configure
  identical things differently too the
  differences might not matter.

18
The Sesame Street Method A Good Tip

• The Sesame Street method is really simple, easy
  to apply, quick, and reasonably effective, if
  comparisons can be made relatively easily.
• To help do this, keep as many things the same as
  possible!
• The more things are the same, the easier the
  comparisons are, and the easier it is to spot
  differences that could explain problems.
• To do this
• Develop standard procedures and scripts for
  configuring hardware, software, and user
  accounts.
• Standardize on a few hardware and software
  platforms from a limited number of vendors unless
  there are compelling reasons not to.

19
The SOAP Method

• The acronym SOAP stands for
• Subjective data. This is what users report as
  their problem in the network, and any symptoms
  of the problem they have observed.
• Objective data. This is what the network manager
  observes either directly, by investigation, or
  through data monitored and collected by
  management tools.
• Analysis. This is what the network manager
  deduces to be the problem based on the above
  data.
• Plan. This is how the network manager will treat
  the problem and perform follow-up.
• With the plan in place, hopefully you are headed
  in the right direction.

20
The SOAP Method

• This is usually a more rigorous and more formal
  method of attacking network problems.
• You may want to try out the other methods first!
• It can involve a lot of writing and tracking to
  record all of the data, analyses and plans.
• Keep in mind that the problem might not be solved
  in the first attempt. The follow-up in the plan
  usually includes another iteration of SOAP if the
  plan does not successfully resolve the problem.
• The SOAP method actually originated from the
  medical profession as a common tool for doctors
  to assist in patient diagnostics.

21
The SOAP Method Things to Keep in Mind

• If the problem is complex, be prepared for
  several iterations of SOAP.
• Sometimes, the plans of the first few runs
  through SOAP will be get more data, and then
  analyze the situation again.
• Do not give up! The right data will eventually
  come by that will help resolve the situation.
• If really stuck, get a consultation with someone
  else that has a fresh view of the problem with no
  preconceptions. Sometimes, you can be too close
  to the problem to effectively solve it.

22
The Simple Simon Approach

• Sometimes, the simplest way to solve a problem
  is to find out if anybody else has had the same
  problem before.
• If so, how did they solve it?
• Every problem has to have some poor soul that is
  the first to discover it. Fortunately, chances
  are usually pretty good that it isnt you!
• This can be a good reason not to live on the
  leading edge of technology sometimes!
• If you are not the first to adopt new technology,
  chances are quite good you will not be the first
  torun into problems using it!

23
The Simple Simon ApproachTechnical Support

• Most vendors have technical support sites with
  great features to help in management problems.
• Searchable knowledge bases. (Do not use the main
  search feature on the vendors home page, as this
  usually hits lots of marketing press releases.
  Go to their technical support area and search
  from there.)
• Frequently asked questions.
• Technical specifications and documentation.
• Downloaded patches and upgrades for various
  problems.
• Discussion forums.
• Contacts for further help.

24
The Simple Simon ApproachTechnical Support

• When those avenues of technical support do not
  work out, sometimes direct contact is necessary.
• Talking with another person can be quite helpful,
  but you have to be prepared first.
• Gather all of your SOAP notes.
• Make sure you have lists of the hardware and
  software involved, network configurations, time
  relationships, reproducibility information, and
  so on.
• If you do not have this information, you are
  likely to get a brush off, and have the blame put
  on yourself, or another vendor involved in the
  problem.

25
The Simple Simon ApproachTechnical Support

• Also keep in mind that some vendors have many
  tiers of technical support.
• Some of it is available for free.
• Some of it is available only to those willing to
  pay for it. Depending on how much you are
  willing to spend, you will get different levels
  of support.
• Depending on budget availability, buying extended
  technical support from a vendor can save you a
  lot of headaches.
• It can also save you time, and consequently
  money.
• This is how you justify the expense when thetime
  comes!

26
The Simple Simon ApproachGoing to the Internet

• The general Internet can be a good source of
  information (and misinformation) when it comes to
  support for network management problems.
• Mailing lists can put you in contact with
  vendors, developers, and other experts that might
  be able to help you out.
• Usenet newsgroups are good discussion forums can
  be a useful source of real world, unsanitized
  information that can help you out.
• Search engines like Google can really help you
  sift through not just web pages, but also
  archived mailing lists, and Usenet too!

27
Scientific Methods

• The previous approaches have been largely
  informal or technology driven.
• The application of more formal, rigorous, and
  scientific methods and tools can also be useful
  in many cases.
• They tend to be very time consuming, and
  therefore expensive, to carry out, but they can
  reveal information otherwise unobtainable.
• For example, a correlation analysis of data
  collected during rigorous experimentation can
  give insight into the dependencies and
  relationships in measured data. This can be used
  to help determine causality which event or
  events introduced the studied problem into the
  network in the first place.

28
Scientific Methods Types of Evaluations

• In formally evaluating a network of computers
  there are a few main techniques
• Theoretical modeling. A mathematical model of
  the network is constructed to assist in reasoning
  about the performance and behaviour of the
  network.
• Simulation. A simulation represents the
  operation and features of a network under a
  variety of different operating conditions.
• Experimentation. A rigorous method of taking
  measurements from a real network environment (or
  real users of the network) under a variety of
  controlled operating conditions.

29
Scientific Methods Trade Offs

• When can the evaluation techniques be used?
• Models and simulations can be used before a
  network is even constructed or deployed.
• Experimentation requires the network to be built
  and deployed for measurements to take place.
• How do the different techniques scale?
• Theoretical models do not scale well at all
  once the network is sufficiently large, modeling
  all of its components becomes impossible.
• Simulations can handle medium sized environments.
  Once they get too large, it becomes difficult to
  accurately represent them in simulations.
• Experimentation can be conducted on any sized
  network, although the volume of measured data in
  large networks can be overwhelming.

30
Scientific Methods Trade Offs

• How much do the different techniques cost?
• Theoretical models and simulations tend not to
  take too much time or money to use.
• Experimentation can take a considerable amount of
  time, and money, since it uses the real network.
  If the network needs to be isolated during
  experimentation, it cannot be used for useful
  work, which again costs money in downtime.
• How complex are the different techniques to
  develop?
• All of the techniques can be reasonably complex
  and take time to develop, if they are to be used
  properly.

31
Scientific Methods Trade Offs

• How accurate are the different techniques?
• Theoretical models tend to have the least
  accuracy, since capturing the behaviour of the
  network is hard, especially when we do not fully
  understand it. Assumptions and simplifications
  tend to be needed.
• Simulations tend to have better accuracy than
  models since they are representing the network in
  a more operational and less mathematical fashion.
  Fewer assumptions and simplifications tend to be
  needed.
• Experimentation tends to have the best accuracy
  overall since you are measuring the real network
  under real conditions. One must remember the
  principle of certainty, though You cannot
  measure the behaviour of the network without
  influencing the behaviour in subtle ways.

32
Scientific Methods A Final Word

• If you are interested in formal evaluations of
  computer systems and networks, you have to read
  the book
• The Art of Computer Systems Performance
  Analysis Techniques for Experimental Design,
  Measurement, Simulation, and Modeling, by Raj
  Jain.
• It is an invaluable resource for formal studies
  of computer network operations.