NOC Theory and Practice - PowerPoint PPT Presentation


PPT – NOC Theory and Practice PowerPoint presentation | free to download - id: 12bcf5-ZGQyM


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

NOC Theory and Practice


NOC Theory and Practice - NANOG 24 - Miami, FL - TCB. NOC Theory and Practice ... We're not kidding. Give some thought to a 24x7 outlook. ... – PowerPoint PPT presentation

Number of Views:90
Avg rating:3.0/5.0
Slides: 34
Provided by: bill512
Learn more at:


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: NOC Theory and Practice

NOC Theory and Practice
  • Part II The Nuts and Bolts
  • (The Practice Of Making a Good NOC)

  • I am not affiliated with any corporation. My
    views are my own.

Intro - Whats In A NOC?
  • Purpose - What Type of Company Are You?
  • People will always be your key asset.
  • Systems make the job of people easier.
  • Managers assist in identifying gaps in both.
  • Processes allow for flexibility in operations
    while still maintaining standards.
  • And, last but not least, the facility is where
    you do your work.

The Facility - Think Data Center
  • Location, location, location.
  • Security
  • Environmental Concerns
  • This applies to both your inside and outside
  • Ergonomics
  • Reliability
  • Sources of Labor

The Goals of a NOC
  • Our conception is
  • At the bare minimum, you will receive requests
    from your customers
  • You will process those requests in some fashion
    (engineering, handoff, ...)
  • You will communicate resolution to the customer,
    and/or keep customer updated on status

Goals II
  • NOC as first, second, or third tier support.
  • Conception of NOC as do everything group
  • The engineers should be smart at designing,
    configuring, etc. - NOC should be divided into
    categories of engineers able to resolve issues
  • A smart issue routing process makes this a very
    effective model

The Process
  • Receive events from your inputs (phone calls,
    network monitoring, etc)
  • Process inputs into a problem to be resolved
  • Route the problem to the correct individual
  • Resolve the problem as appropriate
  • Communicate status continually.
  • Record the problem for later review and training.

The Key to Success Is
  • By constant re-examination of your operation,
    your employees, and what inputs you are getting,
    you can reach incredible efficiency. Communicate
    this to other teams and your operations will

The Facility (cont.)
  • Security applies to keeping other people out, but
    it also applies to making a safe arrival for your
    employees, at any hour.
  • Environmental concerns Just like a datacenter,
    consider climate control and earthquake/act of
    god resistance. Disaster planning rules should
    be in effect.
  • Sources of labor Not just engineers, but

A Word on Portability
  • As Matt suggests, portability is critical.
  • Obvious resources Computers, communications,
  • Not obvious resources Vehicles and quick
    transport/assembly, good cases for shipping or
    relocation, breakdown/reassembly schedules.
  • Multiple everything.
  • Make sure your resources are distributed. If
    your network is large, your footprint of mobility
    should be large.

Hiring the Right People
  • Skillset
  • but more importantly, flexibility
  • Stress
  • Scheduling
  • Stability
  • NOCs are hard things to staff appropriately. And
    good people are hard to retain.

Hiring (cont.)
  • Make sure they have lives. But know what you
    cant ask.
  • Can they talk? How are their soft skills?
  • Do they get flustered easily?
  • And remember, org charts suck. Classify people
    according to their skills, not according to some
    hierarchy which has no meaning outside of some
    middle-management mind.

Keeping Your People Happy
  • Environment
  • Ergonomics
  • Creature Comforts
  • Were not kidding.
  • Give some thought to a 24x7 outlook. This means
    beds, showers, food, and climate.

Environment - People
  • Climate controlled centers get cold at night.
    Make sure you have 24H control over your space.
  • When thinking about climate, keep in mind your
    working conditions are slightly different in a
  • 200AM to 400AM are the critical hours.

Ergonomics - People
  • Hire a good ergonomic consultant.
  • Positions of lights, phones, keyboards, monitors,
    pens - all this matters.
  • Observation is key. Your engineers will show you
    what the problems are. (Cameras)
  • Seating. If youre in a chair for eight hours at
    a stretch, dont you want it to be a comfortable
  • Screen space.

About People. About NOCs.
  • Its a messy business. Its high stress. Its
    easy to let things get out of hand.
  • Good management qualities in a NOC
  • cool heads
  • methodical approach to solving problems
  • having been on the customer side
  • micromanagement sucks.

Lets go over retention.
  • Retention of NOC staff is 100 harder. Very few
    people live for this kind of job.
  • So, innovate. Think outside the box.
  • People want to grow. Know what your people want
    to do when they grow up.
  • Remember your core focus. Do operations, not
    sales, or account services, or anything else.
  • At the same time, make sure your folks are
    informed on everything. Everything comes out in
    the wash. All the dirty clothes come to NOC.

Interviewing NOC folks
  • You are interested in three things.
  • The candidates willingness to learn.
  • The candidates 1-2 year outlook on their career.
  • The candidates ability to work funny schedules.
  • They are interested in three things.
  • Will you offer them growth potential?
  • Can you give them enough perks to enjoy their
  • Do you have your stuff together?
  • Are the procedures well-documented?
  • Will they be called in outside their shift?
  • Will they ever wonder who to go to?

  • Momentum is key. Keeping your people busy is the
    most important thing towards maintaining
  • The other component to momentum is keeping them
    working on different things. This makes them
    better engineers, and helps feed ego.
  • Everything at your fingertips - your engineers
    should never have to make a customer wait more
    than four minutes on hold for anything. If it
    has to wait longer than that, tell the customer
    why, and mark it as something to fix.

The View of the Customer
  • A customer wants
  • Professionalism your engineers must treat the
    customer as if they are paying you a lot of money
    and the engineer wants the customer to continue
    to do that.
  • ETR Your customer wants to know when the problem
    will be fixed. I dont know is not an
    acceptable answer. The ETR is the first thing
    you give to a customer and the one thing you
    update him on at least every hour (or less if
    they prefer).

The View of the Customer II
  • A description Concise or not, the customer
    wants to know what the problem appears to be.
  • Remember your focus. It is not (typically) your
    responsibility to engineer your customers
    network for maximum reliability. Your
    responsibility is the operation of that network.
    The only thing you should do as a NOC is fix
    problems when they arise in the most expedient
    and professional manner possible. A customer
    wants to be comfortable, and assured that that
    will happen.

Things we never, EVER allow a customer to hear
  • I dont know when it will be fixed.
  • I cant help you.
  • Im not qualified to answer that question.
  • Let me transfer you to someone else.

Its not always ideal. But -
  • Where possible, the first person to take the call
    should be the person who follows it to
    completion. The customer hates being thrown
    between different engineers - but not having an
    answer is worse.

A Little About Systems
  • Dont just focus on ticketing.
  • Make sure your people have information on
    everything they need to operate.
  • If all the architects fell off the planet today,
    could you rebuild?
  • Its about agility.

Policies and Procedures
  • If you dont write it down, it isnt a procedure
    or a policy. There are no unwritten rules in a
    NOC. People cycle.
  • Focus on procedures that affect the handling of a
    customer. The soft skills are usually the
    hardest to learn.
  • Full disclosure?
  • Troubleshooting Techniques

And a word on incidents.
  • Incidents, tickets, call it what you will.
  • Whatever happens in the NOC
  • gets summarized to the account exec if the issue
    is of appropriate severity
  • gets recorded - nothing EVER gets deleted from a
    customer log
  • gets archived
  • gets resolved - how much time will you spend on
    issues that never get fixed?

Some final notes - Conclusion
  • Its easy to throw a bunch of people in a room.
    Its hard to build an efficient, productive, and
    well-oiled NOC (the same goes for networks)
  • Record everything. Write everything down.
  • Try to remain as flexible as possible.
  • Look to your employees for cues on what their
    ideal environment should be.
  • Remember your focus, and know your customer.
  • Know what not to say.

  • As far as the practice of NOC building goes, we
    must thank
  • - thanks for great
    private discussions on NOC building
  • nanog subscribers - for talking about operational

And finally
  • Sean Donelan - for giving network operations
    folks an impetus for existing (due to his backhoe
  • Shouts to EFNet IRC nanog you know who you are

Part III Evaluation Criteria
  • Social engineer your own NOC.
  • We like to rate NOC folks on
  • Do they sound professional? Do they have the
  • Do they follow us all the way through to
  • Can they tackle difficult as well as simple

Part III Evaluation Criteria (cont).
  • How many calls a day does your NOC take?
  • Do you survey your customers, Cisco-TAC style?
  • The three Ps
  • Perception
  • Passion (Emotion)
  • Presentation

Evaluation Criteria
  • They (customer) must perceive you as the people
    that will help.
  • You (NOC) must present your solution with total
    assurance that you can handle the problem.
  • You (NOC) must have passion in dealing with the
    customer. Make it friendly, even if its a
    network down emergency. It will come back to pay

  • Your goal is to be professional, calm, cool and
    collected, even in the face of serious adversity.
  • If you dont have an answer, you know where to
    get it and how long it will take.
  • You have to communicate to the customer and keep
    the customers viewpoint in mind, no matter what.
  • As a NOC manager/builder, you have to make sure
    your people have the right tools.