Implementing IPPBX Systems: Lessons from the Trenches Information Technology Conference

1
Implementing IP-PBX Systems Lessons from the
TrenchesInformation Technology Conference

• Michael Weller, Managing Principal
• PlanNet Consulting
• www.plannet.net

2
Session Agenda

• Introductions and Background
• FAQs
• Common Problems
• Areas of Focus
• Pearls and Nuggets

3
Introductions

• PlanNet Consulting
• Background
• Qualifications
• Presenters

4
Framing the Issues

• Focus on Lessons Learned
• Todays orientation on systems, not applications
  and peripherals
• New language for many - Every term we cover has
  different meanings for different people
• Need to view project as broader than a phone
  system installation, not just a point
  implementation

5
State of the State

• Average system size is increasing (well over 150
  stations now)
• More and more large systems now installed
• More deployment of applications such as Automatic
  Call Distribution (ACD), Computer TeIephony
  Integration (CTI) and Unified Messaging
• It seems everybody is selling and integrating
  IP-PBXs!
• Some of the deployment challenges have been slow
  to go away

6
IP PBX Implementation Realities

• IP is the catalyst for convergence
• First implementation will be more difficult than
  with TDM PBXs
• The subsequent implementations will be similar to
  or easier than with TDM PBXs
• Requires an educational investment in the
  technology (learning curve)
• Requires an up-front investment in the technology
  that can be leveraged for subsequent deployments
• BCR survey on drawbacks of deploying VoIP 39
  indicated deployment was more difficult than
  anticipated
• You wont regret doing a pilot in a controlled
  non-production environment

7
Planning
8
Planning

• Plan for more than just a one-for-one switch
  replacement
• Plan for a voice system that works that may not
  necessarily be the latest release of the
  technology
• Planning is not just a project schedule
  resource assignments
• Review how each technology discipline conducts
  planning
• Planning must be performed throughout the project
• Integrate with other plans
• How will success be measured?
• One site or more?

9
Planning Rule of Thumb

• Do as much as you can as early as you can
• Minimize risks seen when tasks run concurrently
• Example Data Network first, then Voice Network
• Integrator proposed plan typically covers 80
• 80/20 rule The last 20 will take 80 of time to
  complete
• 100 hours of planning without boilerplate still
  need 80 hours with boilerplate!
• Planning Design Implementation ratio is
  typically 312.5

10
Requirements Definition

• Why you are doing the project?
• Detailed description of your business needs and
  applications
• Avoid describing needs using features
• Verify that you know what users dont want
• Whats changed since evaluation process
• Compare old and new feature sets
• Revisit call flows

11
Continuous Design

• Minimum of 3 design phases
• Market (Order of Magnitude / -25 to 75
  accuracy)
• Order (Budgetary / -10 to 25 accuracy)
• Implementation (Definitive / -5 to 10
  accuracy)
• Market Design What you went to market for
• Order Design High-level confirmation of order
  framework
• Implementation Design Detailed architectural
  review of network, system, applications, and
  infrastructure

12
Design Rule of Thumb

• QoS end-to-end Just because you already have
  QoS, it doesnt mean you have QOS for voice
• Ignore errors on packet transport
• Packet loss of 1-2
• Round trip delay cant exceed 150 ms
• Jitter in 10-50 ms range
• Echoechoecho
• MOS 4.0
• Search for chokepoints
• Design for manageability and sustainability, not
  just functionality (avoid tendency to view design
  as a one-time activity)

13
Integration

• Where the rubber meets the roadIP PBXs are all
  about integration
• Consider lessons from other integration projects
  such as CRM
• Integration overlaps other phases
• Recognize odds are nobody is exactly like yours

14
Integration

• Keepupgradeor replace peripherals?
• Voice architecture decisions centralized,
  regionalized, distributed
• Technology and business model mirroring each
  other
• Single Vendor or multi-vendor solutions?
• Interoperability between different vendors
• Not just technical issues potential for old
  fashioned finger pointing
• Impact on changing existing apps or vendors?
• Proprietary or Open Standards solutions Are
  they really open?

15
Integration
16
Integration Rule of Thumb

• Budget 32-80 Kbps per voice conversation
  variance based on degree of compression
• Limit the number of switches in tandem to four or
  less
• Check on special phone configurations. Dont risk
  finding out about a special configuration 2-3
  weeks before cutover --- you may not have time to
  resolve it.
• In some cases, it may be prudent to phase in IP
  phones
• Dont connect devices that produce heavy data
  traffic to IP phones the phones (internal)
  switch may drop packets
• Call Processing Servers / Gateways and Messaging
  Servers need to be behind firewalls enable on
  required applications
• Youll be surprised at how much storage voice
  mail/UM requires when it is uncompressed
• Search for chokepoints
• If you have 100MB to desktop and Gbit uplinks,
  network capacity will not be an issue in most
  cases.

17
Installation

• Emphasis on the physical aspects of
  implementation
• Significant risk here of getting locked into old
  practices
• Project may look like a traditional PBX install
• Temptation is to revert to old voice system
  methodologies

18
Installation Rule of Thumb

• Dont do any MAC work on old PBX for at least one
  week prior to cutover of new system
• Test for echo cancellation wherever
  digital-analog conversion occurs
• Make sure all necessary upgrades for LAN, WAN and
  call control server are performed first then do
  a test, then cut to live traffic
• Pre-cut site survey to verify devices also gives
  team time to update cut sheets and identify
  problems
• Include jack and MAC numbers on cut sheets
• Consistent review of old PBX database dumps and
  comparisons with new database minimizes problems
  and surprises
• Make sure you have deleted old and unnecessary
  fax and modem lines, etc.

19
FAQs

• How is planning for VoIP different than other IT
  projects?
• Its (not) just another application!
• It requires more groups to be involved
• A VoIP project typically has not been done before

20
FAQs

• Is my data network infrastructure ready for voice
  integration?
• Yes, if you have looked at and addressed
• Current LAN network diagram
• QoS voice design (ToS, CoS, RSVP, 802.1p/Q,
  DiffServ)
• Traffic shaping WAN traffic
• VLAN / Subnetting designs
• Security
• Network management
• Power for voice
• Installed data equipment (speed, performance,
  routing protocols, link speeds)
• Assumption Cable (physical layer) is ready

21
FAQs

• When you assess the data network electronics,
  what problems are you looking for and where do
  you expect to find them?
• Age and vintage of equipment
• Levels of operating systems on various
  switches/routers
• Bottlenecks and chokepoints (topology mismatch)
• QoS capabilities

22
FAQs

• What installation services should be in scope?
• Design assistance services
• Project planning assistance (all 4 phases)
• Network assessment/validation
• Requirements review/database definitions
• Specific carrier coordination assistance
• Software release coordination
• Pre-cutover testing
• Integration
• Training
• Support 1st day and post-install
• Acceptance testing
• Documentation
• 30-day checkup

23
FAQs
24
FAQs

• How much of my own staff resources do I need to
  provide?

25
Common Problems

• Wide range of expectations
• Not knowing what to ask and uncollected needs
• Resistance to plan development
• Communications among multitude of groups
• Customer gives the integrator incorrect,
  incomplete, or old information (e.g., unknown
  or inaccurate network bandwidth requirements)
• Last minute design and database changes

26
Common Problems (cont.)

• Nailing down integrator scope
• How little vendors/integrators know about their
  products
• Lack of manufacturer involvement
• Integrator does not provide clear and complete
  network requirements checklist
• Interoperability issues (e.g., legacy equipment
  trunk integration)
• Cabling, power, and HVAC
• Dealing with multiple standards

27
Common Problems (cont.)

• LAN/WAN network not ready misconfigured or
  hardware not ready
• Inadequate network and/or network management
  tools
• Underestimating QoS and other quality concerns
• Vendors indicate network is insufficient in
  80-90 of projects
• Customer needs to add services or upgrade the
  network 30-40 of the time
• Carrier delays and incorrect provisioning
• Incompatibility between software releases
• Frequent firmware changes, especially phones

28
Common Problems (cont.)

• Soft phones may not act like youd expect
• IP phones and digital phones with same form
  factor may not have same feature sets.
• IP phones to reboot delays when installed and
  when firmware updates are transmitted.
• Voice/Unified messaging often weak link
• Incomplete or inadequate testing
• User training

29
Common Problems (cont.)

• Call routing failures
• Faxing (analog)
• Staffing for post-cutover
• Project Managers dont have right mix of voice
  and data experience
• Determining what is MAC and what is Help Desk
  work
• Escalation management (internal integrator
  manufacturer)

30
Installation Issues for IP-PBX Systems

• All the same as traditional plus
• Increased environmental (In-line power) plus UPS
  in closets
• Moving from non-production (test) environment to
  production
• Cabling (requires 4-pr. UTP, Cat 5 or better,
  extra patch cords required)
• Carrier issues (if doing IP trunking, need SLA
  for QoS, latency, etc.)
• IP addressing
• Network management
• Security (IP)
• Port misconfiguration
• TFTP requirements (e.g., variance in download
  files, how upgrades will work)
• Network management and QoS mitigation
• Configuration
• Echo (cancellation at end points)
• Silence suppression

31
Baggage Management

• IP PBX decisions may be emotional or even heated
• Some unenthused about the outcome or not totally
  on board
• Deal with the baggage
• This is not a voice or data project it is both!
• Avoid the issues by involving those who can
  negatively affect the outcome of the project

32
Focus Availability and Reliability

• Everybody seems to agree they are different but,
  there is little agreement on each.
• Availability Amount of time a computer/telephone
  system is available for processing transactions
  or telephone calls (Newtons Telecom Dictionary)
• Reliability A measure of how dependable a system
  is once you actually use it (Newton)
• Four types of availability hardware, software,
  power and network
• Understand Mean Time Between Failures (MTBF) and
  Mean Time to Repair/Reboot (MTTR) Figures

33
Focus Quality

• Voice quality is subjective though MOS scoring is
  a good tool
• MOS depends on QoS, delay, and codec(s) that is
  deployed
• Users know when quality isnt right
• Quality is 25 configuration 75 education

34
Focus Why is QoS Important?

• Integrators cite QoS as frequent source of
  problems
• End-to-end QoS considered by most to be the
  single biggest obstacle to voice quality
• Consider QoS to be about performance and capacity
• Five dimensions to QoS Availability, Throughput
  (performance), Delay, Jitter, Loss
• Understand how call volume affects QoS
• Use CAC
• Use Traffic Shaping

35
Focus Too Many Standards!

• Alphabet soup
• Many vendors (currently) pay lip service to
  standards
• Vendors want you on their version or proprietary
  standard
• Most systems still proprietary in handsets,
  applications, network management, feature sets
  and power
• Much of the proprietary content of IP telephony
  systems is not likely to decrease anytime soon
• Waiting for SIP
• At the end of the day, functionality is more
  important than standards

36
Focus Data Network Remediation

• Most data network device configurations contain
  errors
• Traditional IP applications are tolerant Voice
  is NOT!
• Set the port speed and duplex on all routers,
  switches, gateways, servers, etc
• 10/100 half/full auto detect does not work
  properly most of the time
• common that it appears to work, but detailed
  testing shows not
• Investigate ALL methods of QoS
• What worked for someone else may not work for you
• Voice and Data groups should meet to specifically
  address QoS
• Throw out previous assumptions

37
Focus Testing

• Quality is planned not tested in
• Pilot programs in non-critical production
  environments
• Test against pre-established criteria
• Assumes rigorous test plan created earlier
• Plan updated early in testing phase
• Run traffic test/stress scenarios against model

38
Focus Training

• Training more than cycling users through
  15-minute tutorials and sending support staff to
  one-week class
• Need to understand how your organizations people
  learn best
• Remember adult learning principles! (Rule 1
  Short doses frequently repeated. Rule 2 Lots of
  pictures/hands-on)
• Identify all groups that need training
• End users
• System administrators
• Network technicians
• Help Desk staff

39
Focus Cutover

• Schedule shutdown of production environment as
  early as possible
• Cut trunks over at 400 pm (Make sure your
  version of 400 is the same as your carriers)
• Negotiate (or pay) the overtime for carrier
  resources to be available after 500 pm
• Dont schedule critical testing or other
  activities on Sunday - keep Sunday free in case
  you need it. Dont rush!
• There are usually not as many on-site experts
• Initial troubleshooting is more complex dont
  panic
• This is not the time to expand scope!

40
Focus Managing Transition and Acceptance

• If you didnt define acceptance in the contract,
  define it and document it - before cutover and
  obtain the agreement of all parties
• Dont let the integrators PM escape too early
• Dont get turned over to the integrators tech
  support center too early
• Be active in the managing the integrators
  account transition activities

41
What do I need to know if Im a voice person?

• Basic Elements of Packet Switching (as opposed to
  Circuit Switching)
• QoS
• Role of Routers, Switches, Gateways, Security
  Devices
• You will communicate with data staff
• How to isolate problems within a more complex
  environment
• Data specialists themselves will typically not
  differentiate between voice and other network
  applications.

42
What do I need to know if Im a voice person?

• How the data network will prioritize voice
  packets
• How desktop functionality may change
• Impact of voice traffic on data network
• Effect of compressed and uncompressed voice
  (email storage and network performance)
• Details of interoperability between network
  devices (especially unified messaging, gateway,
  and analog endpoints)
• Database integration and standards (LDAP, length
  of fields, display look)
• How the dial plan will work today and in the
  future

43
What do I need to know if Im a data person?

• End user expectations of performance and support
• Basic Elements of Circuit Switching (as opposed
  to Packet Switching)
• You may already understand depending on the type
  of network you have
• Basic Elements of PBX systems and feature sets
  such as dial plans, signaling, etc.
• You will communicate with the voice staff
• The difference between control and bearer traffic
  and their tolerances to delay, jitter (delay
  variance) and packet loss
• How to isolate voice problems on the network
• Voice specialists like other applications
  specialists will see their application as the
  most important on the network

44
What do I need to know if Im a data person?

• Impact of converged traffic on the data network
• How IP voice endpoints will have power
• How security will be managed
• How QoS for voice will be designed
• Details of database interoperability
• Readiness of data network electronics
• Impact on network of IP phone initial boot
• Topology mismatches

45
Pearls and Nuggets

• IP PBX systems typically grow from one site to an
  enterprise application. Plan for this up front!
• Dont recreate your TDM environment!
• Keep an eye out for hidden costs in all areas
• Simplify your architecture wherever possible
• Document all of your assumptions when planning
• Assume even less than you usually do
• Invest in the planning process - plan early and
  often
• Get the right people (not the right groups)

46
Pearls and Nuggets

• Set design objectives recognize that designs
  change
• Dont take simple truths at face value (e.g.,
  physical changes should start at the core of the
  network software changes most often should start
  from the edge.)
• Train technical staff early in the process.
  Dont let them figure it out as they go.
• They may not ask for help!
• Include manufacturer involvement in budget.
• Get their skin in the game!
• Create separate VLANs for voice - insulate voice
  from Internet via proxies
• Take the time to develop thorough call flows

47
Pearls and Nuggets

• Document your design assumptions!
• Dont go just halfway down the IP PBX path If
  youre going to do it, then do it!
• Get the list of materials finalized early
• Designs change fact of life
• Recognize odds are that nobody has a situation
  exactly like yours
• Once implementation design complete, there is a
  tendency to gold-plate the solution
• End user by the ways (last minutes discoveries)
• Headset compatibility issues

48
Pearls and Nuggets

• Voice cutovers have more emotion because of users
  (network cuts dont usually affect users so
  directly)
• Answer user questions promptly, preferably within
  24 hours
• Without proper resource estimation processes and
  tools, FTE needs are only guesses
• Get very specific with carrier requests,
  especially with PRI orders (coding, framing,
  incoming/outgoing outpulse start, outpulsed
  digits, digit manipulation, hunting, NFAS)
• Plan for user access to legacy voicemail for 2
  weeks following cutover
• More powermore heat. This is commonly
  overlooked.
• Plan administrator/tech training well in advance.

49
Questions, Discussion, and Close

• Questions?
• Discussion
• Close