Walking the Tight-Rope

1
Walking the Tight-Rope

• Software Project Managementin Real Life
• Urbán Zoltán, Várszegi György2004

2
Introduction

• There are well established theories about project
  management. In practice they are easiest to
  follow for
• small-sized independent (demo) projects
• government financed mega-projects
• The real challenge is to manage multiple projects
  in parallel in a competitive environment under
  time, budget and resource pressure
• Using well-established general practices can
  still reduce the risks

3
Who we are

• ScanSoft, Inc. is a public company based in
  Boston, MA. With nearly 800 employees worldwide,
  it is the market-leading supplier of speech and
  imaging solutions.
• ScanSoft-Recognita Rt. in Budapest is its only
  imaging development hub. Size of its engineering
  is about 90 heads
• The ratio between development and QA is close to
  21

4
Who we are

• Our product portfolio is
• OmniPage stand-alone OCR
• PDF Converter Pro opening and creating PDF files
  
• PaperPort document management system
• OmniPage SDK imaging development toolkit
• OmniForm form designer and data acquisition
  solution

5
Engineering Process

• We have two basic lines of work
• Retail (boxed) products
• a classic project management concept to deliver
  major product versions
• Customization projects
• a relatively high number of minor projects
  controlled by available resources and priorities
  defined by sales
• We will concentrate on the first type in this
  presentation

6
Roles

• For retail product projects the customer is not
  the contracting party for the development
• Product Delivery Team (PDT)
• Defines and delivers the product
• A cross-functional group of area representatives
• Program manager the coordinator to drive toward
  team consensus
• Product manager, marketing, international
• Project manager, QA Lead, documentation,
  engineering
• Technical support
• Manufacturing
• Product Approval Committee (PAC)
• Approves, finances and supervises the project
• Senior management of the company

7
Engineering Phases / Milestones

• Strategic vision
• Kick-off PAC review
• Product definition
• Product and market definition PAC review
• Product design
• Design PAC review
• Coding
• Alpha acceptance

8
Engineering Phases / Milestones

• Alpha
• UI freeze
• Beta readiness PAC review
• Beta acceptance
• Beta
• First release candidate (RC0)
• Launch PAC review
• Release to manufacturing (RTM)
• Sustaining
• RTM of localized versions, patches, point releases

9
PAC reviews

• Regular PAC reviews
• Planned delimiters of the phases
• PDT demonstrates the current state and the
  successful termination of the previous phase
• PDT presents a detailed plan for the rest of the
  project
• Engineering deliverables (25 varieties),
  schedule (100 date points), resource plan,
  technical overview, quality plan, documentation,
  localization, 3rd party components, risks
• PAC decision can be
• GO (contract for the next phase)
• Redirect (major change in the course of the
  project)
• Retry the review (inadequate input from PDT)
• Cancel the project

10
PAC reviews

• Exception reviews
• Any material change in the course of the project
  that affects the contract for the current phase
  should prompt an exception review
• Unexpected major market changes
• Missed milestones
• Budget overrun
• Usually PDT (Program Manager) initiates them
• The review materials and the PAC decision
  criteria are basically the same as at the
  regular reviews

11
Life Cycle / Definition, Design

• Our approach to these phases is iterative
• Initial Marketing Requirements Document
• Product marketing creates it. MRD0 for a 60
  man-year project is typically less than 10 pages
  with about 60 new product features with very few
  details.
• Functional Specification
• Engineering has pretty wide freedom in
  implementation details
• Feature-based (short and comprehensive
  descriptions, technical details, licensed
  components, test methodology, benchmarking, use
  cases, risks)
• Incremental for existing product lines

12
Life Cycle / Definition, Design

• Feature matrix
• Important communication and decision-making tool
• To set realistic project goals (feedback to MRD)
• To follow the coding progress until Alpha (weekly
  updates)
• Required resources for each feature
• Rows features with owners and links to the
  feature descriptions
• Columns resource names
• Cells necessary man weeks (special skills
  considered)
• Available resources
• Considering holidays, other projects, project
  management
• Padding 25-50
• for target features, unforeseen events,
  underestimated features

13
Life Cycle / Definition, Design

• Feature matrix (cont.)
• Priority of the feature
• Minimal (committed) / Target (may be realized)
• Dropped (by marketing or a target feature during
  coding)
• Feature meetings
• Useful to understand the feature implementation
  details and their effects
• 3-4 features discussed daily with
• Project manager
• Director
• Developer(s) implementing the feature
• QA Lead
• Technical writer

14
Life Cycle / Coding

• Coding (Implementation)
• High risk items first
• New 3rd party items or new versions of them
  should be definitely taken care of first
• Experts with unique knowledge
• Very effective and very vulnerable
• Technology test group within development
• But unit tests are not a general rule
• No obligatory coding standard
• Currently code review only for new hires
• Extension under preparation
• Planned after Alpha, based on bug statistics

15
Life Cycle / Coding

• Coding (cont.)
• Unified installer configurations
• Avoid sharing components run-time between
  separate products
• It increases complexity a lot
• Version control
• Using unified folder structures
• Build process
• In-house automated nightly build process with
  labeling, binary versioning and error reporting
  (including automated test script results after
  Alpha)

16
Life Cycle / Coding

• QA preparation
• Test Case List
• A list of all the project-related planned /
  implemented Test Cases
• Test Case
• One suite of manual test steps to check a
  specific item of product functionality
• We write them in the form of test automation
  scripts
• Test Matrix
• Plan / report about the performed test steps
• who performed the test, which test case, when,
  how long it took, operating system, build id,
  bugs reported

17
Life Cycle / Coding

• QA preparation (cont.)
• Number of Test Cases
• No theoretical upper limit. A higher number
  yields better coverage, but raises costs.
• We plan weekly test cycles. Test cases are sized
  to fit all functionality tests under one
  operating system into one week for one tester.
• We usually test on 5 operating systems
• Basically finalized by mid-Alpha
• Usually tuned even later based on the test
  results, external beta test reports, random tests

18
Life Cycle / Coding

• The trend of the number of man-weeks necessary to
  implement the minimum and target features
  predicts Alpha date

19
Life Cycle / Alpha

• Milestones
• Alpha acceptance test
• Run on Alpha-candidate(s)
• Test cases to check feature availability
• Benchmarked parameters with Alpha thresholds
• About 25 parameters accuracies, speeds,
  stability, leak etc.
• Marketing review
• Last-minute call for marketing to tune the UI and
  the features
• Resources reserved for an iteration cycle
• UI Freeze
• Finalize program resources then user guide and
  help
• These items are usually on the critical path for
  the localized releases
• Localization starts

20
Life Cycle / Alpha, Beta

• The primary goal is to detect and fix bugs
• Based on years of experience in three different
  companies, both the Alpha and the Beta phase
  should be 10 weeks long
• Compression disorganizes the process and finally
  results in at least the same amount of time
• Bug track database is used with a bug fixing
  workflow
• Allowed state transitions per role, change
  history, on-line reports on the intranet

21
Life Cycle / Alpha, Beta

• Tools / QA
• Usually 3 computers for each tester
• Disk images for each platform and hardware
• Test automation
• It has become more relevant recently
• Tests are run on about 20 machines 24 by 7
• Script development
• Starts only in Alpha because it is very sensitive
  to structural changes
• Implemented test steps are commented out from the
  test cases so manual testers do not perform them
• Pretty expensive to prepare them for all kinds of
  failure situations and to give a meaningful test
  report
• During script development a high portion of bugs
  is detected
• Scripts pay off in the regression tests of the
  sustaining phase

22
Life Cycle / Alpha, Beta

• Tools / Developers
• Test machines with disk images and multi-OS
  emulators are used to reproduce bugs without
  disturbing QA
• Special software tools to fix hard-to-isolate
  problems
• Memory over- and underwrites
• Memory leaks
• Threading problems
• Using common components for
• Logging (run-time selectable level)
• Setting debug variables run-time
• Letting system-level exceptions be caught in JIT
  debugger

23
Life Cycle / Beta

• Beta acceptance test
• Run on Beta candidate(s)
• Using all test cases
• Benchmarked parameters with Beta thresholds
• External Beta cycles
• Managed by technical support
• Base test scripts prepared by QA
• We usually have 2-3 external beta cycles
• Important to get test results from
• Real-life, non-clean software environments
• Machines with software and hardware components
  not available in?house

24
Life Cycle / Beta

• The trend of benchmarked parameters
• Predicts dates when Alpha/Beta/RTM thresholds can
  be met
• Especially useful considering historical data
  trends e.g. improvement in the last 10 weeks

25
Life Cycle / Beta

• Statistical analysis of the bugs
• Most useful before RC
• Comparison with historical data results in more
  reliable estimatese.g. turning point in open
  bugs happened 12 weeks before RTM for the
  previous version

26
Life Cycle / Beta

• Bug meetings
• Low-priority bugs are requested to be deferred to
  reduce the number of changes and the associated
  risk
• The PDT (mainly technical support) decides to
  accept or refuse the request
• Daily meetings in the last test cycle(s)
• Release candidate
• Very important milestone
• Does every participant believe that the build,
  with all its known problems, could be released,
  if the next test cycle does not detect any new
  bug?
• Check-in only through the project manager
• Tests performed from the final media

27
Life Cycle / Sustaining

• Deliverables
• Localized versions, trials, point releases,
  patches
• Team
• Separate small team of 2-3 developers
• Very QA-intensive period. Typically 5 operating
  systems and max. 2-3 languages tested in
  parallel.
• Archival
• Seems to be simple so long as no-one tries to
  recreate the build (tools, instructions,
  environment)
• Current version control system is not reliable
  enough
• Sources on DVD and masters on CD kept safe in 2
  copies geographically separated

28
Project control

• ISO 90012000 and TickIT certification
• External audits twice a year
• Internal audits 2-3 times a year
• Control documents on the intranet
• Software Development Handbook
• Describes the development process
• Specifies locations for tools, documents, source
  code, defect database etc.
• Very useful for new hires
• Our project quality plan is expressed in a life-
  cycle form on the intranet with all the
  milestones and the placeholders for the required
  documents

29
Project control

• Build reports
• Informs development and QA about the location and
  status of the current build and the known issues
• Test reports
• QA summarizes its findings in the form of a test
  report about important builds (Alpha, Beta, RC)
• PDT conference calls once a week
• Detailed project status reports every second week

30
Deviations

• Schedule compression
• Usually a time-to-market requirement
• Losing resources
• Mainly to a higher priority project
• Creeping features
• Changes in market conditions or late discoveries
  about cross-effects may cause features to creep
• 3rd parties and outsourcing
• For economical reasons suppliers with much
  inferior quality systems may be selected

31
Deviations

• Dealing with deviations
• Cutting administrative corners temporarily
• No process revisions
• Skipping reports
• Not updating project documents
• Putting archival tasks aside
• Design
• Feature bargaining
• Coding
• Using padding and dropping (target) features
• Alpha and Beta
• Shortening test cycles (but not reducing their
  number)
• Hiring more temps (typically QA)

32
Conclusions

• Our engineering process is far from being optimal
  from a project management point of view
• Other presentations try to describe such ideal
  methods
• Listening to these, youd probably feel guilty as
  we did for not following all their instructions
  and advice
• However, we believe our approach is closer to
  optimal economically
• The evidence for this may be the number of our
  successfully delivered projects
• We know that behind the implementation and
  operation of each small step towards an ideal
  plan there lies blood, sweat and tears