Software initiatives 4 Quality Standards - PowerPoint PPT Presentation


PPT – Software initiatives 4 Quality Standards PowerPoint presentation | free to view - id: 6482d8-YmUxM


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Software initiatives 4 Quality Standards


Software initiatives 4 Quality Standards See Word 97 file Software initiatives 4 Quality (BURKS) The totality of features and characteristics of a product or ... – PowerPoint PPT presentation

Number of Views:53
Avg rating:3.0/5.0
Slides: 76
Provided by: john1149


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Software initiatives 4 Quality Standards

Software initiatives 4Quality Standards
  • See Word 97 file
  • Software initiatives 4

Quality (BURKS)
  • The totality of features and characteristics of a
    product or service that bear on its ability to
    satisfy stated or implied needs. Not to be
    mistaken for "degree of excellence" or "fitness
    for use" which meet only part of the definition.
  • There is no single, unambiguous definition.

Quality Assurance (BURKS)
  • A planned and systematic pattern of all actions
    necessary to provide adequate confidence that the
    product optimally fulfils customer's
  • This term can be used generically to refer to all
    activities that affect quality in any way.
  • It also has a number of more specific
    definitions, such as the label assigned to
    quality assurance departments and to quality
    assurance personnel.

Quality Control (BURKS)
  • The assessment of product compliance.
    Independently finding deficiencies assures
    compliance of the product with stated
  • This term can be used generically for the same
    concepts as "quality assurance" or more
    specifically to specific combinations of
    inspections and test steps applied to software

Quality Standards (BURKS)
  • "The nice thing about standards is that there are
    so many of them to choose from",
  • Standards are necessary for interworking,
    portability, and reusability.
  • They may be de facto standards for various
    communities, or officially recognised national or
    international standards.

  • Every country has a standards organisation.
  • In addition, many industries and technologies
    also have standards bodies, such as the well
    known standards published by the IEEE (Institute
    of Electrical and Electronic Engineers) and the
    European Computer Manufacturers Association
  • Many large corporations have in-house standards.
    Indeed, for software quality the internal
    standards used at companies such as ATT, IBM,
    Hewlett Packard, Motorola, and many others are
    among the most effective in the world.
  • In some cases, the in-house standards pioneered
    entirely new concepts such as Motorola's well
    known Six-Sigma quality standard.

Some standards-issuing organisations
  • Allied Quality Assurance Publications (AQAP)
    produced by NATO
  • American National Standards Institute (ANSI)
  • British Standards Institute (BSI)
  • Department of Defence (DoD) in the United States
  • Department of Defence (Def) in the United Kingdom
  • Deutsches Institut fur Normung (DIN)
  • European Computer Manufacturing (ECMA)
  • European Space Agency (ESA)

Some more standards-issuing organisations
  • Federal Information Process Standards (FIPS)
  • International Organisation for Standards (ISO)
  • Institution of Electrical Engineers (IEE)
  • Institute of Electrical and Electronic
    Engineering (IEEE)
  • Japanese Industrial Standards Committee (JISC)
  • Japanese Union of Scientists and Engineers (JUSA)
  • Software Publishers Association (SPA)
  • Various in-house standards and guidelines by
    specific companies

Oh Dear!
  • On the whole, the software standards associated
    with quality have had only a marginal effect on
  • Generally speaking, software standards are based
    on the subjective views of the standards
    committee, and seldom have even much solid
    empirical data behind them.

European Software Initiatives
  • The combined software population of Western
    Europe and the U.K. is somewhat larger than the
    software population of the United States and very
    energetic in terms of quality initiatives,
    standards, conferences, and Internet
    communication channels.
  • Quality initiatives such as the ISO 9000-9004
    standards, now have global impact.

The Good
  • The same industries tend to achieve the best
    quality results in every industrialised country
    in Europe, North America, South America, the
    Middle East, and the Pacific Rim.
  • These industries share a common characteristic
    that their software supports large and complex
    physical devices which need close to zero-defect
    software in order to operate successfully.

The Good
  • Aerospace manufacturers
  • Commercial systems software manufacturers
  • Computer manufacturers
  • Defence system manufacturers
  • Medical instrument manufacturers
  • Telecommunication manufacturers

The Bad
  • Several industries where software quality lags in
    the United States appear to be rather good in
    much of Europe insurance and banking software,
    for example.
  • In the USA the insurance and banking software
    development practices tend to be somewhat less
    formal and more casual than the six industries
    with the best quality results.

The Bad
  • The financial and insurance industries have also
    been rushing to get into the
  • client/server world,
  • where quality is often marginal and even worse
    than equivalent mainframe applications.
  • By contrast, financial and insurance software in
    Europe has a higher probability of rigor during
    development, more care in quality control, and a
    greater use of non-test defect removal methods
    such as inspections.

The Ugly
  • The quality of government software seems somewhat
    better in several European countries than here in
    the United States.
  • Software produced by some of the civilian
    agencies of the Federal Government in the United
    States tends to lag the private sector in terms
    of defect prevention and defect removal
    operations, and hence gets deployed with
    substantially more errors than commercial norms.

The Ugly
  • By contrast, government-produced software in
    Belgium, Italy, France, Sweden, and the U.K. is
    sometimes equivalent to information systems
    software produced by public company or their
    outsourcing partners.

  • Requirements and design approaches are often more
    formal in Europe, and some of the European
    front-end approaches are technically very
  • On the whole, formal methodologies are used more
    often than in the United States, and much more
    often in the information systems and
    client/server domains where U.S. practices tend
    to be rather careless.
  • The overall result is that European MIS software
    projects often run about 10 to 20 below
    equivalent U.S. projects in requirements and
    design errors.

  • The patterns of programming language usage tend
    to differ between the U.S. and Europe.
  • Logic-based languages such as ALGOL, Prolog, and
    Miranda have a higher frequency of use as do
    strongly-typed languages such as PASCAL and
  • The usage of Ada83 for civilian projects is far
    more common in Europe than in the United States.
  • Programming language research is very strong in
    much of Europe.

  • Usage of commercial software cost estimating
    tools and other software project management aids
    are less common in Europe than in the United
    States, although both sides of the Atlantic are
    now experiencing rapid growth in project
    management technologies.
  • Usage of function point metrics for data
    normalisation is roughly equal between the United
    States and Europe in terms of frequency of use.

  • Membership in various function point users'
    groups is growing at about 50 per year in every
    industrialised country, while usage of "1ines of
    code" is declining by about 10 per year.
  • In Europe, large function point associations have
    been formed in France, Italy, the Netherlands,
    the United Kingdom, and also a multi-country
    Scandinavian function point group.
  • Function point usage is growing but not yet
    formally organised in Austria, Belgium, Germany,
    Greece, Portugal, or Spain.

  • Although the factor is more important for
    productivity studies than for quality studies,
    the rather lengthy European vacation periods
    (running to more than 30 days per year in some
    cases) tend to favour countries with more working
    days per year, such as the United States, Japan,
    and India.

  • Although a difficult topic to explore since many
    tracking systems don't record it, the amount of
    unpaid overtime applied to software projects
    (i.e., unpaid work in the evenings, on weekends,
    and on public holidays) appears low in Europe and
    the United Kingdom compared to the United States
    and Japan.
  • Unpaid overtime is so often part of software
    projects that it can exert an overall
    productivity influence that approaches 20.

  • One of the problems of doing direct international
    comparisons between the United States and Europe
    is that the metrics are sometimes different.
  • A survey of refereed software engineering
    journals carried out a few years ago noted that
  • one-third of the articles using "lines of code"
    were based on physical lines
  • one-third were based on logical statements
  • the remaining third did not even state the basis
    of the lines of code count.
  • almost 20 of the articles did not even identify
    the languages from which the study was based.

  • Of course, there are function point variants too.
  • Between the United States and Europe, at least a
    dozen function point variations can now be
  • On a global basis, however, standard IFPUG
    function points have about 90 penetration in
    terms of published books, articles, and data.

ISO 9000 (BURKS)
  • A set of international standards for both quality
    management and quality assurance that has been
    adopted by over 90 countries world-wide.
  • The ISO 9000 standards apply to all types and
    sizes of organisations.
  • Documentation is at the core of ISO 9000
  • Say what you do.
  • Do what you say.
  • Write it down.

ISO 9000
  • The standards require
  • a standard language for documenting quality
  • a system to manage evidence that these practices
    are instituted throughout an organisation and
  • third-party auditing to review, certify, and
    maintain certification of organisations.

ISO 9000
  • Say what you do.
  • Do what you say.
  • Write it down.

Used where?
  • The ISO standards are used intermittently in the
    United States, Japan and the Pacific Rim, South
    America, the Middle East, and Eastern Europe.
  • The ISO standards have had far and away their
    greatest deployment in Europe and the United
  • Indeed, they are now often required for marketing
    products within the European Community (EC).

Used where?
  • Not every company utilises the ISO standards in
    Europe, but the penetration is probably now in
    excess of 20 in Western Europe and the U.K. as
    opposed to perhaps 3 elsewhere based on
    interviews with software managers and QA
  • A few U.S. companies that produce only domestic
    software respond to questions about ISO standards
    with a question of their own "What are they?"

ISO 9001 and ISO 9000.3
  • ISO 9001 is the primary standard on how products
    are built, and a guideline
  • ISO 9000.3 explains the ISO approach in a
    software context.

But ...
  • So far as can be determined, there is not yet
    any tangible or perceptible difference in the
    software quality levels of companies which have
    been certified when compared to similar companies
    in the same industries which have not been

More But ...
  • Currently the most tangible aspects remain
  • the high costs of the ISO certification process,
  • the large volumes of planning and control
    documents, and
  • the time required to achieve certification.

Motorolas 2 Objections
  • ISO certification was used for marketing purposes
    with an implied assertion that achieving ISO
    certification meant high quality levels
    regardless of the lack of empirical data
    demonstrating that point. Motorola's Vice
    President of Quality, bluntly stated that was
    false advertising.
  • The ISO standards themselves lacked some of the
    key quality elements found in Motorola's own
    quality standards and could be viewed as a step
    back rather than a step forward in terms of
    quality methodology.

  • Some ISO users have observed that it is not the
    place of the ISO 9000-9004 standards to actually
    improve quality.
  • Their purpose is simply to ensure that effective
    practices are followed, and are not in and of
    themselves a compilation of best current

Does ISO 9001 ...
  • reduce software defect potentials below 5 per
    function point?
  • raise defect removal efficiency levels above 95
  • reduce the number of had test cases below 15 ?
  • reduce the rate of "bad fix" injections below 7
  • increase reliability under field conditions?
  • increase customer satisfaction levels?
  • have any affect on data quality?
  • reduce the number of cancelled or delayed
  • reduce the incidence of litigation for breach of
  • reduce the incidence of litigation for poor
    quality control?

  • Software standards groups should consider the
    methods through which medical and pharmaceutical
    standards are set.
  • Although medical standards are not perfect, their
    concept is that a therapy should be proven to be
    effective and its side effects understood before
    it becomes a standard.
  • It would be a strong statement that software is
    approaching the status of a real profession if
    software standards included information similar
    to medical and pharmaceutical standards on
    efficacy, counter-indications, known hazards, and
    potential side effects.

  • Software standards are normally totally published
    in a way that is totally devoid of empirical data
    regarding their effectiveness when the standards
    are first promulgated.
  • The fact that there may possibly be negative or
    harmful side effects from following the standards
    is usually not even discussed, although negative
    side effects are a major section of medical and
    pharmaceutical standards.
  • Standards organisations should collect data, and
    should encourage auditors and assessors to move
    toward quantification and empirical results
    rather than subjective opinions.

  • TickIT is a method of evaluating software quality
    that originated in the late 1980s by the BCS.
  • The TickIT approach is based on ISO 9000.3 and is
    intended to serve as a consistency check on
    auditing ISO standards as applied to software.
  • The TickIT concept includes both training and
    certification of auditors, who are then qualified
    to perform ISO audits.
  • The TickIT approach includes on-site interviews
    and assessments and envisions the production of a
    number of quality-related plans and reports.

  • TickIT is widely used throughout the United
    Kingdom, and especially so for projects involving
    the government.
  • Since the ISO standards did not originate for
    software, there is a strong need to customise or
    tailor the ISO standards and audits to the
    software world. The TickIT concept is to serve as
    a translator of ISO philosophy into the software
  • Empirical data on the effectiveness of the TickIT
    approach is sparse.
  • The TickIT concept and the ISO standards ignore
    both functional metrics and defect removal
    efficiency measurement.

  • "Software Process Improvement and Capability
  • The SPICE approach is a composite software
    assessment method based in part on
  • the Software Engineering Institute (SEI)
    capability maturity model (CMM)
  • the Bell Northern Research TRILLIUM assessment
    approach and
  • the International Standards Organisation (ISO)
    9000-9004 standard series.

  • The SPICE approach is moving toward an
    international standard for software process
    assessments that is intended to improve the
    relationship between software contractors and
  • Many of the SPICE concepts are derived from the
    needs of large corporations and large government
    agencies whose software contracts might run to
    many millions of dollars for key contracts, and
    billions of dollars in aggregate.

  • SPICE is a bold and ambitious program, and like
    many bold programs, also has some risks
    associated with it.
  • It is premature to judge the overall impact of
    SPICE since it is still evolving.
  • The direction is promising, but when so many
    organisations are involved in standards the
    results are often cumbersome and unsuitable for
    small projects and small companies.

Software development in 1984
  • More than half of large software systems were
    over 12 months late.
  • The average costs of large software systems was
    more than twice the initial budget.
  • The cancellation rate of large software systems
    exceeded 35.
  • The quality and reliability levels of delivered
    software of all sizes were poor.
  • Software personnel were increasing by more than
    10 per year.
  • Software was the largest known business expense
    that could not be managed.

Software Engineering Institute
  • In order to explore software issues, and
    especially those topics associated with defence
    contracts, the Department of Defence in 1984
    funded a new software research facility.
  • This was the essential origin of the Software
    Engineering Institute.
  • The SEI is located on the campus of Carnegie
    Mellon University in Pittsburgh.
  • It is a non-profit software research organisation
    funded primarily by the US government through the
    Defence Advanced Research Project agency (DARPA).

SEI Assessment Approach
  • The SEI assessment data is collected by means of
    on-site interviews using both a standard
    questionnaire and also observations and informal
  • Once collected, the assessment data is analysed,
    and used to place a software organisation on one
    of the five plateau of the now well-known SEI
    Capability Maturity Model
  • CMM

System Size?
  • The SEI assessment approach originally dealt
    primarily with the software processes and
    methodologies used by very large companies that
    produced very large systems.
  • It was derived from the best practices used by
    leading corporations such as IBM and ITT which
    employ from 5,000 to more than 25,000 software
    professionals, and which could safely build
    systems in excess of 1,000,000 lines of code or
    10,000 function points.

  • The Software Engineering Institute's model of
    software engineering that specifies five levels
    of maturity of the processes of a software
  • CMM offers a framework for evolutionary process
  • Originally applied to software development
    (SE-CMM), it has been expanded to cover other
    areas including Human Resources and Software

Focii - key process areas Level - 1 Initial
  • Heroes
  • None.

Level 2 - Repeatable
  • Project Management
  • Software Project Planning, Software Project
    Tracking and Oversight, Software Subcontract
    Management, Software Quality Assurance, Software
    Configuration Management, Requirements

Level 3 - Defined
  • Engineering Process
  • Organisation Process Focus, Organisation Process
    Definition, Peer Reviews, Training Program,
    Inter-group Coordination, Software Product
    Engineering, Integrated Software Management.

Level 4 - Managed
  • Product and Process Quality
  • Software Quality Management, Quantitative Process

Level 5 - Optimising
  • Continuous Improvement
  • Process Change Management, Technology Change
    Management, Defect Prevention.

  • The SEI assessment program was criticised for
    concentrating on too narrow a set of topics, and
    ignoring other topics such as personnel,
    compensation, hiring practices, and the like.
  • In 1995 the SEI filled a notable gap by
    publishing a People Capability Maturity Model or
  • The lead researcher on the P-CMM was a cognitive
    psychologist, Dr. Bill Curtis.
  • He had formerly headed the SEI assessment practice

Level 1 initial
  • Characterised by random or chaotic development
    methods with little formality and uninformed
    project management.
  • Some smaller projects may be successful due to
    the capabilities of individual staff members,
    large systems are often failures and the overall
    results are marginal to poor.
  • In terms P-CMM, such organisations are asserted
    to be deficient in training at both the technical
    staff and managerial levels.

Level 1 initial
  • SEI does not recommend any key process areas for
    Level 1 organisations on the grounds that they
    are not sophisticated enough to deploy methods
    consistently and will probably botch things up if
    they try.
  • Moving from Level 1 to Level 2 is the most
    difficult transition point.
  • For a large company with perhaps 1,000 software
    personnel, the move from Level 1 to Level 2 can
    take from 18 to more than 36 months.

Level 2 repeatable
  • Level 2 or Repeatable organisations have
    introduced at least some rigor into project
    management and technical development tasks.
  • Approaches such as formal cost estimating are
    noted for project management, and formal
    requirements gathering are often noted during
  • Compared to the initial level, a higher frequency
    of success and a lower incidence of overruns and
    cancelled projects can be observed.

Level 2 repeatable
  • In terms P-CMM, the Level 2 organisations have
    begun to provide adequate training for managers
    and technical staff.
  • Further, they have become aware of the importance
    of professional growth and the need for selecting
    and keeping capable personnel.

Level 2 repeatable
  • The key process areas which SEI recommends for
    Level 2 organisations include requirements
    management, project planning, resource tracking,
    quality assurance, and configuration management
    plus subcontract management for projects that
    utilise multiple contractors.
  • The key process areas for Level 2 of the People
    CMM include compensation, training, staffing,
    communication, and work environment.

Level 2 to 3?
  • Moving from Level 2 to Level 3 is not as
    difficult as the first move, since organisations
    have usually become used to the idea that change
    will be necessary.
  • However, the same size organisation illustrated
    previously with 1000 software personnel, should
    not expect to go from Level 2 to Level 3 in less
    than a year, and it may take more than 18 months.

Level 3 defined
  • Level 3 or Defined organisations have mastered a
    development process that can often lead to
    successful large systems.
  • Over and above the project management and
    technical approaches found in Level 2
    organisations, the Level 3 groups have a
    well-defined development process that can handle
    all sizes and kinds of projects.

Level 3 defined
  • In terms of the P-CMM, the Level 3 organisations
    have developed skills inventories and are now
    capable of selecting appropriate specialists who
    may be needed for critical topics such as
    testing, quality assurance, web mastery, and the
  • Level 3 organisations should be capable of
    effective software reusability programs.

Level 3 defined
  • The key process areas which SEI recommends for
    Level 3 organisations include establishing an
    effective organisational infrastructure, training
    of both managers and technical staff, inter-group
    co-ordination, and the utilisation of formal
    design and code inspections.
  • The key process areas for Level 3 of the P-CMM
    include career development, competency-based
    practices, work force planning, and analysis of
    the knowledge and skills that are needed by the

Level 3 to 4?
  • Organisations that achieve Level 3 sometimes rest
    on their laurels, since Level 3 is the minimum
    level needed to bid on some U.S. defence
  • Organisations that are pursing "best in class"
    goals will find Level 3 to be only a way station.
  • For these energetic groups, moving from Level 3
    to Level 4 can sometimes be accomplished in less
    than 12 months.

Level 4 managed
  • Level 4 or Managed organisations have established
    a firm quantitative basis for project management
    and utilise both effective measurements and also
    effective cost and quality estimation.
  • In terms of the People CMM, the Level 4
    organisations are able to not only monitor their
    need for specialised personnel, but are actually
    able to explore the productivity and quality
    results associated from the presence of
    specialists in a quantitative way.

Level 4 managed
  • Long-range predictions of future needs are also
    part of the Level 4 P-CMM assertions. In
    addition, Level 4 organisations would make
    extensive use of mentoring.
  • The key process areas which SEI recommends for
    Level 4 organisations include quantification of
    defect levels, productivity levels, and
    activity-based costing.
  • Level 4 organisations should be approaching 50
    levels of software reuse for both design, code,
    and testing materials.
  • The key process areas for Level 4 of the P-CMM
    include mentoring, team building, organisational
    competency, and the ability to predict and
    measure the effect of specialists and teams in a
    quantitative manner.

Level 4 to 5?
  • Moving from Level 4 all the way to Level 5 is not
    easy to quantify because the actual criteria for
    Level 5 are somewhat abstract.
  • There is no reason to assume that a transition
    from Level 4 to Level 5 should take more than 12

Level 5 optimising
  • Level 5 or Optimising organisations are assumed
    to have mastered the current state-of-the-art of
    software project management and development.
  • Such groups are expected to be the pioneers who
    are capable of building entirely new methods that
    advance the state of the art beyond current
  • Although not explicitly stated, Level 5
    organisations should be approaching or exceeding
    a volume of about 75 of reusable design, code,
    and test materials.

Level 5 optimising
  • In terms of the P-CMM, the requirements are an
    extension of Level 4 and hence different more in
    degree than in kind.
  • The P-CMM stresses both coaching and rewards for
    innovation at Level 5.
  • The key process areas recommended for Level 5
    include defect prevention and advancing the
    fundamental software engineering and management
    technologies, plus rapid and effective technology
    transfer and deployment of improvement
  • The key process areas for the P-CMM include
    encouragement of innovation, coaching, and
    personal competency development.

Same again?
  • When the SEI capability maturity model was first
    published it was asserted that both quality and
    productivity levels would improve at the higher
  • This is an area where SEI shares the bad habits
    of the rest of the software industry. The SEI
    assertions about quality and productivity were
    initially made without any kind of study or
    empirical evidence being collected.
  • In fact, even in 1997 the SEI assessment approach
    itself still does not include the collection of
    quantitative information on either productivity
    rates or quality levels.
  • This is a serious flaw in the SEI assessment

Level overlaps
Six-Sigma Quality Levels
  • The Motorola Corporation has received substantial
    interest in its famous six-sigma quality
    program for hardware and manufactured components.
  • The phrase refers to defect densities of 3.4 in
    a million.
  • Since the six-sigma definition is hard to
    visualise for software, an alternative approach
    would be to achieve a cumulative defect removal
    efficiency rate of 99.999999 prior to delivery
    of a product.
  • This method is not what Motorola uses but it
    helps to clarify what would have to be done to
    achieve six-sigma results for software projects.

Six-Sigma Quality Levels
  • Since the current U.S. average for software
    defect removal efficiency is only about 85, and
    quite a few software products are even below 70,
    it may be seen that software producers have some
    serious catching up to do.
  • As it happens, there are a few software products
    which appear to have achieved six-sigma quality
  • In 30 years Capers Jones has observed four
    projects that achieved six-sigma results in their
    first year of deployment out of a total of almost
    10,000 projects.

Six-Sigma Quality Levels
  • Surprisingly, the prognosis for achieving
    six-sigma quality levels for software is fairly
  • The best of the available technologies today can
    approach this level if
  • a full suite of defect prevention and
  • defect removal operations are carried out
  • Indeed, for software projects where reuse of
    certified materials top 75 by volume, and where
    formal inspections are used to augment testing,
    six-sigma quality levels should be achieved

Six-Sigma Quality Levels
  • That does not mean that the software industry is
    ready to try.
  • There are both sociological and technical reasons
    that quality is held back, such as
  • the widespread lack of appreciation of the
    correlation between high quality and short
    schedules and
  • the lack of commercial reusable materials
    certified to zero-defect levels.