Increasing the Business Value of Software Using a Separation of Concerns Approach to Software Testin

1
Increasing the Business Value of Software Using a
Separation of Concerns Approach to Software
Testing

• D. P. Bullington
• Sr. Software Engineer

2
Quality is a Differentiator

• If you can't test it, don't build it. If you
  don't test it, rip it out. - Boris Beizer
• As a discipline, software development has quality
  standards far below those of other engineering
  disciplines.
• Managers and developers are good at rationalizing
  their products' shortcomings
• The functionality they offer is "better than
  nothing" or "better than the former system," they
  say, and there are manual backups in case the
  system crashes. Or, lets just get it done and fix
  the bugs later.
• But would a structural engineer reason away the
  collapse of a building in a similar way? Would an
  aeronautical engineer merely shrug his shoulders
  if a control panel malfunctioned in mid-flight?
• To support a business effectively, software
  engineers must also move toward a zero tolerance
  policy for defects.

3
Perceived Business Value and Software Quality

• If the perceived business value of software were
  expressed as a function, if could be in the form
• Value f (Cost, Scope, Time, Quality)
• Mimics the classic software development triad of
  cost, time, and scope quality being a new
  dimension to the software development equation.
• Produces a four dimensional abstract universe in
  which points can describe potential business
  value.
• Negative Extreme f (high, out-of, inadequate,
  low)
• Positive Extreme f (low, just-in, enough, high)
• We live in the real world where the perceived
  business value of real software projects vary
  from one to another, for reasons as unique as the
  project itself (e.g. poor management, poor
  technical ability, lack of planning, lack of
  vision, etc.)

4
(No Transcript)
5
PBV, Quality, and the Software Developer

• For a developer, more often than not, they have
  limited, if any, direct control over the
  perceived business value of a software project.
• Developers though, are often blamed first when
  software projects go sour.
• What a developer CAN control is the quality input
  which ironically can actually affect the other
  three inputs indirectly.
• To control quality effectively, a well-defined
  process with unanimous buy-in from team members
  is required.
• Quality across the lifecyclefrom start to finish
  and beyond.

6
The Give and Take of Quality

• Quality software can decrease costs by flushing
  out bugs in the active coding phase of the
  development cycle as opposed to the QA phase or
  even worse in production.
• Developers should be the first line of defense in
  bug discovery, not QA.
• Of course, nothing comes without a catch quality
  software can lower the amount of scope you
  address in a given box of time due to the
  development of test suites which could affect
  cost.
• But wait National Institute of Standards and
  Technology (NIST) found that about 80 percent of
  development funds are consumed by software
  developers identifying and correcting defects.
• So we could hypothesize that the give and take of
  software quality is a zero-sum proposition in
  that you spend more on the front end
  (development) to save your butt on the back end
  (QA and production).

7
Here Comes The Oh No Factor

• The mere mention of increasing cost, reducing
  scope and/or increasing time strikes fear in the
  hearts of unenlightened project stakeholders
  because all they see is that they could
  sacrificing something (and quality rarely enters
  their mind).
• The difference between a successful software
  project and one that falls flat on its ass is
  generally how well the business value equation is
  managed.
• If quality is allowed to slip in order to eek in
  more scope in less time at a cheaper cost, then
  failure becomes self fulfilling or at the very
  least maintenance costs in production will be
  high.
• Project stakeholders need to understand that
  quality is a first class business value equation
  variable and must be accounted for in project
  estimates.
• Consider the DDTD strategy for estimating

8
Quality gt Testing

• Sobuilding quality software requires software
  testing.
• Any testing no matter how off-base is better than
  zero testing.
• Without a firm grasp of certain testing concepts
  producing a correct and repeatable test suite
  becomes difficult.
• Just as developers (hopefully) strive to provide
  for the separation of concerns in application
  code via layering, object oriented
  programming/design, etc., so must you also be
  cognizant of and implement a clear-cut separation
  of concerns when it comes to software testing
  strategy.
• To illustrate this, lets look at what I call the
  Levels of Software Testing.

9
Level 0 Syntactical Testing

• Serve as the start of the testing journey.
• If your code does not compile, build, link, or
  interpret then you have just immediately failed
  your test suite.
• Poorly documented, structured, or just plain
  sloppy code fails another test the
  maintainability test.
• On a side note, crashing your compiler also is an
  immediate test suite failure.

10
Level 1 Unit Testing

• Unit testing is a quite simple but very
  misunderstood testing process. The primary goal
  of unit testing is to take the smallest piece of
  testable software in the application, isolate it
  from the remainder of the code, and determine
  whether it behaves exactly as you expect. Each
  unit is tested separately before integrating them
  into modules to test the interfaces between
  modules.
• In order to test the interfaces (i.e. point of
  interconnection) between modules, the use of
  interfaces (i.e. an abstraction of a software
  component), dependency inversion and injection
  principles, and mock and/or stub objects are the
  correct and preferred way to proceed.
• Leveraging these measures also has the desirable
  side effect of ensuring separation of concerns in
  application code as well.

11
Correct Unit Testing

• It is all too unfortunate that many developers do
  not understand what a unit test is designed to
  test and not to test.
• Example
• Because some classes may have references to other
  classes, testing a class can frequently spill
  over into testing another class. A common example
  of this is classes that depend on a database or
  other component in order to test the class, the
  tester often writes code that interacts with the
  database or other component. This is a mistake,
  because a unit test should never go outside of
  its own class boundary. As a result, the software
  developer abstracts an interface around the
  database connection or other component, and
  then implements that interface with their own
  mock object or using a dynamic mock library. By
  abstracting this necessary attachment from the
  code (temporarily reducing the net effective
  coupling), the independent unit can be more
  thoroughly tested than may have been previously
  achieved. This results in a higher quality unit
  that is also more maintainable.

12
Unit Testing Tips

• Examples of good unit test suites include those
  which exemplify
• Are actually unit tests if you see a test that
  connects to a live database or leverages an
  external component then that test is not a unit
  test it is an integration test.
• Are automated and headless execution (no one-off
  Win Forms test harnesses) with code coverage
  enabled. This provides much needed introspection
  into how well the development team is adhering to
  the testing strategy.
• Should test the both positive and negative
  conditions. Failure to test edge cases and
  boundary conditions is a major cause of bugs
  which slip into QA and/or production.

13
Level 2 Integration Testing

• The goal of integration testing is to expose
  defects and inconsistencies in the interfaces and
  interaction between integrated units.
• There are many ways to perform integration
  testing but generally the strategies come down to
  top-down, bottom-up, and functional-driven.
• A top-down approach entails starting your
  integration testing with integrated components as
  high up in the dependency space as feasible and
  working down until you have a full suite a
  bottom-up approach is the opposite in that you
  start your integration testing with integrated
  components as low down in the dependency space as
  feasible and working up until you have a full
  suite.
• A functional-driven approach uses key feature
  code paths as the guiding light in integration
  testing.

14
Integration Testing Tips

• Smaller integration (versus large system tests)
  tests will help narrow the area where the failure
  lies.
• An integration test must be isolated in setup and
  teardown.
• If it requires some data to be in a database, it
  must put it there.
• It must also run fast. If it is slow, build time
  will suffer, and you will run fewer builds -
  leading to other problems.
• Integration tests should be order-independent.
• It should not matter the order you run them they
  should all pass.

15
Level 3 System Testing

• Testing conducted on a complete, integrated
  system to evaluate the system's compliance with
  its specified technical and non-technical
  requirements.
• During system testing, certain behaviors,
  artifacts, and/or aspects of the system are
  usually tested, generally because they are
  properties of the entire system
• Capacity, Scalability, Performance, Load, Volume,
  and Stress
• Security and Error Handling
• Installation, Maintenance and Recovery
• Accessibility, UI, Usability, and Help

16
Level 4 Acceptance Testing

• Acceptance testing involves performing automated
  or manual tests by the end-user, customer, QA
  team, or client to validate whether or not the
  product meets acceptance criteria.
• The end result is a decision to accept the
  product all, some, or none of the product as
  being feature complete and feature correct.
• Acceptance testing can spring a development team
  right back to where they started Testing Level
  0 as undiscovered bugs, faulty interpretation of
  requirements, etc. can cause code changes.

17
What About Regression Testing?Level -1

• Regression bugs occur whenever software
  functionality that previously worked as desired,
  stops working or no longer works in the same way
  that was previously planned.
• Typically regression bugs occur as an unintended
  consequence of program changes or refactoring.
• Regression testing is a cross cutting testing
  aspect, and as such, should really be a side
  effect of all the other level of testing.
• Thus a complete test suite which passes should in
  theory provide a high level of confidence that
  any changes had zero impact assuming a correct
  test suite is developed with a depth and breadth
  that can provide earn said confidence.
• The use of a continuous integration system can
  ensure timely notification of regression bugs
  (and other bugs as well).

18
The Net Result

• The business value and success of software
  projects relies on quality, which in turn
  requires testing, which in turn necessitates a
  solid plan to be successful.
• Focusing the testing starts at smallest concern
  (source code) and continually broadening (units,
  components, system) to the largest concern
  (customer) forms a pyramid like paradigm very
  well suited to structure a winning software
  testing strategy.

19
(No Transcript)
20
Conclusion

• Comments?
• Questions?
• Complaints?

21
References

• various authors, MSDN
• various authors, Wikipedia
• Jeffery Palermo, Headspring Systems
• Geoffrey Bessin, IBM
• Boris Beizer, Software Research