Dave Henderson, Ph.D.

1
Systems Development Methodologies and Information
Systems Audit

• Dave Henderson, Ph.D.
• Assistant Professor of AccountingDepartment of
  Accounting and Legal StudiesSchool of Business
  and EconomicsCollege of Charleston

2
Systems Development Life Cycle (SDLC)

• Generic term for the systems development process
• Used by systems analysts to develop an
  information system
• Several generic stages in systems development

3
General Stages in the SDLC

• Feasibility Study
• -Preliminary study undertaken to determine and
  document a project's viability
• Typically analyses financial and technical
  feasibility how much will it cost to build the
  system?, what is the expected ROI?, does the
  technology exist to build the system?
• Planning
• Establish the plans for creating an information
  system
• Develop the project plan
• Develop the scope of the project
• Analysis
• Users and developers collaborate to derive system
  business requirements
• JAD sessions
• Design
• Create technical blueprint of the system
• Technical architecture
• Client server/web-based system
• Database design

4
General Stages in the SDLC contd

• Development
• Turn the design into a physical system
• Coding phase
• Testing
• Test the system using the established test cases
• Deployment
• Distribute system to end users
• Create user guide
• Provide user training
• Maintenance
• Implement a help desk
• Implement system changes when necessary

5
Formalized Systems Development Methodologies

• Siau and Tan (2005) define a formalized
  information systems development methodology as
• A systematic approach to conducting at least one
  complete phase of information systems
  development, consisting of a recommended
  collection of phases, techniques, procedures,
  tools and documentation aids (p. 863).
• A methodology is a version of the SDLC

6
Methodologies
Siau Tan (2005)
7
Why are methodologies important during an IS
Audit?

• Existence of documented systems development
  methodology suggests a more reliable IT
  operating environment
• Provides documentation for the IT audit
• The presence of proper SDLC documentation
  illustrates the level of application of the best
  practices in SDLC. A review of a sample of the
  documents will provide evidence that the entity
  is using SDLC best practices, which provides some
  assurance that systems are being developed
  efficiently and effectively

Singleton, 2007
8
Why are methodologies important during an IS
Audit? contd

• Methodologies are an important element of the
  organizations internal control structure
• Acquire and Maintain Application Structure (A12)
  in COBIT states the following illustrative
  controls are most relevant to the IT general
  control structure for the A12 process
• The organization has a systems development life
  cycle (SDLC) methodology, which includes security
  and processing integrity requirements of the
  organization
• The SDLC methodology includes requirements that
  information systems be designed to include
  application controls that support complete,
  accurate and authorized and valid transaction
  processing
• The organization acquires/develops application
  systems software in accordance with its
  acquisition, development and planning process

IT Control Objectives for SOX, 2002 Singleton,
2007
9
Why are methodologies important during an IS
Audit? contd

• Methodologies
• Serve as IT governance mechanisms
• Provides assurance that systems are being
  developed effectively and efficiently
• Change management procedures recommended by
  methodologies may deter fraud
• Programming techniques recommended by
  methodologies may deter fraud
• Pair-programming

Singleton, 2007
10
Types of Methodologies

• In-house vs. Commercial
• An in-house methodology is proprietary to an
  organization
• A commercial methodology is used across
  organizations
• Traditional vs. Agile

11
Traditional Methodologies

• Typically guided by a sequential development
  model
• Waterfall model
• Specifies tasks to be performed in each phase
• Full requirements document at end of requirements
  phase
• Produces large amounts of documentation
• Communication with end users is typically
  conducted via documents

12
Example of a Traditional Methodology Waterfall
Paul A. Hoadley, Wikipedia
13
Pros of the Waterfall methodology

• Upfront planning
• A requirements defect left undetected until the
  development or maintenance phase will cost 50 to
  200 times as much to fix as it would have cost to
  fix at the requirements phase
• Documentation
• New team members should be able to familiarize
  themselves by reading design documents

14
Criticisms of the Waterfall methodology

• Requirements frequently change
• Difficult to solidify all requirements in the
  requirement phase
• Does not advocate frequent software builds
• Frequent incremental builds are often needed to
  build confidence for a development team and their
  client.
• Excessive documentation

15
Agile Methodologies

• Many Agile methodologies exist
• Scrum, Extreme Programming, DSDM
• Even though many Agile methodologies exist, they
  share a number of common characteristics
• Iterative development
• Focus on communication
• Eschew documentation
• Software is developed in iterations, which may
  last from one to four weeks
• Each iteration is an entire software project
  including planning, requirements analysis,
  design, coding, testing, and documentation
• Developing in iterations minimizes risk
• Goal is to have an available release (without
  bugs) at the end of each iteration
• At the end of each iteration, the team
  re-evaluates project priorities

16
Agile Manifesto

• Agile Manifesto (http//agilemanifesto.org/) is a
  statement of the principles that underpin agile
  software development
• We are uncovering better ways of developing
  software by doing it and helping others do it.
  Through this work we have come to value
• Individuals and interactions over processes and
  tools
• Working software over comprehensive documentation
  
• Customer collaboration over contract negotiation
• Responding to change over following a plan
• That is, while there is value in the items on the
  right, we value the items on the left more.

17
Principles of Agile Methodologies

• Emphasizes customer satisfaction via rapid,
  continuous delivery of working software
• Working software is delivered frequently (weeks
  rather than months)
• Working software is the principal measure of
  progress
• Even late changes in requirements are welcomed
• Close, daily cooperation between business people
  and developers
• Face-to-face conversation is the best form of
  communication
• Projects are built around motivated individuals,
  who should be trusted
• Continuous attention to technical excellence and
  good design
• Simplicity in design and coding practices
• Self-organizing teams

18
Suitability of Agile methodologies

• Best when requirements are emergent and rapidly
  changing
• Necessary conditions for Agile
• The culture of the organization must be
  supportive of negotiation
• People must be trusted
• Fewer staff, with higher levels of competency
• Senior developers
• Empower developers to make decisions
• Organizations need to have an environment that
  facilitates rapid communication between team
  members

19
Suitability of Agile methodologies contd

• Agile development's applicability to the
  following scenarios is open to question
• Large-scale development efforts (gt20 developers)
• Agile has been modified for large-scale efforts
• Distributed development efforts
• Mission- critical systems
• Command and control company culture

20
Criticisms of Agile Methodologies

• Lack of structure and necessary documentation
• Only works with senior-level developers
• Incorporates insufficient software design
• Requires too much cultural change to adopt
• Drastically increases the chances of scope creep
  due to the lack of detailed requirements
  documentation

Extreme Programming Refactored, Stephens and
Rosenberg
21
Example of an Agile methodology Extreme
Programming (XP)

• Most widely used Agile methodology
• XP values
• Communication
• Focus on face-to-face communication among
  developers and with users
• Eschews documentation
• Simplicity
• Design simplicity
• Feedback
• Encourages frequent feedback from users
• Courage
• Courage to take risks during development

Lindstrom Jeffries, 2004
22
Extreme Programming contd

• Three explicit roles
• Customer
• Provide business requirements
• Developer role
• Implements customer requirements
• Management role
• Allocates resources for the teams
• Manager is a facilitator
• XP has a set of core systems development practices

23
XP Core Practices

• Whole Team
• Customers and developers sit together as one team
• Planning Game
• Emphasis is on steering the project, rather than
  exact prediction of what will be needed and how
  long it will take
• Customer Tests
• Customers define one or more automated acceptance
  tests to show the feature is working
• Small Releases
• XP team releases running, tested software with
  every iteration
• Higher visibility for the software

Lindstrom Jeffries, 2004
24
XP Core Practices contd

• Simple Design
• XP teams build software to a simple design
• You Arent Going to Need It
• Pair Programming
• Has the potential to produce better code
• Communicates knowledge throughout the team
• Potential to reduce fraud
• Test-driven development
• Write test cases in the beginning stages of
  development
• Design improvement
• Constant revision of the code
• Remove duplicate code

25
XP Core Practices

• Continuous integration
• XP teams build the software several times a day
• Collective code ownership
• Any pair of programmers can improve code at
  anytime
• Potential to reduce fraud?
• Coding standard
• Team members follow a common coding standard
• Metaphor
• XP teams develop a common vision of how the
  program works
• This program works like a hive of bees going out
  for pollen and bringing it back to the hive as a
  description for an information retrieval system
• Sustainable Pace
• Recognizes that software development is a
  marathon, not a sprint

Lindstrom Jeffries, 2004
26
Additional XP practices

• Open workspace
• XP team works together in a large room with
  tables in the center
• Retrospectives
• XP advocates taking time after each iteration to
  reflect on progress
• Self-Directed Teams
• Empower developers to make decisions
• Manager is more of a facilitator

Lindstrom Jeffries, 2004
27
Comparing Agile to Traditional Nerur, Mahapatra,
Mangalaraj (2005)
28
Agile vs. Traditional contd

• Agile home ground
• Low criticality
• Senior developers
• Changing requirements
• Smaller projects
• Culture that is supportive of negotiation
• Plan-driven home ground
• High criticality systems
• Stable requirements
• Larger projects
• Command and control culture

Lindvall, Basili, Boehm, Costa, Dangle, Shull,
Tesoriero, Williams Zelkowitz, 2002
29
Methodology usage

• Methodology usage among organizations is low
• Hardy, Thompson and Edwards (1995) surveyed 100
  companies in the United Kingdom.
• Only 18 of organizations use a methodology
• Chatzoglou and Macaulay (1996) surveyed 72
  projects within the United Kingdom.
• Less than half--47--of organizations use a
  methodology
• Russo, Hightower and Pearson (1996) surveyed 92
  organizations as well as developers on individual
  projects.
• 20 of organizations claim to never use a
  methodology
• 46 of organizations conduct systems development
  without a methodology at least occasionally
• 31 of systems development on individual projects
  is performed without any methodology.

30
Methodology usage contd

• Methodology usage by developers is low
• Fitzgerald (1998) surveyed 162 individual
  developers
• 60 of developers do not use a methodology
• Only 6 of developers rigorously follow a
  methodology
• Capability Maturity Model Integration (CMMi)
• CMMi compliance is pushing organizations toward
  greater methodology use

31
Methodology tailoring

• Methodology tailoring is the process of changing
  a methodology to accommodate a specific software
  development project
• Rather than following a method rigorously,
  developers adapt methods based upon the
  characteristics of the development context

Fitzgerald, 1997 Fitzgerald et al., 2002
Beynon-Davies Williams, 2003 Taylor 2000
32
Methodology tailoring contd
Fitzgerald, Russo Stolterman, (2002)
33
Developer experience and methodology use

• Experienced developers perceive methodologies as
  plans or guides to action, rather than as
  deterministic rule-sets to be followed rigorously
  (Kautz et al., 2004 Beynon-Davies Williams,
  2003 Nandhakumar Avison, 1999 Fitzgerald,
  1997)
• Experienced developers are likely to rely more on
  the skills they have obtained through experience
  and find methodologies inhibiting (Fitzgerald,
  1997)

34
Developer experience and methodology use contd

• Inexperienced developers find methodologies to be
  useful guides to help them learn the companys
  development practices (Kautz et al., 2004)
• Inexperienced developers who use methodologies
  are more likely to follow them rigorously to ease
  their lack of self-confidence (Fitzgerald, 1997)

35
Developer experience and methodology use contd
Fitzgerald (1997)
36
Goal Displacement

• Goal displacement is when the developer follows
  the method at the expense of actual development
• Does not purposefully tailor the methodology, but
  rather, rigorously follows every step like a
  recipe
• Inexperienced developers are more prone to goal
  displacement
• Due to lack of self-confidence

(Fitzgerald, 1998)
37
How is a methodology useful to a development team?

• Rational usefulness
• Two requirements for a process to be rational
• Process should have a set of identifiable and
  agreed-upon goals
• Process should be prescribed to meet those goals
• A methodology can be a rational process because
  it helps achieve goals that are agreed upon and
  because it has been prescribed to meet those
  goals
• Degree to which using a methodology results in
  valuable outcomes that are agreed-upon by
  stakeholders on the systems development project
  (Henderson, unpublished dissertation)

38
Dimensions of Rational Usefulness

• Product Usefulness
• Degree to which using a methodology improves the
  quality of the developed system
• Methodologies can be useful for improving system
  quality (Huisman Iivari, 2006 Johnson et al.,
  1999)
• Product Usefulness is a significant factor in
  what makes a methodology useful to developers
  (Henderson, unpublished dissertation)
• Process Usefulness
• Degree to which using a methodology improves the
  productivity of the systems development process
• Methodologies can be useful for improving the
  productivity of the development process (Huisman
  Iivari, 2006 Johnson et al., 1999 Khalifa
  Verner, 2000)
• Process Usefulness is a significant factor in
  what makes a methodology useful to developers
  (Henderson, unpublished dissertation)
• Communication Usefulness
• Degree to which using a methodology improves
  communication with users and other team members
• Communication usefulness is a factor of perceived
  usefulness for a methodology (Johnson et al.,
  1999)
• Communication Usefulness is a significant factor
  in what makes a methodology useful to developers
  (Henderson, unpublished dissertation)

39
How is a methodology useful to a development
team? contd

• Political usefulness
• 2 requirements for a process to be considered
  political
• Motive refers to the existence of two or more
  individuals having different objectives
• Opportunity a situation in which some
  individuals or groups may achieve their own
  objectives to the relative disadvantage of others
• Methodology is a political process because it can
  be used to achieve objectives particular to one
  group or individual to the relative disadvantage
  of other stakeholders (Robey Markus, 1984)
• Degree to which using a methodology results in
  valuable outcomes that are particular to one
  group or individual and may result in the
  relative disadvantage of other stakeholders on
  the systems development project (Henderson,
  unpublished dissertation)

40
Dimensions of Political Usefulness

• Career Usefulness
• Degree to which using a methodology enhances
  career opportunities
• Career Usefulness may be significant for a
  formalized commercial methodology because they
  are used across organizations
• Information Technology professionals are
  motivated by skill acquisition (Ferratt Short,
  1998 Igbaria, Parasuraman Badawy, 1994)
• Career Usefulness is a significant factor in what
  makes a methodology useful to developers
  (Henderson, unpublished dissertation)
• Symbolic Usefulness
• Degree to which using a methodology projects an
  impression of control and professionalism
• Methodologies are useful for portraying
  professionalism and showing that systematic
  processes are being used (Fitzgerald, 1998b
  Fitzgerald et al., 2002) Kautz et al., 2004
  Nandhakumar Avison, 1999Robey Markus, 1984)
• Symbolic Usefulness is a significant factor in
  what makes a methodology useful to developers
  (Henderson, unpublished dissertation)

41
Dimensions of Political Usefulness contd

• Support Usefulness
• Degree to which using a methodology reduces
  developer anxiety
• Systems development is a stressful process
  (Wastell Newman, 1993)
• Methodologies are useful for helping developers
  cope with the stress of systems development
  (Kautz et al., 2004 Wastell, 1996 1999)
• Support Usefulness is a significant factor in
  what makes a methodology useful to developers
  (Henderson, unpublished dissertation)
• Defense Usefulness
• The degree to which using a methodology provides
  protection in case decisions made during systems
  development turn out to be wrong and protects
  against unreasonable user demands.
• Methodologies
• May be used to authorize decisions made during
  systems development (Wastell, 1999)
• Create an audit trail of the development process
  (Fitzgerald, 1998b Fitzgerald et al., 2002)
• Insulate developers from unreasonable user
  demands (Fitzgerald et al., 2002 Kautz et al.,
  2004)
• Defense Usefulness is a significant factor in
  what makes a methodology useful to developers
  (Henderson, unpublished dissertation)

42
Implications for IS Audit Risks during systems
development

• Adoption of inappropriate methodology
• Inadequate controls in the development process
• User requirements not met by application system
• Inadequate stakeholder (including internal audit)
  involvement
• Lack of management support
• Inadequate project management

IS Auditing Guideline, Document G23, ISACA
43
Implications for IS Audit Risks during systems
development contd

• Inappropriate technology and architecture
• Scope variations
• Time and cost over-runs
• Insufficient attention to security and controls
  in the application system
• Insufficient documentation
• Inadequate adherence to chosen methodology
• Inadequate configuration management
• Insufficient planning for data conversion/migratio
  n and system cutover

IS Auditing Guideline, Document G23, ISACA
44
Implications for IS Audit Factors to be
considered during development

• IS auditor should consider the following when
  auditing the development process for an
  application system
• The technology, size, objectives and intended
  usage of the system
• Skill and experience of the project team
• The systems development methodology chosen
• Customization of the methodology
• The current stage of the development process
• Any prior review of earlier stages in the
  development process
• Any prior reviews of the development process of
  similar applications

IS Auditing Guideline, Document G23, ISACA
45
Implications for IS Audit IS Auditor skills

• Possess knowledge of various methodologies being
  used
• IS Auditor may need to seek external resources to
  complement skill availability

IS Auditing Guideline, Document G23, ISACA
46
Implications for IS Audit Aspects to be reviewed

• IS Auditors should review
• Project charter and business case
• Project plan, deliverables and schedule
• Formal project methodology and the tailoring of
  the methodology
• Contracts with suppliers for purchased
  application software
• Contracts with suppliers for outsourced services
• Control processes within the methodology
• Approvals, sign-offs
• Minutes of relevant meetings
• Requirements and design meetings
• Actual deliverables
• Audit trails of sign-offs and approvals
• Project reporting, progress tracking, problem
  escalation procedures

IS Auditing Guideline, Document G23, ISACA
47
Implications for IS Audit Aspects to be reviewed
contd

• Change management
• Configuration management
• Data conversion/migration
• Documentation relating to in-project reviews
  including testing
• Reviews of earlier stages of the development
  process
• Earlier reviews of similar applications
• Relevant legal, regulatory and policy aspects

IS Auditing Guideline, Document G23, ISACA
48
Implications for IS Audit Methodology Usage

• IS Auditors should be aware that methodology use
  is low
• Non-existence of a formal methodology may prompt
  IS auditors to suggest a formal methodology
• Development teams may resist the imposition of a
  methodology
• Developers view the impact of a formalized
  methodology as a negligible factor in the success
  of a development project (Fitzgerald, 1998)
• Methodologies are at odds with preferred ad-hoc
  development practices (Raghaven Chand, 1989)
• Appropriate change management strategies should
  be followed when implementing a new methodology
• Communicate
• Train
• Motivate
• Provide incentives for use

49
Implications for IS Audit Methodology Tailoring

• IS auditors should understand that purposeful
  methodology tailoring is desirable
• Methodology tailoring can lead to a more
  efficient development process
• IS auditors should make sure that general
  guidelines for development are followed
• Deviations from the formal methodology may be
  necessary given the development context
• Methodology tailoring should be done by senior
  team members
• All methodology adaptations should be documented
• Adequate rationale for each adaptation should
  exist

50
Implications for IS Audit Career Usefulness

• Quest for career advancement could lead
  developers to recommend a less than optimal
  methodology
• Recommend an Agile methodology, for example, over
  a more effective traditional methodology
• IS auditors should enquire as to the rationale
  for the methodology choice
• Methodology choice should be documented

51
Implications for IS Audit Support Usefulness

• Using a methodology to ease lack of self
  confidence can lead to goal displacement
• IS auditors should be wary of goal displacement
• Blindly following a methodology can result in
  inefficiencies during the development process

52
Implications for IS Audit Symbolic Usefulness

• Methodology may be used superficially
• Hinder the full potential of the methodology
• IS auditors typically take a sample of documents
  when auditing the use of the SDLC (Singleton,
  2007 SDLC checklist, IT Control Objectives for
  SOX, 2002)
• Data flow diagrams and a systems testing plan,
  for example
• Documentation of a methodology does not mean that
  the methodology is used appropriately
• Documentation can be retrofitted to give
  appearance methodology was used (Fitzgerald,
  1994)
• Design documentation, for example, can be
  retrofitted
• Employ additional data-gathering techniques
• Inquiry
• Interview members of the organization outside the
  development team
• Observation

53
Implications for IS Audit Defense Usefulness

• Using a methodology for its defensive benefits
  may result in a relative disadvantage to users
• For example, developers may use a methodology to
  try to block requirements changes (Wastell, 1996)
  
• Interview members of the organization outside the
  development team
• Ensure that the methodology was not used to block
  requirements changes

54
Concluding Remarks

• Methodologies are important internal controls
• Methodology usage tends to be low
• Methodologies are often tailored to the
  development context
• Rigorous adherence to a methodology can be
  inefficient
• Methodologies are often used for political
  reasons
• Purpose is to call attention to these issues, not
  indict developers
• Many developers recognize the importance of
  methodologies and used them in an informed manner
  (Fitzgerald, 1997)

55
Future Research

• Evaluating Extreme Programming from an IS audit
  perspective
• Pros
• Pair Programming
• Prototyping
• Test driven development
• Prioritization of user requirements
• Cons
• Sparse documentation
• What is your opinion of XP from an IS audit
  perspective?

56
Thank you!