Experiences Gained from Experimentation in Software Courses at the University of Maryland Vic Basili University of Maryland

1
Experiences Gained from Experimentation in
Software Coursesat the University of
MarylandVic Basili University of Maryland
2
Issues

• Courses available for experimentation
• Benefits and Issues with classroom experiments
• Training in experimentation

3
Software Engineering Courses

• There are two major software engineering classes
• CMSC 435 - A software engineering project course
  taught by various people in various ways
  (graduate/undergraduate credit available)
• CMSC 735 A software engineering graduate course
  in modeling measurement, and process improvement
• And one HCI course
• CSMC 435 A course in Computer Human Interfaces
  that also teaches and uses experimental methods
  as part of the curriculum.
• There are no research methods courses in the CS
  department, nor is there likely to be in the near
  future
• Many other courses have projects but they are
  not used to run experiments

4
CMSC 435

• Students work in teams (3 or more) to use the
  technology to build some kind of software
  product
• Have been running experiments since 1975,
  beginning with Bob Rieters thesis
• Early methodology analysis (top down design,
  chief programmer teams, ) to specific technology
  experiments (reading vs. testing, requirements
  reading, OO reading)
• Experiment team is usually two to five people,
  of various levels of experience, interacting and
  reviewing the design and implementation, and one
  or two people analyzing the results.

5
CMSC 735

• Course is directly about model building and
  measurement and learning and experimentation
• Graduate students work on building models rather
  than software systems
• Run experiments here also but mostly in the small
  
• study effects of technique,
• play with experimental methods (observation
  protocols, grounded theory, )

6
Running Classroom Experiments

• Goals for experimenting with the class
• Teaches students in the class
• The study of SE follows an experimental paradigm
• The concepts behind experimentation
• That we do not know the effects of a process and
  the effects of context variables and are
  learning
• How to learn more about the reffects
• Gives graduate students a chance to run an
  experiment
• Gives experimental software engineers
• a set of data points for analysis for mostly
  novice subjects
• a chance to debug protocols and improve the
  process definition
• More confidence and credibility when running of a
  similar experiment with a company

7
Running Classroom Experiments

• Problems with classroom experiments
• Limiting the context of the study
• Selecting process that is relevant to both
  students and professionals
• Dealing with the learning curve
• Dealing with student/classroom conflicts
• Subject experience
• Multiple motivations
• Limited time frame, nature of the problem
• Design options
• Cheating

8
Running Classroom Experiments

• Controlled experiments have limits when done in
  classroom or training situations
• You must train everyone (effects the control
  group issue)
• You cannot teach a new technique and not expect
  people to use it, especially when it is in
  comparison to a non-technique
• You face a variety of threats to validity
• There is high risk spend the time and do not
  get sufficient information/analysis
• This effects the design
• Consider the PBR experimental style design
  (partial factorial) as opposed to the reading vs.
  testing design
• You need to supplement the approach with other
  evaluation techniques
• e.g., a mix of quantitative and qualitative
  analysis, observations, grounded theory

9
Teaching CMSC735

• Besides models and metrics, cover aspects of the
  experimental discipline
• Experimental Paradigms
• Scientific, engineering, empirical, analytic
  methods
• Experimental Classifications
• Level of variable relationship
• Experience of Subjects
• Experimental Setting
• Type of Study
• Types of Analysis
• Factors Jeopardizing Validity
• internal validity, external validity
• Building laboratory Manuals
• Picking a dissertation topic in SE

10
The Experimental Discipline

• Experimental Classifications
• Level of variable relationship
• Descriptive, Correlational, Cause-effect
• Experience of Subjects
• Expert -gt Novice
• Experimental Setting
• in vivo, in vitro
• Type of Study
• experimental (controlled experiment,
  quasi-experimental design, case study, field
  study
• Types of Analysis
• Quantitative Analysis, Qualitative Analysis

11
Writing an Experimental SE DissertationDefinition
s

• RESEARCH
• Diligent search or inquiry scientific
  investigation and study to discover facts.
• SCIENCE
• Systematic knowledge of natural or physical
  phenomena
• Facts ascertained by observation, experiment, and
  introduction
• Ordered arrangement of facts known under classes
  or heads
• Theoretical knowledge as distinguished from
  practical
• Knowledge of principle and rules of invention,
  construction, mechanism, etc.,
• As distinguished from art.
• THEORY/MODEL
• A system for explaining a set of phenomena by
  specifying constructs and the laws that relate
  these constructs to each other.

12
Writing an Experimental SE DissertationDefinition
s

• Fact information obtained through direct
  observation
• Hypothesis educated guess, precedes an
  experiment
• Experiment operation carried out (sometimes
  under controlled conditions) to discover unknown
  effect/law, test/establish hypothesis, illustrate
  a known law
• Theory possible explanation based upon many
  facts/reason
• Law description/observation of behavior used for
  prediction based upon facts and reason
• Model simplified representation of a
  system/phenomenon
• can be a theory or a law
• Paradigm conceptual filter, how we
  perceive/interpret
• Truth what really is

13
RESEARCH APPROACHES

• ESTABLISHED FIELD
• Easier to answer questions
• Areas better defined
• More consensus on the importance of an area
• Standard methods of study
• METHODOLOGICAL APPROACHES
• ANALYSIS
• Build a theory
• Derive properties
• Show boundary conditions and limits
• EXPERIMENTATION
• Formulate hypotheses
• Deduce empirical consequences
• Test the hypotheses by collecting data

14
QUESTIONS for EVALUATING RESEARCH

• IS THERE NEW KNOWLEDGE?
• Were the methods used to obtain the knowledge
  scientifically sound?
• ARE THE RESULTS SIGNIFICANT?
• Do they improve our ability to describe,
  predict, control or explain?
• PICKING A TOPIC
• Build on prior theories
• Fill in gaps in theories
• Create new theories that explain better than old
• Disprove a commonly held proven theory
• CHARACTERISTICS
• Can be neatly packaged
• Focused
• Consistent methodology

15
QUESTIONS for EVALUATING RESEARCH

• What would make the following dissertation
  research?
• Building a descriptive model/theory
• Building a predictive model/theory
• Improving an existing model/theory
• Verifying properties of a model/theory
• Implementing/automating a model/theory

16
THINKING ABOUT THE RESEARCH PROCESS THEORY AND
RESEARCH PERSPECTIVES
17
THINKING ABOUT THE RESEARCH PROCESS PROCEDURAL
PERSEPECTIVE
18
SAMPLE EXPERIEMTNAL DISSERTATION

• CHAPTER 1 Introduction
• A. General statement of the problem
• B. Statement of the hypotheses, objectives, or
  questions
• C. Definitions of terms (assumptions/limitations/
  significance)
• CHAPTER 2 Review of the Literature
• A. Review of previous research
• B. Pertinent opinion
• C. Summary of the state-of-the-art (tie it all
  together)
• CHAPTER 3 Method
• A. Description of the subjects (how chosen)
• B. Research design and procedures
• (overview of statistical procedures)
• C. Description of measures employed

19
SAMPLE EXPERIMENTAL DISSERTATION

• CHAPTER 4 Findings
• A. Description of finding pertinent to each
  hypothesis, objective, or question
• B. Other findings
• CHAPTER 5 Summary and Discussion
• A. Summary of research problem, method, and
  finding
• B. Conclusions
• C. Implications
• D. Suggestions for further research