"Methodology" of research in computer science

1
"Methodology"ofresearch in computer science
Lionel Brunie National Institute of Applied
Sciences (INSA) LIRIS Laboratory/DRIM Team UMR
CNRS 5205 Lyon, France http//liris.cnrs.fr/lione
l.brunie
2
Agenda

• (Disclaimer)
• Discovery vs. invention
• What is a research result in computer science?
• What is a research paper in computer science?
  General structure(s) of a research paper in CS
• Searching for black boxes, traps and black holes
• How is research evaluated?
• And now?

3
Agenda

• ? Disclaimer
• Discovery vs. invention
• What is a research result in computer science?
• What is a research paper in computer science?
  General structure(s) of a research paper in CS
• Searching for black boxes, traps and black holes
• How is research evaluated?
• And now?

3
4
Disclaimer

• This is NOT a lecture !
• I apologize not to be an epistemologist
• I am just a practitioner of computer science that
  have co-authored some research papers and
  read/evaluated several hundreds ones
• This open discussion is absolutely NOT objective
• Goal open the windows and discuss how we
  ACTUALLY work (and not how we should work)
• All questions, remarks, contradictions are
  welcome!!!

5
Agenda

• Disclaimer
• ? Discovery vs. invention
• What is a research result in computer science?
• What is a research paper in computer science ?
  General structure(s) of a research paper in CS
• Searching for black boxes, traps and black holes
• How is research evaluated?
• And now?

6
Discovery vs. Invention

• My feeling is that there are two main ways of
  practicing science discovery vs. invention
• Biologists, physicists, chemists, researchers in
  psychology are discoverers
• (at least, software) Computer scientists,
  researchers in nanotech, researchers in process
  engineering or in industrial engineering are
  inventors
• However inventing often needs a priori
  discoveries (e.g., of your target/domain of
  interest) discovering often needs inventing
  (e.g., measuring instruments)
• Mathematicians are mathematicians -)

7
Discovering (just two words)

• Understanding the world what are atoms
  constituted of, why a disease in inherited, why
  do people have dreams, etc.
• Understanding means first asking questions, then
  observing, inquiring, modeling, evaluating
• At the end of the research process, one has an
  answer to the initial question (question of
  research)
• An answer is the most often not definitive. It is
  an explanation of a small piece of the natural
  world under some hypotheses
• Famous recent example the proof of the
  existence of Higgs boson at CERN/LHC

8
Inventing (1/2)

• Software computer science produces inventions
• Computers do not exist by themselves. They have
  been created by human beings gt there is nothing
  to discover in a computer or in a software
• The objective of research in CS is just to make
  computers/computer networks and computer
  applications more efficient, more easy to use,
  more reliable, more powerful i.e. more
  useable/useful

9
Inventing (2/2)

• As a consequence, a research result in (software)
  CS has no (real) intrinsic value. It has only the
  value that the research community and/or the
  society gives to it. A useless invention has no
  value!
• This has to be moderated when one considers
  research in fundamental and theoretical CS where
  people study formal problems gt context actually
  close to research in maths
• This has also to be moderated when one considers
  social- or bio- or something-inspired CS where
  researchers attempt to understand and model an
  external (social or biological or) phenomenon

10
Agenda

• Disclaimer
• Discovery vs. invention
• ? What is a research result in computer science?
• What is a research paper in computer science?
  General structure(s) of a research paper in CS
• Searching for black boxes, traps and black holes
• How is research evaluated?
• And now?

11
A difficult answer

• The endless debate about fundamental research,
  basic research, applied research, finalized
  research
• A first and definitive answer a result has some
  scientific value as soon as it has been accepted
  by a scientific publication (a scientific
  committee), i.e., research is a social activity
  (hence, fashions, trends)
• A second answer a result has some scientific
  value if it interests people and if it is novel
  (original) (and if it has some genericity) i.e.,
  refer to the scientific method
• The most difficult (and so the most interesting)
  problems have often been raised by applications,
  e.g. medical applications. Investigating these
  problems have allowed going through theoretical
  scientific developments

12
Agenda

• Disclaimer
• Discovery vs. invention
• What is a research result in computer science?
• ? What is a research paper in computer science?
  General structure(s) of a research paper in CS
• Searching for black boxes, traps and black holes
• How is research evaluated?
• And now?

13
Structure of a research paper in CS

• Introduction-Motivations
• State of the art
• Proposal
• Experiments
• Discussion
• Conclusion and future work
• gt Computing the D (or d or e)!

14
3 (1) key sections (1/4)

• Having 1 very good idea in a scientists life is
  very good having two is exceptional
• One paper gt one problem and at most one (two)
  contribution(s) to defend
• A first key section the statement of the problem
• Why is the problem important? Remark notion of
  fashion (e.g., Big Data)
• Why is the problem difficult (? link to the state
  of the art)?
• What are the hypotheses? What is/are the
  context/conditions?
• What are the objectives i.e., what are the
  performance/quality criterions with respect to
  which the contribution should be evaluated?
• Then, add the outline of the contribution

15
3 (1) key sections (2/4)

• A second key section the state of the art
• A state of the art must not be flat or general
  purpose
• It must be focused on your scientific and
  applicative targets and critical and
  compared/taxonomic
• Critical the contribution/limitations of each
  cited paper should be analyzed wrt your problem
• Compared papers should be organized in
  taxonomies
• The state of the art establishes/describes the
  scientific basis wrt which your work should be
  compared, i.e., your ?/?/? will be evaluated
• Ideally, all pertinent papers should be analyzed.
  Practically, it is often untracktable
• Making the bibliography is a key component of
  the scientific method

16
3 (1) key sections (3/4)

• A third key section the discussion
• Goal of the discussion analyzing your
  contribution/proposal
• Wrt to your initial objective/motivation
• Wrt to the state of the art
• Wrt the experiments you have run and have not
  run
• Wrt the implementation choices you have made
• Wrt genericity
• A discussion is not a conclusion
• The discussion is often distributed over the
  paper ( -( ?)
• Do not believe authors make your personal
  discussion!

17
3 (1) key sections (4/4)

• A fourth obvious section your contribution
• Must be sound
• Must be comprehensive and complete
• Should allow people to reproduce the experiments
  you use
• Must be self-contained

18
A short focus on theIntroduction Motivation
section

• Inventing some thing that interests nobody has
  no real sense
• Inventing some thing that addresses a problem
  that has an existing satisfactory solution has a
  limited interest
• To be interesting, a work/paper should address a
  problem
• With a high (scientific or economic or social
  or) value
• And no solution
• (already noted) A paper should list the criteria
  wrt which authors consider that their proposal is
  to be evaluated

19
Experiments (1/3)

• The often dark side of research in CS
• A first reason cultural/educational computer
  scientists have an unsatisfactory education in
  statistics/experiment planning/performance
  evaluation
• However, things become better -)

20
Experiments (2/3)

• A second reason real life experiments are often
  intractable because comparing two inventions is
  often intractable
• Concurrent software are often not available
• Concurrent complex systems/softwares cannot be
  re-develop from scratch at a reasonable price
• Evaluation criteria are different
• Users of your software are reluctant to
  spend/waste their time on alpha versions
• It often exists no benchmarks
• Research works often address very small
  range/specific issues that are part of a complex
  systems but integration is often not realized
• Research teams have neither the manpower nor the
  competence to develop products and limit their
  development to demonstrator or prototypes that
  cannot be tested by real end users

21
Experiments (3/3)

• However experiments are mandatory to evaluate the
  pertinence of your work
• Experiments should allow
• To evaluate the performance/pertinence/efficiency
  of your proposal wrt your initial objective and
  wrt all the possible contexts (not only in the
  best case)
• To evaluate the influence of the various
  parameters on the performance
• To compare your proposal wrt the state of the art
• Experimental conditions must be made completely
  explicit

22
Agenda

• Disclaimer
• Discovery vs. invention
• What is a research result in computer science?
• What is a research paper in computer science?
  General structure(s) of a research paper in CS
• ? Searching for black boxes, traps and black
  holes
• How is research evaluated?
• And now?

23
Reading a paper as an investigation

• A key behavior criticism (cf. methodical doubt
  by Descartes)
• A first obvious step analyze what is written
• is the proposal scientifically sound?
• are the performances good?
• is the problem really important?
• is the state of the art exhaustive?
• etc.
• A second step analyze what is not written

24
Black boxes and assumptions

• Computer systems/applications/software are too
  complex/too large so that a single team/paper can
  address the entire problem
• gt Focus on a small part or specific issue
• gt Assumptions on the other parts
• inputs it exists/will exist (soon) some
  system can can provide your prototype with the
  data/information it needs
• outputs the data produced by your prototype
  could be/will (soon) be used by the end-user side
  of the global system
• Assumptions are not always explicit
• Always ask yourself if these assumptions are
  valid/realistic, what do they imply

25
Analyzing experiments (1/3)

• The border line
• Science definitely condemns treachery and
  falsification modifying the result of an
  experiment is beyond the red line
• Experimental measurements that are presented in a
  paper are (supposed to be) true (I personally
  have always supposed it)
• The question is about omitted/forgotten results
• The question is about uncertainty and statistical
  errors on the measurements
• The question is about some generalization of
  results/experiments
• From a scientific point of view, a negative
  result is often as interesting as a positive
  result
• However, as CS has to deal with invention, a
  negative result is usually considered as a
  depreciated fact

26
Analyzing experiments (2/3)

• So the temptation to avoid including negative
  measurements often exists
• This measurement is not really important but as
  I do not have enough space to explain it, it is
  more important, from a scientific point of view
  (of course), to discuss in deep other more
  important experiments
• The situation that leads to this bad result
  never happens in the real life
• With a slight modification of the prototype, it
  is obvious that one would never get such a
  result, so let us focus on the results concerning
  the core of the prototype
• etc.
• are reasons that a scientist may use to
  self-persuade him that presenting this experiment
  is not pertinent

27
Analyzing experiments (3/3)

• Where is the yellow line? Where is the red line?
• Always
• analyze if experiments are exhaustive
• If not, always analyze if the missing experiments
  are important to evaluate the performance of the
  prototype
• and try to infer what could be the behavior of
  the prototype wrt these experiments

28
Agenda

• Disclaimer
• Discovery vs. invention
• What is a research result in computer science?
• What is a research paper in computer science?
  General structure(s) of a research paper in CS
• Searching for black boxes, traps and black holes
• ? How is a research paper evaluated?
• And now?

29
Evaluation of a research paper

• Is the topic of the paper in the domain of
  interest of the conference/journal?
• (Is the bibliography complete, critical and
  taxonomic?)
• Is the contribution original?
• Is the paper technically sound/correct?
• Is the paper easy to read?
• Is the contribution significant?
• What is the level of expertise of the evaluator /
  what is the confidence of the evaluator in his
  judgement
• Discussion within the program or scientific
  committee

30
Menu

• Disclaimer
• Discovery vs. invention
• What is a research result in computer science?
• What is a research paper in computer science?
  General structure(s) of a research paper in CS
• Searching for black boxes, traps and black holes
• How is research evaluated?
• ? And now?

31
Conclusion?

• Read, read, read, read!
• Be positively critical
• Never admit supposed evidences. Always doubt
  (Descartes)
• All questions are good
• Be modest

32
Bring your creativity!