M2R Scientific Methodology and Performance Evaluation

The coordinator for these lectures is Jean-Marc Vincent . The lecturers are Jean-Marc Vincent and Arnaud Legrand .

Lectures take place on Thursday afternoon from 3:30PM to 5PM, generally in D211.

Here is a map of the campus for those who do not know where the rooms are.

The planning with lecture rooms is available at the ADE website (look for PDES then for Scientific Methodology).

The aim of this course is to provide the fundamental basis for sound scientific methodology of performance evaluation of computer systems. This lecture emphasize on methodological aspects of measurement and on the statistics needed to analyze computer systems. We first sensibilize the audience to the importance of experiment and analysis reproducibility in particular in computer science. Then we present tools that help answering the analysis problem and may also reveal useful for managing the experimental process through notebooks. The audience will be given the basis of probabilities and statistics required to develop sound experiment designs. Unlike some other lectures, our goal is not to provide analysis recipes that people can readily apply but to make the audience truly understand some simple statistical tools on which they can build further.

Here are links to the previous editions of this lecture: 2011-2012, 2012-2013, 2013-2014, 2014-2015, 2015-2016.

See https://github.com/alegrand/SMPE for details of the lectures and additional resources.

1. 29 September 2016: J-M. Vincent A bit of epistemology…
   • Homework: Read the presentation of Karl Popper Science: Conjectures and Refutations.
   • Question: This text has been written before Computer Science was funded. Explain in at most one page, how Popper's theory could apply in the context of Computer Science research now.
2. 6 October 2016: A. Legrand Methodological aspects of science: reproducible research, article vs. laboratory notebookn social aspects of science (peer review, communication, …)
   • Further readings: A Summary of Scientific Method, Peter Kosso, Springer
   • Homework:
     1. Install R and Rstudio to learn litterate programming and check how to create small reproducible analysis and publish them on rpubs with Rstudio.
     2. Setup your own laboratory notebook and start using it to collect information. Emacs org-mode is a great tool for this but if you're more confortable with something else, go for it. Just make sure it's plain text, backed up and easily shared with someone else…
     3. Select a topic on which you would like to apply the techniques we will present you.
3. 20 October 2016: A. Legrand Causality, dependence, correlation, covariance, observational vs. experimental data, observation → hypothesis → experiments, experiment design and begining of Descriptive descriptive statistics of univariate data (part 3).
4. 3 November 2016: A. Legrand Wrap-up on correlation/causation and end of Descriptive statistics of univariate data.
   • Investigating some univariate datasets.
5. 10 November 2016: Arnaud Legrand Descriptive statistics and data visualization
6. 17 November 2016: J.M. Vincent Modeling, basis of probabilities, estimation (confidence and bias)
7. 24 November 2016: J.M. Vincent Checking hypothesis, risk and test.
8. 1 December 2016: J.M. Vincent The linear model and testing adequation
9. 8 December 2016: A. Legrand Optimizing experiment design
10. 15 December 2016: A. Legrand Optimizing experiment design (continuation)
11. 16 January 2016: A. Legrand and J.M. Vincent (from 2PM to 5PM in room D211) Reviewing and brainstorming about some of your own experimental process (see the corresponding pad) and reviewing notions that may not be clear.
12. 16 January 2016: A. Legrand and J.M. Vincent See the line above.

As explained during the lectures, now that we have covered the whole spectrum of experimental study (preliminary analysis and visualization, modeling, experiment planning, corresponding analysis), I want you to put this into practice. The best way for this is to work in small groups, pick a topic of your choice and just report your activity and possibly conclude. The topic and the conclusions you will reach are of little importance. What's important is that you do it in a clean way and allow others to look into your work.

Here is a pad (https://pad.inria.fr/p/5wdFzCQFJ39Rk00v) where you will indicate for each group:

• the names of the members
• the topic in a few words
• and the URL where we can access your work (github, rstudio, …).

The deadline is set to January 19th so that we have enough time to read your work. Note that I have added in this pad a few links that some students returned me a few weeks ago. This is what they did in 5-6 hours max so you should have way enough time to do something pretty clean.