M2R Performance Evaluation
Table of Contents
Sitemap
Performance Evaluation
General Informations
The coordinator for these lectures is Jean-Marc Vincent . The lecturers are Jean-Marc Vincent and Arnaud Legrand .
Lectures take place generally on Monday afternoon generally in PLURIEL 134 (but also twice in H103) from 13:45 to 15:45.
In doubt, the planning with lecture rooms is available here (login/password is voirIMATEL/imatel).
Objectives
The aim of this course is to provide the fundamental basis for performance evaluation of computer systems. Two approaches are developed:
- performance measurement: based on experimental platforms (benchmarks or owner instrumented code execution), how to analyze data and synthesize performance indexes
- performance modeling: from a description of resources and the behavior of applications, how to predict the performance of the application
Here are links to the previous editions of this lecture: 2011-2012, 2012-2013.
Program and expected schedule
23 September 2013 (13:45 - 16:45): Arnaud Legrand and Jean-Marc Vincent
Data presentation, reporting results. Introduction to visualizing data with R and reporting results.Documents:
slides (EP_01_Introduction.pdf and Looking at your data: use R)References:
- R. Jain, The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling Wiley- Interscience, New York, NY, April 1991.
- Brief introduction to visualisation using R, ggplot, knitr/Rmd, Rstudio.
- http://www.rpubs.com/
- http://www.rstudio.com/
- http://www-958.ibm.com/software/analytics/manyeyes/
- http://www.r-bloggers.com/
- Checklist for good graphics, Checklist for good graphics (table version).
To do for the next time
: Present experimental results on quicksort. The archive with all the source files is available here and the resulting caricatured analysis is available on rpubs.
11 October 2013 (13:45 - 15:45): Arnaud Legrand
- We reviewed together the Rpubs document I had submitted as a
naive analysis. Here is the original document on which the
analysis was run, in case it gets updated. Here is a short
summary of what we said about this poor analysis:
- You used a "Dell Latitude 6430u with 16Gb of RAM".
- Unless the size of the workload is huge, the amount of RAM is not really useful.
- You meant GB, right ?
- What kind of processor is there in this machine ? How many cores
? Looking at "/proc/cpuinfo" would indicate that you used an
"
Intel(R) Core(TM) i7-3687U CPU @ 2.10GHz
" and that you have 4 cores. Looking at "/sys/devices/system/node/node0/cpu*/online
" would indicate you that Hyperthreading is actually activated. Ideally, you should use hwloc to obtain all such information as well as how much cache there are. Note that some of you performed the experiment on a single machine but were so convinced that there would be a gain that they claimed there was one. - What was the frequency used ? You should make sure that your cpu
was in the right frequency state e.g., using "
cpufreq-info
" and "cpufreq-set
". In my case, the machine was on battery so the governor was powersave with the minimal frequency (i.e., 754MHz)… - Did you activate compiling optimizations ?
- There are no units on the labels…
- Do you think 10 measurements is really enough ?
- For small arrays, the time taken is around a dozen of micro-seconds. Are your timing functions accurate enough ?
- Why did you plot the mean time ? Couldn't you plot also all the values to get a feeling of variability. If you aim at comparing average response time, maybe confidence intervals would be a good option ?
- It is actually not possible to reproduce your analysis as the measurements.csv file is not available. Ideally, you would have either inlined it in the Rmd file or made it available on the web. This would allow others to try other analysis and plots with the same dataset. Note that many of you explained they changed the code but did not bother to provide the modifications…
- Wouldn't it be interesting to make measurements for larger data sets as well ? Just to see up to how much you manage to gain. Indeed, since this is a log-log scale and complexity should be n.log(n) the slope of the parallel plot is likely to increase.
- You claim that activating parallelism is beneficial as soon as tables comprise 5E05 elements. However this value was determined graphically because you connected your estimations of the expected execution time with lines. Additional measurements should be done around the announced value to check whether this is true or not.
- You used a "Dell Latitude 6430u with 16Gb of RAM".
- Measurement on computer systems (benchmarking, observation,
tracing, monitoring, profiling).
Documents:
slides EP_02_measurements.pdfGoal:
become fully aware of variability, the need to replicate, experimental setup (compiling, machine, input workload), timing resolution, warmup, randomization, …To do for the next time
: Use what you learnt to improve previous work! :) Try to assess the importance of gcc optimization, of loop unrolling, of input workload, … To this end, here is a new version of the code and the corresponding Rpubs document.
- We reviewed together the Rpubs document I had submitted as a
naive analysis. Here is the original document on which the
analysis was run, in case it gets updated. Here is a short
summary of what we said about this poor analysis:
14 October 2013 (13:45 - 15:45): Arnaud Legrand
Esperance and variance estimation, confidence interval, distribution comparison. Checking whether the hypothesis apply or not. Illustration with R.Documents:
slides EP_03_confidence_interval.pdfGoal:
understand how to deal with randomnessTo do for the next time
: Use what you learnt to improve previous work! :) Make sure you do not run into issues such as:- Crappy data
- Inadequate data
- Temporal dependencies
21 October 2013 (13:45 - 15:45): Arnaud Legrand
Using simulation/emulation/real experiments to study large scale distributed systems.Documents:
slides EP_04_simulation_validation.pdf, A summary of Scientific Method by P. Kosso.Goals:
- Every experiment relies on a given model that should be validated/invalidated.
4 November 2013 (13:45 - 15:45): Jean-Marc Vincent
Linear RegressionDocuments:
slides, a sample org-file which comprises a factorial design, the experiment runner and the analysis (I named it as a txt file so that it is not interpreted by my website export mechanism). This is by no mean a perfect example but merely an illustration of how I have proceeded recently.Goals:
- Understand the strengths and limitations of the linear model
To do for the next time
: Check whether Amdahl's law hold true on the Hubble data set. Can you detect packages that behave in a strange way.
18 November 2013 (13:45 - 15:45): Jean-Marc Vincent
Statistical ModelingDocuments:
slides
9 December 2013 (13:45 - 15:45): Jean-Marc Vincent
Design of ExperimentsDocuments:
slidesGoals:
- Understand the difference between observationnal and experimental data.
- Fishbone
- Factorial designs
- Fractional designs
- Saturated designs for screening
To do for the next time
: Use what you learnt to improve previous work! Propose an experimental design.
Using R
Installing R and Rstudio
Here is how to proceed on debian-based distributions:
sudo apt-get install r-base r-cran-ggplot2 r-cran-reshape
Rstudio and knitr are unfortunately not packaged within debian so the easiest is to download the corresponding debian package on the Rstudio webpage and then to install it manually (depending on when you do this, you can obviously change the version number).
wget http://download1.rstudio.org/rstudio-0.97.551-amd64.deb sudo dpkg -i rstudio-0.97.551-amd64.deb sudo apt-get -f install # to fix possibly missing dependencies
You will also need to install knitr. To this end, you should simply run R (or Rstudio) and use the following command.
install.packages("knitr")
If r-cran-ggplot2
or r-cran-reshape
could not be installed for
some reason, you can also install it through R by doing:
install.packages("ggplot2") install.packages("reshape")
Producing documents
The easiest way to go is probably to use R+Markdown (Rmd files) in Rstudio and to export them via Rpubs to make available whatever you want.
We can roughly distinguish between three kinds of documents:
- Lab notebook (with everything you try and that is meant mainly for yourself)
- Experimental report (selected results and explanations with enough details to discuss with your advisor)
- Result description (rather short with only the main point and, which could be embedded in an article)
We expect you to provide us the last two ones and to make them publicly available so as to allow others to comment on them.
Documentation
For a quick start, you may want to look at R for Beginners (French version). A probably more entertaining way to go is to follow a good online lecture providing an introduction to R and to data analysis such as this one: https://www.coursera.org/course/compdata
Bibliography
- R. Jain, The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling, Wiley- Interscience, New York, NY, April 1991.
- Jean-Yves Le Boudec. Methods, practice and theory for the performance evaluation of computer and communication systems, 2006. EPFL electronic book.
- David J. Lilja, Measuring Computer Performance: A Practitioner’s Guide, Cambridge University Press 2005
- R. Nelson, Probability stochastic processes and queuing theory: the mathematics of computer performance modeling. Springer Verlag 1995