howprogrammingworks / benchmark

Performance testing for different techniques

Измерение производительности кода и оптимизация

===============

Оптимизация

• Объекты одинаковой структуры
• Функции одинаковой формы (мономорфные)
• Массивы одинаковых типов

Замер скорости

• Желательно проверять крупные алгоритмы или целые блоки
• Одинаковые условия исполнения для каждой итерации теста
• Единица оптимизации - функция, оборачиваем тестируемый код в функцию
• Сохраняем результаты (вводим оптимизатор в заблуждение)
• Прогрев кода перед тестом (дождаться стабилизации)
• Может получиться инлайнинг функций (можно проверить руками)
• Чтоб не инлайнило, например (можно добавить try {})
• Принимаем во внимание, что константы подставляются в выражения
• Выражения могут быть предварительно вычислены (не на каждой итерации)
• Вынесение инвариантов цикла (в цикле остается только то, на что влияет цикл)
• Мертвый код (который мог бы быть выброшен оптимизатором)
• Внетренние функции v8 --allow-natives-syntax
• Отключение автоматического gc --nouse-idle-notification
• Ручной gc --expose-gc
• Выделение больше памяти --max_old_space_size=2048

benchmark's Issues

variable iterations in Benchmark

@tshemsedinov Hi, What is it used for?

Benchmark/JavaScript/2-benchmark.js

Line 66 in 500596a

const iterations = result.length - PRE_COUNT;

Compare collection/set includes/set

Compare switch, { key: true }.hasOwnProperty(key) and new Set()

Statistical significance of benchmark results

In order for us to be able to make any decisions based on the results of benchmarks, they must be statistically significant. There may be options depending on what a benchmark tries to test, but I'll try to describe the common approach here.

First of all, separate the collection of raw data and the analysis of it. These are completely different things.

What you'll probably want to do to collect the data is to write a generic benchmarking framework that will warm up V8 (hint: it may be done faster using %OptimizeFunctionOnNextCall()), trigger a sensible amount of a benchmarked function runs and calculate the average count of operations per second. It should be possible to run such a script on its own and see the result so it is feasible to make some assumptions about the code immediately without detailed analysis. If we really want to compare the results of two benchmarks and they don't differ in several orders of magnitude, we have no right to do this unless some statistics is involved. In this case the whole process must be repeated some 100 times or so (maybe less, idk, worth playing with it) so that the benchmark runs for a fair amount of time (20–30 minutes would be okay) and writes a series of results using CSV, JSON or other machine-readable format. Then the next part is being involved, the most interesting one for mathematically-minded people out there.

The approach to comparison of results depends on the nature of these benchmarks and their output. If, e.g., we have two series of "ops/s" results from two comparable (i.e., testing the performance of two ways of solving a single problem, like comparing ES2015+ features with their ES5 counterparts) benchmarks, and we want to know which one is faster and how much, it is appropriate to use 2-sample independent Student's t-test with null hypothesis that performance is the same. There's also additional neat stuff we can do with raw data, like plotting graphs and charts. I'd recommend using R or Python with scipy for all of these, but if one wants to do the whole thing in JavaScript and Node, it'll be a great project on its own.

Object instantiation benchmark tests

Prepare execution scripts and run tests with different parameters:

• iteration number (1.000-10.000.000)
• different execution environments:
  • Browsers: Chrome, Firefox
  • Node.js 4.x, 5.x, 6.x, 7.x (a few of them)
  • OS: Fedora, CentOS, Win, FreeBSD, MacOS, etc. (a few of them)
• save results to CSV file and make charts:
  • x (iteration number), y (time)
  • x (iteration number), y (memory allocation)
    That's your task @NikitaRodan

Benchmark Requests

Please propose new benchmarks requests and methods/tools for it as a separate issues.
Everything we need to make decisions, for example:

• We may refactor JSTP and Impress to use classes or leave prototypes
• We may use Map or Set to calculate uniqueness in GlobalStorage
• We may use string concatenation or template literals with function tagging
  etc.
  And it will be good practice for ours young colleagues from 2nd course.