quasilyte / benchstat.el

Proper Emacs Lisp benchmarking made simple.

I worked on NoVerify static analyzer for PHP and KPHP compiler at vk.com. Before that, I was a part of Intel Go team: I added AVX-512 support and some compiler optimizations there, but most of the time I was tinkering the Go x86-64 assembler.

I try to maintain several^[1] good^[2] static^[3] analyzers^[4].
Sometimes I create dev tools like phpgrep (structural PHP code search using AST patterns).

Some facts about me:

• I do not approve the ways of Russia government (wars, crazy mass propaganda, censorship, ...) ⬜️🟦⬜️
• I enjoy 3D-printing and drawing
• I love video games and game development
• My tech talks and articles can be found at github.com/quasilyte/talks
• Co-host of the GolangKazan Go community
• You can call me Alex

benchstat.el's Issues

Consider `dotimes-with-progress-reporter`

Can try to use dotimes-with-progress-reporter instead of dotimes+with-temp-message.

Add to MELPA

This will make package installation easier.

If benchstat.el is accepted and merged, README should be updated to reflect that.

Issue will be closed when MELPA maintainers will make their final decision.

See melpa/melpa#5051.

Negative execution time

Although docs/troubleshooting.md does mention this issue and describes potential solutions, it does not reveal the underlying reason.

Looking at the code of benchmark-run-compiled, I think the problem is clear:

(defmacro benchmark-run (&optional repetitions &rest forms)
  "Time execution of FORMS.
If REPETITIONS is supplied as a number, run forms that many times,
accounting for the overhead of the resulting loop.  Otherwise run
FORMS once.
Return a list of the total elapsed time for execution, the number of
garbage collections that ran, and the time taken by garbage collection.
See also `benchmark-run-compiled'."
  (declare (indent 1) (debug t))
  (unless (natnump repetitions)
    (setq forms (cons repetitions forms)
	  repetitions 1))
  (let ((i (make-symbol "i"))
	(gcs (make-symbol "gcs"))
	(gc (make-symbol "gc")))
    `(let ((,gc gc-elapsed)
	   (,gcs gcs-done))
       (list ,(if (> repetitions 1)
		  ;; Take account of the loop overhead.
		  `(- (benchmark-elapse (dotimes (,i ,repetitions)
					  ,@forms))
		      (benchmark-elapse (dotimes (,i ,repetitions))))
		`(benchmark-elapse ,@forms))
	     (- gcs-done ,gcs)
	     (- gc-elapsed ,gc)))))

These lines attract the attention:

		  ;; Take account of the loop overhead.
		  `(- (benchmark-elapse (dotimes (,i ,repetitions)
					  ,@forms))
		      (benchmark-elapse (dotimes (,i ,repetitions))))

Seems like a bad way to account loop overhead as it can sometimes
result in negative values.

Ignoring negative results is a good start, but maybe the problem
can be remedied in a better way.
Warnings are quite annoying.

Take GC time into account

It needs some investigation, but it seems like benchmark-run does not return
total execution time in list head.

benchstat.el profiles currently ignore time spent in GC as third list value is ignored.

Before submitting a fix, I should be sure that simple addition is appropriate.

Note: documentation for benchmark-run return value is misleading.

Return a list of the total elapsed time for execution, the number of
garbage collections that ran, and the time taken by garbage collection.
See also ‘benchmark-run-compiled’.