bdrung / startup-time

Measure startup time of different programming languages

Startup time of different programming languages

The startup time of a programming language is important for short running programs that are called interactively by the user or are called (many times) by other programs. For example, the version control system Git is written in C and the invoked commands (like git status and git log) execute fast. The version control systems Bazaar and Mercurial are written in Python. They take much longer to execute than Git and they feel slow compared to Git. The startup time of Python contributes an important portion of the execution time.

Several years ago, I wrote a small script named distro-info which provides information about the Debian/Ubuntu releases. This script is called in a bash completion for pbuilder-dist (from the ubuntu-dev-tools package). An Ubuntu user experience a bash startup of 4 seconds with cold cache due to the multiple invocations of distro-info. This made me search a suitable programming language with a fast startup time and I wrote this man page to measure the startup time of many programming languages.

This project consists of many hello world programs in different languages and an ugly Makefile that compiles the programs and runs the benchmark. Each program is run 1000 times in a row (using a tiny run.c program to minimize the overhead for the invocation):

time -f "%e" taskset -c 0 ./run 1000 $program

Usage

Before running the benchmark, install all relevant compilers. On Debian/Ubuntu you can run make install to install the compilers. Then start the benchmark by calling make:

$ make
Run on: Intel(R) Core(TM) i5-2400S CPU @ 2.50GHz | Ubuntu 17.10 | 2018-02-10
C (gcc 7.2.0):            0.26 ms
C++ (g++ 7.2.0):          0.79 ms
Cython (cython 0.25.2):   9.91 ms
Cython3 (cython3 0.25.2): 26.04 ms
D (gdc 7.2.0):            0.57 ms
Go (go go1.8.3):          0.41 ms
Go_GCC (gccgo 7.2.0):     98.26 ms
Haskell (ghc 8.0.2):      0.72 ms
Pascal (fpc 3.0.2):       0.08 ms
Rust (rustc 1.21.0):      0.51 ms
Bash 4.4.12(1):           0.71 ms
CShell 20110502:          3.26 ms
Lua 5.2.4:                0.63 ms
Perl 5.26.0:              0.94 ms
PHP 7.1.11:               8.71 ms
Python-S 2.7.14:          2.91 ms
Python 2.7.14:            9.43 ms
Python3-S 3.6.3:          9.31 ms
Python3 3.6.3:            25.84 ms
PyPy 5.8.0:               27.53 ms
Ruby 2.3.3p222:           32.43 ms
Shell (dash 0.5.8):       0.33 ms
ZShell 5.2:               1.57 ms
C# (mcs 4.6.2.0):         13.37 ms
Java (javac 1.8.0_151):   54.55 ms
Scala (scalac 2.11.8):    310.81 ms

$ make
Run on: Raspberry Pi 3 (arm64) | Debian GNU/Linux buster/sid | 2018-02-10
C (gcc 7.2.0):            2.19 ms
C++ (g++ 7.2.0):          8.24 ms
Cython (cython 0.26.1):   98.71 ms
Cython3 (cython3 0.26.1): 196.36 ms
Go (go go1.9.3):          4.10 ms
Go_GCC (gccgo 7.2.0):     898.30 ms
Haskell (ghc 8.0.2):      9.44 ms
Pascal (fpc 3.0.4):       0.66 ms
Rust (rustc 1.22.1):      4.42 ms
Bash 4.4.12(1):           7.31 ms
CShell 20110502:          10.98 ms
Lua 5.2.4:                6.23 ms
Perl 5.26.1:              8.78 ms
PHP 7.2.2:                98.03 ms
Python-S 2.7.14+:         32.77 ms
Python 2.7.14+:           91.85 ms
Python3-S 3.6.4:          110.02 ms
Python3 3.6.4:            197.79 ms
PyPy 5.8.0:               183.50 ms
Ruby 2.3.6p384:           421.53 ms
Shell (dash 0.5.8):       2.81 ms
ZShell 5.4.2:             11.04 ms
C# (mcs 4.6.2.0):         137.53 ms
Java (javac 1.8.0_151):   566.66 ms
Scala (scalac 2.11.8):    2989.72 ms

If you don't want to test all programming languages, edit COMPILED_LANGS and INTERPRETED_LANGS in Makefile (and maybe edit the all target to not depend on the .exe and .class files). Note: The Makefile should be replaced by something else that is simpler to read and can produce nicer looking output.

Results

Language	version	Intel Core i5 2400S	Raspberry Pi 3
Pascal (fpc)	3.0.2 / 3.0.4	0.08 ms	0.66 ms
C (gcc)	7.2.0	0.26 ms	2.19 ms
Shell (dash)	0.5.8	0.33 ms	2.81 ms
Go (go)	1.8.3 / 1.9.3	0.41 ms	4.10 ms
Rust (rustc)	1.21.0 / 1.22.1	0.51 ms	4.42 ms
D (gdc)	7.2.0	0.57 ms
Lua	5.2.4	0.63 ms	6.23 ms
Bash	4.4.12(1)	0.71 ms	7.31 ms
C++ (g++)	7.2.0	0.79 ms	8.24 ms
Perl	5.26.0 / 5.26.1	0.94 ms	8.78 ms
Haskell (ghc)	8.0.2	0.72 ms	9.44 ms
ZShell	5.2 / 5.4.2	1.57 ms	11.04 ms
CShell	20110502	3.26 ms	10.98 ms
Python (with -S)	2.7.14	2.91 ms	32.77 ms
Python	2.7.14	9.43 ms	91.85 ms
PHP	7.1.11 / 7.2.2	8.71 ms	98.03 ms
Cython	0.25.2 / 0.26.1	9.91 ms	98.71 ms
Python3 (with -S)	3.6.3 / 3.6.4	9.31 ms	110.02 ms
C# (mcs)	4.6.2.0	13.37 ms	137.53 ms
PyPy	5.8.0	27.53 ms	183.50 ms
Cython3	0.25.2 / 0.26.1	26.04 ms	196.36 ms
Python3	3.6.3 / 3.6.4	25.84 ms	197.79 ms
Ruby	2.3.3p222 / 2.3.6p384	32.43 ms	421.53 ms
Java (javac)	1.8.0_151	54.55 ms	566.66 ms
Go (gccgo)	7.2.0	98.26 ms	898.30 ms
Scala (scalac)	2.11.8	310.81 ms	2989.72 ms

Evaluation

I categorize the programming languages in four categories: fast, okay, slow, takes ages.

Fast

Fast startup times are below 1 ms on my desktop (Intel Core i5 2400S) and below 10 ms on slow hardware (Raspberry Pi 3):

• Bash
• C
• C++
• D (gdc)
• Go (go)
• Haskell
• Lua
• Pascal
• Perl
• Rust
• Shell (dash)

Okay

Okayish startup times are between 1 and 5 ms on my desktop (Intel Core i5 2400S) and between 10 and 50 ms on slow hardware (Raspberry Pi 3):

• CShell
• Python 2 (with -S)
• ZShell

Slow

Slow startup times are between 5 and 50 ms on my desktop (Intel Core i5 2400S) and between 50 and 500 ms on slow hardware (Raspberry Pi 3):

• C# (mcs)
• Cython (Python 2)
• Cython3 (Python 3)
• PHP
• Python 2
• Python 3
• Python 3 (with -S)
• PyPy (Python 2)
• Ruby

Takes ages

Some programming languages take ages to start up. The startup times are above 50 ms on my desktop (Intel Core i5 2400S) and over 500 ms on slow hardware (Raspberry Pi 3):

• Java
• Go (gccgo)
• Scala

startup-time's Issues

run.c doesn't compile on OpenBSD

uname
OpenBSD

clang run.c -o run
run.c:36:5: error: use of undeclared identifier 'cpu_set_t'
    cpu_set_t cpu_set;
    ^
run.c:64:5: warning: implicit declaration of function 'CPU_ZERO' is invalid in C99 [-Wimplicit-function-declaration]
    CPU_ZERO(&cpu_set);
    ^
run.c:64:15: error: use of undeclared identifier 'cpu_set'; did you mean 'cgetset'?
    CPU_ZERO(&cpu_set);
              ^~~~~~~
              cgetset
/usr/include/stdlib.h:277:6: note: 'cgetset' declared here
int      cgetset(const char *);
         ^
run.c:65:5: warning: implicit declaration of function 'CPU_SET' is invalid in C99 [-Wimplicit-function-declaration]
    CPU_SET(core, &cpu_set);
    ^
run.c:65:20: error: use of undeclared identifier 'cpu_set'; did you mean 'cgetset'?
    CPU_SET(core, &cpu_set);
                   ^~~~~~~
                   cgetset
/usr/include/stdlib.h:277:6: note: 'cgetset' declared here
int      cgetset(const char *);
         ^
run.c:66:17: warning: implicit declaration of function 'sched_setaffinity' is invalid in C99 [-Wimplicit-function-declaration]
    if(unlikely(sched_setaffinity(getpid(), sizeof(cpu_set), &cpu_set) == -1)) {
                ^
run.c:66:52: error: use of undeclared identifier 'cpu_set'
    if(unlikely(sched_setaffinity(getpid(), sizeof(cpu_set), &cpu_set) == -1)) {
                                                   ^
run.c:66:63: error: use of undeclared identifier 'cpu_set'; did you mean 'cgetset'?
    if(unlikely(sched_setaffinity(getpid(), sizeof(cpu_set), &cpu_set) == -1)) {
                                                              ^~~~~~~
                                                              cgetset
run.c:28:39: note: expanded from macro 'unlikely'
#define unlikely(x) __builtin_expect((x),0)
                                      ^
/usr/include/stdlib.h:277:6: note: 'cgetset' declared here
int      cgetset(const char *);
         ^
3 warnings and 5 errors generated.

egcc run.c -o run
run.c: In function 'main':
run.c:36:5: error: unknown type name 'cpu_set_t'
   36 |     cpu_set_t cpu_set;
      |     ^~~~~~~~~
run.c:64:5: warning: implicit declaration of function 'CPU_ZERO' [-Wimplicit-function-declaration]
   64 |     CPU_ZERO(&cpu_set);
      |     ^~~~~~~~
run.c:65:5: warning: implicit declaration of function 'CPU_SET'; did you mean 'L_SET'? [-Wimplicit-function-declaration]
   65 |     CPU_SET(core, &cpu_set);
      |     ^~~~~~~
      |     L_SET
run.c:66:17: warning: implicit declaration of function 'sched_setaffinity' [-Wimplicit-function-declaration]
   66 |     if(unlikely(sched_setaffinity(getpid(), sizeof(cpu_set), &cpu_set) == -1)) {
      |                 ^~~~~~~~~~~~~~~~~
run.c:28:39: note: in definition of macro 'unlikely'
   28 | #define unlikely(x) __builtin_expect((x),0)
      |                                       ^

Ubuntu 16.04 - all results are "/bin/sh: 1: time: not found ms"

First run of this code, all the results are like this:

Run on: Intel(R) Xeon(R) CPU E5-2470 v2 @ 2.40GHz | Ubuntu 16.04.7 LTS | 2020-12-06
C (gcc 5.4.0):            /bin/sh: 1: time: not found ms
C++ (g++ 5.4.0):          /bin/sh: 1: time: not found ms
Cython (cython 0.23.4):   /bin/sh: 1: time: not found ms
... etc.

time is a shell keyword, requires: sudo apt-get install time to add /usr/bin/time binary.

Might need the Makefile changing TIME := /usr/bin/time -f "%e", I needed that to make it work, but if I take that away, it still works.

Logging so that other people can see a fix; might be good to add time as one of the things installed by make install, though I don't know how.

Adding JavaScript/Nodejs

@bdrung Could you add JavaScript?

I think JavaScript is the most popular programming language.

Thank you.

C code should use `puts`

The current code uses printf which allows for formatted/dynamic output, but I would expect it to use puts for verbatim output.

Add Ruby without RubyGems

Is there any interest in adding Ruby without RubyGems? I can make Pull Request. It’s starting Ruby with --disable=gems.

Adding an Assembly Version

Adding an assembly version as a base benchmark would be nice. I've attached a Assembly Version for x86 assembly, which is slightly better then gcc-musl.

.global _start

.text
_start:
    # Call Write
    movq $1, %rax 
    movq $1, %rdi 
    movq $msg, %rsi 
    movq $13, %rdx 
    syscall

    # Call sys_exit
    movq $60, %rax
    movq $0, %rdi
    syscall

.data
msg:
    .ascii "Hello World!\n"

Add Zig (https://github.com/ziglang/zig)