akihirosuda / buildbench

benchmark tool for Docker, BuildKit, img, Buildah, and Kaniko

Buildbench: benchmark tool for Docker, BuildKit, img, Buildah, and Kaniko

Requirements

• bash
• realpath
• bc
• docker

Usage

$ ./buildbench ./examples/ex01 out.csv 5

• To disable specific builder, please set DISABLE_DOCKER, DISABLE_BUILDKIT, and so on.

buildbench's Issues

Add makisu

https://github.com/uber/makisu

Benchmark result (June 6, 2018)

Travis log available: https://travis-ci.org/AkihiroSuda/buildbench/builds/388656138

Buildbench version

70c8889

Env

TravisCI Ubuntu Trusty ( https://docs.travis-ci.com/user/reference/overview/ )

• Ext4
• 7.5GB RAM
• 2 Cores, bursted

Tested tools

• Docker 18.05-ce
• moby/buildkit@376093d
• genuinetools/img@1a47aed9
• containers/buildah@f90b6c0
• gcr.io/kaniko-project/executor@sha256:501056bf52f3a96f151ccbeb028715330d5d5aa6647e7572ce6c6c55f91ab374 (Uploaded time: May 18, 2018 at 5:13:31 AM UTC+9)

Benchmark: examples/ex01

• foo-1: clean environment
• foo-2: keep foo-1 cache storage, touch buildContext/build-bench-dummy, and then build the image again
• Tested 5 times

ID	Took (seconds)
docker-1	20,15,18,16,17 (avg: 17.2)
docker-2	1,2,2,1,1 (avg: 1.4)
buildkit-1	12, 13, 8, 9, 8 (avg: 10.0)
buildkit-2	2, 1, 1, 1, 1 (avg: 1.2)
img-1	11,11,10,13,14 (avg: 11.8)
img-2	2,2,2,1,2 (avg: 1.8)
buildah-1	15, 15, 13, 21, 14 (avg: 15.6)
buildah-2	11, 11, 15, 14, 17 (avg: 13.6)
kaniko-1	21, 18, 16, 16, 15 (avg: 17.2)
kaniko-2	15, 18, 18 26, 16 (avg: 18.6)

20180606-ex01.txt

Benchmark: moby/moby@fd2f2a919e392b96de74795ae9af2dc5e510bc4c

• The image could not be built with Buildah and Kaniko, due to lack of support for FROM anotherstage (EDIT: fixed typo s/anotherimage/anotherstage/)
• Tested single time only

ID	Took (seconds)
docker-1	387
docker-2	8
buildkit-1	266
buildkit-2	2
img-1	450
img-2	6

20180606-moby.txt

Benchmark result (June 9, 2018)

Full logs and version info: https://travis-ci.org/AkihiroSuda/buildbench/builds/389792477?utm_source=github_status&utm_medium=notification

Description about workloads: #1

CSV columns: ID, took, begun, finished

ex01.csv

docker-1,15.088948494,1528516883.067540964,1528516898.156489458
docker-2,1.264920067,1528516898.162242410,1528516899.427162477
docker-1,14.851893349,1528516900.139559065,1528516914.991452414
docker-2,1.296145993,1528516914.995536683,1528516916.291682676
docker-1,14.634571302,1528516916.982249664,1528516931.616820966
docker-2,1.335567962,1528516931.620468566,1528516932.956036528
docker-1,14.938568557,1528516933.646126876,1528516948.584695433
docker-2,1.339203861,1528516948.588702272,1528516949.927906133
docker-1,15.008774077,1528516950.595051276,1528516965.603825353
docker-2,1.287587080,1528516965.608013027,1528516966.895600107
buildkit-1,7.676590955,1528516970.952694997,1528516978.629285952
buildkit-2,1.220504264,1528516978.633374484,1528516979.853878748
buildkit-1,7.520999153,1528516980.576517629,1528516988.097516782
buildkit-2,1.124634502,1528516988.101464458,1528516989.226098960
buildkit-1,7.678619715,1528516989.962819048,1528516997.641438763
buildkit-2,1.172230003,1528516997.645713939,1528516998.817943942
buildkit-1,8.343999825,1528516999.540903954,1528517007.884903779
buildkit-2,1.136848518,1528517007.892358192,1528517009.029206710
buildkit-1,7.540258448,1528517009.767991265,1528517017.308249713
buildkit-2,1.133150257,1528517017.312111999,1528517018.445262256
img-1,9.916585799,1528517021.603791267,1528517031.520377066
img-2,1.895819467,1528517031.524145433,1528517033.419964900
img-1,9.896674246,1528517033.647922903,1528517043.544597149
img-2,1.972287355,1528517043.548718848,1528517045.521006203
img-1,9.927664649,1528517045.753273162,1528517055.680937811
img-2,1.830990818,1528517055.685087238,1528517057.516078056
img-1,10.094313643,1528517057.753391160,1528517067.847704803
img-2,1.972982689,1528517067.851263908,1528517069.824246597
img-1,10.522893321,1528517070.050599802,1528517080.573493123
img-2,1.912381080,1528517080.579451096,1528517082.491832176
buildah-1,13.357741251,1528517102.141753277,1528517115.499494528
buildah-2,12.167797793,1528517115.503435666,1528517127.671233459
buildah-1,12.777023598,1528517127.720550188,1528517140.497573786
buildah-2,10.884054123,1528517140.501773347,1528517151.385827470
buildah-1,14.075376815,1528517151.443113080,1528517165.518489895
buildah-2,11.159878164,1528517165.522856817,1528517176.682734981
buildah-1,12.498024832,1528517176.736068416,1528517189.234093248
buildah-2,11.252820026,1528517189.238772689,1528517200.491592715
buildah-1,13.086742903,1528517200.543874630,1528517213.630617533
buildah-2,11.105947072,1528517213.634928288,1528517224.740875360
kaniko-1,13.949581683,1528517226.325083879,1528517240.274665562
kaniko-2,13.708101622,1528517240.279425567,1528517253.987527189
kaniko-1,12.888729452,1528517253.992463459,1528517266.881192911
kaniko-2,14.101354493,1528517266.885450815,1528517280.986805308
kaniko-1,13.230798628,1528517280.993682640,1528517294.224481268
kaniko-2,13.271245080,1528517294.228877218,1528517307.500122298
kaniko-1,15.104304591,1528517307.512061163,1528517322.616365754
kaniko-2,13.208975221,1528517322.624875939,1528517335.833851160
kaniko-1,13.893086794,1528517335.839334127,1528517349.732420921
kaniko-2,12.690004360,1528517349.739682189,1528517362.429686549

ex01.txt

moby.csv

docker-1,368.328064583,1528516894.627995325,1528517262.956059908
docker-2,5.913627327,1528517262.963389876,1528517268.877017203
docker-1,346.701621722,1528517271.244334087,1528517617.945955809
docker-2,5.887373814,1528517617.950057810,1528517623.837431624
docker-1,354.015341011,1528517626.326424942,1528517980.341765953
docker-2,5.862704967,1528517980.345517145,1528517986.208222112
buildkit-1,281.376391784,1528516969.148981670,1528517250.525373454
buildkit-2,1.772235548,1528517250.531774709,1528517252.304010257
buildkit-1,281.301812854,1528517255.114552131,1528517536.416364985
buildkit-2,1.822073199,1528517536.420459090,1528517538.242532289
buildkit-1,280.564748838,1528517540.955174297,1528517821.519923135
buildkit-2,1.852769957,1528517821.524235519,1528517823.377005476
img-1,430.453649374,1528516883.837080020,1528517314.290729394
img-2,7.088786477,1528517314.303254456,1528517321.392040933
img-1,431.462889460,1528517323.572703808,1528517755.035593268
img-2,7.318575166,1528517755.039628785,1528517762.358203951
img-1,433.546605852,1528517764.603786101,1528518198.150391953
img-2,7.159197898,1528518198.154567821,1528518205.313765719

moby-docker.txt
moby-buildkit.txt
moby-img.txt

Benchmark result (June 19, 2018)

Full logs and version info: https://travis-ci.org/AkihiroSuda/buildbench/builds/393967682?utm_source=github_status&utm_medium=notification ( 80e6341 )

# May, 2018
DOCKER_IMAGE="docker:18.05-dind"
# BuildKit does not have tagged images; git: moby/buildkit@00140b79929a046614feec5985c8e8f56f7501e7 (June 16, 2018)
BUILDKIT_IMAGE="tonistiigi/buildkit@sha256:38d17109b217c2894063f0bd4b844554901de65add1f9d7cc3b5b73da4f6ab4e"
# June 13, 2018
IMG_IMAGE="r.j3ss.co/img:v0.4.6"
# Buildah does not have tagged images; git: projectatomic/buildah@fc438bb932e891cbe04109cfae1dfbe3c99307a5 (June 15, 2018)
BUILDAH_IMAGE="containerbuilding/buildah@sha256:b03b04c2480521c9ae05c87f28cc2568a347d009de47f4153d30b41f3bf0ccb8"
# May, 2018
KANIKO_IMAGE="gcr.io/kaniko-project/executor:v0.1.0"

Description about workloads: #1

CSV columns: ID, took, begun, finished

ex01.csv

docker-1,18.165849907,1529391592.263071609,1529391610.428921516
docker-2,1.235037047,1529391610.434916377,1529391611.669953424
docker-1,14.818404800,1529391612.367617690,1529391627.186022490
docker-2,1.287132571,1529391627.190408360,1529391628.477540931
docker-1,14.802825026,1529391629.158332039,1529391643.961157065
docker-2,1.228064991,1529391643.965079234,1529391645.193144225
docker-1,15.078891824,1529391645.875215472,1529391660.954107296
docker-2,1.234751728,1529391660.957918448,1529391662.192670176
docker-1,15.178132443,1529391662.856697985,1529391678.034830428
docker-2,1.249452815,1529391678.038773656,1529391679.288226471
buildkit-1,7.861676981,1529391683.423117905,1529391691.284794886
buildkit-2,1.132106418,1529391691.288556542,1529391692.420662960
buildkit-1,7.333715405,1529391693.139716291,1529391700.473431696
buildkit-2,1.153705816,1529391700.477532735,1529391701.631238551
buildkit-1,7.319216463,1529391702.341668182,1529391709.660884645
buildkit-2,1.108104800,1529391709.664805940,1529391710.772910740
buildkit-1,7.638978042,1529391711.533288749,1529391719.172266791
buildkit-2,1.136491177,1529391719.176583838,1529391720.313075015
buildkit-1,9.102700778,1529391721.029759107,1529391730.132459885
buildkit-2,1.183218361,1529391730.136429357,1529391731.319647718
img-1,10.564293780,1529391734.808084287,1529391745.372378067
img-2,1.835848179,1529391745.376254356,1529391747.212102535
img-1,9.769529824,1529391747.431877654,1529391757.201407478
img-2,1.952110230,1529391757.205537429,1529391759.157647659
img-1,9.723758768,1529391759.379980697,1529391769.103739465
img-2,1.817143219,1529391769.107704175,1529391770.924847394
img-1,10.408545686,1529391771.151449561,1529391781.559995247
img-2,1.843239252,1529391781.564312723,1529391783.407551975
img-1,9.514438987,1529391783.633510686,1529391793.147949673
img-2,1.894057609,1529391793.151888472,1529391795.045946081
buildah-1,15.500970782,1529391817.194548251,1529391832.695519033
buildah-2,13.619950535,1529391832.699490829,1529391846.319441364
buildah-1,14.968011827,1529391846.369269058,1529391861.337280885
buildah-2,13.369802949,1529391861.341253218,1529391874.711056167
buildah-1,15.547745947,1529391874.760378952,1529391890.308124899
buildah-2,13.435780378,1529391890.312092187,1529391903.747872565
buildah-1,15.260446171,1529391903.799704630,1529391919.060150801
buildah-2,13.683330997,1529391919.064120785,1529391932.747451782
buildah-1,15.078477477,1529391932.797529318,1529391947.876006795
buildah-2,13.539169131,1529391947.880278848,1529391961.419447979
kaniko-1,14.243428057,1529391963.082480443,1529391977.325908500
kaniko-2,13.409059837,1529391977.330233443,1529391990.739293280
kaniko-1,12.146617865,1529391990.744615692,1529392002.891233557
kaniko-2,12.328629475,1529392002.895359127,1529392015.223988602
kaniko-1,13.466581014,1529392015.228922142,1529392028.695503156
kaniko-2,13.449305711,1529392028.699544314,1529392042.148850025
kaniko-1,13.317310198,1529392042.153942771,1529392055.471252969
kaniko-2,12.848505995,1529392055.475368000,1529392068.323873995
kaniko-1,13.035854787,1529392068.328808640,1529392081.364663427
kaniko-2,13.319246937,1529392081.368680907,1529392094.687927844

ex01.txt

moby.csv

docker-1,353.787756456,1529392007.999389134,1529392361.787145590
docker-2,5.904878890,1529392361.793445013,1529392367.698323903
docker-1,356.152896153,1529392370.380126094,1529392726.533022247
docker-2,5.756538996,1529392726.540641159,1529392732.297180155
docker-1,351.236617027,1529392734.749708553,1529393085.986325580
docker-2,5.694913895,1529393085.990213594,1529393091.685127489
docker-1,346.027762206,1529393094.139444521,1529393440.167206727
docker-2,8.714977267,1529393440.171129941,1529393448.886107208
docker-1,351.950024279,1529393451.371205967,1529393803.321230246
docker-2,5.825282377,1529393803.325100829,1529393809.150383206
buildkit-1,279.197551929,1529392095.673257046,1529392374.870808975
buildkit-2,1.883717697,1529392374.879647411,1529392376.763365108
buildkit-1,279.243992842,1529392379.512312725,1529392658.756305567
buildkit-2,1.844081306,1529392658.760409468,1529392660.604490774
buildkit-1,279.479077391,1529392663.435859295,1529392942.914936686
buildkit-2,1.859363537,1529392942.918906463,1529392944.778270000
buildkit-1,278.989010640,1529392947.566908645,1529393226.555919285
buildkit-2,1.898166498,1529393226.559961109,1529393228.458127607
buildkit-1,277.165037488,1529393231.221658213,1529393508.386695701
buildkit-2,1.872406282,1529393508.390886640,1529393510.263292922
img-1,442.317527554,1529392155.120938513,1529392597.438466067
img-2,7.521936376,1529392597.445835650,1529392604.967772026
img-1,451.095576198,1529392607.323904690,1529393058.419480888
img-2,7.424161858,1529393058.423472851,1529393065.847634709
img-1,447.124890511,1529393068.196446102,1529393515.321336613
img-2,7.677913305,1529393515.325266950,1529393523.003180255
img-1,447.304128181,1529393525.338733986,1529393972.642862167
img-2,7.472302153,1529393972.646785396,1529393980.119087549
img-1,447.636093349,1529393982.401350000,1529394430.037443349
img-2,7.685351348,1529394430.041214689,1529394437.726566037

moby_docker.txt
moby_buildkit.txt
moby_img.txt

Update kaniko with support for cache

Add gnuplot/matplotlib script