Slow Performance about earthly

[brandonkal]

Oh that's interesting.

Here is my system info for when I tested this.
Between it running VSCode server and k3s, RAM used is around 70% and CPU is typically idle before the build.

brandon@dev:~/deno-lib$ sudo lshw -short
[sudo] password for brandon: 
H/W path          Device        Class          Description
==========================================================
                                system         Standard PC (Q35 + ICH9, 2009)
/0                              bus            Motherboard
/0/400                          processor      Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
/0/1000                         memory         8GiB System Memory
/0/1000/0                       memory         8GiB DIMM RAM
/0/0                            memory         96KiB BIOS
/0/100                          bridge         82G33/G31/P35/P31 Express DRAM Controller
/0/100/1                        display        QXL paravirtual graphic card
/0/100/2                        bridge         QEMU PCIe Root port
/0/100/2/0                      generic        Virtio filesystem
/0/100/2/0/0                    generic        Virtual I/O device
/0/100/2.1                      bridge         QEMU PCIe Root port
/0/100/2.1/0                    communication  Virtio console
/0/100/2.1/0/0                  generic        Virtual I/O device
/0/100/2.2                      bridge         QEMU PCIe Root port
/0/100/2.2/0                    network        Virtio network device
/0/100/2.2/0/0    enp3s0        network        Ethernet interface
/0/100/2.3                      bridge         QEMU PCIe Root port
/0/100/2.3/0                    generic        Virtio memory balloon
/0/100/2.3/0/0                  generic        Virtual I/O device
/0/100/2.4                      bridge         QEMU PCIe Root port
/0/100/2.4/0                    storage        Virtio block device
/0/100/2.4/0/0    /dev/vda      disk           107GB Virtual I/O device
/0/100/2.4/0/0/1  /dev/vda1     volume         99GiB EXT4 volume
/0/100/2.4/0/0/e  /dev/vda14    volume         4095KiB BIOS Boot partition
/0/100/2.4/0/0/f  /dev/vda15    volume         105MiB Windows FAT volume
/0/100/7                        bus            82801I (ICH9 Family) USB UHCI Controller #1
/0/100/7/1        usb2          bus            UHCI Host Controller
/0/100/7.1                      bus            82801I (ICH9 Family) USB UHCI Controller #2
/0/100/7.1/1      usb3          bus            UHCI Host Controller
/0/100/7.2                      bus            82801I (ICH9 Family) USB UHCI Controller #3
/0/100/7.2/1      usb4          bus            UHCI Host Controller
/0/100/7.7                      bus            82801I (ICH9 Family) USB2 EHCI Controller #1
/0/100/7.7/1      usb1          bus            EHCI Host Controller
/0/100/7.7/1/1                  input          QEMU USB Tablet
/0/100/1f                       bridge         82801IB (ICH9) LPC Interface Controller
/0/100/1f.2                     storage        82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
/0/100/1f.3                     bus            82801I (ICH9 Family) SMBus Controller
/1                veth396c5200  network        Ethernet interface
/2                vethd9258556  network        Ethernet interface
/3                veth1a9ac58e  network        Ethernet interface
/4                vetha76c9df0  network        Ethernet interface
/5                flannel.1     network        Ethernet interface
/6                vetha2b67664  network        Ethernet interface
/7                vethded0ddcc  network        Ethernet interface
/8                veth6253a200  network        Ethernet interface
/9                veth4a1d04a0  network        Ethernet interface
/a                vethe65f2e38  network        Ethernet interface
/b                vetha4b575a9  network        Ethernet interface
/c                veth97e98bc3  network        Ethernet interface
/d                veth7b423083  network        Ethernet interface
/e                veth890117d5  network        Ethernet interface
/f                veth005ce570  network        Ethernet interface
/10               vethf6340e60  network        Ethernet interface
/11               vethb04b6202  network        Ethernet interface
/12               veth4e9fa542  network        Ethernet interface
/13               veth11081c8c  network        Ethernet interface
/14               vethaef728a4  network        Ethernet interface
/15               veth90885a7c  network        Ethernet interface
/16               vethd8739135  network        Ethernet interface
/17               veth5c549dc4  network        Ethernet interface
/18               vethc5638b56  network        Ethernet interface
/19               veth4d703eab  network        Ethernet interface
/1a               veth60df1ad1  network        Ethernet interface
/1b               veth8042fa81  network        Ethernet interface
/1c               veth6511dfd4  network        Ethernet interface
/1d               veth0f97d916  network        Ethernet interface
/1e               vethc7d77f81  network        Ethernet interface
/1f               veth7e3add9b  network        Ethernet interface
/20               veth10b3cf17  network        Ethernet interface
/21               vethcc27a64f  network        Ethernet interface
/22               veth1f74120d  network        Ethernet interface
/23               veth27ec4539  network        Ethernet interface
/24               veth77da6382  network        Ethernet interface
/25               vethef2e62e5  network        Ethernet interface
/26               vethf582a7c7  network        Ethernet interface
/27               vethd45c09cc  network        Ethernet interface
/28               veth11b3e77f  network        Ethernet interface
/29               veth45419251  network        Ethernet interface
/2a               vethce98c844  network        Ethernet interface
/2b               veth532c6e6a  network        Ethernet interface
/2c               cni0          network        Ethernet interface
/2d               veth9c2454fa  network        Ethernet interface
/2e               veth2c43faee  network        Ethernet interface
/2f               veth5a0494a3  network        Ethernet interface
/30               veth7dbbe4d6  network        Ethernet interface
/31               veth0dfeda6e  network        Ethernet interface
/32               veth6a1de561  network        Ethernet interface
/33               veth637cccfe  network        Ethernet interface
/34               veth1b0413e3  network        Ethernet interface
/35               vethbcb1f8a9  network        Ethernet interface
/36               ztly54ertv    network        Ethernet interface
/37               vethf75bee17  network        Ethernet interface
/38               veth92bb080e  network        Ethernet interface
/39               vethbaf0d6c7  network        Ethernet interface
/3a               docker0       network        Ethernet interface
/3b               veth14cc2266  network        Ethernet interface
/3c               veth5a1f8f19  network        Ethernet interface
/3d               veth9c6a627b  network        Ethernet interface
/3e               vethb45667e4  network        Ethernet interface
/3f               veth3eadb8d9  network        Ethernet interface
/40               veth9b60b956  network        Ethernet interface
/41               vethdde57fc2  network        Ethernet interface
/42               veth3901d092  network        Ethernet interface
/43               vethaa80411e  network        Ethernet interface

[vladaionescu]

Hi Brandon - there is indeed some overhead for pure Go builds - mostly due to Docker layer caching. The build becomes more performant in complex setups, where some parallelism kicks in.

I'll mark this issue as closed as there's nothing specific that can be improved.

[brandonkal]

Naturally I expect some overhead with having to launch containers and pulling layers.
But even just using vanilla buildkit, I see a 2x performance improvement over using earth. It's much closer to the pure Go build.

Note that real is much more than user+sys time. So something is blocking inside of earth more than just buildkit.

I've tried with and without the loopback device. This is on an ubuntu host.

[vladaionescu]

Interesting... then perhaps this needs to be investigated further.

[vladaionescu]

I did a bit of testing and I have a theory as to what the cause is.

For my reference, this is the Dockerfile I used:

FROM golang:1.13-alpine3.11 AS build
WORKDIR /go-example

COPY go.mod go.sum ./
RUN go mod download

COPY main.go .
RUN go build -o build/go-example main.go

FROM golang:1.13-alpine3.11
WORKDIR /go-example
COPY --from=build /go-example/build/go-example .
ENTRYPOINT ["/go-example/go-example"]

It seems that performance is more similar to Dockerfile builds when not outputting the artifact. I suspect it's due to the way Earthly creates these outputs - it asks buildkit for another completely separate solve and relies on caching to get the same artifact from a previous solve. This may have race condition issues beyond just the performance impact. We'd been planning to change this to avoid such issues, but haven't yet gotten around to it. Hopefully performance will be improved too once we address the multi-solve issue.

Curious: What system were you testing this on @brandonkal? The numbers that I got seemed to vary from yours.