openllb / hlb

A developer-first language to build and test any software efficiently

hlb

hlb is a high-level build language for BuildKit.

Describe your build in containerized units of work, and BuildKit will build your target as efficiently as possible.

Getting started with HLB

If you're on a MacOS or Linux (linux-amd64), head on over to Releases to grab a static binary.

Otherwise, you can compile HLB yourself using go:

git clone https://github.com/openllb/hlb.git
cd hlb
go install ./cmd/hlb

Then you can run one of the examples in ./examples:

hlb run ./examples/node.hlb

Bring your own BuildKit

By default, HLB uses the BuildKit embedded in a docker engine. HLB supports BUILDKIT_HOST the same way buildctl does, so you can run BuildKit in a container and connect to it:

docker run -d --name buildkitd --privileged moby/buildkit:master
export BUILDKIT_HOST=docker-container://buildkitd
hlb run ./examples/node.hlb

Language server

If your editor has a decent LSP plugin, HLB does support LSP over stdio via the hlb langserver subcommand.

hlb's Issues

Design questions around caching in HLB

There are three levels of caching in HLB: Cache mounts, FS caching, Build cache.

Cache mounts

Cache mounts is a persistent read-writable mount stored by the BuildKit backend. This is typically used for compiler / package manager caches.

Some caveats:

• Has sharing modes depending on the underlying compiler/package manager. Is concurrent usage safe? It's complicated to know what to pick.
• Ever growing size, how do we prune this? How many projects can we use this with? If its a single cache key for every project, would using FS caching be better because we have more control?
• In a cloud cluster environment, this is not very reliable because you may not hit the same buildkitd.

fs default() {
    image "golang:alpine"
    run "go build xyz" with option {
        mount fs { scratch; } "/root/.cache/go-build" with option {
            cache "someCacheKey" "shared"
        }
    }
}

1. I'm not sure if the scratch fs defined after mount is ever utilized if there is a cache option. Need to investigate this.
2. Instead of nesting as a option::mount cache, can we define it as a option::run cacheMount for UX? Initially it was designed as a mount option because of LLB, but we can change that.
3. In the Dockerfile frontend, I recall that they define cache keys for the user, it's possible we can move the cache key as an option to cacheMount. Need to investigate this.
4. Does the BuildKit build cache export cache mounts as well? Need to investigate this.

FS caching

Rather than a language or backend feature, this is more of a pattern emerging from HLB usage. You are able to export filesystems to various sources (image to DockerHub, pushing to remote git repo, publishing to HTTP server like artifactory), and you can also use these remote sources to mount as a "starting point" or a "primed mount" to speed up the operation.

When running npm install, the current pattern is:

• Mount option::mount cache for npm cache directories. (Likely safe to be shared with multiple projects)
• Mount fs { scratch; } "src/node_modules" because you can also prime the node modules. (Only safe to use for a single project)

And then you can push the node modules mount as an image, then remount it for subsequent runs:

fs default(fs src) {
    image "node:alpine"
    run "npm install" with option {
        dir "/src"
        mount src "/src"
        mount fs { scratch; } "/root/.npm" with option {
            cache "npm-config-cache" "shared"
        }
        mount fs { image "my-node-modules:latest";  } "/src/node_modules" as nodeModules
    }
}

fs snapshotModules() {
    nodeModules
    dockerPush "my-node-modules:latest"
}

Perhaps users will then set up a CRON CI job to run the target snapshotModules once in a while, or on a merge to master.

Build cache

Build cache import/export is a native feature to BuildKit. See: https://github.com/moby/buildkit#export-cache

This is the basic behavior you get when you run a second build through HLB on the same BuildKit backend. Unchanged input will return the same output so they don't need to run it a second time. However when you point to a new BuildKit backend this is lost. Build cache import/export allows you to publish the build cache for a particular build to disk (local), inline (if creating a docker image, it will be embedded), or an OCI image (not an executable image, just a data container for the build cache).

There are two main angles we can tackle this from:

• Backend solution: We develop and maintain infrastructure around BuildKit to distribute build caches between buildkitd nodes.
• Frontend solution: We expose build cache import/export in HLB.

Build cache import/export is per-solve, and we already do multiple solves when executing HLB.

If we want to implement this in the frontend side, we'll need to somehow inform the HLB compiler that this section is using this particular cache. Ideally we want to provide a fs { ... } to be agonistic to the source (whether its local or image), but that will require an upstream change.

Here is an example that somewhat fulfills the requirements but I think is a terrible UX. But perhaps can serve as a starting point for discussion:

fs default() {
    cacheContext fs {
        image "openllb/my-remote-build-cache:latest";
    } fs {
        image "alpine"
        run "running with cache context"
    }  
}

HEREDOC support

Multi-line string support is mainly in 2 forms:

1. Raw string literal like go.
2. HEREDOC has << regular version and <<- where all leading tabs are ignored.

Here are the current proposals so we can get this conversation started.

Proposal 1

fs default() {
	image "alpine"
	run <<-END
	apk add -U foo \
		bar \
		baz
	END <<-END
	apk add -U foo \
		bar \
		baz
	END with option {
		env "key" "value"
	}
}

Proposal 2

fs default() {
	image "alpine"
	run <<-ARG1 <<-ARG2 with option { env "key" "value"; }
	apk add -U foo \
		bar \
		baz
	ARG1
	apk add -U foo \
		bar \
		baz
	ARG2
}

Unit test bundle 2 for older issues

Tests should go in codegen/codegen_test.go. The input program should be as minimal as possible.

• #59 run has new builtin "ignoreCache"
• #52 stack overflow when using alias within same fs
• #40 panic from named option::run

unable to reuse mount within fs block

Input:

fs default() {
	image "busybox"
	run "touch foo" with option {
		dir "/foo"
		mount fs { scratch; } "/foo" as fooMount
	}
	run "ls -l" with option {
		dir "/foo"
		mount fooMount "/foo"
	}
}

Output:

$ hlb run ./bugs.hlb
#1 compiling [default]
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/github.com/openllb/hlb/codegen/decl.go:19
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitAliasDecl
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/github.com/openllb/hlb/codegen/decl.go:81
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemAliasDecl
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/github.com/openllb/hlb/codegen/decl.go:96
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemExpr
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/github.com/openllb/hlb/codegen/expr.go:110
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitExecOptions
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/github.com/openllb/hlb/codegen/codegen.go:1357
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitOptions
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/github.com/openllb/hlb/codegen/codegen.go:866
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitWithOption
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/github.com/openllb/hlb/codegen/codegen.go:322
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemChainStmt
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/github.com/openllb/hlb/codegen/codegen.go:330
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitChainStmt
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/github.com/openllb/hlb/codegen/codegen.go:185
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitBlock
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/github.com/openllb/hlb/codegen/codegen.go:157
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemBlock
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/github.com/openllb/hlb/codegen/codegen.go:272
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/github.com/openllb/hlb/codegen/decl.go:45
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemFuncDecl
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/github.com/openllb/hlb/codegen/decl.go:56
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).Generate
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/github.com/openllb/hlb/codegen/codegen.go:95
#1 0.000 github.com/openllb/hlb.Compile.func1
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/github.com/openllb/hlb/hlb.go:146
#1 0.000 github.com/openllb/hlb/solver.(*progressUI).Write.func2.1
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/github.com/openllb/hlb/solver/progress.go:172
#1 0.000 github.com/openllb/hlb/solver.write
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/github.com/openllb/hlb/solver/progress.go:196
#1 0.000 github.com/openllb/hlb/solver.(*progressUI).Write.func2
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/github.com/openllb/hlb/solver/progress.go:171
#1 0.000 golang.org/x/sync/errgroup.(*Group).Go.func1
#1 0.000 	/go/src/stash.corp.netflix.com/engtools/newt/vendor/golang.org/x/sync/errgroup/errgroup.go:57
#1 0.000 runtime.goexit
#1 0.000 	/go-version/1.14.1/src/runtime/asm_amd64.s:1373
#1 0.000 Caused by: default expected args [], found [fooMount "/foo"]
#1 ERROR: default expected args [], found [fooMount "/foo"]
------
 > compiling [default]:
#1 0.000 Caused by: default expected args [], found [fooMount "/foo"]
------

Defining outputs in HLB rather than in command-line

When we land multi-targets in #47, it is only helpful if we can output the targets to different locations.

1. Output as soon as targets have been solved, without waiting for other parts of the graph
2. Multiple outputs per target, useful when wanting to push to multiple registries

Proposal

HLB CLI

We'll move away from hlb run --download <dest> / hlb run --push <ref> and instead bring it down into hlb functions.

But you'll still be able to specify output flags on the command line:

• hlb run --target foo,download=.

The target is splitted on the sentinel , and the first value is always the target name, followed by key values for output flags.

Builtin Functions

# Outputs the filesystem to external sources. If no options are
# defined, this is a no-op.
#
# @return the filesystem unchanged.
fs (fs) output()

# Pushes the filesystem to a registry following the distribution
# spec: https://github.com/opencontainers/distribution-spec/
#
# @param ref a docker registry reference. if not fully qualified, it will be
# expanded the same as the docker CLI.
# @return an option to push the filesystem to a registry.
option::output push(string ref)

# Downloads the filesystem to a local path.
#
# @param localPath the destination filepath for the filesystem contents.
# @return an option to download a filesystem to the local system.
option::output download(string localPath)

fs default() {
    scratch
    mkfile "/out" 0o644 "hello world"
    output with option {
        push "us-east-1.registry/foo:latest"
        push "us-west-2.registry/foo:latest"
    }
}

By running the default target, we end up solving the graph and pushing it to two different registries.

Note this changes HLB from a compiler to more of an interpreter, unless BuildKit supports these output vertices in LLB

Outputs happen concurrently with rest of graph

Output returns the same input filesystem so you can continue chaining statements afterwards:

fs default() {
    scratch
    mkfile "/out" 0o644 "hello world"
    output with option {
        push "foo:latest"
    }
    env "key" "value"
}

readonlyRootfs prevents `run` from running

Input:

fs default() {
  image "alpine:latest"
  run "echo 1" with option {
    readonlyRootfs  
  }
  run "echo 2"
}

Output:

$ ./hlb run --log-output plain ./tests/bug.hlb
#1 compiling ./tests/bug.hlb
#1 DONE 0.0s

#3 docker-image://docker.io/library/alpine:latest
#3 resolve docker.io/library/alpine:latest
#3 resolve docker.io/library/alpine:latest 0.4s done
#3 DONE 0.4s

#3 docker-image://docker.io/library/alpine:latest
#3 CACHED

#2 /bin/sh -c echo 2
#2 0.101 2
#2 DONE 0.1s

Note: echo 1 is not in output

Add minimal travis yml

We don't have a great test suite atm, but we should have CI to at least make sure the code we're merging continues to compile.

Unit test bundle 1 for older issues

Tests should go in codegen/codegen_test.go. The input program should be as minimal as possible.

• #76 copy missing builtins for "allowWildcards", "allowEmptyWildcard", "chown", "createdTime"
• #71 unable to reuse mount within fs block
• #67 unable to pass anonymous option as argument

Pass in external values into HLB

Want the ability to pass in values from the CLI or something fetch it from inside the .hlb file. Perhaps something like string { getEnv "CWD"; }.

String interpolation of environment variables

It would be nice to have environment variable substitution in Strings, especially in local. Environment variables are commonly used to put strings together or reference where files are on the local filesystem, like $HOME.

In this example, is impossible to generalize for different users without environment variables:

fs homeConfiguration() {
  local "/home/quidryan/.myconfig"
}

The used environment variables would probably have to be considered part of the input for caching.

Support import / export of external HLB modules, paving the way for HLB libraries

There's two main use cases for libraries:

1. Sharing common hlb functions, like a canonical function for building go code, or running npm install, etc.
2. Sharing a hlb function wrapper for an external frontend.

Either way, we want to avoid copy & pasting that happens in Dockerfiles, and since we have function arguments, we can have better reusability than ONBUILD instructions.

Proposal

HLB CLI

HLB is a file-scoped language, so each file represents a module. We introduce a subcommand namespace module to manage your module dependencies:

By default, imports are solved everytime, since solve targets may have sources that change over time. But users can opt in to vendoring their dependencies via the hlb module subcommands.

Module vendoring

hlb module vendor [--target <ident of an import> ...] <*.hlb>

hlb module vendor will solve the imports (all imports if no --target is specified) and export them to your local system under ./.hlb/modules/** and basically freeze a snapshot of that hlb library. Since solve targets may have sources like image "foobar:latest" that may change when you resolve the import, this is "vendors" the import locally.

Running hlb module vendor --target <ident of an import> will update the vendor for a specific import. The path is based on the underlying LLB's vertex digest like ./hlb/modules/sha256/9f/9fabc/module.hlb so it's stable based on the import fs graph.

Module print tree

hlb module tree <*.hlb>

Basically prints out a tree of the imports starting from the hlb module you provided. Useful to understand what are your dependencies.

Cleaning up unused dependencies

hlb module tidy <*.hlb>

When adding and removing imports from a hlb module, the vendor directory will drift and either have unused modules or missing modules. hlb module tidy will fill in the gaps and prune unused modules from the directory.

Import / export declarations

import go from fs {
    image "openllb/golang.hlb"
}

export buildHLB
export leftPadNodeModules

fs buildHLB() {
    go.build fs { local "."; } "./cmd/hlb"
}

fs example() {
    image "node:alpine"
    run "npm install" with option {
        dir "/src"
        mount fs { git "https://github.com/left-pad/left-pad.git" "master"; } "/src"
        mount fs { scratch; } "/src/node_modules" as leftPadNodeModules
    }
}

Imports declare an identifier that shares the same scope as other function declarations. Then you can access exported functions from the import via go.<identifier>. Only functions that are exported may be invoked, so that library authors to define what their public API is.

Exports is a declaration at the top-level scope too, allowing you to export functions and aliases like as leftPadNodeModules. Opted for top-level rather than adding modifiers to declarations like public fs buildHLB or as public leftPadNodeModules so that its clearer what the module import/exports.

HLB format

hlb format is an opinionated way of formatting your .hlb files. With the introduction of import/exports, they will be formatted so all the imports declarations are at the top, then exports, then the rest of the function declarations. This way, users can see at a glance, what is imported/exported in this module.

How libraries are distributed

Since import takes a fs function literal, you can import from theoretically any source, be it from a git repo, an OCI image, or a tarball hosted on some HTTP server.

It could even be dynamic:

import go from fs {
    scratch
    copy fs {
        image "alpine"
        run "apk add -U openssh-client"
        run "scp [email protected]:/home/root/module.hlb /out/module.hlb" with option { ssh; }
    } "/out/module.hlb" "/module.hlb"
}

Import from local hlb modules

Consider the scenario where you import some library and that library is split up into different files, for organizational purposes.

# user.hlb
import go from fs {
    image "openllb/golang.hlb"
}

# golang.hlb
import utils from fs {
    local "./utils.hlb"
}

Then when recursively solving the imports, we'll need to:

1. Solve import go by unpacking the contents of openllb/golang.hlb to local system. (both golang.hlb and utils.hlb)
2. Parse and check that hlb file.
3. Recursively solve golang.hlb's imports.
4. Transfer ./utils.hlb back to BuildKit
5. Download ./utils.hlb again as part of solving an import.

Instead, we could special case local hlb module imports by saying:

# golang.hlb
import utils "./utils.hlb`

Syntactically different because of missing the from, since HLB currently doesn't support polymorphic types so we couldn't overload it there.

Library wrappers for external frontends

When developing a frontend, you need to also write a function in order to consume it inside HLB. For example, the dockerfile frontend:

# dockerfile.hlb
export build

fs build(fs context, fs dockerfile) {
    generate fs {
        image "openllb/dockerfile:experimental" with option { resolve; }
    } with option {
        frontendInput "context" context
        frontendInput "dockerfile" dockerfile
    }
}

Then you can also ship another image openllb/dockerfile.hlb so that users can consume the dockerfile frontend without copy and pasting the dockerBuild function.

import dockerfile from fs {
    image "openllb/dockerfile.hlb"
}

fs default() {
    dockerfile.build fs {
        local "." with option { excludePatterns "Dockerfile"; }
    } fs {
        local "." with option { includePatterns "Dockerfile"; }
    }
}

Implement global option `ignoreCache`

• Define option ignoreCache in language/builtin.hlb and run go generate ./gen
• Not sure if global options are supported yet, may need to complete that implementation

Plumbs this LLB option as a global HLB option:

https://github.com/moby/buildkit/blob/21ed8f5c1d078d1e28eac2d443fc791fff60eef9/client/llb/state.go#L412-L414

Usage:

fs foo() {
  image "alpine" with option {
    ignoreCache
  }
  run "echo hi" with option {
    ignoreCache
  }
}

Unit test bundle for scope and identifiers

Tests should go into compile_test.go, and the input program should be minimal as possible. There may be more than one test case per bullet point.

• Function with duplicate names should produce error (functions, aliases).
• Export name should exist in module scope.
• Function with duplicate fields in the signature should produce error.
• Calling selector that does not exist should produce error.
• Calling selector on name that isn't an import should produce error.

image resolve failing on imported fs

This is the test hlb:

import test "./import.hlb"
fs default() {
	test.output
}

and ./import.hlb looks like:

export output
fs _buildOutput() {
        image "node:alpine" with resolve
        run "touch /out/hello" with option {
                mount scratch "/out" as output
        }
}

Running we get this error:

./hlb run ./bug.hlb  | cat
#1 compiling [default]
#1 1.412 github.com/containerd/containerd/errdefs.init
#1 1.412        /Users/cbennett/gosrc/hlb/pkg/mod/github.com/containerd/[email protected]/errdefs/errors.go:45
#1 1.412 runtime.doInit
#1 1.412        /usr/local/go/src/runtime/proc.go:5414
#1 1.412 runtime.doInit
#1 1.412        /usr/local/go/src/runtime/proc.go:5409
#1 1.412 runtime.doInit
#1 1.412        /usr/local/go/src/runtime/proc.go:5409
#1 1.412 runtime.doInit
#1 1.412        /usr/local/go/src/runtime/proc.go:5409
#1 1.412 runtime.doInit
#1 1.412        /usr/local/go/src/runtime/proc.go:5409
#1 1.412 runtime.doInit
#1 1.412        /usr/local/go/src/runtime/proc.go:5409
#1 1.412 runtime.doInit
#1 1.412        /usr/local/go/src/runtime/proc.go:5409
#1 1.412 runtime.doInit
#1 1.412        /usr/local/go/src/runtime/proc.go:5409
#1 1.412 runtime.main
#1 1.412        /usr/local/go/src/runtime/proc.go:190
#1 1.412 runtime.goexit
#1 1.412        /usr/local/go/src/runtime/asm_amd64.s:1373
#1 1.412 Caused by: import.hlb:4:2: no match for platform in manifest sha256:a8553f21bea4e402dac9618fadb47b179bd956c07f97064263f9522a6e53693a: not found
#1 ERROR: import.hlb:4:2: no match for platform in manifest sha256:a8553f21bea4e402dac9618fadb47b179bd956c07f97064263f9522a6e53693a: not found
------
 > compiling [default]:
#1 1.412 Caused by: import.hlb:4:2: no match for platform in manifest sha256:a8553f21bea4e402dac9618fadb47b179bd956c07f97064263f9522a6e53693a: not found
------

If I remove the with resolve option it works fine:

$ ./hlb run ./bug.hlb
#1 compiling [default]
#1 DONE 0.0s

#2 docker-image://docker.io/library/node:alpine
#2 resolve docker.io/library/node:alpine
#2 resolve docker.io/library/node:alpine 3.4s done
#2 DONE 3.4s

#3 /bin/sh -c 'touch /out/hello'
#3 CACHED

Unit test bundle for codegen basic cases

Tests should go in codegen/codegen_test.go. The input program should be as minimal as possible. There may be multiple test cases per bullet point:

• scratch
• http
• git
• env
• dir
• user
• mkdir
• mkfile
• rm
• copy

Checker must validate LLB errors

There are LLB errors that are only caught when the llb.State is marshalled and the underlying vertices are invoked with (Op).Validate). For example, the ExecOp which maps to run in HLB, must have more than 0 arguments. However, since run expects variadic string args, 0 args are valid for a variadic field.

Unit test bundle for type checking and function signatures

Tests should go into compile_test.go, and the input program should be minimal as possible. There may be more than one test case per bullet point.

• Calling function that is wrong type for the function it is called in should produce error.
• Calling function with args that don't match its signature should produce error (length, basic literal type, function type, func lit type).
• Calling function with option that doesn't match the subtype should produce error.
• Correctly calling function with variadic field in its signature should succeed.

option::copy missing builtin's

Looks like we plumbed it in the codegen, but the builtins map for option::copy does not recognize:
"allowWildcards"
"allowEmptyWildcard"
"chown"
"createdTime"

Shared sessions when solving

Currently, no session is initialized for solves so for every solve, a new session is created and every llb.Local will need to be transferred for that session.

Assuming the llb.Local are the same underlying directories, using a shared session will eliminate these duplicated transfer.

The SharedSession is a *session.Session defined in the client.SolveOpt:
https://github.com/moby/buildkit/blob/6e725bdbb6ce0763cc17074f1c4e86032cee4415/client/solve.go#L43

By default, this is nil and checked here:
https://github.com/moby/buildkit/blob/6e725bdbb6ce0763cc17074f1c4e86032cee4415/client/solve.go#L100-L115

A session is created from session.NewSession:
https://github.com/moby/buildkit/blob/6e725bdbb6ce0763cc17074f1c4e86032cee4415/session/session.go#L45

Regarding the name argument, it looks like default session name is initialized using the base directory of the current working directory, but it looks like it's never used anywhere so it might be fine to leave it as empty string. (Sessions generate their own UUID that you cannot control)
https://github.com/moby/buildkit/blob/6e725bdbb6ce0763cc17074f1c4e86032cee4415/client/solve.go#L359-L365

Regarding the sharedKey argument, it looks used here but seems to be defaulted to empty string anyway:
https://github.com/moby/buildkit/blob/6e725bdbb6ce0763cc17074f1c4e86032cee4415/source/local/local.go#L110

---

From an implementation perspective, we just need to create a session.NewSession that is plumbed to the client.SolveOpt for all the solves in the solve request tree.

Unit test bundle for calling functions

Tests should go in codegen/codegen_test.go. The input program should be as minimal as possible. There may be multiple test cases per bullet point. We should test the available options as well.

• Calling aliased function
• Calling aliased mount
• Function scope variable should be used over function decl of same name.
• Calling option function in an option function literal (should merge all the options)

Consider how to create stable and unique cache keys

A common pattern among all build workflows is using cache mounts. But in order to use it, we need to provide a cache key. If we're using the same HLB for multiple projects, or multiple users using the same BuildKit we need their cache keys to not collide.

For example, we can do:

run "foobar" with option {
    mount fs { scratch; } "/cache" with option {
        cache string { hash string { getEnv "CWD"; }; }
    }
}

`stack overflow` panic when using alias within same `fs`

Input:

fs cpTest() {
	scratch
	mkfile "/foo" 0o644 "foobar" as foo
	copy foo "/foo" "/bar"
}

Error:

#1 compiling [cpTest]
runtime: goroutine stack exceeds 1000000000-byte limit
fatal error: stack overflow

runtime stack:
runtime.throw(0x4fc7627, 0xe)
	/usr/local/go/src/runtime/panic.go:617 +0x72
runtime.newstack()
	/usr/local/go/src/runtime/stack.go:1041 +0x6f0
runtime.morestack()
	/usr/local/go/src/runtime/asm_amd64.s:429 +0x8f

goroutine 1 [running]:
runtime.heapBitsSetType(0xc006c21c20, 0x10, 0x10, 0x4d9a6c0)
	/usr/local/go/src/runtime/mbitmap.go:938 +0xa56 fp=0xc024000350 sp=0xc024000348 pc=0x40187b6
runtime.mallocgc(0x10, 0x4d9a6c0, 0x1, 0x0)
	/usr/local/go/src/runtime/malloc.go:969 +0x51c fp=0xc0240003f0 sp=0xc024000350 pc=0x400e37c
runtime.newobject(0x4d9a6c0, 0x0)
	/usr/local/go/src/runtime/malloc.go:1068 +0x38 fp=0xc024000420 sp=0xc0240003f0 pc=0x400e9d8
github.com/moby/buildkit/client/llb.dirf(0x4fbbf46, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
	pkg/mod/github.com/hinshun/[email protected]/client/llb/meta.go:40 +0x31 fp=0xc024000470 sp=0xc024000420 pc=0x438ebb1
github.com/moby/buildkit/client/llb.dir(...)
	pkg/mod/github.com/hinshun/[email protected]/client/llb/meta.go:37
github.com/moby/buildkit/client/llb.NewState(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	pkg/mod/github.com/hinshun/[email protected]/client/llb/state.go:37 +0xda fp=0xc024000528 sp=0xc024000470 pc=0x43937da
github.com/moby/buildkit/client/llb.Scratch(...)
	pkg/mod/github.com/hinshun/[email protected]/client/llb/source.go:243
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemSourceStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc000382c40, 0xc003eabc20, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:332 +0x14b0 fp=0xc024000878 sp=0xc024000528 pc=0x4c65be0
github.com/openllb/hlb/codegen.(*CodeGen).EmitSourceStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382c40, 0xc000319b66, 0x7, 0xc003eabc20, ...)
	/go/src/hlb/codegen/codegen.go:287 +0x20a fp=0xc024000948 sp=0xc024000878 pc=0x4c6416a
github.com/openllb/hlb/codegen.(*CodeGen).EmitBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc003eabc40, 0x3, 0x4, 0xc003eabc20, ...)
	/go/src/hlb/codegen/codegen.go:170 +0x31a fp=0xc024000bf0 sp=0xc024000948 pc=0x4c6294a
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc003eabc40, 0x3, 0x4, 0xc003eabc20, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:253 +0xd7 fp=0xc024000cf0 sp=0xc024000bf0 pc=0x4c63737
github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004cd9e0, 0x0, 0x0, 0x0, 0xc003eabc20, 0xc0004f0000, ...)
	/go/src/hlb/codegen/decl.go:45 +0x717 fp=0xc024000ea8 sp=0xc024000cf0 pc=0x4c79917
github.com/openllb/hlb/codegen.(*CodeGen).EmitAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0xc0004de138, 0x40ec75a, 0xc0004de0d0, 0x203000)
	/go/src/hlb/codegen/decl.go:81 +0xd7 fp=0xc024000f28 sp=0xc024000ea8 pc=0x4c7a1d7
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/decl.go:94 +0xa8 fp=0xc024001008 sp=0xc024000f28 pc=0x4c7a328
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemExpr(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x0, 0xc0004ce4b0, 0xc003eabbe0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/expr.go:109 +0x3ca fp=0xc024001188 sp=0xc024001008 pc=0x4c7b9da
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eabbe0, 0xc0022eb320, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:681 +0x1b87 fp=0xc0240014f8 sp=0xc024001188 pc=0x4c680f7
github.com/openllb/hlb/codegen.(*CodeGen).EmitChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eabbe0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:228 +0x9c fp=0xc024001568 sp=0xc0240014f8 pc=0x4c634fc
github.com/openllb/hlb/codegen.(*CodeGen).EmitBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc003eabc00, 0x3, 0x4, 0xc003eabbe0, ...)
	/go/src/hlb/codegen/codegen.go:203 +0x703 fp=0xc024001810 sp=0xc024001568 pc=0x4c62d33
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc003eabc00, 0x3, 0x4, 0xc003eabbe0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:253 +0xd7 fp=0xc024001910 sp=0xc024001810 pc=0x4c63737
github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004cd9e0, 0x0, 0x0, 0x0, 0xc003eabbe0, 0xc0004f0000, ...)
	/go/src/hlb/codegen/decl.go:45 +0x717 fp=0xc024001ac8 sp=0xc024001910 pc=0x4c79917
github.com/openllb/hlb/codegen.(*CodeGen).EmitAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0xc0004de138, 0x64fffff, 0xc0004de0d0, 0x4017a7a)
	/go/src/hlb/codegen/decl.go:81 +0xd7 fp=0xc024001b48 sp=0xc024001ac8 pc=0x4c7a1d7
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/decl.go:94 +0xa8 fp=0xc024001c28 sp=0xc024001b48 pc=0x4c7a328
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemExpr(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x0, 0xc0004ce4b0, 0xc003eabba0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/expr.go:109 +0x3ca fp=0xc024001da8 sp=0xc024001c28 pc=0x4c7b9da
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eabba0, 0xc0022eb0e0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:681 +0x1b87 fp=0xc024002118 sp=0xc024001da8 pc=0x4c680f7
github.com/openllb/hlb/codegen.(*CodeGen).EmitChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eabba0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:228 +0x9c fp=0xc024002188 sp=0xc024002118 pc=0x4c634fc
github.com/openllb/hlb/codegen.(*CodeGen).EmitBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc003eabbc0, 0x3, 0x4, 0xc003eabba0, ...)
	/go/src/hlb/codegen/codegen.go:203 +0x703 fp=0xc024002430 sp=0xc024002188 pc=0x4c62d33
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc003eabbc0, 0x3, 0x4, 0xc003eabba0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:253 +0xd7 fp=0xc024002530 sp=0xc024002430 pc=0x4c63737
github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004cd9e0, 0x0, 0x0, 0x0, 0xc003eabba0, 0xc0004f0000, ...)
	/go/src/hlb/codegen/decl.go:45 +0x717 fp=0xc0240026e8 sp=0xc024002530 pc=0x4c79917
github.com/openllb/hlb/codegen.(*CodeGen).EmitAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0xc0004de138, 0x40ec75a, 0xc0004de0d0, 0x203000)
	/go/src/hlb/codegen/decl.go:81 +0xd7 fp=0xc024002768 sp=0xc0240026e8 pc=0x4c7a1d7
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/decl.go:94 +0xa8 fp=0xc024002848 sp=0xc024002768 pc=0x4c7a328
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemExpr(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x0, 0xc0004ce4b0, 0xc003eabb60, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/expr.go:109 +0x3ca fp=0xc0240029c8 sp=0xc024002848 pc=0x4c7b9da
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eabb60, 0xc0022eaea0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:681 +0x1b87 fp=0xc024002d38 sp=0xc0240029c8 pc=0x4c680f7
github.com/openllb/hlb/codegen.(*CodeGen).EmitChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eabb60, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:228 +0x9c fp=0xc024002da8 sp=0xc024002d38 pc=0x4c634fc
github.com/openllb/hlb/codegen.(*CodeGen).EmitBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc003eabb80, 0x3, 0x4, 0xc003eabb60, ...)
	/go/src/hlb/codegen/codegen.go:203 +0x703 fp=0xc024003050 sp=0xc024002da8 pc=0x4c62d33
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc003eabb80, 0x3, 0x4, 0xc003eabb60, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:253 +0xd7 fp=0xc024003150 sp=0xc024003050 pc=0x4c63737
github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004cd9e0, 0x0, 0x0, 0x0, 0xc003eabb60, 0xc0004f0000, ...)
	/go/src/hlb/codegen/decl.go:45 +0x717 fp=0xc024003308 sp=0xc024003150 pc=0x4c79917
github.com/openllb/hlb/codegen.(*CodeGen).EmitAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0xc0004de138, 0x40ec75a, 0xc0004de0d0, 0x203000)
	/go/src/hlb/codegen/decl.go:81 +0xd7 fp=0xc024003388 sp=0xc024003308 pc=0x4c7a1d7
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/decl.go:94 +0xa8 fp=0xc024003468 sp=0xc024003388 pc=0x4c7a328
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemExpr(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x0, 0xc0004ce4b0, 0xc003eabb20, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/expr.go:109 +0x3ca fp=0xc0240035e8 sp=0xc024003468 pc=0x4c7b9da
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eabb20, 0xc0022eac60, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:681 +0x1b87 fp=0xc024003958 sp=0xc0240035e8 pc=0x4c680f7
github.com/openllb/hlb/codegen.(*CodeGen).EmitChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eabb20, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:228 +0x9c fp=0xc0240039c8 sp=0xc024003958 pc=0x4c634fc
github.com/openllb/hlb/codegen.(*CodeGen).EmitBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc003eabb40, 0x3, 0x4, 0xc003eabb20, ...)
	/go/src/hlb/codegen/codegen.go:203 +0x703 fp=0xc024003c70 sp=0xc0240039c8 pc=0x4c62d33
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc003eabb40, 0x3, 0x4, 0xc003eabb20, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:253 +0xd7 fp=0xc024003d70 sp=0xc024003c70 pc=0x4c63737
github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004cd9e0, 0x0, 0x0, 0x0, 0xc003eabb20, 0xc0004f0000, ...)
	/go/src/hlb/codegen/decl.go:45 +0x717 fp=0xc024003f28 sp=0xc024003d70 pc=0x4c79917
github.com/openllb/hlb/codegen.(*CodeGen).EmitAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0xc0004de138, 0x40ec75a, 0xc0004de0d0, 0x203000)
	/go/src/hlb/codegen/decl.go:81 +0xd7 fp=0xc024003fa8 sp=0xc024003f28 pc=0x4c7a1d7
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/decl.go:94 +0xa8 fp=0xc024004088 sp=0xc024003fa8 pc=0x4c7a328
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemExpr(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x0, 0xc0004ce4b0, 0xc003eabae0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/expr.go:109 +0x3ca fp=0xc024004208 sp=0xc024004088 pc=0x4c7b9da
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eabae0, 0xc0022eaa20, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:681 +0x1b87 fp=0xc024004578 sp=0xc024004208 pc=0x4c680f7
github.com/openllb/hlb/codegen.(*CodeGen).EmitChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eabae0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:228 +0x9c fp=0xc0240045e8 sp=0xc024004578 pc=0x4c634fc
github.com/openllb/hlb/codegen.(*CodeGen).EmitBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc003eabb00, 0x3, 0x4, 0xc003eabae0, ...)
	/go/src/hlb/codegen/codegen.go:203 +0x703 fp=0xc024004890 sp=0xc0240045e8 pc=0x4c62d33
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc003eabb00, 0x3, 0x4, 0xc003eabae0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:253 +0xd7 fp=0xc024004990 sp=0xc024004890 pc=0x4c63737
github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004cd9e0, 0x0, 0x0, 0x0, 0xc003eabae0, 0xc0004f0000, ...)
	/go/src/hlb/codegen/decl.go:45 +0x717 fp=0xc024004b48 sp=0xc024004990 pc=0x4c79917
github.com/openllb/hlb/codegen.(*CodeGen).EmitAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0xc0004de138, 0x40ec75a, 0xc0004de0d0, 0x203000)
	/go/src/hlb/codegen/decl.go:81 +0xd7 fp=0xc024004bc8 sp=0xc024004b48 pc=0x4c7a1d7
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/decl.go:94 +0xa8 fp=0xc024004ca8 sp=0xc024004bc8 pc=0x4c7a328
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemExpr(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x0, 0xc0004ce4b0, 0xc003eabaa0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/expr.go:109 +0x3ca fp=0xc024004e28 sp=0xc024004ca8 pc=0x4c7b9da
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eabaa0, 0xc0022ea7e0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:681 +0x1b87 fp=0xc024005198 sp=0xc024004e28 pc=0x4c680f7
github.com/openllb/hlb/codegen.(*CodeGen).EmitChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eabaa0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:228 +0x9c fp=0xc024005208 sp=0xc024005198 pc=0x4c634fc
github.com/openllb/hlb/codegen.(*CodeGen).EmitBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc003eabac0, 0x3, 0x4, 0xc003eabaa0, ...)
	/go/src/hlb/codegen/codegen.go:203 +0x703 fp=0xc0240054b0 sp=0xc024005208 pc=0x4c62d33
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc003eabac0, 0x3, 0x4, 0xc003eabaa0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:253 +0xd7 fp=0xc0240055b0 sp=0xc0240054b0 pc=0x4c63737
github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004cd9e0, 0x0, 0x0, 0x0, 0xc003eabaa0, 0xc0004f0000, ...)
	/go/src/hlb/codegen/decl.go:45 +0x717 fp=0xc024005768 sp=0xc0240055b0 pc=0x4c79917
github.com/openllb/hlb/codegen.(*CodeGen).EmitAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0xc0004de138, 0x40ec75a, 0xc0004de0d0, 0x203000)
	/go/src/hlb/codegen/decl.go:81 +0xd7 fp=0xc0240057e8 sp=0xc024005768 pc=0x4c7a1d7
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/decl.go:94 +0xa8 fp=0xc0240058c8 sp=0xc0240057e8 pc=0x4c7a328
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemExpr(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x0, 0xc0004ce4b0, 0xc003eaba60, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/expr.go:109 +0x3ca fp=0xc024005a48 sp=0xc0240058c8 pc=0x4c7b9da
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eaba60, 0xc0022ea5a0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:681 +0x1b87 fp=0xc024005db8 sp=0xc024005a48 pc=0x4c680f7
github.com/openllb/hlb/codegen.(*CodeGen).EmitChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eaba60, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:228 +0x9c fp=0xc024005e28 sp=0xc024005db8 pc=0x4c634fc
github.com/openllb/hlb/codegen.(*CodeGen).EmitBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc003eaba80, 0x3, 0x4, 0xc003eaba60, ...)
	/go/src/hlb/codegen/codegen.go:203 +0x703 fp=0xc0240060d0 sp=0xc024005e28 pc=0x4c62d33
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc003eaba80, 0x3, 0x4, 0xc003eaba60, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:253 +0xd7 fp=0xc0240061d0 sp=0xc0240060d0 pc=0x4c63737
github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004cd9e0, 0x0, 0x0, 0x0, 0xc003eaba60, 0xc0004f0000, ...)
	/go/src/hlb/codegen/decl.go:45 +0x717 fp=0xc024006388 sp=0xc0240061d0 pc=0x4c79917
github.com/openllb/hlb/codegen.(*CodeGen).EmitAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0xc0004de138, 0x40ec75a, 0xc0004de0d0, 0x203000)
	/go/src/hlb/codegen/decl.go:81 +0xd7 fp=0xc024006408 sp=0xc024006388 pc=0x4c7a1d7
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/decl.go:94 +0xa8 fp=0xc0240064e8 sp=0xc024006408 pc=0x4c7a328
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemExpr(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x0, 0xc0004ce4b0, 0xc003eaba20, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/expr.go:109 +0x3ca fp=0xc024006668 sp=0xc0240064e8 pc=0x4c7b9da
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eaba20, 0xc0022ea360, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:681 +0x1b87 fp=0xc0240069d8 sp=0xc024006668 pc=0x4c680f7
github.com/openllb/hlb/codegen.(*CodeGen).EmitChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eaba20, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:228 +0x9c fp=0xc024006a48 sp=0xc0240069d8 pc=0x4c634fc
github.com/openllb/hlb/codegen.(*CodeGen).EmitBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc003eaba40, 0x3, 0x4, 0xc003eaba20, ...)
	/go/src/hlb/codegen/codegen.go:203 +0x703 fp=0xc024006cf0 sp=0xc024006a48 pc=0x4c62d33
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc003eaba40, 0x3, 0x4, 0xc003eaba20, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:253 +0xd7 fp=0xc024006df0 sp=0xc024006cf0 pc=0x4c63737
github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004cd9e0, 0x0, 0x0, 0x0, 0xc003eaba20, 0xc0004f0000, ...)
	/go/src/hlb/codegen/decl.go:45 +0x717 fp=0xc024006fa8 sp=0xc024006df0 pc=0x4c79917
github.com/openllb/hlb/codegen.(*CodeGen).EmitAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0xc0004de138, 0x40ec75a, 0xc0004de0d0, 0x203000)
	/go/src/hlb/codegen/decl.go:81 +0xd7 fp=0xc024007028 sp=0xc024006fa8 pc=0x4c7a1d7
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/decl.go:94 +0xa8 fp=0xc024007108 sp=0xc024007028 pc=0x4c7a328
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemExpr(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x0, 0xc0004ce4b0, 0xc003eab9e0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/expr.go:109 +0x3ca fp=0xc024007288 sp=0xc024007108 pc=0x4c7b9da
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eab9e0, 0xc0022ea120, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:681 +0x1b87 fp=0xc0240075f8 sp=0xc024007288 pc=0x4c680f7
github.com/openllb/hlb/codegen.(*CodeGen).EmitChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eab9e0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:228 +0x9c fp=0xc024007668 sp=0xc0240075f8 pc=0x4c634fc
github.com/openllb/hlb/codegen.(*CodeGen).EmitBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc003eaba00, 0x3, 0x4, 0xc003eab9e0, ...)
	/go/src/hlb/codegen/codegen.go:203 +0x703 fp=0xc024007910 sp=0xc024007668 pc=0x4c62d33
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc003eaba00, 0x3, 0x4, 0xc003eab9e0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:253 +0xd7 fp=0xc024007a10 sp=0xc024007910 pc=0x4c63737
github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004cd9e0, 0x0, 0x0, 0x0, 0xc003eab9e0, 0xc0004f0000, ...)
	/go/src/hlb/codegen/decl.go:45 +0x717 fp=0xc024007bc8 sp=0xc024007a10 pc=0x4c79917
github.com/openllb/hlb/codegen.(*CodeGen).EmitAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0xc0004de138, 0x40ec75a, 0xc0004de0d0, 0x203000)
	/go/src/hlb/codegen/decl.go:81 +0xd7 fp=0xc024007c48 sp=0xc024007bc8 pc=0x4c7a1d7
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/decl.go:94 +0xa8 fp=0xc024007d28 sp=0xc024007c48 pc=0x4c7a328
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemExpr(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x0, 0xc0004ce4b0, 0xc003eab9a0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/expr.go:109 +0x3ca fp=0xc024007ea8 sp=0xc024007d28 pc=0x4c7b9da
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eab9a0, 0xc0022e3ec0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:681 +0x1b87 fp=0xc024008218 sp=0xc024007ea8 pc=0x4c680f7
github.com/openllb/hlb/codegen.(*CodeGen).EmitChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eab9a0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:228 +0x9c fp=0xc024008288 sp=0xc024008218 pc=0x4c634fc
github.com/openllb/hlb/codegen.(*CodeGen).EmitBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc003eab9c0, 0x3, 0x4, 0xc003eab9a0, ...)
	/go/src/hlb/codegen/codegen.go:203 +0x703 fp=0xc024008530 sp=0xc024008288 pc=0x4c62d33
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc003eab9c0, 0x3, 0x4, 0xc003eab9a0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:253 +0xd7 fp=0xc024008630 sp=0xc024008530 pc=0x4c63737
github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004cd9e0, 0x0, 0x0, 0x0, 0xc003eab9a0, 0xc0004f0000, ...)
	/go/src/hlb/codegen/decl.go:45 +0x717 fp=0xc0240087e8 sp=0xc024008630 pc=0x4c79917
github.com/openllb/hlb/codegen.(*CodeGen).EmitAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0xc0004de138, 0x40ec75a, 0xc0004de0d0, 0x203000)
	/go/src/hlb/codegen/decl.go:81 +0xd7 fp=0xc024008868 sp=0xc0240087e8 pc=0x4c7a1d7
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/decl.go:94 +0xa8 fp=0xc024008948 sp=0xc024008868 pc=0x4c7a328
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemExpr(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x0, 0xc0004ce4b0, 0xc003eab960, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/expr.go:109 +0x3ca fp=0xc024008ac8 sp=0xc024008948 pc=0x4c7b9da
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eab960, 0xc0022e3c80, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:681 +0x1b87 fp=0xc024008e38 sp=0xc024008ac8 pc=0x4c680f7
github.com/openllb/hlb/codegen.(*CodeGen).EmitChainStmt(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc000382e00, 0xc003eab960, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:228 +0x9c fp=0xc024008ea8 sp=0xc024008e38 pc=0x4c634fc
github.com/openllb/hlb/codegen.(*CodeGen).EmitBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0x4fbc146, 0x2, 0xc003eab980, 0x3, 0x4, 0xc003eab960, ...)
	/go/src/hlb/codegen/codegen.go:203 +0x703 fp=0xc024009150 sp=0xc024008ea8 pc=0x4c62d33
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemBlock(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc003eab980, 0x3, 0x4, 0xc003eab960, 0x0, 0x0, ...)
	/go/src/hlb/codegen/codegen.go:253 +0xd7 fp=0xc024009250 sp=0xc024009150 pc=0x4c63737
github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004cd9e0, 0x0, 0x0, 0x0, 0xc003eab960, 0xc0004f0000, ...)
	/go/src/hlb/codegen/decl.go:45 +0x717 fp=0xc024009408 sp=0xc024009250 pc=0x4c79917
github.com/openllb/hlb/codegen.(*CodeGen).EmitAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0xc0004de138, 0x40ec75a, 0xc0004de0d0, 0x203000)
	/go/src/hlb/codegen/decl.go:81 +0xd7 fp=0xc024009488 sp=0xc024009408 pc=0x4c7a1d7
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemAliasDecl(0xc00047e000, 0x51bee80, 0xc0003882c0, 0xc0004d2d80, 0xc0004ce190, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/hlb/codegen/decl.go:94 +0xa8 fp=0xc024009568 sp=0xc024009488 pc=0x4c7a328

goroutine 4 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x5be8220)
	pkg/mod/github.com/golang/[email protected]/glog.go:882 +0x8b
created by github.com/golang/glog.init.0
	pkg/mod/github.com/golang/[email protected]/glog.go:410 +0x272

goroutine 35 [syscall]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:139 +0x9f
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 51 [chan receive]:
github.com/moby/buildkit/util/appcontext.Context.func1.1(0xc000052cc0, 0xc00043c260, 0xc000256320)
	pkg/mod/github.com/hinshun/[email protected]/util/appcontext/appcontext.go:30 +0x38
created by github.com/moby/buildkit/util/appcontext.Context.func1
	pkg/mod/github.com/hinshun/[email protected]/util/appcontext/appcontext.go:28 +0xff

goroutine 52 [select]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000388480)
	pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:69 +0xc1
created by google.golang.org/grpc.newCCBalancerWrapper
	pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:60 +0x16f

goroutine 53 [chan receive]:
google.golang.org/grpc.(*addrConn).resetTransport(0xc00033a000)
	pkg/mod/google.golang.org/[email protected]/clientconn.go:1155 +0x60d
created by google.golang.org/grpc.(*addrConn).connect
	pkg/mod/google.golang.org/[email protected]/clientconn.go:798 +0xc2

goroutine 54 [select]:
github.com/moby/buildkit/util/progress/progressui.DisplaySolveStatus(0x51bee80, 0xc000388500, 0x0, 0x0, 0x0, 0x0, 0x518b540, 0xc000010010, 0xc0002f4300, 0x0, ...)
	pkg/mod/github.com/hinshun/[email protected]/util/progress/progressui/display.go:55 +0x3b8
github.com/docker/buildx/util/progress.NewPrinter.func1(0xc000010010, 0x4fbdc49, 0x5, 0x51bee80, 0xc000388500, 0xc0002f4300, 0xc0003967c0, 0xc0002f4360)
	pkg/mod/github.com/docker/[email protected]/util/progress/printer.go:52 +0x9c
created by github.com/docker/buildx/util/progress.NewPrinter
	pkg/mod/github.com/docker/[email protected]/util/progress/printer.go:46 +0x15c

goroutine 55 [chan receive]:
github.com/openllb/hlb/solver.NewProgress.func1(0x0, 0x0)
	/go/src/hlb/solver/progress.go:107 +0x7f
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0000bfa10, 0xc000396800)
	pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x57
created by golang.org/x/sync/errgroup.(*Group).Go
	pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x66

goroutine 56 [semacquire]:
sync.runtime_Semacquire(0xc0000bfa50)
	/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc0000bfa48)
	/usr/local/go/src/sync/waitgroup.go:130 +0x65
golang.org/x/sync/errgroup.(*Group).Wait(0xc0000bfa40, 0x0, 0x0)
	pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:40 +0x31
github.com/docker/buildx/util/progress.NewMultiWriter.func1(0xc0002f43c0, 0xc0000bfa40, 0x51b29c0, 0xc0003967c0)
	pkg/mod/github.com/docker/[email protected]/util/progress/multiwriter.go:90 +0x42
created by github.com/docker/buildx/util/progress.NewMultiWriter
	pkg/mod/github.com/docker/[email protected]/util/progress/multiwriter.go:88 +0xdf

goroutine 57 [chan receive]:
github.com/openllb/hlb/solver.NewProgress.func2(0x0, 0x0)
	/go/src/hlb/solver/progress.go:120 +0xaa
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0000bfa10, 0xc000396820)
	pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x57
created by golang.org/x/sync/errgroup.(*Group).Go
	pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x66

goroutine 58 [select]:
github.com/docker/buildx/util/progress.(*MultiWriter).WithPrefix.func1(0x8, 0x501f520)
	pkg/mod/github.com/docker/[email protected]/util/progress/multiwriter.go:31 +0x382
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0000bfa40, 0xc0003885c0)
	pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x57
created by golang.org/x/sync/errgroup.(*Group).Go
	pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x66

goroutine 12 [select]:
github.com/docker/buildx/util/progress.(*MultiWriter).WithPrefix.func1(0x8, 0x501f520)
	pkg/mod/github.com/docker/[email protected]/util/progress/multiwriter.go:31 +0x382
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0000bfa40, 0xc0004594c0)
	pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x57
created by golang.org/x/sync/errgroup.(*Group).Go
	pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x66

goroutine 44 [IO wait]:
internal/poll.runtime_pollWait(0x651dd88, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc00043e258, 0x72, 0x8001, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00043e240, 0xc000286000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1f2
os.(*File).read(...)
	/usr/local/go/src/os/file_unix.go:263
os.(*File).Read(0xc0000cc030, 0xc000286000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/file.go:108 +0x70
io.copyBuffer(0x5189600, 0xc0003fa0e0, 0x518b520, 0xc0000cc030, 0xc000286000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/io.go:402 +0x122
io.Copy(...)
	/usr/local/go/src/io/io.go:364
os/exec.(*Cmd).writerDescriptor.func1(0x0, 0x0)
	/usr/local/go/src/os/exec/exec.go:288 +0x63
os/exec.(*Cmd).Start.func1(0xc00011a000, 0xc0003fa140)
	/usr/local/go/src/os/exec/exec.go:409 +0x27
created by os/exec.(*Cmd).Start
	/usr/local/go/src/os/exec/exec.go:408 +0x58f

goroutine 45 [select]:
os/exec.(*Cmd).Start.func2(0xc00011a000)
	/usr/local/go/src/os/exec/exec.go:416 +0xc4
created by os/exec.(*Cmd).Start
	/usr/local/go/src/os/exec/exec.go:415 +0x62d

goroutine 66 [IO wait]:
internal/poll.runtime_pollWait(0x651df28, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc00043e198, 0x72, 0x8001, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00043e180, 0xc00027e000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1f2
os.(*File).read(...)
	/usr/local/go/src/os/file_unix.go:263
os.(*File).Read(0xc0000cc018, 0xc00027e000, 0x8000, 0x8000, 0x0, 0x10401, 0x0)
	/usr/local/go/src/os/file.go:108 +0x70
github.com/docker/cli/cli/connhelper/commandconn.(*commandConn).Read(0xc000118000, 0xc00027e000, 0x8000, 0x8000, 0xc006000030, 0x30, 0x28)
	pkg/mod/github.com/docker/[email protected]/cli/connhelper/commandconn/commandconn.go:196 +0x56
bufio.(*Reader).Read(0xc00043e420, 0xc0004d4e38, 0x9, 0x9, 0xc0004e2000, 0x6151d98, 0x0)
	/usr/local/go/src/bufio/bufio.go:223 +0x23e
io.ReadAtLeast(0x51890e0, 0xc00043e420, 0xc0004d4e38, 0x9, 0x9, 0x9, 0x459fb49, 0xc006000030, 0xc006000004)
	/usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0004d4e38, 0x9, 0x9, 0x51890e0, 0xc00043e420, 0x0, 0x0, 0xc006000030, 0x0)
	pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x88
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0004d4e00, 0xc006000030, 0xc006000000, 0x0, 0x0)
	pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00015a000)
	pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1270 +0x170
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:300 +0xcf0

goroutine 82 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0004cfb30, 0x1, 0x0, 0x0, 0x0, 0x0)
	pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x107
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0005d6000, 0x0, 0x0)
	pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1b6
google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00015a000)
	pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:346 +0x7b
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:344 +0xea8

Support executing multiple targets

There's a few use cases for executing multiple targets:

1. Executing some targets in parallel, some sequentially.
2. Specifying different outputs for each target

Proposal

HLB CLI

You'll be able to specify multiple targets like:

• hlb run --target foo --target bar

Or specify a target which is a group, more on that below.

New type group

Currently, the only available targets are functions that return fs. One common ask to run multiple targets in parallel, and multiple targets sequentially, so for that we propose a new type group which provides that functionality.

All fs can be thought of as a single element group, but group cannot be passed as a fs.

group default() {
    foo
    bar
}

fs foo() {
    # ...
}

fs bar() {
    # ...
}

Like fs, the statements are sequential so foo will execute and complete before we start building the graph for bar.

We also define a group function to execute groups in parallel:

# Runs the given targets in parallel.
#
# @param targets the targets to run in parallel.
# @return a group for parallel execution.
group parallel(variadic group targets)

Using function literals, you can define the build pipeline all in one group:

group foo() {
  parallel group {
    a
    b
  } group {
    c
    d
  }
  e
}

When executing target foo, a and c will start immediately. b will start when a finishes, d will start when c finishes, and e will start when both a, b, c, d have finished.

Since groups are also functions, you can define parameters, like in this concrete example:

group pullrequest() {
	unitTests
	lint
}

group ci() {
	parallel group {
		pullrequest
	} group {
		parallel group {
			integrationTests "staging"
		} group {
			integrationTests "prod"
		}
		publish
	}
}

group integrationTests(string environment) {
	# ...
}

Unfork participle dependency

Currently, we use a fork of participle because we needed to introspect the length of the peeking lexer to calibrate the position of the cursor.

Instead, we should ensure that the parser always try to not error, and with the checker walking the partial CST and producing concrete errors that the report package consumes. Then we never use the peeking lexer for error reporting and thus no need to fork participle. Decoupling will also allow us to implement diagnostics cleanly in the LSP implementation.

hlb faills crossBuild but returns success

From out latest github actions:

So we have two issues, the cacheMounts usage is no longer valid in the checker, and hlb is returning success on checker errors.

unable to use mount as alias with cache directives

Not sure if this is supposed to work or not.

With input:

fs foo() {
	image "busybox"
	run "touch /foo/bar" with option {
		ignoreCache
		mount fs { scratch; } "/foo" with option {
			cache "foo-bar-test" "private"
		} as fooCache
	}
}

If I run target foo it works fine:

$ ./hlb run -t foo ./bugs.hlb
#1 compiling [foo]
#1 DONE 0.0s

#2 docker-image://docker.io/library/busybox:latest
#2 resolve docker.io/library/busybox:latest
#2 resolve docker.io/library/busybox:latest 1.2s done
#2 CACHED

#3 /bin/sh -c touch /foo/bar
#3 DONE 0.2s

However if I try to run the fooCache alias, I get an error:

$ ./hlb run -t fooCache ./bugs.hlb
#1 compiling [fooCache]
#1 0.000 github.com/moby/buildkit/client/llb.(*ExecOp).getMountIndexFn.func1
#1 0.000        /go/pkg/mod/github.com/moby/[email protected]/client/llb/exec.go:399
#1 0.000 github.com/moby/buildkit/client/llb.(*output).ToInput
#1 0.000        /go/pkg/mod/github.com/moby/[email protected]/client/llb/state.go:365
#1 0.000 github.com/moby/buildkit/client/llb.State.Marshal
#1 0.000        /go/pkg/mod/github.com/moby/[email protected]/client/llb/state.go:131
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).Generate
#1 0.000        /go/src/github.com/openllb/hlb/codegen/codegen.go:174
#1 0.000 github.com/openllb/hlb.Compile.func1
#1 0.000        /go/src/github.com/openllb/hlb/hlb.go:91
#1 0.000 github.com/openllb/hlb/solver.(*progressUI).Write.func2.1
#1 0.000        /go/src/github.com/openllb/hlb/solver/progress.go:172
#1 0.000 github.com/openllb/hlb/solver.write
#1 0.000        /go/src/github.com/openllb/hlb/solver/progress.go:196
#1 0.000 github.com/openllb/hlb/solver.(*progressUI).Write.func2
#1 0.000        /go/src/github.com/openllb/hlb/solver/progress.go:171
#1 0.000 golang.org/x/sync/errgroup.(*Group).Go.func1
#1 0.000        /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57
#1 0.000 runtime.goexit
#1 0.000        /go/src/runtime/asm_amd64.s:1373
#1 0.000 Caused by: invalid mount: /foo
#1 ERROR: invalid mount: /foo
------
 > compiling [fooCache]:
#1 0.000 Caused by: invalid mount: /foo
------

Support user defined methods

Currently, users can only define functions:

fs foo() {
    image "busybox"
}

Functions can only be used as source statements (i.e. the first statement in a block surrounded by braces { ... }). Statements following the source modify the value on the stack. For example run "echo hello" is executed with the busybox image. These are known as methods.

We propose an explicit method contract instead of magic first arguments found in some languages. For example, the signature for run is:

fs (fs) run(variadic string args)

The return type is fs and the receiver (fs) is also the same type. Methods must not have any source statements because its assumed that the receiver is already the source.

fs (fs) applyCommonEnv() {
    env "key1" "value1"
    env "key2" "value2"
}

fs default() {
    image "busybox"
    applyCommonEnv
    run "echo $key1"
}

Refactor `report` to consume `checker` errors

Previously the report package consumed errors from participle directly and generated a pretty error report to the user. However, the errors from participle is not stable and positioning is hard to predict and thus prone to error.

We should produce all syntax and semantic errors from the checker and the report produces the pretty print from the errors (which will have lexer.Position) instead.

Call expressions as a shortcut for invoking functions

In the beginning, I designed HLB to invoke functions without parenthesis or commas like so:

fs default() {
    image "alpine"
    env "key" "value"
    run "echo $key > /out/foo" with option {
        mount fs { scratch; } "/out" as fooOut
    }
}

Compare this to invoking with parens:

fs default() {
    image("alpine")
    env("key", "value")
    run("echo $key > /out/foo") with option {
        mount fs { scratch(); } "/out" as fooOut
    }
}

It's a bit verbose and doesn't have the same readability as Dockerfiles so we ended up with what we have now. However, there are some common patterns that emerged:

Running with string interpolated values

run string {
    format "npm run %s" script
} with option {
    ...
}

Mounting local and scratch filesystems

run "go build -o /out/binary ./cmd/my-binary" with option {
    dir "/src"
    mount fs { local "."; } "/src"
    mount fs { scratch; } "/out"
}

Proxying environment variables

option::run awsCredentials() {
    proxyEnv "AWS_ACCESS_KEY_ID"
    proxyEnv "AWS_SECRET_ACCESS_KEY"
    proxyEnv "AWS_SESSION_TOKEN"
}

option::run proxyEnv(string key) {
    env key string { localEnv key; }
}

Having to create a fs { ... } or string { ... } just to invoke one function is often too verbose. It only makes sense if you want to modify something so you end up having multiple statements:

run "make" with option {
    dir "/src"
    mount fs {
        local "."
        rm "./git/objects"
        mkdir "node_modules" 0o755
    } "/src"
}

I propose we support parens-style invocation of functions as an alternative, in order to reduce verbosity in these examples:

Running with string interpolated values

run format("npm run %s", script) with option {
    ...
}

Mounting local and scratch filesystems

run "go build -o /out/binary ./cmd/my-binary" with option {
    dir "/src"
    mount local(".") "/src"
    mount scratch "/out"
}

Proxying environment variables

option::run awsCredentials() {
    proxyEnv "AWS_ACCESS_KEY_ID"
    proxyEnv "AWS_SECRET_ACCESS_KEY"
    proxyEnv "AWS_SESSION_TOKEN"
}

option::run proxyEnv(string key) {
    env key localEnv(key)
}

Unit test bundle for codegen run instruction

Tests should go in codegen/codegen_test.go. The input program should be as minimal as possible. There may be multiple test cases per bullet point. We should test the available options as well.

• shell
• run
• ssh
• secret
• mount
• cache mount

Seg fault on some inputs

This produces a seg fault:

fs default() {
  scratch
  nothing fs { scratch; }
}
fs nothing(fs repo) {
    scratch
}

Remove methods from HLB

Problem

Currently, fs function like the one below:

fs default() {
    image "alpine"
    run "echo hello"
}

Have the following restraints:

• fs body must have at least one statement.
• fs first statement must be a "source" statement (i.e. git, image, http, local, scratch).
• fs statements after the first must be methods. ("source" statements are not allowed).

Contrasted with option blocks:

• option body can be empty.
• option has no "source" statement, any statement is thought of as appending to the list of options.
• option block can invoke an option function, where the options are merged to the list of options.

When also taking #47 into consideration, group blocks are:

• group body can be empty.
• group also has no "source" statement, any statement is thought of as appending to a list of sequential targets.
• group block can invoke a group function, where the groups are merged to the list of sequential targets.

Having different rules for blocks depending on the return type is confusing, and the complexity also shows up in implementation of the compiler, as well as having to explain to users in documentation.

Proposal

I propose we unify these traits in a framework that can be applied to each block. In this proposal, methods are no longer a concept, you can think of every function as a "method" that modifies the value on the stack.

All blocks have the same rules:

• body can be empty
• conceptually, when running a function, the stack has one register which holds the zero-value of that type (fs is llb.Scratch, option is empty list of options). functions modify the value in this register and saves it to the register.

WIP (show examples)

unable to pass anonymous option as argument

fs foo(option::run opts) {
	image "busybox"
	run "echo hi" with opts
}

fs default() {
	foo option::run {
		mount fs { local "."; } "/foo"
	}
}

error:

$ hlb run -t default ./bugs.hlb
#1 compiling [default]
panic: call stmt does not support options

goroutine 1 [running]:
github.com/openllb/hlb/codegen.(*CodeGen).EmitOptions(0xc000128f80, 0x5102020, 0xc0003c8540, 0xc0004d21c0, 0x0, 0x0, 0xc00069a8b0, 0x1, 0x1, 0x4f75f78, ...)
	/go/src/github.com/openllb/hlb/codegen/codegen.go:881 +0xb41
github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncLit(0xc000128f80, 0x5102020, 0xc0003c8540, 0xc0004d21c0, 0xc000128540, 0x0, 0x0, 0x4f75f78, 0x0, 0x1, ...)
	/go/src/github.com/openllb/hlb/codegen/codegen.go:295 +0x3a8
github.com/openllb/hlb/codegen.(*CodeGen).EmitOptionExpr(0xc000128f80, 0x5102020, 0xc0003c8540, 0xc0004d21c0, 0x0, 0x0, 0xc000112eb0, 0xc000231560, 0x8, 0xc0002de2d8, ...)
	/go/src/github.com/openllb/hlb/codegen/expr.go:151 +0xae
github.com/openllb/hlb/codegen.(*CodeGen).ParameterizedScope(0xc000128f80, 0x5102020, 0xc0003c8540, 0xc0004d21c0, 0xc0001e29a0, 0x0, 0x0, 0xc00050e840, 0xc00069a7c0, 0x1, ...)
	/go/src/github.com/openllb/hlb/codegen/decl.go:138 +0x5f9
github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl(0xc000128f80, 0x5102020, 0xc0003c8540, 0xc0004d21c0, 0xc00050e840, 0xc0001e29a0, 0x0, 0x0, 0x4f75f78, 0x0, ...)
	/go/src/github.com/openllb/hlb/codegen/decl.go:22 +0xe8
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemChainStmt(0xc000128f80, 0x5102020, 0xc0003c8540, 0xc0004d21c0, 0x4f127ac, 0x2, 0xc0001e29a0, 0x4f75f78, 0x4ef7080, 0xc000129000, ...)
	/go/src/github.com/openllb/hlb/codegen/codegen.go:822 +0x516
github.com/openllb/hlb/codegen.(*CodeGen).EmitChainStmt(0xc000128f80, 0x5102020, 0xc0003c8540, 0xc0004d21c0, 0x4f127ac, 0x2, 0xc0001e29a0, 0x4f75f78, 0x0, 0xc0002decb0, ...)
	/go/src/github.com/openllb/hlb/codegen/codegen.go:184 +0x9f
github.com/openllb/hlb/codegen.(*CodeGen).EmitBlock(0xc000128f80, 0x5102020, 0xc0003c8540, 0xc0004d21c0, 0x4f127ac, 0x2, 0xc00069a8a8, 0x1, 0x1, 0x4f75f78, ...)
	/go/src/github.com/openllb/hlb/codegen/codegen.go:156 +0x1c5
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemBlock(0xc000128f80, 0x5102020, 0xc0003c8540, 0xc0004d21c0, 0xc00069a8a8, 0x1, 0x1, 0x4f75f78, 0x0, 0x0, ...)
	/go/src/github.com/openllb/hlb/codegen/codegen.go:271 +0xd7
github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl(0xc000128f80, 0x5102020, 0xc0003c8540, 0xc0004d2140, 0xc0002314a0, 0xc0001e2af0, 0x0, 0x0, 0x4f75f78, 0xc000299902, ...)
	/go/src/github.com/openllb/hlb/codegen/decl.go:45 +0x2ac
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemFuncDecl(0xc000128f80, 0x5102020, 0xc0003c8540, 0xc0004d2140, 0xc0002314a0, 0xc0001e2af0, 0x4f75f78, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/openllb/hlb/codegen/decl.go:56 +0xba
github.com/openllb/hlb/codegen.(*CodeGen).Generate(0xc000128f80, 0x5102020, 0xc0003c8540, 0xc00068f630, 0xc00069a898, 0x1, 0x1, 0x405efa0, 0xc0000a6ae0, 0xc0002df308, ...)
	/go/src/github.com/openllb/hlb/codegen/codegen.go:94 +0x3fb
github.com/openllb/hlb.Compile.func1(0xc0000a6ae0, 0xc0002df380)
	/go/src/github.com/openllb/hlb/hlb.go:136 +0xd9
github.com/docker/buildx/util/progress.Write(0x50f6be0, 0xc0004d2300, 0xc0003eb580, 0x13, 0xc0002df798)
	/go/pkg/mod/github.com/docker/[email protected]/util/progress/writer.go:32 +0x257
github.com/openllb/hlb.Compile(0x5102020, 0xc0003c8540, 0xc00069a2d8, 0xc00026aa80, 0xc000699e60, 0x1, 0x1, 0x50cf3e0, 0xc00069a2b0, 0x0, ...)
	/go/src/github.com/openllb/hlb/hlb.go:148 +0x962
github.com/openllb/hlb/cmd/hlb/command.Run(0x5102020, 0xc0003c8540, 0xc00069a2d8, 0x50e4ae0, 0xc00069a2b0, 0x0, 0xc00046e340, 0x1, 0x1, 0x0, ...)
	/go/src/github.com/openllb/hlb/cmd/hlb/command/run.go:148 +0x6bc
github.com/openllb/hlb/cmd/hlb/command.glob..func5(0xc0003c84c0, 0x0, 0x0)
	/go/src/github.com/openllb/hlb/cmd/hlb/command/run.go:55 +0x415
github.com/urfave/cli/v2.(*Command).Run(0x59bfbc0, 0xc0003c80c0, 0x0, 0x0)
	/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:161 +0x4e0
github.com/urfave/cli/v2.(*App).RunContext(0xc000622300, 0x5102060, 0xc00019e008, 0xc0001a2000, 0x5, 0x5, 0x0, 0x0)
	/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:302 +0x814
github.com/urfave/cli/v2.(*App).Run(...)
	/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:211
main.main()
	/go/src/github.com/openllb/hlb/cmd/hlb/main.go:12 +0x67

Add string function to join variadic strings

Implement the following function:

# Join concatenates elems to create a single string. The separator string sep
# is placed between elements in the resulting string.
#
# @param sep a separator string placed between elements.
# @param elems the elements to concatenate together.
string join(string sep, variadic string elems)

How to use option::run secret?

From the run example:
https://openllb.github.io/hlb/reference/#fs-fs-runstring-arg

It appears like I should be able to do:

fs default() {
  image "busybox:latest"
  dir "/dir"
  run "ls" with option {
    secret "./tests" "/dir"
  }

The usage is:

option::run secret(string localPath, string mountPoint)

but there is no local context associated with this function, so I am not sure where it would be loading localPath from. The error message implies that the first argument is some label but I am unclear how to set this up:

$ ./hlb run --log-output plain ./tests/bug.hlb
#1 compiling ./tests/bug.hlb
#1 DONE 0.0s

#2 docker-image://docker.io/library/busybox:latest
#2 resolve docker.io/library/busybox:latest
#2 resolve docker.io/library/busybox:latest 1.1s done
#2 CACHED

#3 ls
#3 ERROR: secret ./tests not found: not found

Allow invoking functions with zero args without function literal wrapper.

Currently, when you want to pass the result of a function as an argument to another you will have to wrap it in a function literal:

fs foo() {
    image "alpine"
}

fs default() {
    scratch
    copy fs { foo; } "/etc/os-release" "/os-release"
}

I propose that for functions that have zero arguments, you can call them without the wrapper. Of course for functions that have arguments, we need the function literal in order to disambiguate which arguments belong to which call expression.

fs foo() {
   image "alpine"
}

fs default() {
   scratch
   copy foo "/etc/os-release" "/os-release"
}

Discussion around optional arguments to functions

As we develop common modules in: https://github.com/openllb/modules

We want to expose as many knobs to ensure that it is reusable. However, at the same time we want to limit the number of required arguments so that you can invoke the module without being exposed to the full spectrum of options.

There's two approaches I've been thinking about:

1. Optional arguments
2. Option signatures

Optional arguments

Starting with some implementations from some languages:

# Named arguments in ruby
def test(var1: "var1", var2: "var2", var3: "var3")
  puts "#{var1} #{var2} #{var3}"
end

test(var3:"var3-new", var1: 1111, var2: 2222) # ok => 1111 2222 var3-new

# Default values for arguments in ruby
def test(var1="var1", var2="var2", var3="var3")
  puts "#{var1} #{var2} #{var3}"
end

test(var3:"var3-new", var1: 1111, var2: 2222) # ok but ... {:var3=>"var3-new", :var1=>1111, :var2=>2222} var2 var3

or python optional arguments

def info(object, spacing=10, collapse=1):
  # ...

info(odbchelper)
info(odbchelper, 12)
info(odbchelper, collapse=0)
info(spacing=15, object=odbchelper)

The key features extracted from the examples is:

• Positional arguments strictly before named arguments
• Named arguments can be in any order
• Values can be assigned to arguments even if they don't have default values set

In HLB, this may look like:

fs foo(string first) {
    # ...
}

fs bar(string first, string second="world") {
    # ...
}

group default() {
    foo "hello"
    foo first="hello"
    bar "hello"
    bar second="world" first="hello"
}

Option signatures

Currently, builtin functions like run can take options that add optional behavior to a function. And you can define functions that wrap these builtin options too:

option::run commonOptions() {
    env "key" "value"
    dir "/src"
}

fs default() {
    image "alpine"
    run "echo foo" with commonOptions
}

User defined functions cannot have options, which makes for a odd experience. Here's some explorations into how we can make this work:

Idea 1

fs foo(string requiredArg) {
    image baseImage="alpine:3.9"
}

fs default() {
    foo "hello" with option {
        baseImage "alpine:3.9"
    }
}

Idea 2

# Looking awful like optional arguments
fs foo(string requiredArg) optional (string baseImage="alpine:3.9") {
    image baseImage
}

fs default() {
    foo "hello" with option {
        baseImage "alpine:3.9"
    }
}

Resolve image configs server side

Currently, compiling HLB takes a while because if resolve is specified like:

fs default() {
    image "alpine" with option { resolve; }
}

Then the image config is fetched from the registry on the client side. This is due to the implementation of llb.Image and the imagemetaresolver.WithDefault we use.

We should be able to pass in a gateway client as a the resolver, allowing the server side (buildkitd) to resolve, and also have server side caching.

Add `auto` log-output for non-console environments.

Some environments like Jenkins, doesn't have a tty available. We need an auto log-outputter like BuildKit, which will switch between tty and plain depending if a tty is available on stdin.

variadic option arguments not working

This works without variadic arguments:

fs foo(option::run opts) {
	image "busybox"
	run "echo hi" with option {
		opts
	}
}

fs default() {
	foo option::run {
		ignoreCache
	}
}

However when I introduce variadic I cannot call foo with or without arguments. Here is an example without arguments:

fs foo(variadic option::run opts) {
	image "busybox"
	run "echo hi" with option {
		opts
	}
}

fs default() {
	foo
}

Produces:

#1 compiling [default]
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl
#1 0.000 	/go/src/github.com/openllb/hlb/codegen/decl.go:19
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemChainStmt
#1 0.000 	/go/src/github.com/openllb/hlb/codegen/codegen.go:923
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitChainStmt
#1 0.000 	/go/src/github.com/openllb/hlb/codegen/codegen.go:335
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitBlock
#1 0.000 	/go/src/github.com/openllb/hlb/codegen/codegen.go:301
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemBlock
#1 0.000 	/go/src/github.com/openllb/hlb/codegen/codegen.go:422
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl
#1 0.000 	/go/src/github.com/openllb/hlb/codegen/decl.go:47
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemFuncDecl
#1 0.000 	/go/src/github.com/openllb/hlb/codegen/decl.go:58
#1 0.000 github.com/openllb/hlb/codegen.(*CodeGen).Generate
#1 0.000 	/go/src/github.com/openllb/hlb/codegen/codegen.go:156
#1 0.000 github.com/openllb/hlb.Compile.func1
#1 0.000 	/go/src/github.com/openllb/hlb/hlb.go:91
#1 0.000 github.com/openllb/hlb/solver.(*progressUI).Write.func2.1
#1 0.000 	/go/src/github.com/openllb/hlb/solver/progress.go:172
#1 0.000 github.com/openllb/hlb/solver.write
#1 0.000 	/go/src/github.com/openllb/hlb/solver/progress.go:196
#1 0.000 github.com/openllb/hlb/solver.(*progressUI).Write.func2
#1 0.000 	/go/src/github.com/openllb/hlb/solver/progress.go:171
#1 0.000 golang.org/x/sync/errgroup.(*Group).Go.func1
#1 0.000 	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57
#1 0.000 runtime.goexit
#1 0.000 	/usr/local/go/src/runtime/asm_amd64.s:1373
#1 0.000 Caused by: foo expected args [variadic option::run opts], found []
#1 ERROR: foo expected args [variadic option::run opts], found []
------
 > compiling [default]:
#1 0.000 Caused by: foo expected args [variadic option::run opts], found []
------

And here is an error with arguments:

fs foo(variadic option::run opts) {
	image "busybox"
	run "echo hi" with option {
		opts
	}
}

fs default() {
	foo option::run {
		ignoreCache
	}
}

Produces:

./bugs.hlb:9:6: expected arg to be type option, found option::run

hlb exits with 0 status when building target that does not exist

$ ./hlb --addr docker-container://buildkitd run --log-output plain --target bogus source.hlb
#1 compiling source.hlb
#1 ERROR: unknown target "bogus"
------
 > compiling source.hlb:
------
$ echo $?
0

When building bogus target that does not exit, the program is send to buildkit. This should be a checker error and should exit non-zero.

hlb parse error hidden in output

When testing errors in parsing, I noticed when the log-output is tty, the error message is masked:

$ ./hlb --addr docker-container://buildkitd run ./tests/bug.hlb
[+] Building 0.2s (1/1) FINISHED
 => ERROR compiling ./tests/bug.hlb                                                                                                                                                 0.0s
------
 > compiling ./tests/bug.hlb:
------

But we can see it when log-output is plain:

$ ./hlb --addr docker-container://buildkitd run --log-output plain ./tests/bug.hlb
#1 compiling ./tests/bug.hlb
#1 ERROR: ./tests/bug.hlb:3:3: unexpected filesystem statement: "nothing"
------
 > compiling ./tests/bug.hlb:
------

Split off `codegen.Debugger` into its own package

Currently, the debugger sits inside the codegen package. We should refactor it out to a debug package.

panic from named option::run

Input:

fs default() {
  image "alpine:latest"
  run "pwd" with myopt
}

option::run myopt() {
  dir "/tmp"
}

Error:

$ ./hlb run ./tests/bug.hlb
panic: unknown with option kind

goroutine 1 [running]:
github.com/openllb/hlb/codegen.(*CodeGen).EmitWithOption(0xc000433f40, 0x1ac19c0, 0xc000452940, 0xc00059b620, 0xc000443810, 0xc000453b40, 0x19c79a8, 0xc00025ea28, 0x101671f, 0xd, ...)
        /go/src/hlb/codegen/codegen.go:429 +0x206
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemChainStmt(0xc000433f40, 0x1ac19c0, 0xc000452940, 0xc00059b620, 0x198a535, 0x2, 0xc000443810, 0x19c79a8, 0xc000595ec0, 0x0, ...)
        /go/src/hlb/codegen/codegen.go:440 +0xa5
github.com/openllb/hlb/codegen.(*CodeGen).EmitChainStmt(0xc000433f40, 0x1ac19c0, 0xc000452940, 0xc00059b620, 0x198a535, 0x2, 0xc000443810, 0x19c79a8, 0x0, 0x19c79a8, ...)
        /go/src/hlb/codegen/codegen.go:196 +0x9c
github.com/openllb/hlb/codegen.(*CodeGen).EmitBlock(0xc000433f40, 0x1ac19c0, 0xc000452940, 0xc00059b620, 0x198a535, 0x2, 0xc000433ff0, 0x2, 0x2, 0x19c79a8, ...)
        /go/src/hlb/codegen/codegen.go:171 +0x611
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemBlock(0xc000433f40, 0x1ac19c0, 0xc000452940, 0xc00059b620, 0xc000433ff0, 0x2, 0x2, 0x19c79a8, 0x0, 0x0, ...)
        /go/src/hlb/codegen/codegen.go:221 +0xd7
github.com/openllb/hlb/codegen.(*CodeGen).EmitFuncDecl(0xc000433f40, 0x1ac19c0, 0xc000452940, 0xc00059b5e0, 0xc00047bf80, 0xc00025f8c8, 0x0, 0x0, 0x19c79a8, 0xc00044ab50, ...)
        /go/src/hlb/codegen/decl.go:45 +0x2ac
github.com/openllb/hlb/codegen.(*CodeGen).EmitFilesystemFuncDecl(0xc000433f40, 0x1ac19c0, 0xc000452940, 0xc00059b5e0, 0xc00047bf80, 0xc00025f8c8, 0x19c79a8, 0x0, 0x0, 0x0, ...)
        /go/src/hlb/codegen/decl.go:56 +0xba
github.com/openllb/hlb/codegen.(*CodeGen).Generate(0xc000433f40, 0x1ac19c0, 0xc000452940, 0xc00025f8c8, 0xc00030ae10, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/hlb/codegen/codegen.go:71 +0x2dc
github.com/openllb/hlb.Compile.func1(0xc0000a0cc0, 0xc00025f628)
        /go/src/hlb/hlb.go:74 +0x12a
github.com/docker/buildx/util/progress.Write(0x1abb240, 0xc00059b780, 0xc000370f20, 0x19, 0xc00025f830)
        /go/pkg/mod/github.com/docker/[email protected]/util/progress/writer.go:32 +0x257
github.com/openllb/hlb.Compile(0x1ac19c0, 0xc000452940, 0xc000010868, 0xc000464360, 0x198d93b, 0x7, 0x1aa9ec0, 0xc000010838, 0x0, 0x0, ...)
        /go/src/hlb/hlb.go:86 +0x637
github.com/openllb/hlb/cmd/hlb/command.Run(0x1ac19c0, 0xc000452940, 0xc000010868, 0x1ab5e40, 0xc000010838, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/hlb/cmd/hlb/command/run.go:143 +0x2cd
github.com/openllb/hlb/cmd/hlb/command.glob..func5(0xc000452900, 0x0, 0x0)
        /go/src/hlb/cmd/hlb/command/run.go:71 +0x53c
github.com/urfave/cli/v2.(*Command).Run(0x2073fa0, 0xc0004525c0, 0x0, 0x0)
        /go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:161 +0x4e0
github.com/urfave/cli/v2.(*App).RunContext(0xc000001c80, 0x1ac1a00, 0xc0000380c8, 0xc0000321b0, 0x3, 0x3, 0x0, 0x0)
        /go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:302 +0x814
github.com/urfave/cli/v2.(*App).Run(...)
        /go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:211
main.main()
        /go/src/hlb/cmd/hlb/main.go:12 +0x67

Concurrent map write issue when running multi targets

There seems to be a concurrent map write panic coming from the BuildKit golang packages. Update this issue with the panic stack trace when we see it again.

Running HLB from stdin using relative imports fails

$ cat ./source.hlb | hlb --addr docker-container://buildkitd run --target lint
/dev/stdin:1:1: no such file ./go.hlb

Add test suite for `checker`

The checker package has no tests at the moment. This is brittle and we should write a test suite asap. Ideally, the test suite is in raw string of HLB source code, and we are checking for the right checker error to be returned.

Add `--version/--v` flag to show version.

We should add a package version with some golang constants that we replace at build time using ldflags.

Similar to: https://github.com/containerd/containerd/blob/master/version/version.go

Add integration tests for HLB

Perhaps we can borrow the integration package to create BuildKitd sandboxes locally and run integration tests via go test.

support doublestar syntax for includePatterns and excludePatterns

Use case: I want to import only .go files from a build tree to run go build.

Suggestion:

include "." with option {
  includePatterns "**/*.go"
}

that would include all files in the current directory and down that match *.go. I don't think llb supports this directly but I think can emulate it fairly efficiently in hlb.

Currently, if I wanted to implement something like this I think it would have to be:

include "." with option {
  includePatterns "*.go" "*/*.go" "*/*/*.go" "*/*/*/*.go" "*/*/*/*/*.go" "*/*/*/*/*/*.go" "*/*/*/*/*/*/*.go" "*/*/*/*/*/*/*/*.go" "*/*/*/*/*/*/*/*/*.go"
}