Support import / export of external HLB modules, paving the way for HLB libraries about hlb

[hinshun]

• What format is the lockfile? Is it intended to be parsed by 3rd party tooling, or should read operations run through the HLB tool?

The lockfile for hlb is more like vendoring directories but I wasn't sure if the term vendoring only exists in golang? When you lock your modules down, you are basically creating a vendored copy into your repository which you will check in. Since HLB dependencies can only be retrieved by solving, you can't regenerate an identical vendor directory unless you control all the remote sources (docker registries, http servers, git repositories, etc).

When vendored, it may look something like:

❯ tree .hlb
.hlb
└── modules
    └── sha256
        └── 9f
            └── 9ff8d8578c06000a3aa7027e00129524bd3460df2bb6ea3dde0f3bbfd90e4ecc
                └── module.hlb

• If a single module locks its dependencies, how can that lock be shared? Should it be committed alongside the .hlb file?

The .hlb/modules directory is rooted in the same working directory as the .hlb file it was locked for. If that .hlb file has siblings, and they have an import with the same fs function literal (the vertex hashes to the same digest) then the locked import will be shared.

Yes, the .hlb/modules directory should be committed alongside.

• Given import foo from fs { image "myimage"; } use the module.hlb file from myimage; would locks also be shipped in that image and used for second-level dependencies?

Good point to discuss. In languages like go or node, the common practice is to not lock library dependencies, but it is an option to the library author. Perhaps the convention here should be the same; i.e. HLB libraries should try not to ship with the .hlb/modules directory so that the end-user can decide how to lock them but if they did, their dependencies will be using their own locked modules.

[slushie]

A few questions about the lockfile:

• What format is the lockfile? Is it intended to be parsed by 3rd party tooling, or should read operations run through the HLB tool?
• If a single module locks its dependencies, how can that lock be shared? Should it be committed alongside the .hlb file?
• Given import foo from fs { image "myimage"; } use the module.hlb file from myimage; would locks also be shipped in that image and used for second-level dependencies?

[slushie]

I think vendoring is a generic enough term. And I think I conflated the "lock" concept with "lock files" in my mind -- it makes much more sense to me to think of it as a vendor directory.

Is it possible to store metadata about the actual source for each locked module? In other words, if I later run hlb module tree would it show me eg, import foo from fs { image "bar"; } as the source of some particular digest hash? Will it also include eg, resolved image "bar:latest" to "bar@sha256:deadbeef"?

[hinshun]

I think vendoring is a generic enough term. And I think I conflated the "lock" concept with "lock files" in my mind -- it makes much more sense to me to think of it as a vendor directory.

How about hlb module vendor then?

Is it possible to store metadata about the actual source for each locked module? In other words, if I later run hlb module tree would it show me eg, import foo from fs { image "bar"; } as the source of some particular digest hash? Will it also include eg, resolved image "bar:latest" to "bar@sha256:deadbeef"?

We could show the whole inlined import but it may get pretty long. The current idea is to just show the import name go and the digest of the vertex:

.
└── [go]  sha256:9ff8d8578c06000a3aa7027e00129524bd3460df2bb6ea3dde0f3bbfd90e4ecc

Unfortunately, the function literal is opaque so unless I start introspecting the LLB, I wouldn't know what the source is, be it image or git, etc. Also I'm not sure how to extract cache keys of specific vertices atm.

[hinshun]

Changed hlb mod lock to hlb mod vendor. I think its more intuitive.