brioche-dev / brioche

A delicious package manager

Brioche

Brioche is a package manager and build tool for building and running your most complex software projects.

import * as std from "std";
import { cargoBuild } from "rust";

export default () => {
  // Build a Rust project
  const server = cargoBuild({
    crate: Brioche.glob("src", "Cargo.*"),
  });

  // Put it in a container image
  return std.ociContainerImage({
    recipe: server,
    entrypoint: ["/bin/server"],
  });
}

Features

• Caching - All build artifacts are saved and re-used between builds, saving you time when only a part of your project changes. You can even share the cache between machines using a self-hosted registry
• Lockfiles - All of your dependencies are automatically saved in a lockfile, making your builds super reliable for everyone on your team
• TypeScript - Build scripts are written in TypeScript, giving you the flexibility of a familiar full programming language (with great type checking and editor completions!)
• Cross-ecosystem - Build your project regardless of language, and easily mix different languages and ecosystems in one project
• Cross-compilation (work in progress) - Easily build your project from any platform, to any platform
• Dev environments (work in progress) - Set up a suite of tools to help onboard new team members quickly

Installation

Run this in your terminal for quick installation:

curl --proto '=https' --tlsv1.2 -sSfL 'https://brioche.dev/install.sh' | bash

...or check out the official docs on Installation more installation options.

To install from source, simply check out this repo and run cargo install --locked --path .. This will install Brioche into ~/.cargo/bin. You can also run it as a normal Rust project using cargo run.

Packages

See the brioche-packages repo for Brioche packages

brioche's Issues

Add option to force-rerun build

All build outputs, and while that's convenient in most cases, sometimes it would be really handy to rerun an already-cached build.

The are some challenges with this feature:

1. Deciding what to re-run. You wouldn't ever want to rebuild std from source if you can avoid it, so there needs to be some way to decide which things to rebuild and which things not to rebuild
   • We do track "metadata" for most/all recipes, so deciding what to rebuild will likely involve inspecting the recipe metadata, and ignoring the cache if it meets some condition
2. How to treat successes or failures. If a build succeeds but leads to a different output, do we prefer the old output or the new output? The old output is more logically consistent, but picking the new output is probably what most people would expect

A quick-and-dirty workaround that I've done a lot: just add a simple .env({ WATERMARK: "asdf" }) when you want to force-rerun a process. Using any random env value will lead to a new process recipe, which will force it to build. If the output of that process is the same, then anything dependent on it won't rebuild, unless you add more new env vars.

macOS

Would the maintainers be open to patches working towards implementing support on macOS?

Perhaps a good first issue to work on would be a version of the sandboxing mechanism or the packed executable mechanism that works on macOS?

Add support for unarchiving `.zip` archives

The unarchive recipe (e.g. std.download({ ... }).unarchive("tar", "gzip")) supports multiple compression formats, but currently only supports tar archives. Zip archives are also pretty common, so we should update the recipe to also support zip as an archive type

const source = std
  .download({
    url: "https://github.com/brioche-dev/brioche/archive/refs/tags/v0.1.1.zip",
    hash: std.sha256sum("1e98a694fe530de5d641ecfa03c0f2f61ba7eba020866a1876ab2c4eaffb1525"),
  })
  .unarchive("zip");

Zip files can themselves be compressed, so there isn't an extra layer of decompression to worry about like you often see with .tar files. In other words, unarchiving a zip file will usually not specify a compression format. However, for consistency, it might still make sense to allow specifying a decompression format, even though I wouldn't expect it to be used in practice

Add support for more project metadata

The top-level export const project in a project can currently include the fields name and version (extra fields can be included, but aren't semantically meaningful). We should support some other useful fields too:

• website: Link to official project website
• license: SPDX license specifier

There might be some other fields like repository, issueTracker, authors, etc that might also be worth adding, but website and license are a minimum good starting point as many users are likely going to want to know these details at a glance. Adding these details now will also be helpful if/when the registry has an interface for exploring packages, where listing the license and website details up-front will be helpful

Licensing details can also get complicated, such as ffmpeg builds that can be legally-encumbered and not redistributable depending on what codecs are enabled at build time. We can punt on handling these kinds of edge cases for now, and just say that the license field should be set based on the project's license and source code, and should only be set if it's unambiguous what the resulting license will be for every build

Test Issue 1

Testing Discord hook

[LSP server] Add inlay hint

Add support for Inlay hint request on server side.

More and more LSP servers support this feature overtime, and it's definitively a killer feature for the end user, since it embeds the types of most symbols directly in the file viewer.

[CLI] Ability to provide multiple packages

I was wondering if it would be possible to extend the current command line to set the parameter '-p, --project' multiple times:

> brioche install -p packages/openssl -p packages/rust
error: the argument '--project <PROJECT>' cannot be used multiple times

Installing more than one package at a time would be simplified, and prevent calling brioche more than needed. Few similar package managers allow this pattern, such as Nix:

> nix-shell -p oniguruma -p pssh

[LSP server] Add a new rule to detect mismatch project name

When creating a new Brioche project, it could be nice to have a rule that checks if the project.name corresponds to the file name:

• Should warn:

// foo.bri
export const project = {
  name: "bar",
  version: "0.0.1",
};

• Should not warn:

// foo.bri
export const project = {
  name: "foo",
  version: "0.0.1",
};

[Errors] Add more context when file permission cannot be set

I was trying to build locally the toolchain, but I'm facing an issue which prevents the output file to have the correct permission to be set:

> brioche build -p packages/std -e toolchain -o toolchain
[100%] Fetched 3 blobs + 6 recipes from registry
Build finished, completed 1 job in 10.46s
Result: bb53690d859a874183e8a2b071eadbe73c5402bf33bf8359c2f1cc701ee13d54
Writing output
Error: failed to set output file permissions

Caused by:
    Permission denied (os error 13)

I could use strace to trace the system call, and see where it fails (which I'll do), but for convenience, this message should embed more context about which file is being impacted here.

Network errors when installing hellow world

If I try

cargo run -r  -- install -r hello_world

I get a timeout:

        ERROR run_install:bake:bake_inner:run_bake: brioche_core::bake: error=Request error: error sending request for url (https://registry.brioche.dev/v0/blobs/85b898d9dda3158822f5a0c25e71bc5dceb83c06620b4a0f7bf40370a362556f.zst?brioche=0.1.0): error sending request for url (https://registry.brioche.dev/v0/blobs/85b898d9dda3158822f5a0c25e71bc5dceb83c06620b4a0f7bf40370a362556f.zst?brioche=0.1.0): operation timed out scope=Project { project_hash: ProjectHash(Hash("d0aeff98c15b839e2d99e04c023e84414c48e249eb6e2c695696685c42e3d355")), export: "default" } recipe_hash=39cf103288f270b8c774b4d9c5d5b4d10fff20c5da3faf8a0e2721257cc4773e recipe_kind=Process recipe_hash=39cf103288f270b8c774b4d9c5d5b4d10fff20c5da3faf8a0e2721257cc4773e recipe_kind=Process

If I manually bump in-code timeouts to 60s from 10s, I then get some error about "temporary DNS failure" (sadly, lost the exact text of error somewhere in the git history)

Use FUSE filesystem to speed up sandboxed Linux builds

When a sandboxed Linux process runs, all of its inputs are written to disk first so that they can be bind-mounted into the directory used as the rootfs for the process. Even though we use hardlinks as much as possible, the process of creating these local directory structures can take a significant amount of time.

Here's a test case today:

// project.bri
import * as std from "std";

export default () => {
  return std.runBash`
    echo "$large_input"
    touch "$BRIOCHE_OUTPUT"
  `.env({
    large_input: std.directory({
      a: std.toolchain(),
      b: std.toolchain(),
    }),
  });
};

Building this project locally takes about 20 seconds when uncached for me (this assumes std.toolchain() already exists as a local but large_input does not). The majority of that time is just creating the directory structure for large_input, even though all files within that directory are hardlinks to existing blobs.

We definitely need to work on optimizing this, and there are probably lots of low-hanging fruit in how we create these directory structures.

But, one future optimization we can do is to use a FUSE filesystem to avoid needing to create this directory structure entirely. That is, we have a small FUSE filesystem (likely a custom implementation) where, when reading a path like ${large_input}/a/bin/gcc, it forwards that request to ~/.local/share/brioche/blobs/SOME_HASH. This has several advantages:

• The directory structure never needs to get written to disk
• Avoid creating executable/non-executable copies of blobs (e.g. identical blobs that differ only by the .x prefix and the executable permission)
• As a future improvement, we can even map other paths not in the blobs directory into the filesystem, e.g. mapping Brioche.includeFile files directly without copying into a blob

FUSE has some issues, namely that I/O performance is worse than a normal filesystem, it's Linux-only (effectively), and it's not enabled by default in all Linux distros. For these reasons, if/when we implement FUSE, we'll still need to fallback to the current "create then bind-mount a local directory" implementation that we use today.

[CLI] Print files that were just formatted

As of today, to know which files are formatted by brioche fmt, you need to run first the command with the check argument, or you could rely on some version control tools to see the diff.

Symmetrical to what is printed when the brioche fmt --check is run and there are files that are not formatted:

The following files are not formatted:
- foo
- bar

We could have:

The following files have been formatted:
- foo
- bar

Or at least a number of files that were formatted: 1/32 files have been formatted.

What I wanted is just a way to inform that the formatting did actually format some files.

Consider Zulip/Discourse for communication?

I see that brioche is using discord for communication, and I am wondering how strong is that decision. If you are confident in this choice, then I'll adapt! But if you are open to reconsidiration, I'll suggest taking a closer look at Zulip and Discourse.

I am not sure whether is specific to me personally, or is a reflection of a wider demographics, but I can say that:

• I participate in a fair number of Zulip and discourse instances
• I mostly avoid Discord and matix, unless I absolutely have to

The main issue with Discord/IRC/matix for me is that I am really not a fan of real-time communication, both because I am not in a US timezone, and because it's just psychologically easier for me to batch my communication.

The secondary issue is that Discord just has too many features --- it's an advanced app which does all kinds of things, while, for technical communication, I really do prefer just to have a markdown text area.

Overall, my personal experience sums up to maybe 0.9 level of confidence that Zulip in particular fosters better technical communication. What I don't know though, is the strength of network effects. Maybe everyone except me is on Discrord already, which would lower the barrier to entry

I probably should have opened a discussion about this on Discord, but, for some reason, it fails to log me in 🤷

[LSP server] Question: how imports should be formatted ?

I encountered both import statements in the repository https://github.com/brioche-dev/brioche-packages:

import { cargoBuild } from "rust";

// vs

import cargoBuild from "rust";

Both are valid, the point is more on: should we enforce one?

Fix last line of output sometimes getting eaten

See #86 (comment)

Fully static build by default

Trying out brioche on NixOS gives the expected result:

λ ~/.local/bin/brioche 
Could not start dynamically linked executable: /home/matklad/.local/bin/brioche
NixOS cannot run dynamically linked executables intended for generic
linux environments out of the box. For more information, see:
https://nix.dev/permalink/stub-ld
λ ldd ~/.local/bin/brioche
	linux-vdso.so.1 (0x00007ffd7b584000)
	libc.so.6 => /nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/libc.so.6 (0x00007f805a413000)
	/lib64/ld-linux-x86-64.so.2 => /nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib64/ld-linux-x86-64.so.2 (0x00007f805fa59000)
	liblzma.so.5 => not found
	libgcc_s.so.1 => /nix/store/1q9vc0lq7qjlfjz47mfmlzdf86c543jy-xgcc-13.2.0-libgcc/lib/libgcc_s.so.1 (0x00007f805fa32000)
	libm.so.6 => /nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/libm.so.6 (0x00007f805a330000)

For the kind of tool brioche is, it seems that providing a fully statically-liked executable might be beneficial?

Upgrade Deno Core and Rust toolchain

Brioche is currently pinned to Rust 1.77 due to a bug in Deno Core that has been solved upstream (in denoland/deno_core#735). We should upgrade to the latest version of Deno Core (or at least a new enough version to incorporate that fix), which will then allow us to upgrade the Rust toolchain used for Brioche

Add `Brioche.download` to download URLs without hard-coding a SHA hash

Currently, downloading a file with Brioche means manually including the hash of the file's contents in Brioche TypeScript explicitly. In practice, I tend to just fill in a dummy hash, run the CLI, then copy/paste the "real" hash from the CLI's error message. This is not a good UX

Instead, we could leverage the lockfile to automatically record downloads with an unknown hash. There are some nuances that make this more complicated than it first sounds, but nothing insurmountable.

The core idea is that we could update Brioche TypeScript to support "hard-coded" download expressions like this:

const example = Brioche.download("http://example.com");

When resolving a project, we should find any occurrences of Brioche.download("<string>") in the AST, then pre-emptively download the provided URL. Once we download it, we should then record the hash in brioche.lock associated to the given URL, and additionally save the result in the bakes table.

In the runtime, Brioche.download(…) will be implemented with a new op that takes the requested URL and the caller's URL (retrieved using a thrown exception), and returns the URL's hash as an object. The Brioche.download(…) function will take this returned hash and construct a download recipe with it.

The implementation and type definition for Brioche.download will live in std, but will be exported as a global. This is so that Brioche itself can hard-code the name Brioche without needing to determine how std itself is imported. Using the global another way (window["Brioche"]["download"] or similar) will not be supported.

Implementation-wise, this scheme matches how Brioche.includeFile / Brioche.includeDirectory / Brioche.glob all work. The internal name for this type of function is called a "static"

Fix terminal output getting cut-off by one character

Here's a screenshot from a failed build showing what renders in my terminal (set with a width of 80 characters)

Note the message in the middle, which clearly has a letter cut off in the word "backtrace":

note: run with `RUST_BACKTRACE=1` environment variable to display a
acktrace

There is likely an off-by-one error somewhere in in the console reporter code (permalink). The console reporter uses the superconsole crate under the hood. This doesn't affect the plain terminal output (e.g. when running brioche build 2>&1 | cat)

Add a recipe to glob a directory

The Brioche.glob function allows pulling in files into Brioche, grabbing a subset of inputs from the project directory. There is currently no way to do something similar for recipes themselves: you can get individual files from a directory recipe with the .get() method, but there is no way to get part of a directory. This also means there's no way to optionally get a file from a directory recipe: if the given path doesn't exist, .get() will fail

We should add a new recipe type called glob, which takes a directory recipe and a list of glob patterns as inputs, and bakes to a directory. This recipe will be exposed through a new std.glob function.

const thing = std.directory({
  bin: std.directory({
    "foo": std.file("..."),
    "bar.sh": std.file("..."),
  }),
  lib: std.directory({
    "libbaz.so": std.file("..."),
  }),
});

return std.glob(thing, ["bin/**.sh", "lib/**/*"]);
// Bakes to a directory containing the following paths:
// - bin/bar.sh
// - lib/libbaz.so

[CLI] Format a file/folder instead of a project

Usually, formatters take as input a list of files, or folders, or even a glob pattern. It enables granularity and lets the end user choose between formatting a whole folder recursively or just a single file.

The brioche formatter takes another approach, which is to format per project (and without giving any granularity):

Format the Brioche files in a project

Usage: brioche fmt [OPTIONS] --project <PROJECT>

Would it make sense to switch to a file/folder formatter, or should we continue to format per project? The discussion starts here (#75) could also be extended to this issue if the option 2 is chosen.