jotform / zenith

Lightning-fast javascript monorepo build tool with remote/local caching ability /w pnpm

Zenith: Lightning-Fast JavaScript Monorepo Build Tool

Zenith transforms the complexity of managing JavaScript monorepos into a simple, fast, and productive experience. By smartly caching without .git reliance, Zenith slashes CI pipeline times, letting you focus solely on coding.

Quick Start

• Install: pnpm add @jotforminc/zenith
• Use: pnpm zenith --target=<command> --project=<name>

Configuration with zenith.json

Create a zenith.json in your project root including projects and buildConfig to guide Zenith's operation.

Features

• Local & Remote Caching: Deploy faster using less bandwidth.
• No .git Required: Ideal for team-based monorepo development.
• Versatile Commands: Supports build, test, and more.

---

Detailed Setup

Table of contents

• What is Zenith? What is its aim?
• Installation
• zenith.json: What is it and why is it required?
• Required Parameters
  • Environment Variables
  • Params
• Optional Parameters
• Debugging

What is Zenith? What is its aim?

Zenith is a fast build tool used for both remote and local caching. Its speed comes from the ability to cache files remotely while not using .git files, which makes Zenith a good choice when developing monorepos with teams. It provides build cache and test cache functionalities, which can be used to deploy a large application much quicker than its competitors.

Zenith requires pnpm as the node package manager.

Installation

pnpm add @jotforminc/zenith

From the terminal, run:

pnpm zenith --target=("build" | "test") --project=("all" | <project_name>)

Target and project arguments are required for now. Without them, the tool will not work.

zenith.json: What is it and why is it required?

Zenith looks for a file named "zenith.json" in the same folder where your root package.json file is. This file is used to determine the behavior of Zenith. It MUST include 'projects' and 'buildConfig' keys, and MAY include 'ignore' and 'appDirectories' keys. An example of usage is as follows.

{
    "projects": {
        "@jotforminc/app1": "projects/applications/app1",
        "@jotforminc/app2": "projects/applications/app2",
        "@jotforminc/lib1": "projects/libraries/lib1"
    },
    "buildConfig": {
        "cachePath": ".customCache",
        "appConfig": {
            "build": {
                "script": "build",
                "outputs": [
                    "build"
                ]
            },
            "lint:js": {
                "script": "lint:js",
                "outputs": [
                    "stdout"
                ],
                "constantDependencies": [
                    "@jotforminc/lib1"
                ]
            }
        }
    },
    "mainConfig": {
        "build": {
            "script": "build",
            "outputs": [
                "build"
            ]
        },
        "lint:js": {
            "script": "lint:js",
            "outputs": [
                "stdout"
            ],
            "constantDependencies": [
                "@jotforminc/lib1",
                "@jotforminc/lib2"
            ]
        }
    },
    "ignore": [
        "node_modules",
        ".gitignore",
        "build",
        "lib",
        "dist",
        ".DS_Store",
        "test-results"
    ],
    "appDirectories": [
        "/apps/"
    ]
}

Required Parameters

The project uses several required environment variables and params. Without them, the tool will not work as intended.

Environment Variables

- CACHE_TYPE (string): One of ['local', 'remote', 'local-first', 'remote-first'], 'local' by default. If 'remote', S3 environment variables are required.
- S3_ACCESS_KEY (string): Access key to be used to get objects from and write objects to the buckets.
- S3_SECRET_KEY (string): Secret key to be used to get objects from and write objects to the buckets.
- S3_BUCKET_NAME (string): Bucket name to be written and read from.
- S3_REGION (string): AWS S3 region to connect to.
- ZENITH_DEBUG_ID (string): A string that will be used to determine the debug file name.

Params

-t, --target <"build" | "test">: Target command to be used.
-p, --project <"all" | [string]>: Project to be built. "all" value builds every project the tool finds in the subfolders.

Optional Parameters

The following parameters are not required to work, but can be used to modify the tool's behavior.

-h, --help: Show parameter information.
-d, --debug: If given, enters debug mode. Usage is provided in the [debugging](#debugging) section.


-c, --compareWith <string>: Compares and calculates the difference between a given json file and the current build.


-dl, --debugLocation <string>: Debug Location: sets the prefix of the debug location. By default, it is "debug/", and its usage is as follows: \n {target}/{debugLocation}debug.{ZENITH_DEBUG_ID}.json


-w, --worker <number>: Worker Number (default = 6): sets the maximum number of workers that run concurrently.


-l, --logLevel <1 | 2 | 3>: Sets the log level. 1=silent mode. 2=default mode, which only shows errors and statistics after completion. 3=verbose mode, logs cache hits, misses, and recoveries.


-ch, --noCompareHash: default: false. If false, will compare remote folders\' and local folders\' hash and execute target if hashes are not the same.


-la, --logAffected: default: false. If true, will log outputs of ONLY missed caches\' executes.

Debugging

If -d param is given, the tool outputs a file named "debug.{ZENITH_DEBUG_ID}.json". This file will contain hashes of every file that is used in hashing. It can be used to see if the tool is working as intended, and which folders are being hashed. By also setting -c param, you can compare builds and see which files are new, and changed or removed.

zenith's Issues

Migrate to pnpm

Let's convert zenith itself to pnpm instead of yarn

Eslint typescript fixes

As I see the complete conversion to TS is not fully complated. Lets spend a day and make this project fully TS.

Bug: empty files change hashes of the projects

Possibly ignore empty files while hashing

[Performance]: implement local cache

I see some PRs are already successfully built when they are in queue, when it gets to "merging" status it downloads all previously builded files from aws which takes almost 1mins. We can reduce this to milliseconds by using a local-first cache. We know the pr is already built on the local folder in the same agent-machine.

What do you think?

feat: introduce .zenithignore file

We should add an ignore file just like .gitignore. This file should be used while comparing caches and the contents in this file should not affect cache algorithm.

Zero Config Start

Instead of creating a config file manually what about introducing something like

npx zenith setup

which will do the init phase for the related project so anyone can use zenith with low effort

[Security Issue]: Remove zip-local

zip-local is using an old version of jszip which has secuirty issue, please fix it. Instead maybe you can use jszip latest directly

ESLint Configuration

We should have ESLint configured and lint accordingly. Readability should improve as a result.

Documentation Improvements

Please update documentation of Zenith, use Readme file as documentation.
Following must be in the documentation

• Philosophy: What is Zenith? What is its aim
• Installation: How to use it in your javascript monorepo
• Projects.json: what is it and what is the schema
• What is required parameters (s3 configs, CLI params etc)
• How to debug if a cache is missing? (--debug flag)

TS Conversion

Let's convert to TS.

Better Documentation

Current documentation seems a bit complicated and does not cover for our recent features.
Let's improve it.

Dependency Graph (like NX did)

Title: Allow seeing dependency graph in an easy way
Details: We need to see our dependency graph with some small commands like zenith graph. So developers and we can see our package relations then simplify them.
Related work pieces:

• Create a graph command
• Give an interactive graph output
• Allow some filtering

tests: add unit tests

We should increase our coverage,
I think we can start with

• Hashing algorithm
• Config read

rest is up to you.

Rollup: conversion to TS

Rollup was temporarily unused while converting to TS. Configure it according to TS so it can be used again.