franela / pocketci

pocketci

This is very much a work in progress, expect things to be rough and potentially break!

pocketci is the first small and trully portable CI engine. This project leverages the Dagger SDK to wrap webhook with ability to automatically clone repositories. It allows you to run dagger commands (and any other kind of commands) in one of two ways:

• Portable CI runner: By starting the service as is you can hook it to a repository through webhooks and run commands in the context of your repo (like doing a checkout on github actions first!)
• Webhook server: By providing a hooks.yaml on startup you can leverage Dagger modules to hook them into specific webhooks (like a Slack Webhook)

Next steps

Using webhook allowed us to quickly test the idea of a CI engine that by leveraging Dagger in a native way becomes truly portable. The use of this project means we are limited by the definitions of the hooks.yaml format which, while being very useful, does not integrate in a native way with Dagger. We want pocketci to integrate seamlessly with Dagger and Dagger modules. You can join the discussion on how to build that here.

Portable CI runner

1. Start the service: dagger up -m github.com/franela/pocketci/module serve -p 8080:8080
2. Expose it to the internet (you can use the likes of ngrok to expose local endpoints)
3. Connect your repository by setting up a webhook in your VCS (github is the only supported at the moment)
4. Write a hooks.yaml file that triggers a dagger command on every commit to the main branch. For example on a github repo:

- id: git-push
  execute-command: "/bin/dagger"
  include-command-output-in-response: true
  command-working-directory: "/hooks-test"
  pass-arguments-to-command:
  - source: string
    name: call
  - source: string
    name: container-echo
  - source: string
    name: "--string-arg"
  - source: payload
    name: commits.0.message
  trigger-rule:
    and:
    - match:
        type: payload-hmac-sha1
        secret: itsasecret
        parameter:
          source: header
          name: X-Hub-Signature
    - match:
        type: value
        value: refs/heads/main
        parameter:
          source: payload
          name: ref

The working directory of your command will contain the entire contents of your repository at the specified ref so you could run commands from your own dagger module.

By default it proxies requests in a synchronous way, meaning that the execution of your commands will block the incoming requests. This could be a problem if you are exposing this to a provider such as a Github that has a timeout of 10 seconds. To force every request to be processed in an async way, you can pass the flag --proxy-async.

Webhook server

Write a hooks.yaml. For example one that runs a dagger call on every single request:

- id: webhook
  execute-command: dagger
  include-command-output-in-response: true
  response-headers:
    - name: "Access-Control-Allow-Origin"
      value: "*"
  pass-arguments-to-command:
    - source: "string"
      value: "--debug"
    - source: "string"
      value: "--progress=plain"
    - source: "string"
      value: "call"
    - source: "string"
      value: "-m"
    - source: "string"
      value: "github.com/shykes/daggerverse/hello"
    - source: "string"
      value: "Hello World!"

Run it with: dagger up -p 8081:8080 -m github.com/franela/pocketci/module serve --hooks hooks.yaml. You can validate that your webhook is triggered: curl http://localhost:8080/hooks/webhook. You should see a response such as:

1: load call
1: loading module
1: loading module [6.57s]
1: loading objects
1: loading objects [3.02s]
1: traversing arguments
1: traversing arguments [0.00s]
1: load call DONE

2: dagger call message
DEBUG: executing query="query{hello{message}}"
Hello, World!
2: dagger call message DONE

pocketci's Issues

Allow to specify a folder instead of a single file

ref: 5196327#diff-c345af598aafba13762f9c44d620ece05e3fff9f2bc5b6727eee2febc7169844R22

we currently support loading a single hooks file. Eventually it'd be nice to also have a --hooks-dir flag so you can set a dir in case you have multiple hooks definitions scattered across multiple files.

The hooks file format does not integrate seamlessly with Dagger modules

Wrapping webhook with some git niceties and Dagger allowed us to quickly validate that a CI engine that leverages Dagger in a native way is very portable. However, the way that users interact with pocketci today is with the hooks.json defined by webhook itself. This format, while incredibly useful to get started, is limiting us in a few ways:

• Verbose: there is a lot of config one must write in order to trigger a call to a dagger module, this introduces some friction and makes the tool not feel like it's built to integrate easily integrate with the idea of dagger modules. Additionally, the rules-engine that match incoming webhooks with commands, while providing a lot of flexibility, requires users to write a lot in order to match an action with a simple commit being pushed.
• Low-level: when writing hooks.json you are exposed to a lot of webhook-specific details that are not necessarily useful for a CI engine. I'm not sure yet if we want to allow this or not, but if we do it shouldn't make writing a workflow more complicated like it does today

---

A few things to ask ourselves while building this new file format:

• Do we want to maintain the webhook-only mode? If so, should this new file format include it or could we just refer back to hooks.json for this specific use case?

go.mod requires go >= 1.22 (running go 1.21.12; GOTOOLCHAIN=local)

Problem

When I run this:

dagger call -m github.com/franela/pocketci/module serve --src https://github.com/dagger/dagger up

I get this error:

✘ Pocketci.serve(
    src: ✔ GitRef.tree: Directory! 0.0s
  ): Service! 8.5s
! call function "Serve": process "/runtime" did not complete successfully: exit code: 2
  ✘ Container.file(path: "pocketci"): File! 7.1s
  ! process "go build -ldflags -s -w -o pocketci ./proxy" did not complete successfully: exit code: 1
    ✘ exec go build -ldflags -s -w -o pocketci ./proxy 0.2s
    ! process "go build -ldflags -s -w -o pocketci ./proxy" did not complete successfully: exit code: 1
    ┃ go: go.mod requires go >= 1.22 (running go 1.21.12; GOTOOLCHAIN=local)

Error: response from query: input: pocketci.serve resolve: call function "Serve": process "/runtime" did not complete successfully: exit code: 2

Stdout:
marshal: json: error calling MarshalJSON for type *dagger.Service: input: container.from.withDirectory.withWorkdir.withEnvVariable.withMountedCache.withMountedCache.withExec.file resolve: process "go build -ldflags -s -w -o pocketci ./proxy" did not complete successfully: exit code: 1

Stderr:
go: go.mod requires go >= 1.22 (running go 1.21.12; GOTOOLCHAIN=local)