A small library that composes typed units of work into a pipeline.

go get github.com/veggiemonk/workflow

It needs Go 1.27, because it uses generic methods.

The program in examples/basic builds a pipeline from five small steps. Each step has its own input type and its own output type, and Then and Par change the type of the chain at each link.

package main

import (
	"context"
	"fmt"
	"log"
	"strings"

	wf "github.com/veggiemonk/workflow"
)

// Report is what the pipeline returns. Every step between the input and the
// report has its own output type, so no single struct carries the whole run.
type Report struct {
	Words  int
	Unique int
	Head   string
}

func main() {
	// A step declares its input type and its output type. Pure is for a
	// function that cannot fail.
	clean := wf.Pure("Clean", strings.TrimSpace)     // string   -> string
	split := wf.Pure("Split", strings.Fields)        // string   -> []string
	count := wf.Pure("Count", func(w []string) int { // []string -> int
		return len(w)
	})
	unique := wf.Pure("Unique", func(w []string) int { // []string -> int
		seen := make(map[string]struct{}, len(w))
		for _, word := range w {
			seen[strings.ToLower(word)] = struct{}{}
		}
		return len(seen)
	})
	head := wf.Pure("Head", func(w []string) string { // []string -> string
		return strings.Join(w[:min(3, len(w))], " ")
	})

	// Count, Unique and Head read the same words. Par runs two steps at the
	// same time and joins their results with a function you supply. The join
	// is typed, so the compiler will not let a result be dropped.
	analyse := count.
		Par(unique, func(n, u int) ([2]int, error) { return [2]int{n, u}, nil }).
		Par(head, func(nu [2]int, h string) (Report, error) {
			return Report{Words: nu[0], Unique: nu[1], Head: h}, nil
		})

	// Then links the steps. The chain changes type at every link.
	pipeline := clean.Then(split).Then(analyse) // Step[string, Report]

	fmt.Println("🚀 Starting basic workflow example...")
	fmt.Println()

	report, err := pipeline.Run(context.Background(), "  the quick brown fox jumps over the lazy dog  ")
	if err != nil {
		log.Fatalf("Pipeline failed: %v", err)
	}

	fmt.Println("📊 Results:")
	fmt.Printf("Words:  %d\n", report.Words)
	fmt.Printf("Unique: %d\n", report.Unique)
	fmt.Printf("Head:   %q\n", report.Head)

	fmt.Println("\n🌳 Pipeline structure:")
	fmt.Println(pipeline)
}

Run it with cd examples/basic && go run ..

Up to v0.3.0 a step was Step[T]: it read a T and returned a T. One type had to carry the whole pipeline. That single decision caused everything else:

• Parallel branches all returned the same struct, so the library needed a merge function. It used reflection, and it dropped a result without a word.
• Branches shared the struct, so the library needed a deep copy. The copy fell back to a shallow copy in silence.
• Middleware was applied while the pipeline ran, by writing to its own step slice. A second run added a second layer.

Now a step declares its own input type and its own output type. The compiler does the work that reflection did, and the three faults above cannot happen.

The library is below v1.0.0, so v0.4.0 takes the break instead of adding a /v2 import path. The import path does not change. Pin v0.3.0 if you need the old API. CHANGELOG.md lists every change.

type Step[I, O any] struct{ /* unexported */ }   // a value, not an interface

func (s Step[I, O]) Run(ctx context.Context, in I) (O, error)

Step is a concrete struct, not an interface. That is deliberate. Go 1.27 allows type parameters on a method only when the receiver is a concrete type:

interface method must have no type parameters

A generic method is what lets a chain change its type:

func (s Step[I, O]) Then[P any](next Step[O, P]) Step[I, P]

To write a step on a type that holds state, implement Runner — a plain interface, so it stays legal — and wrap it with Of:

type Runner[I, O any] interface{ Run(context.Context, I) (O, error) }

step := workflow.Of("Cache", myCache)   // Step[Key, Value]

Each is a function, not a method. A method returning Step[[]I,[]O] would make the compiler instantiate Step[[][]I,[][]O], and so on without end. The compiler calls that an instantiation cycle and rejects it.

analyse := count.
    Par(uniq,  func(n, u int) ([2]int, error)            { return [2]int{n, u}, nil }).
    Par(upper, func(nu [2]int, s string) (Report, error) { return Report{nu[0], nu[1], s}, nil })

pipeline := parse.Then(analyse)      // Step[Doc, Report]
batch    := workflow.Each(8, pipeline) // Step[[]Doc, []Report]

A step is a value. Every method returns a new step and leaves the receiver unchanged. Middleware is applied once, when you build the step, so a second run can never add a second layer.

step := fetch.
    Retry(workflow.RetryConfig{MaxAttempts: 3}).
    Timeout(2 * time.Second).
    Breaker(breaker).
    Log(logger)

Three notes on behaviour:

• Log never records the payload. A payload can hold a secret, and formatting it costs more than a short step.
• Timeout starts no goroutine. Go cannot stop a goroutine from outside, so the step must honour the context. A step that does no I/O must test ctx.Err() itself. v0.3.0 raced a goroutine and leaked it.
• A CircuitBreaker is an explicit value. You decide what shares it. In v0.3.0 the state sat in a closure, so every step the middleware wrapped shared one circuit without saying so.

Par, Fan and Each run every branch to the end and return every error, joined with errors.Join. They do not stop at the first failure.

Each of them also turns a panic in a branch into a *PanicError, because a panic in a goroutine would otherwise stop the program. A step that runs in sequence is left alone: its panic travels up your own stack, as Go intends. Call Recover() when you want that converted too.

On an Apple M4 Max, Go 1.27:

BenchmarkRun-16                296803701     4.158 ns/op     0 B/op   0 allocs/op
BenchmarkThenChain-16           32746808    36.06  ns/op     0 B/op   0 allocs/op
BenchmarkSeq-16                 45185329    28.23  ns/op     0 B/op   0 allocs/op
BenchmarkPar-16                  1583120   758.9   ns/op   288 B/op   6 allocs/op
BenchmarkEach100-16                30480 39491     ns/op 20422 B/op 104 allocs/op
BenchmarkMiddlewareStack-16      4575774   262.6   ns/op   272 B/op   4 allocs/op

The sequential path allocates nothing. Only the concurrent combinators allocate, for the goroutines and the result slices.

A v0.3.0 pipeline that keeps one type throughout maps to Seq[T] with no other change in shape, because Step[T] is Step[T,T].

This version has no dependency outside the standard library. v0.3.0 needed mergo, google/uuid and x/sync.

It is an in-memory combinator library. It holds no state between runs. It cannot resume after a crash, and it has no general graph with shared dependencies: a pipeline is a tree of sequence and fan-out. Use Temporal or Cadence when you need a durable workflow.

• examples/: four programs, from a sentence counter to a data pipeline with bounded concurrency.
• docs/architecture.md: what the library is made of, and where to extend it.
• docs/best-practices.md: the patterns that hold up, and the ones that do not.
• docs/llms.md: the generated API reference.