jctanner/markov

★ 2Forks 0GoGitHub ↗Compare

README

markov-logo

Markov

A Go-based YAML workflow engine for Kubernetes. Define workflows declaratively, and Markov executes them as K8s Jobs, HTTP requests, or shell commands with built-in concurrency, conditionals, and checkpoint/resume.

Why "Markov"?

A Markov chain is a system that transitions between states based on its current state. Markov workflows work the same way: gate steps evaluate rules against the current state, decide whether to continue or pause for explicit external input, and workflows can recursively invoke themselves — looping until the rules say to stop. The result is a declarative state machine where each iteration's next move depends only on the facts right now, not the history of how it got there.

Features

  • Declarative YAML workflows — multiple workflows per file with an entrypoint
  • User-defined step types — compose reusable types from engine primitives (k8s_job, http_request, shell_exec, script_exec, prompt, load_artifact)
  • Fan-out / fan-in — for_each with sliding-window concurrency control (forks)
  • Sub-workflows & recursion — invoke named workflows inline; workflows can call themselves to loop, controlled by gate rules
  • Conditionals — when: expressions to skip or run steps
  • Failure handling — workflow-level rescue and always handlers for recovery and teardown
  • Template rendering — Jinja2-compatible (pongo2) for params and expressions
  • Artifact loading — load YAML, markdown, and markdown table files from local or K8s volumes; use parsed data in conditions
  • set_fact — compute and store variables from expressions or table lookups for use in downstream steps
  • assert — validate conditions and fail the workflow with a message if any are false
  • Rule engine / gates — define named rules with salience-based priority; gate steps evaluate rules via Grule, set facts, and can pause durably for explicit resume input
  • Checkpoint/resume — SQLite state store; resume failed runs from the last successful step
  • Jev decisions — native Jev/Kev API steps, durable inline conditions, and decision-backed gate facts; see Jev reference
  • K8s native — creates batch/v1 Jobs directly (no Argo dependency)

Documentation

Full documentation is available at docs/, including:

How is this different from X?

See docs/design/project-comparisons.md for detailed comparisons with Ansible, Argo Workflows, Jenkins, Tekton, Kestra, CrewAI, and LangGraph.

Quick Start

go build -o markov ./cmd/markov

# Validate a workflow file
markov validate examples/k8s-job-test.yaml

# Validate a directory workflow
markov validate examples/dir-based-hello-world

# Run a workflow
markov run examples/k8s-job-test.yaml --namespace markov-test --verbose

# Run a directory workflow
markov run examples/dir-based-hello-world --verbose

# Check status
markov status <run-id> --steps

# Resume a failed run
markov resume <run-id>

# Generate a Mermaid diagram of a completed run
markov diagram <run-id>

Example

entrypoint: main
namespace: markov-test
forks: 2

vars:
  greeting: "hello from markov"
  items: ["alpha", "bravo", "charlie"]

step_types:
  echo_job:
    base: k8s_job
    job:
      image: alpine:3.19
      command: ["/bin/sh", "-c"]

workflows:
  - name: main
    steps:
      - name: hello
        type: echo_job
        params:
          args: ["echo '{{ greeting }}'"]

      - name: fan-out
        for_each: "items"
        as: item
        workflow: per-item
        vars:
          value: "{{ item }}"

  - name: per-item
    vars:
      value: null
    steps:
      - name: process
        type: echo_job
        params:
          args: ["echo 'processing: {{ value }}'"]

CLI Flags

Flag Description
--var key=value Override workflow vars (repeatable)
--workflow name Run a specific workflow instead of the entrypoint
--forks N Override concurrency limit
--namespace ns Override K8s namespace
--kubeconfig path Path to kubeconfig
--state-store path SQLite state file (default: ./markov-state.db)
--verbose Show detailed execution output
--steps Show per-step status (with status command)

Project Structure

cmd/markov/          CLI entrypoint
pkg/engine/          Workflow execution, gate evaluation, artifact loading, facts
pkg/parser/          YAML parsing, validation, rule loading
pkg/executor/        Step executors (k8s_job, shell_exec, script_exec, prompt, http_request)
pkg/state/           Checkpoint store (SQLite)
pkg/template/        Pongo2 template rendering
examples/            Example workflow files
docs/                Documentation (reference, guides, design docs)

License

TBD

Contributors

jctanner

Issues