A Go library for debugging Kubernetes controllers built with controller-runtime. This library enables step-through debugging of reconcile loops by wrapping workqueues to control when items are processed.

5-Minute Quick Start Guide - Get started immediately with a working example!

• Manual Reconcile Stepping: Step through reconciles at your own pace
• Selective Debug Mode: Choose which resources reconcile automatically vs manually
• Resource Type Resolution: Use kind, plural, or singular names (e.g., "website", "websites", or "Website")
• kubectl-Style CLI: Intuitive command-line interface for debugging workflows
• Queue State Inspection: View pending and in-progress reconciliations
• HTTP REST API: Query state and step through reconciles programmatically
• Multi-Controller Support: Debug multiple controllers in a single process; if multiple controllers manage the same resource type, all are stepped
• Thread-Safe: Safe for concurrent use
• Structured Logging: Uses slog for clean, parseable logs

go get github.com/georgemac/looppedal

See the Website Controller Example for a complete, working example of a custom controller with looppedal integration.

The example demonstrates:

• A custom Website resource and controller
• How to integrate looppedal into your controller
• Real debugging workflows with loopctl
• Pausing resources, stepping through reconciles, and viewing timelines

# Navigate to the example
cd examples/website-controller

# Install the CRD
kubectl apply -f config/crd/website.yaml

# Run the controller with debugging enabled
go run main.go --debug

# In another terminal, use loopctl
loopctl list
loopctl pause Website default/my-blog
loopctl watch

See the full example README for detailed instructions.

Here's the basic pattern for integrating looppedal into your controller:

import (
    "github.com/georgemac/looppedal/pkg/api"
    "github.com/georgemac/looppedal/pkg/debug"
    "github.com/georgemac/looppedal/pkg/looppedal"
)

func main() {
    // Create debugger instance
    debugger := debug.NewSimpleDebugger(mgr.GetClient())
    defer debugger.Stop()

    // Start HTTP API server
    apiServer := api.NewSimpleServer(debugger, ":8080")
    go apiServer.Start()

    // Define your resource's GVK
    gvk := schema.GroupVersionKind{
        Group:   "demo.looppedal.dev",
        Version: "v1alpha1",
        Kind:    "Website",  // Your custom resource kind
    }

    // Build controller with debug-wrapped queue
    builder.ControllerManagedBy(mgr).
        For(&websitev1alpha1.Website{}).  // Your custom resource
        WithOptions(controller.Options{
            NewQueue: func(controllerName string, rateLimiter workqueue.TypedRateLimiter[reconcile.Request]) workqueue.TypedRateLimitingInterface[reconcile.Request] {
                // Create standard queue
                underlying := workqueue.NewTypedRateLimitingQueueWithConfig(
                    rateLimiter,
                    workqueue.TypedRateLimitingQueueConfig[reconcile.Request]{
                        Name: controllerName,
                    },
                )

                // Wrap with looppedal - registration happens automatically!
                return looppedal.WrapQueue(
                    debugger,
                    controllerName,
                    gvk,
                    underlying,
                    mgr.GetClient(),
                )
            },
        }).
        Complete(&YourReconciler{})

    mgr.Start(ctrl.SetupSignalHandler())
}

Note: looppedal.WrapQueue() handles controller registration automatically, so you don't need to call RegisterController() separately. This prevents ordering mistakes.

The loopctl CLI provides a kubectl-style interface for debugging controllers.

go build -o bin/loopctl ./cmd/loopctl

# List all controllers
loopctl list

Output:

CONTROLLER   GROUP                 VERSION    KIND      QUEUED   IN PROGRESS
website      demo.looppedal.dev    v1alpha1   Website   2        0

# Pause a specific resource (using resource type)
loopctl pause website default/my-blog

Output:

Paused website default/my-blog

The resource will not reconcile until you run:
  loopctl step website default/my-blog

# Step through a single reconcile
loopctl step website default/my-blog

loopctl play website default/my-blog

# View detailed queue state for a controller
loopctl queue website

Output:

Queue for controller: website

NAMESPACE   NAME       STATE      ADDED     REQUEUES
default     my-blog    pending    14:20:15  0
default     prod-site  waiting    14:19:30  2

For programmatic access or custom tooling, use the HTTP API directly.

curl http://localhost:8080/api/v1/controllers

curl -X POST http://localhost:8080/api/v1/paused \
  -H "Content-Type: application/json" \
  -d '{
    "resourceType": "website",
    "namespace": "default",
    "name": "my-blog"
  }'

curl -X DELETE http://localhost:8080/api/v1/paused \
  -H "Content-Type: application/json" \
  -d '{
    "resourceType": "website",
    "namespace": "default",
    "name": "my-blog"
  }'

curl -X POST http://localhost:8080/api/v1/step \
  -H "Content-Type: application/json" \
  -d '{
    "resourceType": "website",
    "namespace": "default",
    "name": "my-blog"
  }'

• GET /api/v1/controllers - List all registered controllers
• GET /api/v1/controllers/{id} - Get queue state for a specific controller

• GET /api/v1/paused - List all paused resources
• POST /api/v1/paused - Pause a resource (requires: resourceType, namespace, name)
• DELETE /api/v1/paused - Resume (play) a resource (requires: resourceType, namespace, name)

• POST /api/v1/step - Manually step through one reconciliation (requires: resourceType, namespace, name)

• GET /api/v1/health - Health check endpoint

The loopctl command-line tool provides an intuitive interface for controller debugging.

• --api-url string - URL of the debugger API server (default: http://localhost:8080)

List all registered controllers and their queue state.

Example:

loopctl list

Pause reconciliation of a resource. The resource will wait for manual stepping.

Arguments:

• resource-type - Type of resource (kind, plural, or singular - e.g., "website", "websites", or "Website")
• namespace/name - Resource identifier (e.g., default/my-blog)

Example:

loopctl pause website default/my-blog

Step through one reconcile iteration for a resource that is paused.

Arguments:

• resource-type - Type of resource
• namespace/name - Resource identifier

Example:

loopctl step website default/my-blog

Resume automatic reconciliation of a resource that was paused.

Arguments:

• resource-type - Type of resource
• namespace/name - Resource identifier

Example:

loopctl play website default/my-blog

Show the queue state for a specific controller.

Arguments:

• controller-id - ID of the controller (from loopctl list)

Example:

loopctl queue website

1. Workqueue Wrapping: The DebugWorkqueue wraps controller-runtime's standard workqueue, intercepting the Get() method where workers pull items to process.

2. Paused Check: When a worker calls Get(), the wrapper checks if the resource is paused:

   • Not paused: Item is returned immediately for normal reconciliation
   • Paused: Worker blocks until a manual step is received

3. UID Tracking: Resource UIDs are fetched from the API server and cached with a 5-minute TTL. This ensures debug mode doesn't persist incorrectly when resources are deleted and recreated with the same name.

4. Step Channels: Manual steps use Go channels to signal blocked workers. When you step a reconcile via the API, the corresponding channel is closed, unblocking the worker.

5. Graceful Shutdown: Context cancellation ensures blocked workers are released during shutdown.

HTTP API Server (REST endpoints)
    |
Debugger (Central Registry)
    |
Controller Registry Entry (per controller)
    |
DebugWorkqueue (wraps workqueue.Interface)
    |
Original controller-runtime Workqueue

• UID fetching requires API calls (cached with 5-minute TTL to minimize overhead)
• Workers block while waiting for steps (ensure adequate worker pool sizing)
• Timeline tracks reconcile events but not individual client operations (Get/Update/Patch)
• Pausing must be done on specific resources (no pattern matching yet)

• Client Operation Tracking: Intercept Get/Update/Patch operations to track resource state before and after each operation
• Diff View: Show diffs between before/after state for each reconcile
• Pattern-Based Pausing: Support wildcards like default/* or */prod-*
• Conditional Pausing: Pause only when certain conditions are met
• TUI Interface: Full-screen terminal UI for interactive debugging
• Browser-Based UI: Web UI embeddable in controllers with real-time updates
• Recording/Replay: Record reconcile flows and replay them for testing
• Advanced Selectors: Label and annotation-based resource selection
• Prometheus Metrics: Expose debugging metrics for observability
• Pause Persistence: Save paused resources across restarts

Contributions are welcome! Please open issues and pull requests on GitHub.

MIT License - see LICENSE file for details