hatchet-dev / hatchet Goto Github PK

Support a postgres-backed task queue to remove the RabbitMQ requirement.

We marshal to/from a bunch of types during the lifecycle of a step run. We should cache the raw JSON bytes and converted objects, wherever possible, and consider an alternative like go-json to speed up decoding.

• #2
• #4

Right now, panic in a method results in the entire worker crashing. Should add recovery to each method so that the panic is sent to the Hatchet engine without killing the worker.

Like this but or workflow names

If there are > 50 events, the html body will overflow and cause scroll behavior:

127.0.0.1 app.dev.hatchet-tools.com

We can just set this as an actual DNS entry I think

Steps can be assigned to workers which have disconnected, which isn't ideal. Stale workers should automatically be removed.

For example, an anonymous function in the folder github.com/testing/workers will be given the name com/testing/workers which isn't a valid Hatchet name.

This is mainly to install dev deps using npm i instead of installing them separately.

My recommendation is to set up golangci-lint with a few reasonable linters to check for code issues early directly in the editor, and run CI to block merging if it fails.

Tickers, dispatchers and workers all use heartbeats to report their status. While the tickers/dispatchers are designed to gracefully shut down after SIGTERM, there's no recovery if the tickers/dispatchers are unexpectedly terminated. The engine needs a core service which can handle components which aren't reporting their status.

It's pretty ugly that workflows get registered with the anonymous reference, like MyService-func1.

Readiness probe should check for database and AMQP connections, liveness should just return 200 when ready.

Getting error Error running worker: could not register workflow for <event>: action <action> already registered when trying to re-use an action across multiple workflows.

Right now, python _grpc.py files need to be fixed manually, for example:

from .workflows_pb2 import workflows__pb2

Needs to be replaced with:

from . import workflows_pb2 as workflows__pb2

We should do this in the generation script.

It's necessary to support steps which can:

1. Execute the same step with different inputs in parallel.
2. Execute different steps with the same input in parallel.

From a client perspective, the Go SDK should be able to detect parallel steps based on the output from a previous step and the input to the current step. For example, if one step has the following output:

func stepOne(ctx context.Context) ([]stepOneOutput, error)

And the second step has the following input:

func stepTwo(ctx context.Context, in *stepOneOutput) error

Then stepTwo should be run for each array element resulting from stepOne.

Jobs should be separate from steps, or at least not in the same list/table

This means it can't be used in worker.On properly.

This should return forbidden only when the query can't find the tenant or the authentication is wrong. Otherwise, the error message should be different and an internal or unavailable error should be returned. This can happen when the database query returns an error (e.g. due to it being unavailable).

An important part is this line

as this needs to be checked for a NotFound error, and then it should return a forbidden error if something is off with the jwt, and in all other cases a generic error with a 500 status code

If errors occur while queueing a step run, the step run will be stuck in a pending state until timed out. The internal error should cause step-run-finished to be triggered with an internal error.

I had 2 workflows with step1, then deleted 1, but it's still registered on the server so step 1 was being dispatched twice regardless.

In other words, if a workflow is put and then the code is removed but the step is still registered in another workflow this step will still be invoked twice.

List of goroutine leaks which are currently ignored, but should be investigate if they could become a problem.

Related code:

Goroutine leaks:

• go.opencensus.io/stats/view.(*worker).start

  [Goroutine 6 in state select, with go.opencensus.io/stats/view.(*worker).start on top of the stack:
  go.opencensus.io/stats/view.(*worker).start(0xc0002feb80)
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/[email protected]/stats/view/worker.go:292 +0x128
  created by go.opencensus.io/stats/view.init.0 in goroutine 1
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/[email protected]/stats/view/worker.go:34 +0xf4
  

• google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

   Goroutine 23 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
  google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000373500, {0x102fde440, 0xc00004aaf0})
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x150
  created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 54
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x1f8
  

• google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

   Goroutine 24 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
  google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000373530, {0x102fde440, 0xc00004ab40})
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x150
  created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 54
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x1f8
  

• google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

   Goroutine 25 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
  google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000373560, {0x102fde440, 0xc00004ab90})
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x150
  created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 54
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x1f8
   Goroutine 40 in state IO wait, with internal/poll.runtime_pollWait on top of the stack:
  internal/poll.runtime_pollWait(0x14d67ce38, 0x72)
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/runtime/netpoll.go:343 +0xa0
  internal/poll.(*pollDesc).wait(0xc00052c0a0, 0xc0000dd200?, 0x0)
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_poll_runtime.go:84 +0xb8
  internal/poll.(*pollDesc).waitRead(...)
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_poll_runtime.go:89
  internal/poll.(*FD).Read(0xc00052c080, {0xc0000dd200, 0x900, 0x900})
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_unix.go:164 +0x2e0
  net.(*netFD).Read(0xc00052c080, {0xc0000dd200, 0x900, 0x900})
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/net/fd_posix.go:55 +0x48
  net.(*conn).Read(0xc00051e018, {0xc0000dd200, 0x900, 0x900})
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/net/net.go:179 +0x8c
  crypto/tls.(*atLeastReader).Read(0xc00069a870, {0xc0000dd200, 0x900, 0x900})
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:805 +0x7c
  bytes.(*Buffer).ReadFrom(0xc00060a2a8, {0x102fd69b8, 0xc00069a870})
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/bytes/buffer.go:211 +0xf4
  crypto/tls.(*Conn).readFromUntil(0xc00060a000, {0x14d67cf30?, 0xc00051e018}, 0x5)
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:827 +0x18c
  crypto/tls.(*Conn).readRecordOrCCS(0xc00060a000, 0x0)
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:625 +0x480
  crypto/tls.(*Conn).readRecord(...)
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:587
  crypto/tls.(*Conn).Read(0xc00060a000, {0xc0005e2000, 0x8000, 0x50000c0003bfb38?})
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:1369 +0x1c4
  bufio.(*Reader).Read(0xc000594d80, {0xc000596200, 0x9, 0x9})
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/bufio/bufio.go:244 +0x390
  io.ReadAtLeast({0x102fd5e58, 0xc000594d80}, {0xc000596200, 0x9, 0x9}, 0x9)
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/io/io.go:335 +0xcc
  io.ReadFull(...)
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/io/io.go:354
  golang.org/x/net/http2.readFrameHeader({0xc000596200, 0x9, 0x9}, {0x102fd5e58, 0xc000594d80})
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x68
  golang.org/x/net/http2.(*Framer).ReadFrame(0xc0005961c0)
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0xbc
  google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00057c240, 0xc00060a000?)
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1587 +0x1e8
  created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 26
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:398 +0x2234
  

• google.golang.org/grpc/internal/transport.(*controlBuffer).get

   Goroutine 41 in state select, with google.golang.org/grpc/internal/transport.(*controlBuffer).get on top of the stack:
  google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0004e4460, 0x1)
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:418 +0x120
  google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0000e77a0)
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:552 +0xf0
  google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:452 +0x114
  created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 26
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:450 +0x288c
  

• google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

   Goroutine 55 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
  google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00012b920, {0x102fde440, 0xc00004ac30})
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x150
  created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 10
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x1f8
  

• google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

   Goroutine 56 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
  google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00012b950, {0x102fde440, 0xc00004ad70})
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x150
  created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 10
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x1f8
  

• google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

   Goroutine 57 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
  google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00012b980, {0x102fde440, 0xc00004adc0})
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x150
  created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 10
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x1f8
   Goroutine 61 in state IO wait, with internal/poll.runtime_pollWait on top of the stack:
  internal/poll.runtime_pollWait(0x14d67cd40, 0x72)
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/runtime/netpoll.go:343 +0xa0
  internal/poll.(*pollDesc).wait(0xc00021dda0, 0xc0000ddb00?, 0x0)
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_poll_runtime.go:84 +0xb8
  internal/poll.(*pollDesc).waitRead(...)
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_poll_runtime.go:89
  internal/poll.(*FD).Read(0xc00021dd80, {0xc0000ddb00, 0x900, 0x900})
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_unix.go:164 +0x2e0
  net.(*netFD).Read(0xc00021dd80, {0xc0000ddb00, 0x900, 0x900})
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/net/fd_posix.go:55 +0x48
  net.(*conn).Read(0xc00051e0d0, {0xc0000ddb00, 0x900, 0x900})
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/net/net.go:179 +0x8c
  crypto/tls.(*atLeastReader).Read(0xc000155d10, {0xc0000ddb00, 0x900, 0x900})
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:805 +0x7c
  bytes.(*Buffer).ReadFrom(0xc00060b0a8, {0x102fd69b8, 0xc000155d10})
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/bytes/buffer.go:211 +0xf4
  crypto/tls.(*Conn).readFromUntil(0xc00060ae00, {0x14d67cf30?, 0xc00051e0d0}, 0x5)
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:827 +0x18c
  crypto/tls.(*Conn).readRecordOrCCS(0xc00060ae00, 0x0)
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:625 +0x480
  crypto/tls.(*Conn).readRecord(...)
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:587
  crypto/tls.(*Conn).Read(0xc00060ae00, {0xc000440000, 0x8000, 0x50000c0005258b8?})
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:1369 +0x1c4
  bufio.(*Reader).Read(0xc00018f140, {0xc0000d63c0, 0x9, 0x9})
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/bufio/bufio.go:244 +0x390
  io.ReadAtLeast({0x102fd5e58, 0xc00018f140}, {0xc0000d63c0, 0x9, 0x9}, 0x9)
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/io/io.go:335 +0xcc
  io.ReadFull(...)
    /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/io/io.go:354
  golang.org/x/net/http2.readFrameHeader({0xc0000d63c0, 0x9, 0x9}, {0x102fd5e58, 0xc00018f140})
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x68
  golang.org/x/net/http2.(*Framer).ReadFrame(0xc0000d6380)
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0xbc
  google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00057cd80, 0xc00060ae00?)
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1587 +0x1e8
  created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 14
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:398 +0x2234
  

• google.golang.org/grpc/internal/transport.(*controlBuffer).get

   Goroutine 62 in state select, with google.golang.org/grpc/internal/transport.(*controlBuffer).get on top of the stack:
  google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00004b540, 0x1)
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:418 +0x120
  google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000520d20)
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:552 +0xf0
  google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:452 +0x114
  created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 14
    /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:450 +0x288c
  ]
  

From the dashboard, step runs which time out after 5 minutes are incorrectly reporting the following:

This step was cancelled because it exceeded its timeout of 60s

Instead of computing the default timeout on the frontend as well, the backend should just populate the default timeout, so we keep the defaults as a single source of truth.

Support middleware that can wrap each action with access to the function's context.Context.

It should be possible to register this globally with worker.Use, and per-service with service.Use.

For example:

worker.Use(func (ctx context.Context, next func(ctx context.Context) error) error {
  // can set values in context, handle and throw errors, etc
  next(ctx)
  return nil
})

Panic middleware (#71) should be present by default, but overwritten as needed.

Unclear why as the API shows these have the same workflow ID (unless this is memory pointer in for loop issue):

Problem: existing code in the admin client has incorrect logic for determining the strategy for putting and versioning a workflow.

If auto_version isn't set, should_put should be true if the workflow.version does not equal existing_workflow.versions[0].version, or if there is no existing_workflow. So perhaps something like:

def determine_workflow_update(auto_version, workflow, existing_workflow):
    if not auto_version and existing_workflow and existing_workflow.versions:
        return workflow.version != existing_workflow.versions[0].version, workflow.version
    if workflow.version == "":
        return True, "v0.1.0"
    if existing_workflow and existing_workflow.versions:
        new_version = bump_minor_version(
            existing_workflow.versions[0].version)
        should_put = new_version != workflow.version
        return [should_put, new_version]

    return [True, workflow.version]

This seems to match up much better with the Go SDK:

shouldPut := opts.autoVersion

	if err != nil {
		// if not found, create
		if statusErr, ok := status.FromError(err); ok && statusErr.Code() == codes.NotFound {
			shouldPut = true
		} else {
			return fmt.Errorf("could not get workflow: %w", err)
		}

		if workflow.Version == "" && opts.autoVersion {
			req.Opts.Version = "0.1.0"
		}
	} else {
		// if there are no versions, exit
		if len(apiWorkflow.Versions) == 0 {
			return fmt.Errorf("found workflow, but it has no versions")
		}

		// get the workflow version to determine whether to update
		if apiWorkflow.Versions[0].Version != workflow.Version {
			shouldPut = true
		}

		if workflow.Version == "" && opts.autoVersion {
			req.Opts.Version, err = bumpMinorVersion(apiWorkflow.Versions[0].Version)

			if err != nil {
				return fmt.Errorf("could not bump version: %w", err)
			}
		}
	}

Originally posted by @abelanger5 in #122 (comment)

Should be quite similar to the cron job implementation, except unassigning the ticker after execution.

Code blocks can overflow their container horizontally if the JSON strings are long enough.

Relative dates are intuitive, but it would be great to support a hoverable component which shows the precise timestamp. Could replace all relativeDate calls with a shared component which shows the relative date by default, and the precise timestamp on hover. The precise timestamp should also be copyable, it sucks when they're not.

It should be possible to register a single method to convert to a workflow definition, via:

worker.On(
  worker.Event("user:create"),
  func (ctx context.Context) error {
    return nil
  },
)

Utilize the CloudEvents Go SDK instead of relying on tasktypes like we're currently doing

Related issues:

• #85
• #87
• #88
• #89

When deleting an API token or performing other destructive actions, we should have users type out delete or <object-name> in order to confirm deletion.

Problem: We're invalidating the workflow run when replaying an individual step, but the refetched data workflow-runs/$run/index.tsx:34 does not properly represent the state (always succeeded in the case of a succeded run). Because of this, the refetchInterval is never set to

I've temporarily hotfix in #159 by always setting the refetch interval to 1s, but this is not a good solution in the long run.

Anonymous step IDs can get ugly and it's not reasonable to ask users to always create a non-anonymous function, so would be great to have the option of changing the generated step ID.

This can be either a new scheduling state or a specific timeout of type ScheduleTimeout

For example, this will fail because the - is not permitted as a field reference:

name: example-workflow
version: 0.1.0
triggers:
  events:
    - user:create
jobs:
  post-user-create:
    steps:
      - id: example-step
        action: default:action1
        with:
          object: '{{ .input.json }}'
      - id: example-step-2
        action: default:action2
        with:
          object: '{{ .steps.example-step.json }}'

I've been running gosec locally occasionally to check for unhandled errors and for loop pointer bugs, would be great to have this as part of CI.

fix all data races

Need a better way to manage the assignment of jobs and step runs to tickers and dispatchers, respectively.

Here's an example of how a ticker is currently assigned to a job run.

There are a few problems with this:

1. The ticker is assigned in the database before the ticker has acknowledged that it's registered that job run/started a timeout process.
2. The JobsController shouldn't need to be aware of how to assign tickers to a job or select a ticker for the job run. The jobs controller should message the task queue that a job run is available, and it should be picked up by an available ticker.

The same problems exist for scheduled workflows, crons, and step runs. In particular, the admin service shouldn't be registering/unregistering tickers for old and new workflow versions.

Workflows are fully versioned and considered immutable once written on the backend, but this doesn't work for development when you're iterating on workflows. Additionally, there's no option in the Go SDK to update the workflow version after it's written.

There's no mechanism to use the On method in the Go SDK to register multiple events or crons.