bug: after restarting the engine, the same step gets executed twice, leading to unexpected behavior about hatchet

[abelanger5]

Hey @rbtbar, this is not expected behavior. When we lose connection with the worker, we will reassign work after a visibility timeout of 30 seconds. It seems that when the engine is starting up, we are checking the reassignment immediately, which will always be triggered since the workers haven't connected yet. We'll patch a fix to set a delay on checking for reassignments to give the workers a chance to reconnect.

A few additional questions:

1. We've been planning to have reassignments trigger the retry loop (they currently bypass the retry loop), which means we'll only reassign work if you have set retries greater than 0. Thoughts on the ideal behavior here?
2. Just to be sure, which version of the hatchet-engine are you running in docker compose?

Additionally, the latest version of the Python SDK is 0.26.1 (though this shouldn't impact the behavior here)

[rbtbar]

1. If I was setting retries to 0, then my intention would be to not waste resources, so it makes sense to only reassign tasks when there's still a possibility of success.
2. v0.28.2

Version 0.26.1 exhibits the same behavior.