[Core feature] Allow sub workflows created by `admin-launcher` to run on a different flytepropeller shard about flyte

[dosubot]

Hey @Tom-Newton, great to see you diving deep into Flyte's propeller again! 🚀

Thank you for bringing up this important issue related to the distribution of workflows across shards in Flyte. Your observation about the potential for a single shard to become overloaded due to the current shard-key inheritance behavior is valid, and your proposal to generate independent shard-keys for each subworkflow launched by the admin-launcher could indeed help to alleviate this issue.

To implement this, you would need to modify the shard-key generation logic in the Flyte system. Currently, the shard-key is generated based on a range-bounded hash over various components of the FlyteWorkflow metadata, with the keyspace range defined as [0,32). This deterministic approach ensures that each FlyteWorkflow evaluation is handled by a specific FlytePropeller instance, based on the project, domain, and shard-key labels.

However, as you've pointed out, this can lead to an uneven distribution of workflows across shards, especially in cases where a large workflow spawns many subworkflows. To achieve a more uniform distribution, the shard-key generation logic could be updated to include a dynamic component, such as a UUID or a timestamp, in addition to the existing metadata components. This would ensure that each new workflow gets a unique shard-key, leading to a more even distribution across shards.

Here's a conceptual approach to implementing this:

1. Modify the Shard-Key Generation Logic: Update the logic that generates the shard-key for each FlyteWorkflow to include a dynamic component. For example, you could concatenate the current hash components with a UUID or the current timestamp at the moment of workflow creation.

2. Update the FlytePropeller Manager Configuration: Ensure that the FlytePropeller Manager is configured to handle the updated range of shard-keys. Since the dynamic component could potentially increase the variability of shard-keys, you might need to adjust the shard-count or the keyspace range to accommodate the new distribution pattern.

3. Test the Distribution: After implementing the changes, it's crucial to test the new shard-key generation mechanism to ensure that it achieves a more uniform distribution of workflows across shards. You could simulate a large number of workflow creations and analyze the resulting shard-key distribution to verify its uniformity.

Here is a pseudo-code example illustrating how you might implement the dynamic shard-key generation:

import (
    "crypto/sha256"
    "encoding/hex"
    "time"
)

func generateShardKey(workflowMetadata string) string {
    // Generate a dynamic component, e.g., a timestamp
    timestamp := time.Now().UnixNano()
    dynamicComponent := fmt.Sprintf("%d", timestamp)
    
    // Concatenate the workflow metadata with the dynamic component
    combinedKey := workflowMetadata + dynamicComponent
    
    // Hash the combined key to generate the shard-key
    hasher := sha256.New()
    hasher.Write([]byte(combinedKey))
    shardKey := hex.EncodeToString(hasher.Sum(nil))
    
    // Optionally, you could further process the shardKey to fit the desired keyspace range
    // For simplicity, this step is omitted
    
    return shardKey
}

This example combines the workflow metadata with a dynamic component (a timestamp in this case) to generate a unique shard-key for each workflow. You would need to integrate this logic into the part of the Flyte system responsible for initializing FlyteWorkflow CRDs, ensuring that each new workflow gets a unique shard-key based on this mechanism.

Please note that this is a conceptual approach and might require adjustments based on the specific requirements and constraints of your Flyte deployment. Also, keep in mind that any changes to the shard-key generation logic should be thoroughly tested to ensure they don't introduce new issues or regressions.

As for your offer to contribute a Pull Request, that's greatly appreciated! However, as a bot, I'm unable to create or manage Pull Requests. I recommend discussing your proposal and the potential changes with the Flyte maintainers directly to ensure they align with the project's goals and guidelines. They can provide valuable feedback and guidance on how to best implement and test these changes.

I hope this information is helpful and provides a good starting point for addressing this issue. If you have any further questions or need additional clarification, please don't hesitate to ask.

Sources

• flytepropeller/manager/doc.go
• flytepropeller/pkg/controller/nodes/subworkflow/launchplan/admin.go

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Tom-Newton]

I think the bot hasn't really understood #5174 should resolve it.

[kumare3]

I agree this should use a different shard key