Expose init action API to a higher level provisioner / orchestration library? about initialization-actions

[chimerasaurus]

Hello!

Apologies for the delayed response! We've been hard at work on some new features and our 12/16 release. Thanks for positive feedback - it's always awesome to hear things are working well.

In the long-term, we have a few things in mind. Unfortunately, I cannot go too much into our long-term roadmap (timing or commitment) but we've definitely discussed a few things:

• When initialization actions should run (before daemons, after, both?)
• Parameters to init actions
• Alternatives to initialization actions
• Templatizing clusters

I think we'll likely expand the richness of init actions (parameters, etc.) and they will be a core part of Dataproc in the long term. We are also thinking about ways we can expand deployment and provisioning in the future, especially if other frameworks (like Flink) become popular to use alongside Spark/Hadoop (Yarn + HDFS.)

So, the tl;dr is it's something we have in mind. You're going to see better initialization action support in the future and in the longer-term, some more thinking (and possibly features) around deployment. You can always ping us at dataproc-feedback@google with ideas and suggestions - the dev team and I are on the list and always appreciate feedback, insights, and ideas.

Cheers!

James
(Product Manager, Cloud Dataproc)

[nehalecky]

Hi James (@evilsoapbox), thank you for this thoughtful reply, and I look forward to what you are developing—sounds exciting! Hopefully along the way the open source community can also help contribute to expanding functionality of dataproc (through init actions or more). As dataproc already provides a solid service for computation using open computational frameworks like Spark, it just make sense to help out!

Just to clarify, would you prefer that future ideas and suggestions be emailed instead of posted as issues here? For instance, I was going to open an issue to ask for automated testing and had a few ideas for that. :)

Thanks again and cheers!