civitaspo / digdag-operator-emr_fleet

digdag plugin for operating a cluster with instance fleets on Amazon Elastic Map Reduce.

Hi there 👋

I'm a Data Reliability Engineer having specialties of Distributed Processing, Data Engineering, and Big Data Processing. And I'm also committing some OSS projects like Embulk, Digdag, and so on.

Career

• 2022/07~Current LayerX Inc.
• 2022/01~2022/07 Ubie, Inc.
• 2021/10~2021/12 ZOZO, Inc.
• 2020/06~2021/09 ZOZO Technologies, Inc.
• 2017/10~2020/05 Gunosy Inc.
• 2013/04~2017/09 DeNA Co.,Ltd.

Sponsor

I'm looking for sponsors! => https://github.com/sponsors/civitaspo

digdag-operator-emr_fleet's Issues

Detect subnet id and security group id by thier name

Current subnet_ids and (master|slave)_security_groups options are not human-friendly because the ids are auto-generated and themselves have no meaning. Thier ids have the name we named, so this plugin should detect them by thier name.

Add `content` and `from_local_file` option for `bootstrap_actions`

Current bootstrap_actions option assume to locate that the bootstrap action script locates on S3 or EMR Server local. But I want to define the bootstrap action script inside dig file and digdag executor server local, either. So add content and from_local_file option for bootstrap_actions.