Storing Custom Dataproc images for fast cluster provisioning about initialization-actions

[chimerasaurus]

Accidental send. Good news, this is a public (beta) feature. Please test it out and provide feedback. https://github.com/GoogleCloudPlatform/cloud-dataproc/tree/master/custom-images

…

On Mon, Mar 5, 2018 at 10:37 PM James Malone ***@***.***> wrote: Yep. :) On Mon, Mar 5, 2018 at 9:42 PM Nicholaus Halecky ***@***.***> wrote: > Hi @evilsoapbox <https://github.com/evilsoapbox> and all. Is this PR > related to this issue GoogleCloudDataproc/cloud-dataproc#44 > <GoogleCloudDataproc/cloud-dataproc#44>? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#26 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAZ_R2XSCdUTBW10HIOjBBsPqLlQrniJks5tbiG4gaJpZM4HL50U> > . >

[astanway]

I see - thanks. Any chance we can fix this eventually? I’d love to be able to completely destroy the cluster when I’m not using it and then quickly bring it back up with the same image. However, it would be a chore to have to regenerate the same image every 30 days.

…

On Mar 7, 2018, at 2:29 PM, Yifei Yao ***@***.***> wrote: @astanway The custom image itself is saved into your GCE images. It will not be deleted in 30 days. However, you cannot create a new cluster with an custom image that is more than 30 days old. Any cluster created before the 30 day is not affected. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[chimerasaurus]

This is a great suggestion and has been something which has been on our radar for awhile.

Would you mind also shooting a copy of this to [email protected]? That list has many customers on it and I'd like to see if anyone else also has a +1 (since many of them do not check out GitHub.) I am also happy to send it and attribute it as well if you'd prefer.

Cheers!

[romankor]

Should the imageUrl supposed to work when using workflows ?
I see it in the documentation , but its not visible in the api explorer.

            "managedCluster": {
                "clusterName": "export",
                "config": {
                    "masterConfig": {
                        "machineTypeUri": type,
                        "imageUri": "dataproc-export-image"

                    }

When using imageUri nothing gets created , and i dont seem to find any error in any of the logs
@evilsoapbox

Edit:
My bad , had to use v1beta API and use the full imageUri of the image

[tartavull]

+1

[spark2ignite]

Sure, go ahead and send it yourself ;-) Thanks!

[chimerasaurus]

Will do!

[MarkDunne]

+1

Any updates on this issue?

[snafis]

+1

[jrussek]

@evilsoapbox did this get any traction?

[chimerasaurus]

Hi all, yes. We are still planning on this feature. No firm ETA but it's under active investigation.

[saleh-sedighi-ck]

+1

[chimerasaurus]

Still under active investigation. This feature is very high on our feature list for 2018.

[astanway]

+1, subscribing for updates. Strongly desire this

[nehalecky]

Hi @evilsoapbox and all. Is this PR related to this issue GoogleCloudDataproc/cloud-dataproc#44?

[chimerasaurus]

Yep. :)

…

On Mon, Mar 5, 2018 at 9:42 PM Nicholaus Halecky ***@***.***> wrote: Hi @evilsoapbox <https://github.com/evilsoapbox> and all. Is this PR related to this issue GoogleCloudDataproc/cloud-dataproc#44 <GoogleCloudDataproc/cloud-dataproc#44>? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#26 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZ_R2XSCdUTBW10HIOjBBsPqLlQrniJks5tbiG4gaJpZM4HL50U> .

[astanway]

Nice! Two questions: how do we use the image to build a cluster? The only option we have is to pick an image-version on the gcloud cli, but putting in the image name doesn't seem to work.

Also, it looks like the image will only last for a month - why is there an expiration date on it?

[takeon8]

@astanway
Hi! you can use the new BETA flag "--image" in gcloud 190.0.0.

gcloud beta dataproc clusters create <cluster_name> --image <your_custom_image_name>

[takeon8]

@astanway
The custom image itself is saved into your GCE images. It will not be deleted in 30 days. However, you cannot create a new cluster with an custom image that is more than 30 days old. Any cluster created before the 30 day is not affected.