Questions about pipelines/cf-deployment.yml about cf-deployment

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/140168053

The labels on this github issue will be updated when the story is started.

[dsabeti]

Hey @Amit-PivotalLabs.

I'm assuming you're talking about pipeline configuration from runtime-ci. It sounds like you'd like to re-use this pipeline configuration for your own concourse, but I don't think we've ever considered this yaml file to be part of a product interface. I'm hesitant to treat it like one since each team has pretty different needs, but it would be great if you could elaborate on what you need from the RelInt team -- we may be able to help some other way.

Here are the specific answers to your questions. They might help explain why I'd rather not productize this pipeline yamls:

1. We could try to parameterize the pipeline, but we'd have to do it once for each domain that we use. If, in the future, we decide to add a 4th environment to our pipeline, we'd parameterize that, too, and anyone consuming pipelines/cf-deployment.yml would have to deploy the new environment, and fly set-pipeline with a new variable.
2. cf-deployment-concourse-tasks is the repo where we try to keep tasks that other teams will likely use (things like bosh-deploy and upload-stemcell-from-cf-deployment). By and large, tasks in runtime-ci are tasks that we don't think other teams would benefit from. The runtime-ci task that you cited (for uploading stemcells) is used only for deploying the windows cell, which is something we actually probably should ship in cf-deployment-concourse-tasks. Let me know if there are any other tasks that you think would be of use more widely.

[dsabeti]

Hey @Amit-PivotalLabs, I'm going to close the issue, but feel free to re-open if you have more questions.

[Amit-PivotalLabs]

Don't think you should treat the pipelines as being products for others to consume, I agree with you there.

For your own purposes, DRYing up "hermione.cf-app.com" would save anyone on your team from having to know all the correct places to change that value. When trying to make a whole bunch of changes, it was easy to rely on concourse complaining that some variable is missing (which we'd get if it were a variable) and harder (marginally) to make sure we remembered to change that value everywhere. It's a small cognitive overhead in isolation, but when we were trying to make a whole bunch of changes to easily adapt it to our needs, it adds up a little. Overall the ability to take the pipeline and chop it up to suit our needs was very very smooth.

The other task worth sharing would be run-bosh-cleanup. Everyone's pipeline is eventually going to break because their director will eventually fill up, unless they run this.

[dsabeti]

I see. I think that might make sense. @anEXPer, any thoughts on this point about parameterizing non-credential values as a way to DRY up config?

Also, here's a story for running bosh cleanup: https://www.pivotaltracker.com/story/show/141382703

[anEXPer]

I'd sooner use anchors than variables for this. Our variables are private. Anything we put in them is hidden. I'd rather hide as little as possible about our pipelines. Also, it reduces dev load to have similar rules about vars for our pipes and our bosh manifests.

Honestly, I'd sooner use find and replace than anchors, too, unless this becomes painful for some reason.

I've seen great complication arise from unnecessary parameterization.