eirini-persi-related packaging scripts are broken about eirini-bosh-release

[mudler]

I think you can workaround it as you did already for opi as was showing the same issue during build in #7 which was then merged in #8 .

Note, aside then the build step, the same probably needs to be applied also to pre_packaging scripts of both *-persi: https://github.com/cloudfoundry-community/eirini-bosh-release/blob/master/packages/eirini-persi-broker/pre_packaging and https://github.com/cloudfoundry-community/eirini-bosh-release/blob/master/packages/eirini-persi/pre_packaging (which is the point of divergence from opi)

[mudler]

we are working on a separate branch to integrate another component, we will try to bring the workaround with those changes

[mudler]

Hi @jspawar @jrussett ,

The *-persi pre_packaging scripts require in the host which is building the bosh release a configured golang environment (for development). This is because we are using go mod and we rely on the go.sum to build the vendor/ folder without having to distribute all the project dependencies in the code. To build it, we have a special docker image with all the requirements met ( a configured go>=1.12 env, https://hub.docker.com/r/splatform/bosh-cli ).

That being said, I think we are all open to other solutions. And I suspect we are already switching to submodules for vendoring the dependencies as the cf-operator is having a separate repo for it cc: @viovanov @jimmykarily wdyt?

[jspawar]

@mudler Thanks for the heads up on the new component and taking care of the fix. I was wondering if you could clarify your first comment a bit more though, specifically in the context of moving towards using the BOSH Golang vendor package instead of manually uploading the Golang blob.

I understand the need for Go 1.12 to use gomodules and vendor in dependencies, but I don't quite understand what this Docker image (https://hub.docker.com/r/splatform/bosh-cli) you mentioned is being used for. What exactly is it being using to build?

You are still using the Golang blob that is configured here to actually compile the various binaries, correct? If so, the BOSH vendor packages should continue to support this. We would just prefer to use a standardized Golang blob and not have to continue updating it ourselves as well as apply all these fun new configurations Golang will mandate as it updates.

Not as important, but curious now: you also called out the pre_packaging scripts specifically, I just realized I didn't notice they existed for those persi-related jobs. Is there any specific reason why? We would prefer to minimize usage of pre_packaging scripts as much as possible as well since the BOSH team advocates for not using them: cloudfoundry/bosh#1344 (comment)

Thank you (and apologies for the wall of text!),
Jwal

[mudler]

@mudler Thanks for the heads up on the new component and taking care of the fix. I was wondering if you could clarify your first comment a bit more though, specifically in the context of moving towards using the BOSH Golang vendor package instead of manually uploading the Golang blob.

I understand the need for Go 1.12 to use gomodules and vendor in dependencies, but I don't quite understand what this Docker image (https://hub.docker.com/r/splatform/bosh-cli) you mentioned is being used for. What exactly is it being using to build?

You are still using the Golang blob that is configured here to actually compile the various binaries, correct? If so, the BOSH vendor packages should continue to support this. We would just prefer to use a standardized Golang blob and not have to continue updating it ourselves as well as apply all these fun new configurations Golang will mandate as it updates.

Sorry for the confusion, we use that docker image to build the eirini-bosh-release itself (we call bosh inside that image).

The vendor approach I'm talking about is at the code repositories - if each of those projects on github (eirini-persi, -persi-broker, and -loggregator-bridge) would ship its vendors the problem wouldn't exist on the first place.

My proposal is to ship that folder with all the project vendors as a submodule pointing to a new github repo which is actually containing the generated vendor dependency of each project (as the cf-operator does). So in this way we won't have to care at the bosh-release level, and we always fetch the code as it should be compiled (with all the deps bundled).

Not as important, but curious now: you also called out the pre_packaging scripts specifically, I just realized I didn't notice they existed for those persi-related jobs. Is there any specific reason why? We would prefer to minimize usage of pre_packaging scripts as much as possible as well since the BOSH team advocates for not using them: cloudfoundry/bosh#1344 (comment)

We knew that the usage is discouraged, but it was meant as a workaround to allow us to go ahead. We had to take that approach exactly for the same reason I'm now proposing to handle the vendor's deps directly in each project codebase. The pre_packaging forces who run bosh to create a release to generate the vendor of each project, instead of shipping the whole dep tree with the source code (and pollute project git history)

[jspawar]

Hey @mudler,

Thanks again for further clarifying, very much appreciate it!

I understand more clearly now what you were talking about regarding vendoring the source dependencies and I think, gut feeling at least, that sounds reasonable to move towards. I think I have also realized though that the "vendoring" I initially discussed in the original issue comment is different than the "vendoring" we are discussing now, although I am glad we have had this discussion to consider those "vendoring" changes.

I was originally proposing moving towards removing the golang blob that we currently have and instead use the standard golang blob that BOSH provides here: https://github.com/bosh-packages/golang-release. Using golang-release is what I meant to say would mitigate any more golang environment/compilation issues in the future if we introduce more jobs or need to upgrade golang. By using the golang-release we would simply standardize our packaging scripts (and probably no longer need pre-packaging scripts) to something along the lines of:

#!/bin/bash -eu
source /var/vcap/packages/golang-1.12-linux/bosh/compile.env

pushd ...
  go mod vendor # as needed
  go build ...
popd

Obviously if we do end up moving towards including the vendor dependencies as submodules then we should not need that go mod line, but overall should be simpler and more standardized with other CF golang-based projects.

Does that sound about right and reasonable then? If indeed we were discussing different kinds of "vendoring", do you think it would be valuable for us to open a separate issue around including the golang dependencies as submodules? Not pressing, but we can just leave it open for us all to discuss and come back to when it makes sense.

Thanks (sorry again for a lengthy comment),
Jwal

[jspawar]

Hey @mudler, have you all had a chance to consider the changes we've proposed for using golang-release or your vendoring strategy for your persi-related packages?

Also just wanted to check in again because the pre_packaging scripts continue to remain a problematic issue for our team.

Thanks,

Jwal + @jrussett