cloudfoundry-community / eirini-bosh-release Goto Github PK

Hello Eirini friends!

We on the CAPI team have started running Eirini along with cf-deployment in our CI pipeline. As part of the pipeline, we also have regular cert rotation so that we never get into a situation where our certs have expired.

What we observe is that after certs are rotated, we can no longer retrieve app logs until we restart the eirini-loggregator-fluentd pods. The new certs do get properly propagated to the eirini-loggregator-fluentd pods system running in k8s, however both fluentd and loggregator-agentkeep merrily running along with the old certs already loaded into memory. In a bosh-deployed world, when configuration changes occur, the jobs are stopped and started, so they will pick up the new config. However, since the eirini-loggregator-fluentd pods exist outside of the bosh lifecycle, they are not restarted.

One option I see would be to add a command to the configure-eirini-bosh errand that will restart the pods. That would suit us just fine in our CI, since we don't care about logs until we run tests. However, for folks hoping to run this in production, presumably log downtime between then the doppler instance group gets updated and the post-deploy errand is probably unacceptable.

Maybe there's a more k8s-native way to tell the pods to be rolling-restarted when new certs show up? Can you trigger things on changing secrets, for example?

Hi @viovanov and others!

A new release of eirinifs that uses cflinuxfs3 instead of cflinuxfs2 was just made available: https://github.com/cloudfoundry-incubator/eirinifs/releases/tag/v34.0.0

Is there a pipeline to automatically pull new releases of eirinifs and create a new blob or is this done by hand? If it's automatic, feel free to close this issue. If it's done by hand, than this is a request to update that blob :)

Thanks!
Jen + @jspawar

Hi @viovanov and team,

We would like to get an eirini-bosh-release version that has an eirinifs blob with a cflinuxfs3 Digest in the release notes (currently the minimum version of eirinifs that has this is 52.0.0). Can you please bump the eirinifs blob?

Thanks!
@jpalermo && @madamkiwi

https://github.com/cloudfoundry-incubator/eirini/blob/1d06a7f96d97d9dbac27860b784fed2e5ff9d7d3/models.go#L54

https://github.com/cloudfoundry-community/eirini-bosh-release/blob/master/jobs/opi/templates/opi.yml.erb#L18

We'd just make a PR to fix this, but it would have to be two separate PRs with a dependency so we thought it might be easier for you to just fix and push.

The latest eirini source requires Go 1.12, since it relies on the new strings.ReplaceAll() function that was introduced in that version.

This is the error that appears during a deploy with the latest eirini source:

Task 1123 | 21:01:36 | Error: Action Failed get_task: Task 910fac1c-cb8a-4d76-7cb6-63601fb253f3 result: Compiling package eirini: Running packaging script: Running packaging script: Command exited with 2; Stdout: , Stderr: + mkdir -p /var/vcap/packages/eirini/src
+ cp -a . /var/vcap/packages/eirini/src
+ export GOPATH=/var/vcap/packages/eirini
+ GOPATH=/var/vcap/packages/eirini
++ readlink -nf /var/vcap/packages/golang
+ export GOROOT=/var/vcap/data/packages/golang/cd90ae9ceb26b29ea43c07634c8f103943d9751c
+ GOROOT=/var/vcap/data/packages/golang/cd90ae9ceb26b29ea43c07634c8f103943d9751c
+ export PATH=/var/vcap/data/packages/golang/cd90ae9ceb26b29ea43c07634c8f103943d9751c/bin:/var/vcap/bosh/bin:/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin
+ PATH=/var/vcap/data/packages/golang/cd90ae9ceb26b29ea43c07634c8f103943d9751c/bin:/var/vcap/bosh/bin:/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin
+ pushd /var/vcap/packages/eirini/src/code.cloudfoundry.org/eirini/cmd/opi
+ go build -o /var/vcap/packages/eirini/bin/opi
# code.cloudfoundry.org/eirini/k8s/utils
../../k8s/utils/names.go:10:19: undefined: strings.ReplaceAll

Maybe this is a good opportunity to switch to vendoring in the Golang package: https://bosh.io/docs/package-vendoring/

Hello,

We are attempting to pre-compile the BOSH release and are running into errors from the eirini-persi and eirini-persi-broker packaging scripts. Specifically, there are issues related to Golang-configuration for these packaging scripts:

build cache is required, but could not be located: GOCACHE is not defined and neither $XDG_CACHE_HOME nor $HOME are defined

This is not the first time we've had to address this issue and we have discussed using the Golang vendor package to mitigate these concerns on our ends in the past. It seems y'all were open to using it; however, this is something we cannot PR from our end since using vendor packages requires write access to the release's blobstore. We do not own those credentials of course, so we would need whoever has that access to vendor the Golang package.

We'd be happy to help with this of course. Also open to possibly moving the BOSH release's blobstore to somewhere that is shared by both of us, such as the environment where we will eventually be deploying our shared CI.

Thank you,
Jwal + @jrussett

If kubectl returns an error in run.erb the script is not failing. For instance we could set -e in the beginning of the script to make the job fail when kubectl fails.

This is related to cloudfoundry-incubator/quarks-operator#648

The blobs uploaded when the bot created final releases 0.0.2 and 0.0.3 are not publicly accessible.

This is the error we see when running bosh create-release:

  - Downloading blob 'ccc9c234-0c56-4db7-4c04-62027d8e712b' with digest string '6ad65d60cc92e238e613de4b9f315b361b32d838':
      Getting blob from inner blobstore:
        Getting blob from inner blobstore:
          AccessDenied: Access Denied
	status code: 403, request id: 84C400AE8C4AC796, host id: WdfmY09B8l3wV6xA/p5Me1KE/0kxxqVKBgUAJ9M3XzX42SLAFFNXiUDBW2ZUAiBVvJtuszAqcb0=
  - Downloading blob 'f389d93f-4cae-4e16-7f6c-7166ab7d6249' with digest string 'aeac1adbb1032ff27914f4c7339ce682a284bc57':
      Getting blob from inner blobstore:
        Getting blob from inner blobstore:
          AccessDenied: Access Denied
	status code: 403, request id: 36E5C00FA708DCA2, host id: xHNaIApDJFFLRs5ibJNHFfsX0DEUhDcp58q6WbcsEB8gQtLhMr+YdkEo0TEI55iW9Fctc5C86+M=
  - Downloading blob 'fdf559fe-e37a-4d37-57a9-aeb5ac26f11f' with digest string 'd0a8376aa9bc09107523bb5f2fe6d786dc38f51b':
      Getting blob from inner blobstore:
        Getting blob from inner blobstore:
          AccessDenied: Access Denied
	status code: 403, request id: CD90DB6053770D1F, host id: k1b9qEEXCKHrhMymHiczb+a0pkBu6vVDGx+A9GKkJ4pW5Ek565bYhASJunxcvuEYcouk83/JXUg=
  - Downloading blob '5082062c-d97a-4426-6b7f-a29a610ba026' with digest string 'be35d0970f2d56023002fbfe86c3df4a4689c931':
      Getting blob from inner blobstore:
        Getting blob from inner blobstore:
          AccessDenied: Access Denied
	status code: 403, request id: F73454F92188B8F7, host id: 3H08EluvmLgxITQm+f/LeSmn2uUbfPmf8lghuAUKorheiIPklxsDpE6dlGmVShudiIVujQvknps=

Exit code 1

After talking with @viovanov, it seems that the bucket itself is public, but the objects that the bot are creating are not.

Hi!

There were some changes that went into the eirini master branch very recently that replace the use of heapster with the k8s-built-in metrics server. Heapster has been deprecated, so this bit of code will make metrics collection work on newer k8s versions.

Could you please bump the Eirini submodule to this commit cloudfoundry/eirini@c745c47 or later?

Note: I've submitted a PR that also includes a bump of the Eirini submodule, but that bump is farther in the past than the metrics server work. If you don't have any problems with the PR, you may want to merge that in before bumping the Eirini submodule.

Thanks!
Jen