Access to classes from secrets fails about telegraf-operator

[wojciechka]

Hi @wojciechka
I think using isRegular() in your PR will not work as at first level, there are only links
[...]
That's why I suggested to exclude ..data/ instead of changing for isRegular().

This is why I am calling os.Stat - so that it dereferences the symlink.

I put more details in my comment in #31.

[gitirabassi]

Hello @nsteinmetz. we have an automatic build every time we merge into master.
quay.io/influxdb/telegraf-operator:5ec384ff5e09430187d174e342257693eaf5c2cf is the one from yesterday

I'll take a look at why the certificate is broken

[nsteinmetz]

I'm not a golang dev but maybe this should be enough adding && !strings.Contains(file.Name(), "..data")

	for _, file := range files {
		if !file.Mode().IsDir() && !strings.Contains(file.Name(), "..data") {
			data, err := ioutil.ReadFile(filepath.Join(c.TelegrafClassesDirectory, file.Name()))
			if err != nil {
				c.Logger.Info(fmt.Sprintf("unable to retrieve class data from file %s: %v", file.Name(), err))
			} else {
				filesAvailable = true
				if _, err := toml.Parse(data); err != nil {
					c.Logger.Info(fmt.Sprintf("unable to parse class data %s: %v", file.Name(), err))
					classDataValid = false
				}
			}
		}
	}

[wojciechka]

Hi Nicolas,

I'm working on improving it, but I do not think this is preventing the telegraf-operator from working properly.

Currently telegraf-operator tries to read too many items in the telegraf classes directory, but it only reports errors, since it did not report anything in the logs related to infra, I believe that file was loaded properly.

Could you confirm that assigning the class infra works properly for you?

[nsteinmetz]

Hi @wojciechka

I don't think it works as I don't see anything going to influxdb using the infra class. So from my understanding, the infra class was never read nor taken into account.

I think using isRegular() in your PR will not work as at first level, there are only links

cf

ls /etc/telegraf-operator/ -al
total 4
drwxrwxrwt    3 root     root           100 May 14 14:13 .
drwxr-xr-x    1 root     root          4096 May 14 14:13 ..
drwxr-xr-x    2 root     root            60 May 14 14:13 ..2020_05_14_14_13_56.563495811
lrwxrwxrwx    1 root     root            31 May 14 14:13 ..data -> ..2020_05_14_14_13_56.563495811
lrwxrwxrwx    1 root     root            12 May 14 14:13 infra -> ..data/infra

That's why I suggested to exclude ..data/ instead of changing for isRegular().

[nsteinmetz]

@wojciechka thanks for the precision ! Indeed I missed the stat part ;-)

If I can test a build, let me know !

[wojciechka]

@wojciechka thanks for the precision ! Indeed I missed the stat part ;-)

If I can test a build, let me know !

The changes are now merged into the code base. However, I am not sure we will do a release of telegraf-operator just for this change, but you should be able to build the container on your own and push it to your DockerHub account - such as by doing:

docker build -t docker.io/(username)/telegraf-operator:latest .
docker push docker.io/(username)/telegraf-operator:latest

You will need docker (either natively installed in your Linux, or using Docker for Mac or Docker for Windows).

I am not sure this would be related to your original issue, though. Based on the logs it seems like telegraf-operator simply reported some of the items in the /etc/telegraf-operator cannot be read as files, but it should have read the remaining items - such as infra.

Is the original issue that you are facing that you have successfully deployed the development version of telegraf-opetator and have set appropriate annotation (like telegraf.influxdata.com/class: infra) for your pod, but are not seeing the sidecar container?

If so, can you share how exactly you are deploying telegraf-opeator and the logs from the telegraf-operator after applying a deployment/statefulset/pod with the appropriate annotation?

Can you also share the YAML files for your object definitions so we can confirm everything is defined properly?

[rawkode]

@wojciechka Shouldn't we automate the publish of a new image on every merge to master?

[nsteinmetz]

@wojciechka i know docker, I was just asssuming there were an automated build available somewhere once PR is merged. So I will build my own image later today.

I deployed the telegraf-operator through the helm chart - I reported an issue here

I just added to override default valutes:

classes:
  secretName: "telegraf-operator-classes"
  default: "infra"
  data:
    infra: |+
      [[outputs.influxdb_v2]]
        urls = ["https://in2.monitoring.cerenit.fr"]
        token = "XXX"
        organization = "XXX"
        bucket = "XXX"
      [[outputs.file]]
        files = ["stdout"]
      [global_tags]
        hostname = "$HOSTNAME"
        nodename = "$NODENAME"

and logs are:

kubectl logs telegraf-operator-55c9788d8b-4fcc52020-05-15T07:19:28.672Z        INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
2020-05-15T07:19:28.750Z        INFO    setup.entrypoint        setting up webhook server
2020-05-15T07:19:28.750Z        INFO    setup.entrypoint        registering webhooks to the webhook server
2020-05-15T07:19:28.750Z        INFO    setup.podInjector       validating class data from directory /etc/telegraf-operator
2020-05-15T07:19:28.750Z        INFO    setup.podInjector       unable to retrieve class data from file ..data: read /etc/telegraf-operator/..data: is a directory
2020-05-15T07:19:28.753Z        INFO    controller-runtime.webhook      registering webhook     {"path": "/mutate-v1-pod"}
2020-05-15T07:19:28.753Z        INFO    setup   starting manager
2020-05-15T07:19:28.755Z        INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
2020-05-15T07:19:28.949Z        INFO    controller-runtime.webhook.webhooks     starting webhook server
2020-05-15T07:19:28.950Z        INFO    controller-runtime.certwatcher  Updated current TLS certificate
2020-05-15T07:19:28.950Z        INFO    controller-runtime.webhook      serving webhook server  {"host": "", "port": 9443}
2020-05-15T07:19:28.953Z        INFO    controller-runtime.certwatcher  Starting certificate watcher
2020/05/15 07:22:13 http: TLS handshake error from 51.75.199.67:59436: remote error: tls: bad certificate
2020/05/18 14:08:32 http: TLS handshake error from 51.75.199.67:35288: remote error: tls: bad certificate
2020/05/18 14:08:33 http: TLS handshake error from 51.75.199.67:35300: remote error: tls: bad certificate
2020/05/18 17:18:19 http: TLS handshake error from 51.75.199.67:36062: remote error: tls: bad certificate
2020/05/18 17:18:25 http: TLS handshake error from 51.75.199.67:36132: remote error: tls: bad certificate
2020/05/18 17:18:25 http: TLS handshake error from 51.75.199.67:36134: remote error: tls: bad certificate
2020/05/18 17:18:32 http: TLS handshake error from 51.75.199.67:36234: remote error: tls: bad certificate
2020/05/18 17:18:32 http: TLS handshake error from 51.75.199.67:36238: remote error: tls: bad certificate

nsteinmetz@crnt-nst-lnx  ~/Documents/Projets/k8s/ovh   master ●  kubectl logs telegraf-operator-55c9788d8b-kkxdb                                  
2020-05-15T07:19:24.662Z        INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
2020-05-15T07:19:24.662Z        INFO    setup.entrypoint        setting up webhook server
2020-05-15T07:19:24.662Z        INFO    setup.entrypoint        registering webhooks to the webhook server
2020-05-15T07:19:24.662Z        INFO    setup.podInjector       validating class data from directory /etc/telegraf-operator
2020-05-15T07:19:24.662Z        INFO    setup.podInjector       unable to retrieve class data from file ..data: read /etc/telegraf-operator/..data: is a directory
2020-05-15T07:19:24.663Z        INFO    controller-runtime.webhook      registering webhook     {"path": "/mutate-v1-pod"}
2020-05-15T07:19:24.663Z        INFO    setup   starting manager
2020-05-15T07:19:24.749Z        INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
2020-05-15T07:19:24.764Z        INFO    controller-runtime.webhook.webhooks     starting webhook server
2020-05-15T07:19:24.764Z        INFO    controller-runtime.certwatcher  Updated current TLS certificate
2020-05-15T07:19:24.765Z        INFO    controller-runtime.webhook      serving webhook server  {"host": "", "port": 9443}
2020-05-15T07:19:24.765Z        INFO    controller-runtime.certwatcher  Starting certificate watcher
2020/05/16 01:00:07 http: TLS handshake error from 51.75.199.67:37166: remote error: tls: bad certificate
2020/05/17 01:00:03 http: TLS handshake error from 51.75.199.67:56402: remote error: tls: bad certificate
2020/05/18 01:00:05 http: TLS handshake error from 51.75.199.67:44220: remote error: tls: bad certificate
2020/05/18 14:08:23 http: TLS handshake error from 51.75.199.67:35196: remote error: tls: bad certificate
2020/05/18 14:08:32 http: TLS handshake error from 51.75.199.67:35288: remote error: tls: bad certificate
2020/05/18 17:18:32 http: TLS handshake error from 51.75.199.67:36234: remote error: tls: bad certificate
2020/05/19 01:00:05 http: TLS handshake error from 51.75.199.67:40506: remote error: tls: bad certificate
2020/05/19 01:01:06 http: TLS handshake error from 51.75.199.67:41170: remote error: tls: bad certificate
2020/05/20 01:01:05 http: TLS handshake error from 51.75.199.67:36054: remote error: tls: bad certificate
2020/05/20 01:01:05 http: TLS handshake error from 51.75.199.67:36058: remote error: tls: bad certificate

 nsteinmetz@crnt-nst-lnx  ~/Documents/Projets/k8s/ovh   master ●  kubectl logs telegraf-operator-55c9788d8b-wpmt7                                  
2020-05-15T07:19:27.222Z        INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
2020-05-15T07:19:27.222Z        INFO    setup.entrypoint        setting up webhook server
2020-05-15T07:19:27.222Z        INFO    setup.entrypoint        registering webhooks to the webhook server
2020-05-15T07:19:27.222Z        INFO    setup.podInjector       validating class data from directory /etc/telegraf-operator
2020-05-15T07:19:27.222Z        INFO    setup.podInjector       unable to retrieve class data from file ..data: read /etc/telegraf-operator/..data: is a directory
2020-05-15T07:19:27.223Z        INFO    controller-runtime.webhook      registering webhook     {"path": "/mutate-v1-pod"}
2020-05-15T07:19:27.223Z        INFO    setup   starting manager
2020-05-15T07:19:27.224Z        INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
2020-05-15T07:19:27.325Z        INFO    controller-runtime.webhook.webhooks     starting webhook server
2020-05-15T07:19:27.327Z        INFO    controller-runtime.certwatcher  Updated current TLS certificate
2020-05-15T07:19:27.328Z        INFO    controller-runtime.webhook      serving webhook server  {"host": "", "port": 9443}
2020-05-15T07:19:27.328Z        INFO    controller-runtime.certwatcher  Starting certificate watcher
2020/05/18 17:18:25 http: TLS handshake error from 51.75.199.67:33920: remote error: tls: bad certificate
2020/05/18 17:18:29 http: TLS handshake error from 51.75.199.67:33966: remote error: tls: bad certificate
2020/05/19 01:01:06 http: TLS handshake error from 51.75.199.67:38954: remote error: tls: bad certificate
2020/05/20 01:00:02 http: TLS handshake error from 51.75.199.67:33152: remote error: tls: bad certificate

[nsteinmetz]

Ok thanks - just pushed my image on docker-hub - will for next time.

We can chat on slack if you need for debug & troubleshooting.

[nsteinmetz]

Ok so with the image I built:

Pods starts correctly but tls issue is still present:

kubectl logs telegraf-operator-f68dd659d-9td9g                                 
2020-05-20T14:04:54.426Z        INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
2020-05-20T14:04:54.427Z        INFO    setup.entrypoint        setting up webhook server
2020-05-20T14:04:54.427Z        INFO    setup.entrypoint        registering webhooks to the webhook server
2020-05-20T14:04:54.427Z        INFO    setup.podInjector       validating class data from directory /etc/telegraf-operator
2020-05-20T14:04:54.427Z        INFO    controller-runtime.webhook      registering webhook     {"path": "/mutate-v1-pod"}
2020-05-20T14:04:54.427Z        INFO    setup   starting manager
2020-05-20T14:04:54.513Z        INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
2020-05-20T14:04:54.613Z        INFO    controller-runtime.webhook.webhooks     starting webhook server
2020-05-20T14:04:54.614Z        INFO    controller-runtime.certwatcher  Updated current TLS certificate
2020-05-20T14:04:54.614Z        INFO    controller-runtime.webhook      serving webhook server  {"host": "", "port": 9443}
2020-05-20T14:04:54.615Z        INFO    controller-runtime.certwatcher  Starting certificate watcher
2020/05/20 14:05:02 http: TLS handshake error from 51.68.114.159:33990: remote error: tls: bad certificate
2020/05/20 14:05:02 http: TLS handshake error from 51.68.114.159:33992: remote error: tls: bad certificate

I deployed the redis stateful set from your examples and I have no trace of it in the pod logs, nor on influxdb side.

[gitirabassi]

This was fixed in latest version of helm chart