Kubernetes Envvar not interpolating correctly about nri-flex

[Kav91]

I believe this requires NRIA_PASSTHROUGH_ENVIRONMENT configured.

In your dockerfile, for which ever environment variables you wish to pass through the integration, can you passthrough as similar to the below example and then try again.

Example:
https://github.com/newrelic/nri-flex/blob/master/configs/Dockerfile-linux#L21

Doc:
https://docs.newrelic.com/docs/infrastructure/install-configure-manage-infrastructure/configuration/infrastructure-configuration-settings#integrations-variables

[j0sh3rs]

Attempting to add that in both the container build and in kubernetes template still results in the same problem: no matter how I setup the flex config, it does not inherit the envvar.

Dockerfile:

FROM newrelic/infrastructure:1.7.6

# debuging
#ENV NRIA_LOG_FILE=/etc/newrelic-infra/nr.log
#ENV NRIA_VERBOSE=0

ENV NRIA_IS_FORWARD_ONLY=true
# create some needed default directories for flex
RUN mkdir -p /var/db/newrelic-infra/custom-integrations/flexConfigs/
# RUN mkdir -p /var/db/newrelic-infra/custom-integrations/flexContainerDiscovery/
# if using container discovery configs uncomment this section
# ADD flexContainerDiscovery /var/db/newrelic-infra/custom-integrations/flexContainerDiscovery/
ENV NRIA_PASSTHROUGH_ENVIRONMENT="NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME,NRIA_IS_FORWARD_ONLY"
# copy config/definition/binary over
COPY ./nri-flex-config.yml /etc/newrelic-infra/integrations.d/
COPY ./nri-flex-definition.yml /var/db/newrelic-infra/custom-integrations/nri-flex-definition.yml
COPY ./nri-flex /var/db/newrelic-infra/custom-integrations/nri-flex
RUN chmod 744 /var/db/newrelic-infra/custom-integrations/nri-flex
COPY ./flexConfigs* /var/db/newrelic-infra/custom-integrations/flexConfigs/

# # add netcat if needed
# RUN apk add --update netcat-openbsd && rm -rf /var/cache/apk/*

I've tried using newrelic infra 1.7.6, as well as 1.5.0 (per your link). I've tried many different options for the flex config, including with/without brackets.

I see in the kubernetes config that the envvar exists on the podspec:

containers:
  - env:
    - name: NRIA_IS_FORWARD_ONLY
      value: "true"
    - name: NRIA_PASSTHROUGH_ENVIRONMENT
      value: NRIA_IS_FORWARD_ONLY,NRIA_LICENSE_KEY,NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME
    - name: NRIA_LICENSE_KEY
      valueFrom:
        secretKeyRef:
          key: NRIA_LICENSE_KEY
          name: newrelic-license
    - name: NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME
      value: k8s.mytestcluster.tld
    - name: NEW_RELIC_METADATA_KUBERNETES_NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: NEW_RELIC_METADATA_KUBERNETES_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name

And I see the passthrough var in the pod logs (as well as am able to printenv and see it from exec-ing into the pod):

$ kubectl logs nri-linkerd-gf8nx
[WARN  tini (25455)] Tini is not running as PID 1 and isn't registered as a child subreaper.
Zombie processes will not be re-parented to Tini, so zombie reaping won't work.
To fix the problem, use the -s option or set the environment variable TINI_SUBREAPER to register Tini as a child subreaper, or run Tini as PID 1.
time="2019-12-20T15:00:58Z" level=info msg="runtime configuration" agentUser=root component="New Relic Infrastructure Agent" executablePath= maxProcs=1 pluginDir="[/etc/newrelic-infra/integrations.d /var/db/newrelic-infra/integrations.d]"
time="2019-12-20T15:00:58Z" level=info msg="Creating Service" component="New Relic Infrastructure Agent" elapsedTime="982.457µs" version=1.5.0
time="2019-12-20T15:00:58Z" level=info msg="Agent service manager started successfully" component=AgentService elapsedTime=1.010244ms service=newrelic-infra
time="2019-12-20T15:00:58Z" level=info msg="Checking network connectivity..." component=AgentService service=newrelic-infra
time="2019-12-20T15:00:58Z" level=info msg=Initializing component=AgentService elapsedTime=135.191836ms service=newrelic-infra
time="2019-12-20T15:00:58Z" level=info msg="New Relic Infrastructure Agent Running" component=AgentService elapsedTime=243.94468ms service=newrelic-infra
time="2019-12-20T15:00:58Z" level=info msg="Agent plugin" plugin=metadata/attributes
time="2019-12-20T15:00:58Z" level=info msg="Integration info" arguments="map[container_discovery:true]" command=metrics commandLine="[nri-flex --metrics]" env-vars="map[VERBOSE:0 CONTAINER_DISCOVERY:true NRIA_IS_FORWARD_ONLY:true NRIA_LICENSE_KEY:<HIDDEN> NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME:k8s.mytestcluster.tld PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin]" instance=nri-flex integration=com.newrelic.nri-flex interval=30 labels="map[]" os=linux prefix=integration/com.newrelic.nri-flex protocolVersion=2 workingDir=/var/db/newrelic-infra/custom-integrations
time="2019-12-20T15:00:58Z" level=info msg="Integration health check starting" instance=nri-flex integration=com.newrelic.nri-flex prefix=integration/com.newrelic.nri-flex working-dir=/var/db/newrelic-infra/custom-integrations
time="2019-12-20T15:00:58Z" level=info msg="Integration health check finished with success" instance=nri-flex integration=com.newrelic.nri-flex prefix=integration/com.newrelic.nri-flex working-dir=/var/db/newrelic-infra/custom-integrations

But my queries for this value in insights continue to only produce the placeholder result.

[j0sh3rs]

@Kav91 any update on the above? We're still facing this problem and unable to appropriately leverage envvars in our flex config.

[Kav91]

hey @j0sh3rs apologies for the delay, I am putting some time aside tomorrow to try and replicate and debug this.

I do have a suspicious it could not be Flex related, and it could entirely be that Flex was not passed all the envs from the agent for some reason, but I need to confirm.

[Kav91]

Discovered the issue, the env substitution occurs correctly for standard flex configs, but not ones generated by container discovery.

Looking into applying a fix now, should be a quick one hopefully 🤞

[Kav91]

@j0sh3rs fix is applied in the 0.8.9 release -> https://github.com/newrelic/nri-flex/releases
If you still experience any issues, feel free to reopen the issue.
Thanks for catching this one!