riskfocus / helm-charts-public

Riskfocus Helm Charts

This repo includes Apache Flink helm chart which can be used for developing purposes.

helm-charts-public's Issues

flink dashboard ui no stdout log

Problem:
When task manager prints log to stdout , there is nothing in 'Stdout' tab of FlinkUI.

By the way, Using 'kubectl logs' can get it work, but is not convenient for I need to figure out which pod to select.

Error: no matches for kind "ServiceMonitor"

Hi! I'm trying to install Flink on my Kubernetes cluster with this command:
helm install my-flink --namespace my-namespace riskfocus/flink --version 0.1.10

And I'm receiving the error below:

Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

Support EBS CSI driver and fix ingress

I used this chart install flink but got error

Error: render error in "flink/templates/ingress.yaml": template: flink/templates/ingress.yaml:26:15: executing "flink/templates/ingress.yaml" at <.host>: can't evaluate field host in type interface {}

and i use eks with EBS CSI driver, chart do not support. And several warnings

2020/05/28 15:51:38 Warning: Merging destination map for chart 'flink'. Cannot overwrite table item 'tolerations', with non table value: map[]

2020/05/28 15:51:38 Warning: Merging destination map for chart 'flink'. Cannot overwrite table item 'tolerations', with non table value: map[]

I submit PR #10

Detailed Documentation

Hi Team,

Thanks for the Flink Chart. It would be highly appreciated if you can write detailed documentation.

Wrong keys in initContainer

helm-charts-public/flink/templates/jobmanager.yaml

Line 50 in b6376f0

- name: {{ $ct }}

I think it should be $params.name ? Same for taskmanager

Flink ingress

Hello Team,

I'm unable to create Flink ingress through its chart! so far I've tried several ways to do it by following the template pattern of ingress.yaml but no success.

Please there is away to provide a sample of the required configuration in order to get it properly created? so far tried as follow:

ingress:
  enabled: false
  annotations: {}
  path: /
  hosts: [develop.mydomain.com]
  tls: [tls-ingress]

ingress:
enabled: false
annotations: {}
path: /
hosts: develop.mydomain.com
tls: tls-ingress

ingress:
  enabled: false
  annotations: {}
  path: /
  hosts: "develop.mydomain.com"
  tls: tls-ingress

ingress:                       
  enabled: true                
  annotations: {}              
  path: /                      
  hosts:                       
  tls:                         
    - hosts:                   
      -develop.mydomain.com
      secretName: tls-ingress
  rules:                       
    - host: develop.mydomain.com
      http:                    
        paths:                 
          - backend:           
              service:         
                name: flink-job
                port:          
                  number: 8081 
            path: /            
            pathType: Prefix   

The question about highAvailability

In HA configuration https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/jobmanager_high_availability.html#standalone-cluster-high-availability
{{- if .Values.jobmanager.highAvailability.enabled }}
masters: |
{{ include "flink.fullname" . }}-jobmanager
{{- end }}
What's meaning?

Missing capital letter in "zookeeperRootpath" value

Hi, I've found that during the start of containers there is an error regarding parameter high-availability.zookeeper.path.root.

kubectl logs pod/flink-taskmanager-0

<snip>
TM_RESOURCES_JVM_PARAMS extraction logs:
<snip>
 - Error while trying to split key and value in configuration file /opt/flink/conf/flink-conf.yaml:13: "    high-availability.zookeeper.path.root: "
<snip>

The issue is caused by the following line in _flink_config.tpl:

high-availability.zookeeper.path.root: {{ .Values.jobmanager.highAvailability.zookeeperRootpath }}

As you can see there should be capital letter in zookeeperRootpath (in the values.yaml it is spelled correctly).

I've prepared fix for this issue here: #13

storageClass is not respected

I am trying to change the storageClass for persistant volumes for both the Job Manager and Task Manager

flink:
  image.repository: flink
  workers: 1
  taskmanager:
    replicaCount: 3
    persistent:
      enabled: true
      storageClass: <storageClassName>
      size: 7Gi
  jobmanager:
    persistent:
      enabled: true
      storageClass: <storageClassName>

However, no matter what I do, the default storage class is always used.

HA Zookeeper doesn't work with values.yaml

When running the HA set up Flink won't properly be able to connect to ZK.

This is what is ran:

helm install my-flink --namespace flink . --set \
zookeeper.enabled=true,jobmanager.replicaCount=2,jobmanager.highAvailability.enabled=true,jobmanager.highAvailability.storageDir=s3://bucket_name/flink/jobmanager

The issue is with the values.yaml:
highAvailability.zookeeperConnect: zookeeper:2181

As ZK will be installed as a dependency chart it'll also inherit the prefix of the chart name.

The ZK service won't be called zookeeper but instead in this example my-flink-zookeeper

Trick to use Logback instead of log4j (+ send to graylog)

I struggled to accomplish this with your chart so I thought about sharing, I can't do a PR because our code has derived a bit or maybe later.

https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/best_practices.html#use-logback-when-running-flink-on-a-cluster

# Insert all jars required said in the doc (logback-classic.jar,logback-core.jar,log4j-over-slf4j.jar)
rm -rf /opt/flink/lib/*log4j* /opt/flink/conf/*log4j*

That should make Flink use logback.

Pushing to Graylog with https://github.com/osiegmar/logback-gelf:

<?xml version="1.0" encoding="UTF-8" ?>
  <configuration debug="true">
    <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator"/>

    <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
      <encoder>
        <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{60} %X{sourceThread} - %msg%n</pattern>
      </encoder>
    </appender>

    <appender name="text" class="de.siegmar.logbackgelf.GelfTcpTlsAppender">
      <graylogHost>MY_GRAYLOG_HOST</graylogHost>
      <graylogPort>MY_GRAYLOG_PORT</graylogPort>
    </appender>

    <appender name="async-text" class="ch.qos.logback.classic.AsyncAppender">
      <appender-ref ref="text" />
    </appender>

    <root>
      <level value="${test.log.root.level:-debug}"/>
      <appender-ref ref="async-text"/>
      <appender-ref ref="console"/>
    </root>
</configuration>

Open source license

Hi! I was curious if you'd be willing to add an open source license to this Helm chart. It looks very useful. No worries if not.

Error: missing required field "serviceName"

Hi! I'm trying to install Flink on my Kubernetes cluster with this command:
helm install my-flink --namespace my-namespace riskfocus/flink --version 0.1.9

And I'm receiving the error below:

Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(StatefulSet.spec): missing required field "serviceName" in io.k8s.api.apps.v1.StatefulSetSpec

How can I get rid from this error?

ServiceMonitor selector label default value update

In The chart 0.1.12 values.yaml file, tehere is a field:
prometheus.serviceMonitor.selector.prometheus: kube-prometheus
The comment is not clear about the usage of this field and also no documentation about. This field should be the value from
kubectl get prometheus -o yaml -n <prometheus namespace> serviceMonitorSelector.matchLabels.

In the case prometheus-operator, the value should be release: prometheus-operator.

Then the servicemonitor could be discovered by prometheus in targets

Hadoop Error On Startup

When starting the helm the job manager complains in the k logs of some issues related to hadoop.

2020-04-28 19:09:05,604 WARN  org.apache.hadoop.util.NativeCodeLoader                       - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2020-04-28 19:09:05,730 ERROR org.apache.flink.runtime.security.modules.HadoopModuleFactory  - Cannot create Hadoop Security Module.
java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/HdfsConfiguration
	at org.apache.flink.runtime.security.modules.HadoopModuleFactory.createModule(HadoopModuleFactory.java:50)
	at org.apache.flink.runtime.security.SecurityUtils.install(SecurityUtils.java:64)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.installSecurityContext(ClusterEntrypoint.java:200)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:166)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:518)
	at org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint.main(StandaloneSessionClusterEntrypoint.java:64)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.HdfsConfiguration
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 6 more