lyft / flinkk8soperator

Kubernetes operator that provides control plane for managing Apache Flink applications

Flinkk8soperator

FlinkK8sOperator is a Kubernetes operator that manages Flink applications on Kubernetes. The operator acts as control plane to manage the complete deployment lifecycle of the application.

Project Status

Beta

The operator is in use for some less-critical jobs at Lyft. At this point the focus is on testing and stability. While in Beta, we will attempt to limit the number of backwards-incompatible changes, but they may still occur as necessary.

Prerequisites

• Version >= 1.10 of Kubernetes (versions < 1.13 require --feature-gates=CustomResourceSubresources=true)
• Version >= 1.7 of Apache Flink.

Overview

The goal of running Flink on Kubernetes is to enable more flexible, lighter-weight deployment of streaming applications, without needing to manage infrastructure. The Flink operator aims to abstract out the complexity of hosting, configuring, managing and operating Flink clusters from application developers. It achieves this by extending any kubernetes cluster using custom resources.

The Operator creates flink clusters dynamically using the specified custom resource. Flink clusters in kubernetes consist of the following:

• JobManager Deployment
• TaskManager Deployment
• JobManager Service
• JobManager Ingress for the UI (optional)

Deploying and managing Flink applications in Kubernetes involves two steps:

• Building Flink application packaged as a docker image: A docker image is built containing the application source code with the necessary dependencies built in. This is required to bootstrap the Jobmanager and Taskmanager pods. At Lyft we use Source-To-Image S2I as the image build tool that provides a common builder image with Apache Flink pre-installed. The docker image could be built using any pre-existing workflows at an organization.

• Creating the Flink application custom resource: The custom resource for Flink application provides the spec for configuring and managing flink clusters in Kubernetes. The FlinkK8sOperator, deployed on Kubernetes, continuously monitors the resource and the corresponding flink cluster, and performs actions based on the diff.

Documentation

• Quick start guide
• User guide
• Flink application custom resource
• Operator state machine

Community

• Join our Slack channel.
• Check out who is using FlinkK8sOperator.

We welcome you to contribute and make the operator better! For questions, and changes please create an issue or submit a pull request.

flinkk8soperator's Issues

kubectl apply -f on running cluster

Hello!
I have next FlinkApplication:
flink-operator-demo-yaml.txt
When I tried to update the configuration and run the command "kubectl apply -f flink-operator-demo-yaml.txt", I assumed that my flink cluster would restart with creating a savepoint and restore from the savepoint, but this did not happen.
Log of flinkk8operator:
log.txt

When i run "kubectl delete -f flink-operator-demo-yaml.txt" and then "kubectl create -f flink-operator-demo-yaml.txt" with "savepointPath", the flink cluster started with restoring from savepoint.

Is it correct to try to update flink-cluster "on the fly" using "kubectl apply -f" or is this not supported ?

Question: why are the Pods not deleted after the job has finished?

I successfully ran the wordcount example job by following the quickstart guide, but I noticed the JobManager and TaskManager Pods are not deleted after the job status has changed to FINISHED. I assume the Operator created a Flink Job Cluster, so I expect Pods are deleted.

$ kubectl get pods -n flink-operator

NAME                                                      READY   STATUS    RESTARTS   AGE
flinkoperator-667bdd8486-v6gl4                            1/1     Running   0          32m
wordcount-operator-example-f5b2ddb2-jm-b8cbc7777-9qg84    1/1     Running   0          25m
wordcount-operator-example-f5b2ddb2-tm-6d958d5999-9v2bc   1/1     Running   0          25m
wordcount-operator-example-f5b2ddb2-tm-6d958d5999-p7nts   1/1     Running   0          25m

$ kubectl get flinkapplication.flink.k8s.io -n flink-operator wordcount-operator-example -o yaml
...

status:
  clusterStatus:
    availableTaskSlots: 4
    health: Green
    healthyTaskManagers: 2
    numberOfTaskManagers: 2
    numberOfTaskSlots: 4
  deployHash: f5b2ddb2
  failedUpdateHash: ""
  jobStatus:
    completedCheckpointCount: 0
    entryClass: org.apache.flink.WordCount
    failedCheckpointCount: 0
    health: Green
    jarName: wordcount-operator-example-1.0.0-SNAPSHOT.jar
    jobID: 4e38eedd78e36bb97eb5d6c95ead624a
    jobRestartCount: 0
    lastCheckpointTime: null
    lastFailingTime: null
    parallelism: 3
    restorePath: ""
    restoreTime: null
    startTime: "2019-06-07T22:46:59Z"
    state: FINISHED
  lastUpdatedAt: "2019-06-07T22:46:59Z"
  phase: Running

Flink cluster got into a bad state

I'm not able to delete the FlinkApplication CR object:

$ $ oc delete FlinkApplication wordcount-operator-example
flinkapplication.flink.k8s.io "wordcount-operator-example" deleted
$ oc get FlinkApplication
NAME                         AGE
wordcount-operator-example   28m

As shown above, the delete command did nothing.

The pods associated with the FlinkApp are not cleaned up either:

Out-of-date work count Docker image in Docker hub

The word count docker image has not been updated (~1 month ago) since #45 was merged (~12 days ago), and hence the quick start guide is broken (unless people build their own image against their own Docker registries, at which point the guide is no longer "quick" :-) ). Besides updating the example image in Docker hub, it would also be useful to tag it, so as to prevent this type of issues in the future.

Upgrade controller-runtime to 0.2.0

The current version is 0.2.0-beta.3. We should upgrade to 0.2.0 now that it has been released.

Sidecar container for Apache Ignite

I think this is somewhat related to #102. There is a suggestion about using mutating admission webhooks for this, but given that the sidecar pattern is so widely used I wonder if it would be worth adding support for it on the CRD directly. I wonder if there are other uses cases that may benefit from this too. Mutation admission webhooks are awesome, but probably overkill for this?

Flink Application failing with hash collision message

The flink application is failing with this message what could be the reason? how to debug?

{"json":{"app_name":"streaming-job","ns":"flink-operator","phase":"ClusterStarting"},"level":"info","msg":"Handle state skipped for application, lastSeenError UnknownMethod call failed with status FAILED and message []: found hash collision for deployment, you must do a clean deploy","ts":"2019-08-07T04:21:52Z"}
{"json":{"app_name":"streaming-job","ns":"flink-operator","phase":"ClusterStarting"},"level":"info","msg":"Handle state skipped for application, lastSeenError UnknownMethod call failed with status FAILED and message []: found hash collision for deployment, you must do a clean deploy","ts":"2019-08-07T04:21:52Z"}

Task Manager failing after upgrading the image to 0.1.3

After upgrade the docker image to docker.io/lyft/flinkk8soperator:v0.1.3 facing the following error in the task manager pod. After changing the image to 0.1.1 this works fine.

2019-07-31 22:17:25,980 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.host, $HOST_IP
2019-07-31 22:17:25,985 INFO  org.apache.flink.core.fs.FileSystem                           - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
2019-07-31 22:17:26,054 INFO  org.apache.flink.runtime.security.modules.HadoopModuleFactory  - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2019-07-31 22:17:26,074 INFO  org.apache.flink.runtime.security.SecurityUtils               - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.
2019-07-31 22:17:26,391 INFO  org.apache.flink.configuration.Configuration                  - Config uses fallback configuration key 'jobmanager.rpc.address' instead of key 'rest.address'
2019-07-31 22:17:26,395 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - Using configured hostname/address for TaskManager: $HOST_IP.
2019-07-31 22:17:26,397 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Trying to start actor system at $host_ip:0
2019-07-31 22:17:26,438 ERROR org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - TaskManager initialization failed.
java.lang.Exception: Could not create actor system
	at org.apache.flink.runtime.clusterframework.BootstrapTools.startActorSystem(BootstrapTools.java:267)
	at org.apache.flink.runtime.clusterframework.BootstrapTools.startActorSystem(BootstrapTools.java:153)
	at org.apache.flink.runtime.clusterframework.BootstrapTools.startActorSystem(BootstrapTools.java:112)
	at org.apache.flink.runtime.clusterframework.BootstrapTools.startActorSystem(BootstrapTools.java:87)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils.createRpcService(AkkaRpcServiceUtils.java:84)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.createRpcService(TaskManagerRunner.java:412)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.<init>(TaskManagerRunner.java:134)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:332)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner$1.call(TaskManagerRunner.java:302)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner$1.call(TaskManagerRunner.java:299)
	at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.main(TaskManagerRunner.java:299)
Caused by: com.typesafe.config.ConfigException$Parse: String: 55: Expecting a value but got wrong token: 'h' ('$' not followed by {, 'h' not allowed after '$') (if you intended 'h' ('$' not followed by {, 'h' not allowed after '$') to be part of a key or string value, try enclosing the key or value in double quotes, or you may be able to rename the file .properties rather than .conf)
	at com.typesafe.config.impl.ConfigDocumentParser$ParseContext.parseError(ConfigDocumentParser.java:201)
	at com.typesafe.config.impl.ConfigDocumentParser$ParseContext.parseError(ConfigDocumentParser.java:197)
	at com.typesafe.config.impl.ConfigDocumentParser$ParseContext.parseValue(ConfigDocumentParser.java:251)
	at com.typesafe.config.impl.ConfigDocumentParser$ParseContext.parseObject(ConfigDocumentParser.java:422)
	at com.typesafe.config.impl.ConfigDocumentParser$ParseContext.parseValue(ConfigDocumentParser.java:247)
	at com.typesafe.config.impl.ConfigDocumentParser$ParseContext.parseObject(ConfigDocumentParser.java:405)
	at com.typesafe.config.impl.ConfigDocumentParser$ParseContext.parseValue(ConfigDocumentParser.java:247)
	at com.typesafe.config.impl.ConfigDocumentParser$ParseContext.parseObject(ConfigDocumentParser.java:405)
	at com.typesafe.config.impl.ConfigDocumentParser$ParseContext.parseValue(ConfigDocumentParser.java:247)
	at com.typesafe.config.impl.ConfigDocumentParser$ParseContext.parseObject(ConfigDocumentParser.java:405)
	at com.typesafe.config.impl.ConfigDocumentParser$ParseContext.parseValue(ConfigDocumentParser.java:247)
	at com.typesafe.config.impl.ConfigDocumentParser$ParseContext.parseObject(ConfigDocumentParser.java:405)
	at com.typesafe.config.impl.ConfigDocumentParser$ParseContext.parse(ConfigDocumentParser.java:595)
	at com.typesafe.config.impl.ConfigDocumentParser.parse(ConfigDocumentParser.java:14)
	at com.typesafe.config.impl.Parseable.rawParseValue(Parseable.java:260)
	at com.typesafe.config.impl.Parseable.rawParseValue(Parseable.java:248)
	at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:180)
	at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:174)
	at com.typesafe.config.impl.Parseable.parse(Parseable.java:299)
	at com.typesafe.config.ConfigFactory.parseString(ConfigFactory.java:1046)
	at com.typesafe.config.ConfigFactory.parseString(ConfigFactory.java:1056)
	at org.apache.flink.runtime.akka.AkkaUtils$.getRemoteAkkaConfig(AkkaUtils.scala:601)
	at org.apache.flink.runtime.akka.AkkaUtils$.getAkkaConfig(AkkaUtils.scala:221)
	at org.apache.flink.runtime.akka.AkkaUtils.getAkkaConfig(AkkaUtils.scala)
	at org.apache.flink.runtime.clusterframework.BootstrapTools.startActorSystem(BootstrapTools.java:247)
	... 11 more

jobmanager.rpc.address doesn't get injected into application

While attempting to run a Flink job on an AWS EKS cluster using the operator, the task managers start up, but fail immediatly due to jobmanager.rpc.address not being set:

2019-06-05 13:10:48,097 ERROR org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - TaskManager initialization failed.
org.apache.flink.util.ConfigurationException: Config parameter 'Key: 'jobmanager.rpc.address' , default: null (fallback keys: [])' is missing (hostname/address of JobManager to connect to).
	at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.getJobManagerAddress(HighAvailabilityServicesUtils.java:151)
	at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:87)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.<init>(TaskManagerRunner.java:129)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:332)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner$1.call(TaskManagerRunner.java:302)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner$1.call(TaskManagerRunner.java:299)
	at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.main(TaskManagerRunner.java:299)

When describing the pod for both the jobmanager and taskmanager, I can see the jobmanager.rpc.address set under OPERATOR_FLINK_CONFIG:

OPERATOR_FLINK_CONFIG: blob.server.port: 6125

high-availability.cluster-id: foo-59fd197e
jobmanager.rpc.address: foo-59fd197e
jobmanager.heap.size: 512
jobmanager.rpc.port: 6123
jobmanager.web.port: 8081
metrics.internal.query-service.port: 50101
query.server.port: 6124
taskmanager.numberOfTaskSlots: 2

Everything is running under the flink-operator namespace.

When the taskmanager loads, it emits the following:

Starting Task Manager
config file:
jobmanager.rpc.address:
jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
taskmanager.heap.size:
taskmanager.numberOfTaskSlots:
parallelism.default: 1
metrics.internal.query-service.port:
taskmanager.host :
blob.server.port: 6124
query.server.port: 6125
used deprecated key `taskmanager.heap.mb`, please replace with key `taskmanager.heap.size`

Operator description:

apiVersion: flink.k8s.io/v1alpha1
kind: FlinkApplication
metadata:
  name: foo
  namespace: flink-operator
  annotations:
  labels:
    environment: production
spec:
  image: {{ printf "%s:%s" .Values.image.name .Values.image.tag }}
  flinkConfig:
    taskmanager.heap.size: 200
    state.backend.fs.checkpointdir: {{ .Values.job.checkpointDir }}
    state.checkpoints.dir: {{ .Values.job.checkpointDir }}
    state.savepoints.dir: {{ .Values.job.savepointDir }}
  jobManagerConfig:
    envConfig:
      env:
      - name: BAR
        value: "0"
      - name: BAZ
        value: "1"
    resources:
      requests:
        memory: "1Gi"
        cpu: "2"
    replicas: 1
  taskManagerConfig:
    envConfig:
      env:
        - name: BAR
          value: "0"
        - name: BAZ
          value: "1"
    taskSlots: 2
    resources:
      requests:
        memory: "4Gi"
        cpu: "1"
  flinkVersion: "1.8"
  jarName: "foo.jar"
  parallelism: 4
  entryClass: {{ .Values.job.entryPoint }}
  programArgs: {{ .Values.job.args }}

If I bash into the jobmanager pod and printenv, I definitely see the values being
set in the OPERATOR_FLINK_CONFIG

OPERATOR_FLINK_CONFIG=blob.server.port: 6125
jobmanager.heap.size: 512
jobmanager.rpc.port: 6123
jobmanager.web.port: 8081
metrics.internal.query-service.port: 50101
query.server.port: 6124
taskmanager.heap.size: 2048
taskmanager.numberOfTaskSlots: 2

high-availability.cluster-id: foo-59fd197e
jobmanager.rpc.address: foo-59fd197e

But they don't get injected into flink-conf.yaml:

#==============================================================================
# Common
#==============================================================================

# The external address of the host on which the JobManager runs and can be
# reached by the TaskManagers and any clients which want to connect. This setting
# is only used in Standalone mode and may be overwritten on the JobManager side
# by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable.
# In high availability mode, if you use the bin/start-cluster.sh script and setup
# the conf/masters file, this will be taken care of automatically. Yarn/Mesos
# automatically configure the host name based on the hostname of the node where the
# JobManager runs.

jobmanager.rpc.address:

# The RPC port where the JobManager is reachable.

jobmanager.rpc.port: 6123

Question: why do we have to build job JAR into the image?

Just wondering about the design, why do we have to build job JAR into the image? Wouldn't it be more user-friendly, if we use the same Flink image with job JAR downloaded dynamically? Local JAR could be an option as well.

Is there any technical difficulty preventing us from doing this?

Bare Flink cluster deployment mode

The operator is designed to manage a single job. There are use cases where it is desirable to only have the Flink cluster stood up and manually manage the job. For example, during testing I may want to just modify the job jar locally and repeatedly upload to the cluster or use a Flink deployment for experimentation purposes.

Maybe the CRD could have a field that indicates "cluster only" or "user managed" deployment that short circuits the job related state machine?

Docker build in examples failing with error

When I try to build an docker image in examples folder throws me an error.

docker build -t "onemoretest:1.0" .

Step 19/25 : RUN set -ex; wget -nv -O flink.tgz "$FLINK_TGZ_URL"; tar -xf flink.tgz --strip-components=1; rm flink.tgz; cp ./opt/flink-s3-fs-presto-1.8.1.jar ./lib/; cp ./opt/flink-s3-fs-hadoop-1.8.1.jar ./lib/; chown -R flink:flink .;
---> Running in b908558f3836

• wget -nv -O flink.tgz https://mirrors.ocf.berkeley.edu/apache/flink/flink-1.8.1/flink-1.8.1-bin-scala_2.12.tgz
  https://mirrors.ocf.berkeley.edu/apache/flink/flink-1.8.1/flink-1.8.1-bin-scala_2.12.tgz:
  2019-10-04 22:11:18 ERROR 404: Not Found.

Looks like the path if no more available https://mirrors.ocf.berkeley.edu/apache/flink/flink-1.8.1/

Add a Helm Chart

It would be useful to include a flinkk8soperator Helm Chart to simplify its deployment to k8s clusters.

Replace environment variable for configuration injection

Environment variable OPERATOR_FLINK_CONFIG should be replaced with FLINK_PROPERTIES to support the stock Flink Docker images.

docker-flink/docker-flink#82

We should also update the example to use that image.

Non-streaming jobs, Beam and Checkpointing

Hello,

I'm trying to use the flinkk8soperator with Beam (with Flink being the runner).

The operator is able to launch the Job Manager and Task manager pods and can submit the job as well. It works fine for streaming applications.

However, when I try to run a batch application, it turns out that Beam does not enable checkpointing in Flink.

The k8s operator, however, assumes that checkpointing is turned on, and throws an error as the checkpoint API returns a HTTP 404.

flinkk8soperator/pkg/controller/flink/flink.go

Line 535 in f499e7f

checkpoints, err := f.flinkClient.GetCheckpointCounts(ctx, getURLFromApp(app, hash), app.Status.JobStatus.JobID)

https://github.com/apache/flink/blob/7aafb248770070f0fc1bb2bd49d7bbffbb873699/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/job/checkpoints/CheckpointingStatisticsHandler.java#L94

https://github.com/apache/beam/blob/7b3a3fa6c9291692b56dbc358dfc075724b993b6/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java#L223

Is it possible to let the operator know somehow that checkpoints are not enabled, and that a 404 error on the checkpoint API is not fatal?

Thanks!

Configuring Prometheus metric reporter

Good day.
We use Flink Prometheus metric reporter (https://ci.apache.org/projects/flink/flink-docs-release-1.9/monitoring/metrics.html#prometheus-orgapacheflinkmetricsprometheusprometheusreporter) in conjunction with Prometheus Operator ServiceMonitor (https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md#related-resources) to collect Flink cluster metrics. But when we tried to swithc to flinkk8soperator we encountered some difficulties.

To collect metrics we need to expose metric reporter port 9249 on both services- JobManager and TaskManager, but FlinkApplication configuration does not allow configuring container ports, and flinkk8soperator creates a Service only for JobManager.

Can we configure FlinkApplication to collect metrics without changing the source code of flinkk8soperator?

Support liveness and readiness probes

Supporting the injection of liveness and readiness probes for both job and task managers would protect against hanging processes and allow the running of network services like the Queryable State API on task managers.

Integrating operator with AWS role/KIAM

Hello guys,

I watched your talk @ Flink Forward and I thought it was a great addition for the community, but at the time then it was not publicly available, since it is now, i have a question

When you create the Flink job cluster for instance in EKS @ AWS, how do you put a role for the pods so they only access what the role permits them and not the whole AWS.

We are doing that with KIAM for our ordinary deployments, but the operator does not seem to have such addition, can you elaborate on the case?

Much Thanks,

Mario

TaskExecutor could not resolve address.

Error Log

2019-09-25 07:45:37,610 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Could not resolve ResourceManager address akka.tcp://flink@wordcount-operator-example-68077756:6123/user/resourcemanager, retrying in 10000 ms: Ask timed out on [ActorSelection[Anchor(akka.tcp://flink@wordcount-operator-example-68077756:6123/), Path(/user/resourcemanager)]] after [10000 ms]. Sender[null] sent message of type "akka.actor.Identify"..
2019-09-25 07:45:57,639 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Could not resolve ResourceManager address akka.tcp://flink@wordcount-operator-example-68077756:6123/user/resourcemanager, retrying in 10000 ms: Ask timed out on [ActorSelection[Anchor(akka.tcp://flink@wordcount-operator-example-68077756:6123/), Path(/user/resourcemanager)]] after [10000 ms]. Sender[null] sent message of type "akka.actor.Identify"..
2019-09-25 07:46:17,680 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Could not resolve ResourceManager address akka.tcp://flink@wordcount-operator-example-68077756:6123/user/resourcemanager, retrying in 10000 ms: Ask timed out on [ActorSelection[Anchor(akka.tcp://flink@wordcount-operator-example-68077756:6123/), Path(/user/resourcemanager)]] after [10000 ms]. Sender[null] sent message of type "akka.actor.Identify"..
2019-09-25 07:46:37,719 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Could not resolve ResourceManager address akka.tcp://flink@wordcount-operator-example-68077756:6123/user/resourcemanager, retrying in 10000 ms: Ask timed out on [ActorSelection[Anchor(akka.tcp://flink@wordcount-operator-example-68077756:6123/), Path(/user/resourcemanager)]] after [10000 ms]. Sender[null] sent message of type "akka.actor.Identify"..
2019-09-25 07:46:57,759 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Could not resolve ResourceManager address akka.tcp://flink@wordcount-operator-example-68077756:6123/user/resourcemanager, retrying in 10000 ms: Ask timed out on [ActorSelection[Anchor(akka.tcp://flink@wordcount-operator-example-68077756:6123/), Path(/user/resourcemanager)]] after [10000 ms]. Sender[null] sent message of type "akka.actor.Identify"..
2019-09-25 07:47:17,779 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Could not resolve ResourceManager address akka.tcp://flink@wordcount-operator-example-68077756:6123/user/resourcemanager, retrying in 10000 ms: Ask timed out on [ActorSelection[Anchor(akka.tcp://flink@wordcount-operator-example-68077756:6123/), Path(/user/resourcemanager)]] after [10000 ms]. Sender[null] sent message of type "akka.actor.Identify"..
2019-09-25 07:47:37,819 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Could not resolve ResourceManager address akka.tcp://flink@wordcount-operator-example-68077756:6123/user/resourcemanager, retrying in 10000 ms: Ask timed out on [ActorSelection[Anchor(akka.tcp://flink@wordcount-operator-example-68077756:6123/), Path(/user/resourcemanager)]] after [10000 ms]. Sender[null] sent message of type "akka.actor.Identify"..
2019-09-25 07:47:57,849 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Could not resolve ResourceManager address akka.tcp://flink@wordcount-operator-example-68077756:6123/user/resourcemanager, retrying in 10000 ms: Ask timed out on [ActorSelection[Anchor(akka.tcp://flink@wordcount-operator-example-68077756:6123/), Path(/user/resourcemanager)]] after [10000 ms]. Sender[null] sent message of type "akka.actor.Identify"..

deploy yaml

apiVersion: flink.k8s.io/v1beta1
kind: FlinkApplication
metadata:
  name: wordcount-operator-example
  namespace: flink-operator
  annotations:
  labels:
    environment: development
spec:
  image: lyft/wordcount-operator-example:500fe6bd40da8efca4a48bbb1104896be2c1fae8
  flinkConfig:
    taskmanager.heap.size: 200
    state.backend.fs.checkpointdir: file:///checkpoints/flink/checkpoints
    state.checkpoints.dir: file:///checkpoints/flink/externalized-checkpoints
    state.savepoints.dir: file:///checkpoints/flink/savepoints
  jobManagerConfig:
    resources:
      requests:
        memory: "200Mi"
        cpu: "0.2"
    replicas: 1
  taskManagerConfig:
    taskSlots: 2
    resources:
      requests:
        memory: "200Mi"
        cpu: "0.2"
  flinkVersion: "1.8"
  jarName: "wordcount-operator-example-1.0.0-SNAPSHOT.jar"
  parallelism: 3
  entryClass: "org.apache.flink.WordCount"

does operator support multi containers in one pod?

My origin flink job need to access mysql instance by deploy a gcp-sql-proxy container with my job together, so does flinkk8soperator have ability to add another container ?

this is my deployment.yaml

spec:
      containers:
      - name: job-v1-dev
        image: registry.*****.com/job:dev
        imagePullPolicy: Always
        resources:
          ...
        volumeMounts:
          ...
        env:
          ...
      - name: gcp-sql-proxy
        image: gcr.io/cloudsql-docker/gce-proxy:1.11
        ...

Running an Apache Beam pipeline

What would be the correct way of launching an Apache Beam pipeline using the operator? This is the documentation provided by Apache Beam, but I'm not sure how to map it to an application on the operator.

What ingress suffix should I specify?

I was following the Quickstart Guide to try out the operator, it says

Replace the {ingress_suffix} to indicate your cluster's ingress url.

I am using GKE, I couldn't figure out what {ingress_suffix} should be. Could someone give me a pointer? Thanks!

Questions

Hi authors of the operator,

We are a few engineers from Lightbend Inc. As operating Flink on K8s is part of our product goals, we are actively reviewing the potential usage of your Flink operator in our projects. We have a few questions about your implementation.

1. In the job manager config, the operator allows the user to configure the number replicas in the job manager Deployment. Could you elaborate on why this is allowed? For election of a new job manager, the k8s Deployment will automatically create a new job manager if the previous one is gone/dead. So it seems making the number of replicas always 1 suffices. Any scenarios when multiple replicas are needed in the Deployment?

2). The way the operator launches a Flink job is through uploading the application jar to a Flink cluster, which is created first. We are wondering why you don't simply tell the user to put the application jar on the Flink classpath (e.g. $FLINK_HOME/lib) when building the Docker image? Then there's no need for a separate step to upload the jar. It seems more straightforward that way.

Another issue that we've been looking at is Flink doesn't have native support for checkpointing on K8s. On K8s, we think the better alternative to do checkpointing (compared to using ZooKeeper) is to use a shared Persistent Volume Claim (PVC). From the standpoint of the job manager and task manager pods, checkpointing to a mounted PVC is the same as writing to a local file system. The PVC can be any technology that supports sharing (e.g. NFS, GlusterFS, and Portworx). But Flink itself doesn't yet support checkpointing to filesystem. We have done a PoC implementation for this feature. We will be looking at contributing it back to Flink.

Would you guys be interested in having a conversation with us on potential collaborations on the operator and Flink on K8s?

Support zero-downtime deployments

The Dual deployment mode still incurs downtime while the old job is savepointed and then restored onto the new cluster. Some jobs can tolerate having more than one job instance running concurrently. Supporting a Concurrent deployment mode which savepoints the original job without cancelling, launches the savepoint onto the new cluster, and then tears down the old cluster once the new job is running would allow zero-downtime deployments for jobs that support multiple concurrent instances.

bad state after delete issued

similar issue to #13

basically, there was an access issue to the save point location (i was using S3), and after seeing the logs that something was wrong kubectl delete -f myFlinkApp.yaml

however the jobs would not delete, and the following error message is looped infinitely:
{"json":{"app_name":"davis-wordcount-operator-example","ns":"flink-operator","phase":"Deleting"},"level":"info","msg":"Logged Warning event: SavepointFailed: Failed to take savepoint {java.util.concurrent.CompletionException java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: org.apache.flink.runtime.checkpoint.CheckpointException: Failed to trigger savepoint. Failure reason: Not all required tasks are currently running.\n\tat org.apache.flink.runtime.scheduler.LegacyScheduler.lambda$triggerSavepoint$0(LegacyScheduler.java:510)\n\tat java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)\n\tat java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)\n\tat java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)\n\tat org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:397)\n\tat org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:190)\n\tat org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)\n\tat org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)\n\tat akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)\n\tat akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)\n\tat scala.PartialFunction.applyOrElse(PartialFunction.scala:123)\n\tat scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)\n\tat akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)\n\tat scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)\n\tat scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)\n\tat scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)\n\tat akka.actor.Actor.aroundReceive(Actor.scala:517)\n\tat akka.actor.Actor.aroundReceive$(Actor.scala:515)\n\tat akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)\n\tat akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)\n\tat akka.actor.ActorCell.invoke(ActorCell.scala:561)\n\tat akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)\n\tat akka.dispatch.Mailbox.run(Mailbox.scala:225)\n\tat akka.dispatch.Mailbox.exec(Mailbox.scala:235)\n\tat akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)\n\tat akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)\n\tat akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)\n\tat akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)\nCaused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.checkpoint.CheckpointException: Failed to trigger savepoint. Failure reason: Not all required tasks are currently running.\n\tat java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)\n\tat java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)\n\tat java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)\n\tat java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:614)\n\tat java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1983)\n\tat org.apache.flink.runtime.scheduler.LegacyScheduler.triggerSavepoint(LegacyScheduler.java:504)\n\tat org.apache.flink.runtime.jobmaster.JobMaster.triggerSavepoint(JobMaster.java:638)\n\tat sun.reflect.GeneratedMethodAccessor113.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:279)\n\tat org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:194)\n\t... 22 more\nCaused by: org.apache.flink.runtime.checkpoint.CheckpointException: Failed to trigger savepoint. Failure reason: Not all required tasks are currently running.\n\tat org.apache.flink.runtime.checkpoint.CheckpointCoordinator.triggerSavepointInternal(CheckpointCoordinator.java:428)\n\tat org.apache.flink.runtime.checkpoint.CheckpointCoordinator.triggerSavepoint(CheckpointCoordinator.java:377)\n\tat org.apache.flink.runtime.scheduler.LegacyScheduler.triggerSavepoint(LegacyScheduler.java:503)\n\t... 28 more\n}","ts":"2019-10-10T21:16:34Z"}

it may be nice to have pre-emptive test, or configurable number of retrys on failure.

---

after manually issuing kubectl delete deployment $nameOfFlinkAppDeployment the flinkoperator gets into a bad state trying to manage the jobs. issuing kubectl delete flinkapplication $nameOfFlinkApp results in a failure message in the operator:
{"json":{"app_name":"davis-wordcount-operator-example","ns":"flink-operator","phase":"Deleting"},"level":"warning","msg":"Failed to reconcile resource flink-operator/davis-wordcount-operator-example: GetJobs call failed with status FAILED and message []: Get http://davis-wordcount-operator-example-0a501337.flink-operator:8081/jobs: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)","ts":"2019-10-10T21:26:47Z"}
The workaround was to delete the finalizer when kubectl edit flinkapplication $nameOfFlinkApp

Wordcount example tags not accessible

The quick start guide mentioned that all tags for the wordcount image can be found at https://cloud.docker.com/u/lyft/repository/registry-1.docker.io/lyft/wordcount-operator-example/tags. But it's not accessible to the public. I got a 404. Is this intended? On the other hand, I do have access to this list of tags. Did you mean to put this link in the doc?

Flink Job is not started automatically on deployment of Flink job cluster

I tried to deploy the wordcount-operator-example to my Kubernetes cluster. This works so far as all pods are running and I can visit the Web UI of the Flink cluster.

However, the Flink job in the JAR placed under /opt/flink/flink-web-upload in the Docker image is not started and also not listed under "Submit a new Job". Is this normal behaviour?

I'm using Kubernetes 1.14 and version 0.3.0 of the flinkk8soperator.

Please find the output of kubectl describe flinkapplication.flink.k8s.io -n flink-operator wordcount-operator-example attached.
wordcount-operator-example.txt

The log of the operator contains many lines of the shaped attached below (note that the wordcount-operator-example was successfully deleted as a custom resource before).
flinkk8soperator.log

Question: volumes/volumeMounts in the CRD

I'm trying to understand volumes/volumeMounts in the CRD. If I specify volumes/volumeMounts, are they available in every Pod including JobManager and TaskManager? What are the typical use cases? Can I specify different volumes/volumeMounts for my application, JobManager and TaskManager?

transfer a running job to flinkk8soperator

Is it possible to tranfer a running flink job to flinkk8soperator seamlessly, especially recover from a savepoint?

Question: How can I work with savepointing periodically?

Thank you for developing and open-sourcing this operator!
I'm a data engineer
we're reviewing the potential usage of your Flink operator in our projects.
we have a few question!!

1. we want to run savepointing periodically (ex . every 1 hour)
2. for data recovery or data reload ...
   we want to run a new cluster that run as a savepoint at a specific point in time

Is this possible ?

Or what is the best way to do this?

Question: Gitter/Slack chat room?

Is there a Gitter or Slack chatroom for asking questions and/or sharing information?

Question: why is flink-conf.yaml needed for building job image?

I found there are 2 ways I can specify Flink config,

1. one is the flinkConfig property of the custom resource, for example:

spec:
  image: docker.io/lyft/wordcount-operator-example:{sha}
  flinkConfig:
    taskmanager.heap.size: 200
    state.backend.fs.checkpointdir: file:///checkpoints/flink/checkpoints
    state.checkpoints.dir: file:///checkpoints/flink/externalized-checkpoints
    state.savepoints.dir: file:///checkpoints/flink/savepoints

2. flink-conf.yaml which is copied into the job Docker image.

So, my question is why there are 2 ways of specifying Flink config? In particular, why is the 2nd needed?

Configure ingress class

Many clusters have several ingress classes deployed and it would be quite useful to have ability to specify ingress class in flinkk8soperator config.

Extract Go client for Flink into a separate project?

I noticed there is a client wrapper around the Flink REST API in this project. Wondering is it possible to extract the Go client into a separate project? That would be useful for many projects.

Adding new service port to Job manager service

Hi,
We are trying to expose metrics vis Prometheus reporter. We added prometheus reporter and port details to flink config. We want this port to be exposed from service. But I see only below mentioned ports exposed.

flinkk8soperator/pkg/controller/flink/job_manager_controller.go

Lines 216 to 237 in f687fe5

	func getJobManagerPorts(app *v1beta1.FlinkApplication) []coreV1.ContainerPort {
	return []coreV1.ContainerPort{
	{
	Name: FlinkRPCPortName,
	ContainerPort: getRPCPort(app),
	},
	{
	Name: FlinkBlobPortName,
	ContainerPort: getBlobPort(app),
	},
	{
	Name: FlinkQueryPortName,
	ContainerPort: getQueryPort(app),
	},
	{
	Name: FlinkUIPortName,
	ContainerPort: getUIPort(app),
	},
	{
	Name: FlinkInternalMetricPortName,
	ContainerPort: getInternalMetricsQueryPort(app),
	},

Is there a way we can add additional ports to the spec. ?

Support securityContext settings in jobManager and taskManager spec

It would be great if the jobManager and taskManager specs in the CRD supported setting other Pod specs, like securityContext. For some environments there are requirements to runAsNonRoot or running as a specific uid. Since the operator handles the pod and container creation, there is no way at the moment to provide those specs to the generated objects.

Cannot delete application under certain conditions

Trying to run kubectl delete on a previously created application (using the word count example) will hang forever under certain conditions. I run into this situation completely by mistake: I added zookeeper support to my FlinkApplication resource and made the mistake of pointing high-availability.storageDir to a local directory that did not exist on the container, causing the Job Manager to fail when trying to create the directory (probably because of a lack of permissions). Once in this state, the operator cannot upload tasks to the Job Manager.

I naturally decided to run kubectl delete on the application and fix my mistake, but at this point kubectl will simply wait forever and do nothing. Looking at the deployments, everything seems to be intact (i.e., nothing is being deleted). I even tried to manually delete everything, to no avail: Kubernetes says that the application is in the process of being deleted and refuses to let go, even after trying to force deletion. I'd be more than happy to provide logs upon request.

Update: I can create and delete new applications (without the mistake I mention above) that work just fine, but I just can't delete the faulty one

Add this project to https://flink-packages.org/

Hi,
We are about to announce https://flink-packages.org/, a website dedicated to the broader Flink ecosystem. We want to give open source projects around Flink more visibility.
Could somebody from the project add the package on the site?

How to add a jar from local path

Hello,

If for instance you want to add jar from local path, how would you do it? It appears that it is not possible at the moment. You have to have the code for the jar located because the Dockerfile actually mvn packages the whole thing?
For instance if i want to run the flink-examples, i'd have to clone the code, rather than download the jar or even refer it from mvnrepository.

Thanks!

Provide a README and/or example showing how to use this with Beam

In addition, a Dockerfile demonstrating how to combine flink and beam dependencies would be very informative.

Sidecar container for Beam SDK

The proposal to support Beam Python applications with the Flink k8s operator requires the addition of a sidecar container for the Beam SDK worker pool to the Flink task manager pod.

The sidecar container needs to be specified with image, resources and entry point, but generally it is just another container that has access to external resources similar to the task manager container.

Since there isn't anything Beam specific per se, should the CRD offer something like a taskManagerSideCar section that could also be used for other containers the user may wish to run alongside a task manager in the same pod?

flinkapplication conflict with other deployment which describe in flink doc

I have several job deploy in k8s with flink doc described way, they work correctly. And now I deploy a job by flinkk8soperator, but it is conflict with others. The jobmanager deployed by k8soperator connect to the taskmanager which is another deployment of flink job. All deployment are zookeeper ha enable.
How to solve this problem? thanks.

Replace dep with Go module

This repo currently uses dep to manage its package dependencies. The dep functionality is now legacy and has been replaced with Go modules. The repo should be converted to a Go module.

In addition, this repo uses a makefile to build its executable. It appears that this is unneeded. Normal use of Go tooling should be sufficient to build its executable.

"kubectl -n flink-operator apply -f" does not work as expect

hi! i have a yaml like this

`apiVersion: flink.k8s.io/v1beta1
kind: FlinkApplication
metadata:
name: hotitems-dev
namespace: flink-operator
annotations:
labels:
environment: development
spec:
image: ***/hotitems:dev
imagePullPolicy: Always
imagePullSecrets:

• name: regcred
  flinkConfig:
  taskmanager.heap.size: "4096m"
  state.backend.fs.checkpointdir: hdfs:///checkpoints-backend
  state.checkpoints.dir: hdfs:///checkpoints
  state.savepoints.dir: hdfs:///savepoints
  taskmanager.network.memory.min: "64mb"
  taskmanager.network.memory.max: "1gb"
  taskmanager.network.memory.fraction: 0.125
  jobManagerConfig:
  resources:
  requests:
  memory: "1Gi"
  cpu: "4"
  replicas: 1
  nodeSelector:
  attr: logic
  taskManagerConfig:
  taskSlots: 12
  resources:
  requests:
  memory: "12Gi"
  cpu: "4"
  nodeSelector:
  attr: logic
  flinkVersion: "1.9.1"
  jarName: "HotItems-1.0-SNAPSHOT.jar"
  parallelism: 1
  savepointPath: hdfs:///savepoints
  entryClass: "***.HotItems"
  deleteMode: Savepoint`,

and when i execute the cmd like this "kubectl -n flink-operator apply -f ", it print "flinkapplication.flink.k8s.io/hotitems-dev configured" ,but the operator does not submits a new , what's happen?

thanks!

Print in the Flink application does not show up in the container logs

Print statements like this one is supposed to print to stdout. But from the taskmanager container logs, I don't see it. Is it some configuration that I'm missing? Any help getting it to show?

Support specifying a service account for JM and TM deployments

The Flink operator deployment itself has a configurable service account. But the generated JM and TM deployments both run using an unspecified (i.e. default) service account. It will be useful to allow the user to specify a service account to be used for JM and TMs.

Question: is savepointing required?

Thank you for developing and open-sourcing this operator! Maintaining Flink jobs with state on Kubernetes without it has been error prone. Quick question: I'd also like to use this operator for Flink jobs without a persistent state requirement. Is it possible to deploy a new version of a job without restoring from a savepoint when state doesn't matter?

Fractional {job,task}manager.heap.size crashes Flink

The offHeapMemoryFraction can cause fractional values to be written to flink-conf.yaml, which crashes Flink on startup:

/opt/flink/bin/config.sh: line 261: [ERROR] >> 20: syntax error: operand expected (error token is "[ERROR] >> 20")
[ERROR] Configured JobManager memory size is not a valid value. Please set 'jobmanager.heap.size' in flink-conf.yaml.

For example, the following FlinkApplication...

  jobManagerConfig:
    resources:
      requests:
        memory: "4Gi"
        cpu: "1.0"
    offHeapMemoryFraction: 0.15
    replicas: 1
  taskManagerConfig:
    taskSlots: 1
    resources:
      requests:
        memory: "4Gi"
        cpu: "1.0"
    offHeapMemoryFraction: 0.3

... causes the following to be generated:

jobmanager.heap.size: 3481.6
taskmanager.heap.size: 2867.2

Above is using v0.2.0

Have a way to disable creation of ingress url in config.yaml

Question: where did the example submit the job?

I find following description in example/README.md

The operator submits the flink job through the REST API in the Jobmanager. For this to work, the jar file of the application needs to be present in the folder as indicated by the config value web.upload.dir.

it says that flink k8s operator first deploy the job manager and then submit the job via REST API, but I cannot find related code or command.