Prepare documentation for the official image about docker-flink

[iemejia]

@patricklucas I just pushed the docs, can you take a look and tell me if you are OK with it for a first version.
https://github.com/docker-flink/docs/tree/master/flink

[patricklucas]

Added a few comments, otherwise looks great.

[patricklucas]

Actually realized I can just PR it, so I'll do that. :)

[patricklucas]

docker-flink/docs#1

[patricklucas]

Now I suppose we just wait for 1.2.1 to come out, after which I'll get rid of 1.1 and add 1.2.1, and send a PR to add the generated stackbrew library to docker-library/official images.

I have the branch ready here, ready for the release: docker-flink/official-images@5d10693

[iemejia]

Excellent, I agree 100% 1.2.1 + rid of 1.1 and your branch and go !
Ping me in any case if you need me to fix/or do something extra (oups I have to fix the update also).
This is getting quite nice, once this is official we have to announce it in the mailing list :)

[iemejia]

Two other subjects to add to the documentation (non-blocking for the official image PR):

• Add info on how to conf for HA with the zookeeper image, I have this in my TODO.
• How to configure backends for the checkpoints.
  Maybe I will create independent JIRAs for those, WDYT ? or other ideas of things to add there?

[patricklucas]

FYI I've just been trying to use these images to deploy a cluster on Kubernetes and found I can't actually run a job. It seems the jobmanager picks an arbitrary port by default for the blob server, and taskmanagers can't connect to download the jar.

Another thing is that I'm still not totally convinced the logic we have in docker-entrypoint.sh should be in this repo vs. apache/flink. Many other projects simply proxy commands to a binary from the actual release. Another option, used by docker-solr, is to put scripts on the PATH—in our case, an option would be to have a dir on the PATH with scripts named 'jobmanager', 'taskmanager' and so forth.

[patricklucas]

I've solved the blob server problem (I think). I'll send a PR soon.

[iemejia]

Great! I had kubernetes as next in my TODO list just after the docker flink image got official, so maybe we can add that doc too, are you aware of this JIRA ?
https://issues.apache.org/jira/browse/FLINK-5966
Can you share me somewhere your k8s scripts (or even better in the flink repo) if you had success please. I am a noob in kubernetes but I need to get up to speed for another task I have in my TODO too (this time for Apache Beam).

[iemejia]

About the docker-entrypoint we are not so far of this, the current entrypoint makes two things:

1. It changes the configuration based on the hostnames (so the servers meet each other).
2. It starts the respective daemons.

We can cover (2) without problem just by letting the users invoke the full command (we already share FLINK_HOME/bin to the PATH if I remember well. This will work with the start-foreground out of the box e.g. docker run -it flink flink-daemon.sh start-foreground jobmanager

This is less user friendly but still valid, , The only thing we would need is a proper way to cover (1). But I have not looked enough to other docker-entrypoints for ideas, do you have any ideas for this?

[patricklucas]

I suppose my comment is more about seeing what of that logic could go in scripts in apache/flink rather than this repo.

Another kind of weird thing is how, since Flink doesn't include all possible config vars in the shipped flink-conf.yaml, we can't just sed—we have to append sometimes too.

[patricklucas]

Also, does it really even make sense to set jobmanager.rpc.address to hostname -f in Docker? Most of the time you'll get some autogenerated thing that is pretty much useless. Maybe we should default to 'jobmanager' instead.

[iemejia]

Thinking about the config vars, I wonder how many of those are already configurable via ENV variables, because this could be also a nice contribution to do if there is interest, because in the container world all those configurations can be changed from ENV, are you aware of any ongoing work on this, or do you think it makes sense to even ask maybe in the Mailing List ?

[iemejia]

For the hostname -f, yes it is kinda crappy, but it is the only way that I know to set the address in the taskmanagers if somebody for some crazy decision decides to change the name of the jobmanager image.

[patricklucas]

Not sure of ongoing discussion but I'm not sure I totally like the idea, simply from the perspective of having too many ways to specify vars.

Right now there are already config files, some env vars interpreted by flink, some env vars interpreted by scripts, and command line arguments to some scripts. I don't think any of these mechanisms actually have parity with each other, with some options duplicated and others disjoint.

Another issue is not being able to override the flink-conf.yaml location—you can only override the conf dir location (via an env var).

In my experimental Kubernetes configs (which I'll send you at some point) I'm actually doing that, but I have to write out all the configs I might need into the new directory (namely flink-conf.yaml, and log4j/logback configs) so as not to conflict with anything else. I'd also have to support hdfs and krb5 configs somehow, but I haven't gotten there yet.

[iemejia]

@patricklucas (sorry for asking this in this jira but didn't know how to contact you directly)
Question what are we missing to have the image in hub.docker.com?
I just saw your announcement but wanted to be sure about this, what do we need to use it now? just to do:

docker pull flink:1.2.1

[patricklucas]

Whoops, I realize now that I didn't make it clear in the mailing list post that this isn't actually available yet: the PRs for the images and docs are currently open on the docker-library project.

docker-library/official-images#2895
docker-library/docs#893

[iemejia]

Ok. I will follow the PRs and then test once it is ready.

[patricklucas]

You can also check out the images in the plucas/flink repo—they are identical to the ones that will be produced once they merge the PRs.

[iemejia]

I am closing this since this is ok now (please reopen if it is not the case).