How to configure persistent directories ? about docker-flink

[patricklucas]

Can you be more specific? It likely has to do with how you are running/using the Flink Docker image.

[jurek7]

I'd like to understand how to configure persistent directories for:

• logs
• job upload directories
• working directory

What I'd need to achieve is the single node fail-over in case of machine restarts etc.
Currently it's hard to guess how to do this since I cannot find any documentation.

My questions is similar to #10

[patricklucas]

Except for logs, it's up to the platform you're using the image with. Docker Compose will be different than Kubernetes will be different from bare Docker.

As for logs, the image is configured to send all logs to stdout (for capture by the Docker daemon), so making the logs dir persistent isn't useful.

HA mode is configured in basically the same way as Flink itself; there's not a lot different when using this image, except for differences between container platforms.

[jurek7]

I am not exactly agree about comment on different platforms because once you have reference configuration in docker-compose it's fairly easy to translate to Kubernetes configurations.

In my opinion it'd would be nice to have such reference. Please take a look on other frameworks like storm(mentioned in #10): https://hub.docker.com/_/storm/

[witzatom]

A way you could do this in a docker compose is:

version: '3'
services:
jobmanager:
image: flink:${FLINK_VERSION}
expose:
- "6123"
ports:
- "8081:8081"
volumes:
- ./flink-conf.yaml:/opt/flink/conf/flink-conf.yaml
- ./log:/opt/flink/log
environment:
FLINK_ENV_JAVA_OPTS: -Dlog.file=/opt/flink/log/jobmanager
command: jobmanager

taskmanager:
image: flink:${FLINK_VERSION}
expose:
- "6121"
- "6122"
volumes:
- ./flink-conf.yaml:/opt/flink/conf/flink-conf.yaml
- ./log:/opt/flink/log
environment:
FLINK_ENV_JAVA_OPTS: -Dlog.file=/opt/flink/log/taskmanager
depends_on:
- jobmanager
command: taskmanager
links:
- "jobmanager:jobmanager"

using a custom built image with the following logback.xml - idk how to do this with log4j, but it should not be hard to change to log4j

<timestamp key="datetime" datePattern="yyyyMMdd'T'HHmmss"/>
<property name="logfile" value ="${log.file:-./log/flink}}-${datetime}.log" />

<appender name="FILE" class="ch.qos.logback.core.FileAppender">
    <file>${logfile}</file>
    <encoder>
        <pattern>%date [%level] from %logger{10} in %thread - %message%n%xException</pattern>
    </encoder>
</appender>

<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
        <pattern>%date %highlight([%level]) from %highlight(%logger{10}) [%thread] - %message%n%xException</pattern>
        <!-- in %thread -->
    </encoder>
</appender>

<appender name="ASYNCFILE" class="ch.qos.logback.classic.AsyncAppender">
    <appender-ref ref="FILE"/>
</appender>

<appender name="ASYNCSTDOUT" class="ch.qos.logback.classic.AsyncAppender">
    <appender-ref ref="STDOUT"/>
</appender>

<root level="INFO">
    <appender-ref ref="ASYNCFILE"/>
    <appender-ref ref="ASYNCSTDOUT"/>
</root>

Some explanation.

1. You set a java property log.file - it is a path and file prefix.
2. You use that to configure logback to log to generate a file name with a timestamp marker.
3. You mount the folder from the containers to the host.

[jurek7]

I tried similar and I think that it's not enough to guarantee fail-over and survive machine restarts.