logs question about docker-flink

[kuraninja]

I think you can set the log4js to output into console in that case
log4j.rootLogger=DEBUG, console, file
@patricklucas I still have the issue to pull the log for web ui for flink as well....
I tried to modify the log4j config
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=/opt/flink/log/output.log
Can you help me further ?
Thanks

Budi

[patricklucas]

Hi JP,

This is a known but not very well documented consequence of trying to make Flink more "Dockery": the expectation for Docker images that obey best practices is to do all their logging to stdout/stderr so that the output can be picked up by the Docker daemon and become accessible via docker logs, kubectl logs, a log-collecting Kubernetes DaemonSet etc.

One possibility I've considered is to make the Log4j config included in the Docker images write to the console as well as the log directory, but with aggressive size-based rotation of the files on disk. That way, the most recent logs could remain accessible from the UI without risking using too much space.

[jdevoo]

Thank you Patrick!
I should have mentioned that my test task uses System.out.println - if I can see the output on the console that would be OK too. But nothing is output.
And I didn't see from the Dockerfile definition what prevents log files from being fetched. I still tried setting the log.file through the YAML file but no luck.

[kuraninja]

I put
log4j.rootLogger= INFO, stdout
#define stdout file
log4j.appender.stdout=org.apache.log4j.FileAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.file=/opt/flink/log/output.out

and in flink-config i put
jobmanager.web.log.path: /opt/flink/log/output.log
taskmanager.log.path: /opt/flink/log/output.out

And the flink on kubernetes starts outputting the logs and stdout

Not sure if its the correct approach
@patricklucas can you help us further
Thanks

Budi

[jdevoo]

Following-up on my case. I have followed Budi's advice and simply substituted by old System.out.println() for LOG.info() calls so no change to the default log4j.properties.
That said, I am still struggling as my sample code stops logging to console after a certain call. I raised it to the flink-user list (permalink) as well as to the project (AMIDST, issue here)

[jdevoo]

the image seems to use log4j-console.properties (flink-console.sh) and I can see LOG.info() statements on the console. Still no way to get anything created under /opt/flink/log - tried setting all sorts of environment and JVM variables;
@kuraninja 's setting above (along with a custom log4j.properties) did not help me get any output under /opt/flink/log so I reverted to
just adding FLINK_ENV_JAVA_OPTS=-Dlog.file=/opt/flink/log/taskmanager.log to docker-compose.yaml and using the default log4j-console.properties
Running a task and inspecting the Logs under TaskManagers in the UI throws an exception
java.util.concurrent.CompletionException: java.io.IOException: TaskManager log files are unavailable. Log file could not be found at /opt/flink/log/taskmanager.log.
I just wanted to share this before closing because it does not seem related to this image.
What worked for me was to created a log4j.properties on the host handling console and file logging and exposing that file to the image under /opt/flink/conf/log4j-console.properties which is used by flink-console.sh

[jdevoo]

I am still struggling with logging under Docker. I have not restricted myself to a docker-compose file with one taskmanager and one jobmanager. Then, I simply upload WordCount through the console.
As soon as it hits a System.println.out() I get the following entry
taskmanager_1 | The operator name DataSource (at getDefaultTextLineDataSet(WordCountData.java:70) (org.apache.flink.api.java.io.CollectionInputFormat)) exceeded the 80 characters length limit and was truncated.

[jdevoo]

Not sure what the above message means but it doesn't seem specific to println.
I fail to understand why the following snippet of code fails to output clear statements of progress:

public class ParallelMLExample {
        private static final Logger LOG = LoggerFactory.getLogger(ParallelMLExample.class);
        public static void main(String[] args) throws Exception {
                final ExecutionEnvironment env;
                env = ExecutionEnvironment.getExecutionEnvironment();
                DataSet<Integer> amounts = env.fromElements(1, 29, 40, 50);
                LOG.info("########## BEFORE UPDATEMODEL ##########");
                List<Integer> collect = amounts.filter(a -> a > 30).reduce((integer, t1) -> integer + t1).collect();
                LOG.info("########## AFTER UPDATEMODEL ##########");
                LOG.info(collect.get(0).toString());
        }
}

all I get on the terminal is the following:

jobmanager_1   | Starting Job Manager
jobmanager_1   | config file: 
jobmanager_1   | jobmanager.rpc.address: jobmanager
jobmanager_1   | jobmanager.rpc.port: 6123
jobmanager_1   | jobmanager.heap.mb: 1024
jobmanager_1   | taskmanager.heap.mb: 1024
jobmanager_1   | taskmanager.numberOfTaskSlots: 1
jobmanager_1   | taskmanager.memory.preallocate: false
jobmanager_1   | parallelism.default: 1
jobmanager_1   | web.port: 8081
jobmanager_1   | blob.server.port: 6124
jobmanager_1   | query.server.port: 6125
jobmanager_1   | Starting jobmanager as a console application on host b66ca9ea48e2.
taskmanager_1  | Starting Task Manager
taskmanager_1  | config file: 
taskmanager_1  | jobmanager.rpc.address: jobmanager
taskmanager_1  | jobmanager.rpc.port: 6123
taskmanager_1  | jobmanager.heap.mb: 1024
taskmanager_1  | taskmanager.heap.mb: 1024
taskmanager_1  | taskmanager.numberOfTaskSlots: 4
taskmanager_1  | taskmanager.memory.preallocate: false
taskmanager_1  | parallelism.default: 1
taskmanager_1  | web.port: 8081
taskmanager_1  | blob.server.port: 6124
taskmanager_1  | query.server.port: 6125
taskmanager_1  | Starting taskmanager as a console application on host 5f6223cd3e71.
jobmanager_1   | Slf4jLogger started
taskmanager_1  | Slf4jLogger started
jobmanager_1   | ########## BEFORE UPDATEMODEL ##########
taskmanager_1  | The operator name DataSource (at main(ParallelMLExample.java:30) (org.apache.flink.api.java.io.CollectionInputFormat)) exceeded the 80 characters length limit and was truncated.

Happy to close this if it's not strictly Docker related but as noob in Flink, it's hard to tell. Logging or debugging more complex algorithms has proven so far impossible.

[jdevoo]

closing as unrelated to logging. I can successfully see program output using start-cluster.sh but not using the docker images. Still figuring out.

[wcwang07]

@jdevoo have you had success logging using flink docker image, I am trying to configure logback.xml by specifying logger class in the the configuration, will try to see if it can enable logging from other class added during job running.

[jdevoo]

sort of; logging goes to the console; see example attached using cmder on Windows. Started Flink with docker-compose (1xTM, 1xJM) on the left. On the right, I run a third docker image to execute the job which collects output (docker run -t).
So yes, I stuck using these docker images just with a different entry point (unrelated I believe - it's been a while).

[diman82]

This might help.

[charybr]

One workaround is
https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/kubernetes.html
Modify
then tail -f -n +1 log/*taskmanager*.log;
to
tail -F log/*taskmanager*.log;

Refer #87