Container output should (optionally) be captured and sent to CloudWatch or S3 about amazon-ecs-agent

[damnhandy]

@nletourneau While #123 is necessary and takes priority over this issue, I'd be using the Syslog driver to send my logs to CloudWatch Logs anyway. What I'd be interested in is being able to bypass the middleman and have a CloudWatch Logs driver that does direct and declare something like:

docker run --log-driver=cloudwatch myapp/foo

[seiffert]

I've removed my vote and exchange it for a 👎. My reasons for that are:

1. there is an awslogs Docker logs driver (see https://docs.docker.com/engine/admin/logging/awslogs/)
2. the ECS agent itself should not need to care about container logs because this would violate separation of concerns. If the question would be "should AWS extend ECS and somehow collect Docker logs from ECS cluster instances?", my answer would be: yes please, but maybe rather with another tool 😄

[josh-padnick]

+1 on this feature.

[seiffert]

+1!

[rpiskule]

+1

[nritholtz]

+1

[tj]

and/or syslog driver then you can just syslog -> {loggly, papertrail, *}

[damnhandy]

Docker 1.6 added pluggable drivers, it would be interesting if there could be a CloudWatch Logs driver in addition to the existing Console and Syslog drivers

[nickblt]

#123 is a bit more concise and would use the built in docker features instead of something amazon specific.

[edevil]

+1

[samuelkarp]

I've written an experimental logging driver for Amazon CloudWatch Logs and sent a pull request to the Docker project: moby/moby#15495.

[felipefdl]

+1

[damnhandy]

👍 Nice work @samuelkarp . It's probably better to have this be a part of Docker core instead.

[hirokiky]

+1

[IPvFletch]

@samuelkarp Hey there - I have tried out your new driver, using today's docker-1.9.0-dev build. It seems to work for the most part, great job!

I do see an occasional error from my daemon:

ERRO[1839] Failed to put log events                      errorCode=InvalidSequenceTokenException logGroupName=awslogs logStreamName=apps message=The given sequenceToken is invalid. The next expected sequenceToken is: 49540112654804690997104352997714685391863631165190586194 origError=<nil>
INFO[2031] GET /v1.17/version                           
ERRO[2033] Failed to put log events                      errorCode=InvalidSequenceTokenException logGroupName=awslogs logStreamName=apps message=The given sequenceToken is invalid. The next expected sequenceToken is: 49540112654804690997104353009076170244601927144195506002 origError=<nil>

There's a bunch of these in my docker daemon output.. Not sure what this means though, any ideas??

[samuelkarp]

@IPvFletch Can you open a separate issue for troubleshooting the new logging driver? I'd like to see how you're running the containers (specific log-opt arguments you're passing to the daemon and docker run).

[ChristerWendel]

+1

[astewart-twist]

+1

[asmarques]

+1

[owaaa]

+1

[gabadi]

+1

[dannymcpherson]

+1

[jfgreen]

+1

[sjevs]

👍

[bkyarger]

👍

[nicolai86]

👍

[mattsaddo]

+1

[tom-haines]

👍

[seungha-kim]

+1

[gabadi]

Maybe... but it would like to have an awslogs option in ecs, like i have a
fluentd, syslog, etc

2016-03-11 5:24 GMT-03:00 Paul Seiffert [email protected]:

I've removed my vote and exchange it for a [image: 👎]. My reasons for
that are:

1. there is an awslogs Docker logs driver (see
   https://docs.docker.com/engine/admin/logging/awslogs/)
2. the ECS agent itself should not need to care about container logs
   because this would violate separation of concerns. If the question would be
   "should AWS extend ECS and somehow collect Docker logs from ECS cluster
   instances?", my answer would be: yes please, but maybe rather with another
   tool [image: 😄]

—
Reply to this email directly or view it on GitHub
#9 (comment).

[lox]

@seiffert a few points on that:

• awslogs have a [fixed rate limit](The maximum rate of a PutLogEvents request is 5 requests per second per log stream. The maximum batch size of a PutLogEvents request is 1MB.) that can cause problems with container output. Secondly, the logs are stored in cloudwatch logs under that container id.
• As far as I can tell there is no way short of access to the ecs introspection api to match an ECS job id to a container id.

A key part of the docker experience is being able to get streaming output from containers, IMO. Especially for one-off tasks, this is critical. Our particular use case is continuous deployment and applying database migrations. There are ways to work around this, but it's annoying, and divergent from the rest of the docker ecosystem.

For those that need something workable now, checkout https://github.com/Signiant/logspout-ecs

[seiffert]

@lox you're completely right, gathering logs is extremely important. My argument was just that the ECS agent is the wrong place to implement this logic.
Docker has a plugin-system for logging drivers. In our setup, we're using a software called dockerlogstream that receives logs from the fluentd logging driver. In the dockerlogstream configuration, we're parsing the container name and add it as metadata to the log lines. (The difference to your setup is that we're streaming logs to Papertrail)
First, we used logspout as well. However, we encountered a couple of problems with the stability of logspout. Maybe, this issues have been fixed by now, I don't know.

[samuelkarp]

The awslogs driver is supported with our 1.9.0 release and lets you send container output to CloudWatch Logs. Please see the documentation for details on how to use it.

[gregeinfrank]

@samuelkarp - Is it possible to override the awslogs-group in a one-off invocation of an ECS task?

[samuelkarp]

@gregeinfrank It's not currently possible to override log options through the RunTask or StartTask API.