Sematext Docker Agent - host + container metrics, logs & event collector
Home Page: https://sematext.com/docker
License: Apache License 2.0
You probably want to check out:
Hi
I've tried to use the agent for monitoring in a docker swarm, sending to a local elasticsearch cluster. I keep getting http errors about bulk insert:
""reason" : "Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found ".
This seems to be because the agent posting to es lacks the "action_and_meta_data\n" as line 1.
Now I've tried this with 1.31.62 against es 6.4.2 and 5.6.12 with the same problem. My question is if this setup has worked before or am I on a lonely road here?
If this is a bug, what previous release of the agent could I try instead?
Regards
/Petter
Am using tutum to deploy my containers
My container is not forwarding logs am only seeing this errors been forwarded:
{
"_index": "10896b9b-cefa-4920-b710-5b962fab388a_2015-12-25",
"_type": "weaveworks_weave_1_4_0",
"_id": "AVHar0dJK_rZzSXGHpZI",
"_score": null,
"_source": {
"@timestamp": "2015-12-25T19:48:13.629Z",
"level": "info",
"host": "sematextAgent-4",
"ip": "172.17.0.1",
"message": "ERRO: 2015/12/25 19:48:13.627792 ->[10.16.0.12:6783] error during connection attempt: dial tcp4 10.16.0.12:6783: no route to host",
"@source": "855d7c5696a1/weaveworks/weave:1.4.0",
"ts": null,
"container_id": "855d7c5696a1",
"image_name": "weaveworks/weave:1.4.0",
"container_name": "weave",
"_type": "weaveworks_weave_1_4_0"
},
"fields": {
"@timestamp": [
1451072893629
]
},
"sort": [
1451072893629
]
}
This are the logs that I see in sematext container
2015-12-25T20:02:53.197311447Z Tutum Node Hostname: 4daee7bc-tumacredo
2015-12-25T20:02:55.983962793Z No pattern file for log parsing found /etc/logagent/patterns.yml -> using default patterns
2015-12-25T20:02:55.985644158Z Use -v /mypattern/patterns.yml:/etc/logagent/patterns.yml for custom log parser definitions.
2015-12-25T20:02:56.319912147Z 2015-12-25T20:02:56.318Z - INFO - Listening for "journalctl -o json | ncat localhost PORT" on exposed port 9000
2015-12-25T20:02:56.672860493Z {"_type":"dockerEvent","@timestamp":1451073776,"message":"Docker Event: attach weaveworks/weave:1.4.0 6ebebdc07ee8","container_id":"6ebebdc07ee8","image_name":"weaveworks/weave:1.4.0","event_status":"attach"}
2015-12-25T20:02:56.675326276Z {"_type":"dockerEvent","@timestamp":1451073776,"message":"Docker Event: attach postgres:9.5 4cad2fb9c043","container_id":"4cad2fb9c043","image_name":"postgres:9.5","event_status":"attach"}
2015-12-25T20:02:56.676451651Z {"_type":"dockerEvent","@timestamp":1451073776,"message":"Docker Event: attach tumacredo/nginx:staging 5a5745331b6b","container_id":"5a5745331b6b","image_name":"tumacredo/nginx:staging","event_status":"attach"}
2015-12-25T20:02:56.677269535Z {"_type":"dockerEvent","@timestamp":1451073776,"message":"Docker Event: attach rabbitmq:3.5.7-management 0345dbaf0d3b","container_id":"0345dbaf0d3b","image_name":"rabbitmq:3.5.7-management","event_status":"attach"}
https://www.npmjs.com/package/node-diskfree
I would like to get a warning when my host is running out of disk space
There is support for SPM-only token already, so we need to do the same for Logsene-only.
So this how am running the containter
sematext-agent-docker:
image: 'sematext/sematext-agent-docker:latest'
deployment_strategy: every_node
environment:
- LOGSENE_TOKEN=XXXXX
- SPM_TOKEN=xxxxxx
privileged: true
restart: always
tags:
- production
volumes:
- '/var/run/docker.sock:/var/run/docker.sock'
So if I check at the logs am getting this:
Tutum Node Hostname: ccf49792-tumacredo
2015-12-17T08:35:24.820472622Z No pattern file for log parsing found /etc/logagent/patterns.yml -> using default patterns
2015-12-17T08:35:24.822308761Z Use -v /mypattern/patterns.yml:/etc/logagent/patterns.yml for custom log parser definitions.
2015-12-17T08:35:24.933736811Z Listening for "journalctl -o json | ncat localhost PORT" on exposed port 9000
2015-12-17T08:35:24.955305427Z Thu, 17 Dec 2015 08:35:24 GMT uncaughtException: connect ENOENT /var/run/docker.sock
2015-12-17T08:35:24.955449467Z Error: connect ENOENT /var/run/docker.sock
2015-12-17T08:35:24.955462697Z at Object.exports._errnoException (util.js:874:11)
2015-12-17T08:35:24.955467953Z at exports._exceptionWithHostPort (util.js:897:20)
2015-12-17T08:35:24.955472717Z at PipeConnectWrap.afterConnect as oncomplete
2015-12-17T08:35:24.955857719Z Thu, 17 Dec 2015 08:35:24 GMT uncaughtException: connect ENOENT /var/run/docker.sock
2015-12-17T08:35:24.955988595Z Error: connect ENOENT /var/run/docker.sock
2015-12-17T08:35:24.955999644Z at Object.exports._errnoException (util.js:874:11)
2015-12-17T08:35:24.956004458Z at exports._exceptionWithHostPort (util.js:897:20)
2015-12-17T08:35:24.956008948Z at PipeConnectWrap.afterConnect as oncomplete
The agent should not query logs from Docker API for containers having LOGSENE_ENABLED flag set to false (either by LOGSENE_ENABLED_DEFAULT=false or explicit set of LOGSENE_ENABLED=false per container).
Origin: #21 (comment)
I am trying to configure Logsene to handle multiline logs / stack dumps, but I can’t figure out where to start. We are using docker cloud, and configured Logsene according to this blog-post: https://sematext.com/blog/2016/04/04/docker-cloud-monitoring-logging/
I am at a loss as to where I would find the files described in this blog post: https://sematext.com/blog/2015/05/26/handling-stack-traces-with-logstash
I have attached a file with several examples of a stack-dumps one log per line:
We are not maintaining Sematex Agent Docker anymore.
Please use sematext/agent for monitoring and sematext/logagent for log collection.
Both images are available as Docker certified images:
See the following posts for more details:
i.e. one should be able to change the Receiver URL. Or maybe we just need to document where/how to do that.
The SPM UI provides filters for host, image and container. In Kubernetes environments it would make more sense to group containers by pod instead of grouping by image name.
Suggestion:
SDA should detect automatically containers managed by Kubernetes, instead activating Kuberntes support via KUBERNETES=1 setting.
SDA container exits with:
2016-08-05T02:19:08.445432842Z 2016-08-05T02:19:08.339Z - ERROR - Please contact [email protected] to report the error:
2016-08-05T02:19:08.446000871Z 2016-08-05T02:19:08.393Z - ERROR - UncaughtException:Error: (HTTP code 404) no such container - No such container: 3edaeb2c7a3c2354eb85ec675c65b45c053d61b21435ea066f49f8d9c82dd148
2016-08-05T02:19:08.446010838Z
2016-08-05T02:19:08.446014472Z Error: (HTTP code 404) no such container - No such container: 3edaeb2c7a3c2354eb85ec675c65b45c053d61b21435ea066f49f8d9c82dd148
2016-08-05T02:19:08.446017831Z
2016-08-05T02:19:08.446020873Z at /usr/lib/node_modules/sematext-agent-docker/node_modules/docker-modem/lib/modem.js:229:17
2016-08-05T02:19:08.446024036Z at getCause (/usr/lib/node_modules/sematext-agent-docker/node_modules/docker-modem/lib/modem.js:259:7)
2016-08-05T02:19:08.446027352Z at Modem.buildPayload (/usr/lib/node_modules/sematext-agent-docker/node_modules/docker-modem/lib/modem.js:228:5)
2016-08-05T02:19:08.446030815Z at IncomingMessage.<anonymous> (/usr/lib/node_modules/sematext-agent-docker/node_modules/docker-modem/lib/modem.js:204:14)
2016-08-05T02:19:08.446045087Z at emitNone (events.js:91:20)
2016-08-05T02:19:08.446048170Z at IncomingMessage.emit (events.js:185:7)
2016-08-05T02:19:08.446051030Z at endReadableNT (_stream_readable.js:926:12)
2016-08-05T02:19:08.446053815Z at _combinedTickCallback (internal/process/next_tick.js:74:11)
2016-08-05T02:19:08.446056675Z at process._tickCallback (internal/process/next_tick.js:98:9)
This is the :latest container.
I believe the selector is obsolete, as I get this error:
error validating data: found invalid field selector for v1.PodSpec; if you choose to ignore these errors, turn validation off with --validate=false
But if I remove it, the DaemonSet starts without errors, but I see nothing coming in...
This is from sematext/spm-agent-docker#5
If I have 2 containers, one running Apache and the other one running Elasticsearch, just for example, I may actually want to send their logs to 2 different Logsene apps.
I have more or less the same issue as this: #12
My log source is a golang package called logrus and issues single line json that looks like this:
{"context":["STRING","STRING","STRING"],"level":"info","msg":"MESSAGESTRING","stacktrace":["BLAH1", "BLAH2", "BLAH3"],"time":"2018-03-06T12:24:48Z"}
I also cant seem to map the "time" to "@timestamp" using patterns.yml. My patterns file looks like this:
patterns:
- # Web Logs
sourceName: !!js/regexp .*gateway.*
match:
- regex: !!js/regexp ^\{.*
type: absgateway_logs
fields: [context,level,msg,stacktrace,ts]
dateFormat: YYYY-MM-DDTHH:mm:ssZ
However, in my logsene dashboard, the only json that gets reported is like this:
{
"@timestamp": "2018-03-06T12:24:48.412Z",
"message": "{"context":["STRING"],"level":"info","msg":"MESSAGESTRING","stacktrace":["BLAH1", "BLAH2", "BLAH3"],"time":"2018-03-06T12:24:48Z"}",
"severity": "info",
"host": "ip-1-2-3-4",
"ip": "1.2.3.4",
"logSource": "ab.cde.io/abs/absgateway:staging_r-absStagingStack-absGatewayStaging-1-hash_hash",
"container_id": "hash",
"image_name": "ab.cde.io/abs/absgateway:staging",
"container_name": "r-absStagingStack-absGatewayStaging-1-hash",
"logType": "docker",
"container_hostname": "hash",
"@timestamp_received": "2018-03-06T12:25:25.055Z",
"logsene_original_type": "docker"
}
Maybe its relevant, I am using rancher.
Note that it works fine in logagent doing an echo "{json: blah blah}" | logagent --yaml . It parses it correctly. However the patterns file still doesnt work. "time" just gets pasted as a string AND a new @timestamp is added where it is just simply the time I typed the logagent command in.
Lastly, there's also this block of example surrounding json which is entirely confusing. Am I supposed to use the json block in tandem with the patterns block? Does the json block apply to ALL patterns regardless of logSource?
Not sure what I am doing wrong. Any help would be appreciated.
I tried to plug this in the docker /etc/logagent/patterns.yml - nope doesn't do anything. Works perfectly on my command line using normal logagent.
json:
enabled: true
autohashFields:
time: true
removeFields:
- stacktrace
transform: !!js/function >
function (sourceName, parsedObject, config) {
for (var i=0; i<config.removeFields.length; i++) {
console.log('delete ' + config.removeFields[i])
delete parsedObject[config.removeFields[i]]
}
}
Then I tried to attack the message field that shows up in the logsene like this
json:
enabled: true
autohashFields:
message: true
Vanilla nginx logs work perfectly fine. These are non-json and simply space delimited. In logsene, I can see it fully structured with method, user_agent etc. etc. I just used the default patterns.yml file (i.e. I dont pass any pattern file argument during docker run).
Nope, logsene shows non-hashed message.
The following regex application worked finally:
patterns:
- # Gateway Logrus Logs
sourceName: !!js/regexp .*gateway.*
match:
- regex: !!js/regexp \{"context":(.*),"level":(.*),"msg":(.*),"stacktrace":(.*),"time":(.*)\}
type: absgateway_logs
fields: [context,level,msg,stacktrace,ts]
dateFormat: YYYY-MM-DDTHH:mm:ssZ
So essentially we mis-assumed that json structured logs are automatically parsed but instead they are to be treated like any other single line? It doesn't sound like that in the documents we have read so far.
The documentation for the flag is missing in https://sematext.com/docs/sematext-docker-agent/configuration/#log-routing
A blog post exists:
https://sematext.com/blog/docker-log-management-enrichment/
Hi, I had used sematext with docker swarm setup and I found out that sematext-agent-docker doesn't provide service_name to sematext. I think it would be nice to add that field.
The current workaround for this is to filter based on container_name: "container.*".
The default regex for Nginx access logs doesn't seem to be correct.
The regex ends after the user agent, even though the default nginx.conf includes "$http_x_forwarded_for" right after that. At least on Ubuntu and the nginx alpine docker image.
Default Log format:
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
Most of the time, this will just appear as a "-" in the log files.
The support told me, that the whole log line has to match to get properly parsed.
Is there a way to log to a local elasticsearch container?
Thank You!
After detecting k8s via #30.
It's causing the build to fail
Write an input plugin, wich receives influx line protocol (text via HTTP). Influx format needs to be converted to JSON and emmited to Logagent eventEmitter.
SDA enriches logs with metadata fields to Docker Swarm and kubernetes logs such as swarm_node, swarm_task_id, swarm_service, Kubernetes namespace, pod name, UUID etc.
It should be possible to remove unwanted fields from log output to save storage:
-e REMOVE_LOG_FIELDS: logSource, swarm_node, swarm_task_id, image_name, container_name
An alternative would be dedicated switches to enable/disable log enrichment:
- e ENABLE_SWARM_ENRICHEMENT=true, -e ENABLE_K8S_ENRICHMENT
I'd like to run Sematext Docker Agent so that it always includes certain tags as fields in all logs it collects. I'd like to be able to do something like this:
Docker Daemon settings:
dockerd --label environment=PROD --log-driver=json-file --log-opt env=com.example.service,com.example.environment
Example service create:
docker service create --constraint engine.labels.environment==PROD --replicas 2 --name hw --env "com.example.service=FOO" --env "com.example.environment=PROD" alpine ping docker.com
This would then create log entries with field com.example.service with a value of FOO and com.example.environment with a value of PROD.
Hi guys,
I work at SBB with @szakasz. We have an unicode character that gets added to the logs some times. Example see here (https://apps.sematext.com/ui/logs/10050/0?endDate=1490856600000&query&queryFilters=%5B%7B%22f%22%3A%22kubernetes.namespace.raw%22%2C%22q%22%3A%5B%22zvs-orchestrierung-zvs-inte%22%5D%7D%5D&startDate=1490770200000). See the message from "08:04:00.000" in field severity: IN��FO.
I already debugged this issue up the dependency chain up to dockerode itself. I think it has something to do with the demultiplexing of the stream (See moby/moby#7375). I also tried with the docker-modem parser of dockerode, does not help (https://github.com/SchweizerischeBundesbahnen/sematext-file-agent/blob/master/lib/loghose/loghose.js#L66).
I also noticed, that sometimes the actual full length of the chunk is smaller than the value that's specified in the 4-bytes of the header. In the end I think it mixes up logs & header somehow and reads a part of the header as logs which results in those characters. Any ideas? Heard anything like it? We're using docker 1.10.3.
When I run the Sematext Agent on Kubernetes syslog is getting spammed on each host. The logs look like this:
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.213671166Z" level=error msg="collecting stats for 008d535a91cfd52241e3fdfc597146025f845857f120590bfd8fcffa87d7d84a: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.216322127Z" level=error msg="collecting stats for 3d8790fb03976f6216f5426ccef2370eb4146ad5f5982760d65eee11040c495a: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.361887789Z" level=error msg="collecting stats for b101b6f121ce5940ed60eb4954cca91a188337778911b477a29a9359859e9b3d: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.366114299Z" level=error msg="collecting stats for 83ed2f66a7837810c0bee3e2513bedd1ae0bc863dd82792c32e7759a9f7e3bd9: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.368616832Z" level=error msg="collecting stats for 833db20caaa9577329252adb59bc50797553f777527b3e2e45e2dcd2e5cff012: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.371344527Z" level=error msg="collecting stats for 556cfdc49652f96bc4c61f7feb88e4566271350dafd93c3a484637c43e5bf95e: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.373829194Z" level=error msg="collecting stats for f5c9c0056932667ed5d147b91139ea24883d19f61b03471e13713f67aa0a2026: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.473436115Z" level=error msg="collecting stats for 5ee2e88ece23c22bf29f3ed13d61386158658e3d4c643d606834f6d26833f706: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.476147641Z" level=error msg="collecting stats for 4294341560fdc45d31e92570afde7b676a82c76701dd1c4fc156d0676df28c1a: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.484470147Z" level=error msg="collecting stats for 18c56821fc92e316b3e39fb6200a391dffee2b225ae0e3660e4034d8a623b644: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.622034331Z" level=error msg="collecting stats for b101b6f121ce5940ed60eb4954cca91a188337778911b477a29a9359859e9b3d: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.670685643Z" level=error msg="collecting stats for 83ed2f66a7837810c0bee3e2513bedd1ae0bc863dd82792c32e7759a9f7e3bd9: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.673390768Z" level=error msg="collecting stats for 833db20caaa9577329252adb59bc50797553f777527b3e2e45e2dcd2e5cff012: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.675899686Z" level=error msg="collecting stats for f5c9c0056932667ed5d147b91139ea24883d19f61b03471e13713f67aa0a2026: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.678534432Z" level=error msg="collecting stats for 556cfdc49652f96bc4c61f7feb88e4566271350dafd93c3a484637c43e5bf95e: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.713712092Z" level=error msg="collecting stats for 5ee2e88ece23c22bf29f3ed13d61386158658e3d4c643d606834f6d26833f706: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.715560882Z" level=error msg="collecting stats for 4294341560fdc45d31e92570afde7b676a82c76701dd1c4fc156d0676df28c1a: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.718324232Z" level=error msg="collecting stats for 18c56821fc92e316b3e39fb6200a391dffee2b225ae0e3660e4034d8a623b644: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.757697729Z" level=error msg="collecting stats for e60ab9cce62b6060fa5ac06ceb5c4939528d06705ce04f15d6a8657442347a6c: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.809740245Z" level=error msg="collecting stats for 1f6aaec1658016194004064150be811849117c7327b8c51bedb75012b2ad707a: invalid id: "
Aug 17 19:40:22 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:22.982380108Z" level=error msg="collecting stats for 96045e7afd3ca551e1b582037bb3b43dc3a8fd169ae92c0f7ab351eca538f63b: invalid id: "
Aug 17 19:40:23 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:23.038991487Z" level=error msg="collecting stats for 379205c02518930cbd2d3d588d1c532acc52cc84a7e7f70d6b6708e057498c44: invalid id: "
Aug 17 19:40:23 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:23.105729662Z" level=error msg="collecting stats for 3159cb1247ca615a5c4e5e81d094903b52676a267af0800ed684540a9922b776: invalid id: "
Aug 17 19:40:23 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:23.164253373Z" level=error msg="collecting stats for 29ee26c4768e114e020faa9dbe143e3d7460c1f3348332665574bb20f231e962: invalid id: "
Aug 17 19:40:23 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:23.193878337Z" level=error msg="collecting stats for 3dcf53191563bc316a3133fb963aa5076fab5ea002bbe52cdc5d0b55d970bc01: invalid id: "
Aug 17 19:40:23 ip-172-20-208-90 docker[1124]: time="2016-08-17T19:40:23.195985819Z" level=error msg="collecting stats for 68938cd3fe4188133d7d724cf6b8180a7680a0bd4f64093f95407325a33cc429: invalid id: "
Relevant information, I'm running k8s 1.3.4, provisioned with kops on AWS. My daemonset config is basically straight from Sematext and looks like:
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: sematext-agent
spec:
template:
metadata:
labels:
app: sematext-agent
spec:
nodeSelector: {}
dnsPolicy: "ClusterFirst"
restartPolicy: "Always"
containers:
- name: sematext-agent
image: sematext/sematext-agent-docker:latest
imagePullPolicy: "Always"
env:
- name: SPM_TOKEN
value: "redacted"
- name: LOGSENE_TOKEN
value: "redacted"
- name: KUBERNETES
value: "1"
volumeMounts:
- mountPath: /var/run/docker.sock
name: docker-sock
- mountPath: /etc/localtime
name: localtime
securityContext:
privileged: true
volumes:
- name: docker-sock
hostPath:
path: /var/run/docker.sock
- name: localtime
hostPath:
path: /etc/localtime
Any help here would be appreciated, thanks!
For this work on my environment I had to use
hostNetwork: true
inside spec of the daemon set
My environment:
Azure
CoreOs 1185.2
Rancher 1.1.4
Kubernetes
With the installation of the newest helm chart, I will receive the following error.
setting LOGSENE_ENABLED=true
Possibly Unhandled Rejection at: Promise Promise {
TypeError: Cannot read property 'REMOVE_FIELDS' of undefined
at removeFields (/usr/local/lib/node_modules/@sematext/logagent/lib/plugins/output-filter/kubernetes-enrichment.js:58:33)
at processAnnotations (/usr/local/lib/node_modules/@sematext/logagent/lib/plugins/output-filter/kubernetes-enrichment.js:129:5)
at /usr/local/lib/node_modules/@sematext/logagent/lib/plugins/output-filter/kubernetes-enrichment.js:189:9
at getPodSpec (/usr/local/lib/node_modules/@sematext/logagent/lib/plugins/output-filter/kubernetes-enrichment.js:47:5)
at process._tickCallback (internal/process/next_tick.js:68:7) } reason: TypeError: Cannot read property 'REMOVE_FIELDS' of undefined
at removeFields (/usr/local/lib/node_modules/@sematext/logagent/lib/plugins/output-filter/kubernetes-enrichment.js:58:33)
at processAnnotations (/usr/local/lib/node_modules/@sematext/logagent/lib/plugins/output-filter/kubernetes-enrichment.js:129:5)
at /usr/local/lib/node_modules/@sematext/logagent/lib/plugins/output-filter/kubernetes-enrichment.js:189:9
at getPodSpec (/usr/local/lib/node_modules/@sematext/logagent/lib/plugins/output-filter/kubernetes-enrichment.js:47:5)
at process._tickCallback (internal/process/next_tick.js:68:7)
When somebody uses multiple SDA instances in different Kubernetes namespaces the containers metrics/logs can be filtered with MATCH_BY_NAME and SPM_MATCH_BY_NAME.
Events are not filtered by those filter options (container events could be independent from containers, like network create), nevertheless, it should be possible to filter container events for specific namespaces.
Hi there,
We recently had a big issue with the sematext docker agent on our OpenShift platform. This issue might be tricky - I was not yet able to reproduce it on purpose.
The issue occurs on our platform with 220 containers on one host system when a lot of those containers get high load and high log output at the same time. Those containers are not even logging to sematext, but the agent still is causing the problem. We were able to "fix" the problem with SKIP_BY_NAME=.zvs.|.nova. (the problematic containers).
So I checked the source code. It seems like SKIP_BY_NAME totally ignores those containers. Otherwise those containers are still attached and thus streamed, even if not necessary:
dockerLogsense.js > docker-loghose > docker-allcontainers: Then the containers are skipped directly & never attached & streamed
dockerLogsense.js > docker-loghose > docker-allcontainers:
As said above, I am not yet able to reproduce the issue on purpose, but skipping those containers "fixes" is. The flow without SKIP_BY_NAME seems not ideal to me, as every container is attached & streamed even if he is not enabled for logsene. Would it be possible to find another way of doing this? Maybe just return all container ids, then inspect them in logsense and only attach to them if necessary?
LOGSENE_ENABLED_DEFAULT=false
LOGSENE_ENABLED=true
Thank you in advance,
Regars Reto
@szakasz: FYI.
docker ps -s
docker image ls
docker system df -v
If limits are reached SDA should probably not log things like this every single time:
"Error sending event to SPM {"error":"Limits reached, aborting request execution","errorId":"1794547510039","status":"403"}"
...could probably reduce logging this sort of stuff to logging at most every N minutes.
Is it possible to filter/edit data in the docker agent before it is sent out to Sematext? Also, is there a way to handle multi line logs (i.e. a stack trace)? Is there a better place to ask questions such as these?
Thanks!
Saw this from the :dev container:
sematext-agent-docker-1: Container stopped with exit code 2
2016-08-04T14:19:03.835216077Z at emitNone (events.js:91:20)
The container automatically restarted. Is there a way to see the logs of the stopped container so I can provide the full stack trace?
Actually we have six of these over six nodes, and all of them are throwing this error within a few minutes of each other.
Hi,
Can we use this agent on more than 3 hosts (limit of the free SPM plan) without a paid subscription to SPM, only to route logs to logscene ?
I'm using sematext/sematext-agent-docker:latest and my logs I'm getting
7/21/2017 11:09:18 AMDocker Hostname: REMOVED
7/21/2017 11:09:19 AMdocker_id=REMOVED:REMOVED:REMOVED
7/21/2017 11:09:19 AMdocker_hostname=REMOVED
7/21/2017 11:09:19 AMNODE_VERSION=8.1.4 MAXMIND_DB_DIR=/usr/src/app/ SPM_LOG_TO_CONSOLE=true YARN_VERSION=0.24.6 HOSTNAME=REMOVED SHLVL=1 HOME=/root SPM_REPORTED_HOSTNAME=REMOVED LOGSENE_ENABLED_DEFAULT=true LOGSENE_TMP_DIR=/logsene-log-buffer TERM=xterm SPM_LOG_LEVEL=error SPM_TOKEN=REMOVED CONFIG_FILE=/run/secrets/sematext-agent PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin NPM_CONFIG_LOGLEVEL=info SPM_RECEIVER_URL=https://spm-receiver.sematext.com:443/receiver/v1/_bulk ENABLE_AUTODETECT_SEVERITY=true MAX_CLIENT_SOCKETS=1 GEOIP_ENABLED=false SPM_TRANSMIT_INTERVAL_IN_MS=10000 ENABLE_LOGSENE_STATS=false DOCKER_HOST=unix:///var/run/docker.sock SPM_COLLECTION_INTERVAL_IN_MS=10000 PWD=/usr/src/app DOCKER_PORT=2375 LOGSENE_TOKEN=
7/21/2017 11:09:19 AMhttps://spm-receiver.sematext.com:443/receiver/v1/receiver/custom/receive.json?token=REMOVED
7/21/2017 11:09:20 AM2017-07-21T18:09:20.002Z - error: Cannot initialize linux agent:SyntaxError: missing ) after argument list
7/21/2017 11:09:20 AM2017-07-21T18:09:20.005Z - INFO - No logs will be collected: missing -e LOGSENE_TOKEN=YOUR_LOGSENE_TOKEN
What gives?
Hi Stefan,
We have the problem, that when we set LOGSENE_ENABLED as an environment variable then apparently for some reason, it doesn't get recognized. Here's an example output from 'docker inspect' where this was the case:
[
{
"Id": "55da0101e9e7f77175fb374012621847ad77f4b584376350b5d5100bcfb27f49",
"Created": "2016-12-12T07:54:20.452304595Z",
"Path": "/bin/sh",
"Args": [
"-c",
"java -javaagent:/opt/newrelic/newrelic.jar -Dnewrelic.config.license_key=${NEWRELIC_KEY} -Dnewrelic.config.app_name=${NEWRELIC_APPNAME} -Dnewrelic.config.proxy_host=webproxy.sbb.ch -Dnewrelic.config.proxy_port=8080 -jar /deploy/api-gateway.jar"
],
...
"Config": {
"Hostname": "api-gateway-15-ui8us",
"Domainname": "",
"User": "1001100000",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"8080/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"ELAZ_CONFIG_LABEL=integration",
"ELAZ_CONFIG_SERVER=http://elaz-config-server:8888",
"ELAZ_PROFILES_ACTIVE=int",
"JAVA_TOOL_OPTIONS=-Xms512m -Xmx3072m",
"LOGSENE_ENABLED=false",
"NEWRELIC_APPNAME=int-api-gateway",
...In case of Labels, this Feature functions correctly, however the quotation marks are set differently - might that be the root cause?
...
"Labels": {
"LOGSENE_ENABLED": "false",
"Name": "rhel7/rhel",
"Vendor": "Red Hat, Inc.",
"Version": "7.2",
"io.kubernetes.container.hash": "655c979",
"io.kubernetes.container.name": "akka-singleton",
...To collect only error messages it would be nice to specify a regular expression to limit the log collection. The filter should apply before logs are parsed to save CPU cycles for parsing non-relevant logs.
E.g.
docker run -e LOG_FILTER="error|exception|timeout" ... sematext/sematext-agent-docker
The following log only had its first two lines (up to and including "java.lang.NullPointerException: null" but nothing after that) ingested into sematext docker agent, without the rest of the stack trace (and these were ingested as two separate log entries):
18/07/18 20:57:49,248 ERROR [ProcThread-11] EventProcessingThread - Exception occured while processing rule:5ae0fe5711048b00073572d1 on event:Eve
nt [id=null, companyID=97b47e4d-578a-41cd-a396-ce8fc94c869f, environment=production, entityType=businessDocument, bodID=Amazon_856_000007195_856_
40116, systemID=null, timestamp=Wed Jul 18 20:57:44 GMT 2018, type=advance_ship_notice_received, description=Incoming document is processed, data
=null, laVersion=4, meta={}]
java.lang.NullPointerException: null
at com.coenterprise.syncrofy.rules.ParameterEvaluationUtils.evaluate(ParameterEvaluationUtils.java:577) ~[app.jar:?]
at com.coenterprise.syncrofy.rules.EventProcessingThread.evaluateParameters(EventProcessingThread.java:286) ~[app.jar:?]
at com.coenterprise.syncrofy.rules.EventProcessingThread.applyRule(EventProcessingThread.java:190) ~[app.jar:?]
at com.coenterprise.syncrofy.rules.EventProcessingThread.run(EventProcessingThread.java:132) [app.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_151]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
I have attached my patterns.yml (compressed into a zip for github) which does contain a custom pattern for our backend service which was derived for us by Radu Gheorghe a few months ago to allow that service's logs' severity to be parsed (lines 12-20 in the file).
You still provide an alternative Dockerfile for an image based on Alpine Linux.
However the Dockerfile.alpine doesn't have a related tag on Docker Hub, meaning that users have to build and maintain the image themselves.
Could you therefore set up an additional automated build on Docker hub with an alternative tag, e.g. sematext/sematext-agent-docker:alpine-1.29.30?
Thanks for your consideration!
... so Kubernetes-specific data can be collected.
Or maybe this is already doable?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.