This repository is no longer used to generate the official Beats Docker image from Elastic.
To build Beats docker images for pre-6.6 releases, switch branches in this repo to the matching release.
Official Beats Docker images
Auditbeat does not currently have a Docker image available, it would be great to have one.
Please post all questions and issues on https://discuss.elastic.co/c/beats
before opening a Github Issue. Your questions will reach a wider audience there,
and if we confirm that there is a bug, then you can open a new issue.
For security vulnerabilities please only send reports to [email protected].
See https://www.elastic.co/community/security for more information.
For confirmed bugs, please report:
---> Running in 1772f6cccee3
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
The command '/bin/sh -c curl -Lso - https://artifacts.elastic.co/downloads/beats/auditbeat/auditbeat-6.3.1-linux-x86_64.tar.gz | tar zxf - -C /tmp && mv /tmp/auditbeat-6.3.1-linux-x86_64 /usr/share/auditbeat' returned a non-zero code: 2
make: *** [auditbeat] Error 2```
Starting an ICMP monitor when running the heartbeat docker image results in the following error:
2017/02/24 02:53:36.459997 loop.go:330: INFO IPv4 ICMP not supported: listen ip4:icmp : socket: operation not permitted
2017/02/24 02:53:36.460013 loop.go:330: INFO IPv6 ICMP not supported: listen ip6:ipv6-icmp : socket: operation not permitted
2017/02/24 02:53:36.460129 manager.go:125: ERR failed to load monitor tasks: failed to initiate IPv4 support when initializing monitor icmp(0)
This can be fixed by setting the network capabilities not only for packetbeat, but also for heartbeat.
As a workaround, the following custom docker image can be used:
FROM docker.elastic.co/beats/heartbeat:5.2.1
USER root
RUN setcap cap_net_raw,cap_net_admin=eip /usr/share/heartbeat/heartbeat
USER heartbeat
The Metricbeat Docker didn't work with system.hostfs
:
docker run -it -e ELASTICSEARCH_URL=http://elasticsearch:9200 -v=/proc:/hostfs/proc:ro -v=/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro -v=/:/hostfs:ro docker.elastic.co/beats/metricbeat:5.3.0 -system.hostfs=/hostfs
Error:
docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \"-system.hostfs=/hostfs\": stat -system.hostfs=/hostfs: no such file or directory".
Please post all questions and issues on https://discuss.elastic.co/c/beats
before opening a Github Issue. Your questions will reach a wider audience there,
and if we confirm that there is a bug, then you can open a new issue.
For security vulnerabilities please only send reports to [email protected].
See https://www.elastic.co/community/security for more information.
For confirmed bugs, please report:
docker run -v /root/filebeat-docker/filebeat.yml:/usr/share/filebeat/filebeat.yml docker.elastic.co/beats/filebeat:6.2.4
I get docker: Exiting: error unpacking config data: more than one namespace configured accessing 'output' (source:'filebeat.yml')
filebeat run on k8s in ds , using the yaml file:
https://raw.githubusercontent.com/elastic/beats/6.4/deploy/kubernetes/filebeat-kubernetes.yaml
but error:
/usr/local/bin/docker-entrypoint: line 8: exec: filebeat: not found
I get error while executing the below command -
sudo docker run -v /etc/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml docker.elastic.co/beats/filebeat:6.4.0
container_linux.go:262: starting container process caused "process_linux.go:339: container init caused "rootfs_linux.go:57: mounting \"/etc/filebeat/filebeat.yml\" to rootfs \"/app/docker/devicemapper/mnt/e9a0e8d10a77d4f658c82511f60be50f4afe5598df8bb2f9c168f15d52ff0fd1/rootfs\" at \"/app/docker/devicemapper/mnt/e9a0e8d10a77d4f658c82511f60be50f4afe5598df8bb2f9c168f15d52ff0fd1/rootfs/usr/share/filebeat/filebeat.yml\" caused \"not a directory\"""
docker: Error response from daemon: oci runtime error: container_linux.go:262: starting container process caused "process_linux.go:339: container init caused "rootfs_linux.go:57: mounting \"/etc/filebeat/filebeat.yml\" to rootfs \"/app/docker/devicemapper/mnt/e9a0e8d10a77d4f658c82511f60be50f4afe5598df8bb2f9c168f15d52ff0fd1/rootfs\" at \"/app/docker/devicemapper/mnt/e9a0e8d10a77d4f658c82511f60be50f4afe5598df8bb2f9c168f15d52ff0fd1/rootfs/usr/share/filebeat/filebeat.yml\" caused \"not a directory\"""
: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.
ERRO[0000] error waiting for container: context canceled
Kubernetes' MustRunAsNonRoot option requires a numeric USER
.
Running metricbeat in kubernetes, beats.hostname
is set to the container name, e.g. "metricbeat-v5.2.1-3bhfr".
When running metricbeats in multiple hosts, you will not be able to distinguish one host from another. Furthermore, if metricbeat dies for any reason, it will be assigned a new container name, e.g. "metricbeat-v5.2.1-XXXXX"
Provide a nice ENTRYPOINT, so that people can do things like:
docker run docker.elastic.co/beats/metricbeat:5.3.0 -system.hostfs=/hostfs
See also: #5.
I forked the repo and converted it to Ubuntu 16.04 base image:
https://github.com/apolloclark/packer-elk-docker
What’s the best way to open a PR for this?
Hi,
I'm running MetricBeat as explained here: https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-docker.html#monitoring-host
On a (hardened) Debian 8 machine from OVH French provided.
In MetricBeats' log I get a lot of errors like the following:
WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [system.process.cgroup.memory.memsw.limit.bytes]","caused_by":{"type":"json_parse_exception","reason":"Numeric value (18446744073709551615) out of range of long (-9223372036854775808 - 9223372036854775807)
Same error occurs for:
system.process.cgroup.memory.kmem_tcp.limit.bytes
system.process.cgroup.memory.memsw.limit.bytes
system.process.cgroup.memory.stats.hierarchical_memory_limit.bytes
For confirmed bugs, please report:
Hello, I cannot use docker's filebeat to get my logs.
[machine+version] centos6 filebeat6.3
service filebeat start
]Non-zero metrics
ticking every 30s, I would like to what caused my error. Hopeful you can help me out. Thanks!Hey Elastic!
I wasn't sure where a good place for this was, but I thought other users might benefit from how I setup packetbeat in k8s.
Here is my daemonset configuration. Note that using a daemonset instead of a Pod ensures that each physical host gets a packetbeat container
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: packetbeat-v5.2.1
namespace: kube-system
labels:
k8s-app: packetbeat
kubernetes.io/cluster-service: "true"
version: v5.2.1
spec:
template:
metadata:
labels:
k8s-app: packetbeat
kubernetes.io/cluster-service: "true"
version: v5.2.1
spec:
hostNetwork: true
containers:
- name: packetbeat
image: docker.elastic.co/beats/packetbeat:5.2.1
command:
- '/bin/sh'
- '-c'
- '/usr/share/packetbeat/packetbeat -e -c /etc/packetbeat/packetbeat.yml -path.logs /var/log/packetbeat'
securityContext:
#privileged: true
capabilities:
add:
- NET_ADMIN
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlogpacketbeat
mountPath: /var/log/packetbeat
- name: packetbeatconf
mountPath: /etc/packetbeat
readOnly: true
terminationGracePeriodSeconds: 30
volumes:
- name: varlogpacketbeat
hostPath:
path: /var/log/packetbeat
- name: packetbeatconf
configMap:
name: packetbeat-conf
One of the things I've done for metricbeat is create an put my ElasticSearch endpoint in an ExternalName Service, so that in my metricbeat.yml, I can point my elasticsearch destination to this dns name rather than the actual dns name. This is the preferred pattern for routing network traffic to destinations outside your k8s cluster. This works great for metricbeat, however, breaks for packetbeat. I suspect, because packetbeat requires hostNetwork: true
, it bypasses kube-dns
, which holds k8s service information. The work around is to point packetbeat directly to your elasticsearch, not using k8s services. Here is my k8s service for reference if someone is able to figure out how to use k8s services on the host network, that would be fantastic.
apiVersion: v1
kind: Service
metadata:
name: elasticsearch-logging
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
kubernetes.io/name: Elasticsearch
spec:
type: ExternalName
externalName: your-elasticsearch-hostname.us-west-2.es.amazonaws.com
ports:
- name: esrest
port: 9200
protocol: TCP
targetPort: 80
Then in your metricbeat.yml/packetbeat.yml configuration
output.elasticsearch:
hosts: ["elasticsearch-logging:"]
I am using a lightly customized installation of all 6.0.0-beta1 beats.
However, When attempting to start them, they all spew this error:
2017/08/22 19:58:16.455956 client.go:465: WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_default_]: Mapping definition for [error] has unsupported parameters: [properties : {code={type=long}, message={norms=false, type=text}, type={ignore_above=1024, type=keyword}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Mapping definition for [error] has unsupported parameters: [properties : {code={type=long}, message={norms=false, type=text}, type={ignore_above=1024, type=keyword}}]"}}
I have set setup.template.overwrite: true
in all of the config files, and I have also run docker run --add-host "elasticsearch:xxx.xxx.xx.xx" docker.elastic.co/beats/filebeat:6.0.0-beta1 setup --template
I am not sure how to interpret this error at all.
Running on amazonlinux, docker 17.03.1-ce, elasticsearch 5.
The Auditbeat image currently fails with 'operation not permitted' even when:
--privileged
--cap-add=ALL
)# docker run --privileged -u root --cap-add=ALL --rm -it docker.elastic.co/beats/auditbeat:6.0.0-beta2
2017/09/06 04:16:05.881553 beat.go:430: INFO Home path: [/usr/share/auditbeat] Config path: [/usr/share/auditbeat] Data path: [/usr/share/auditbeat/data] Logs path: [/usr/share/auditbeat/logs]
2017/09/06 04:16:05.881593 beat.go:457: INFO Beat metadata path: /usr/share/auditbeat/data/meta.json
2017/09/06 04:16:05.881623 metrics.go:23: INFO Metrics logging every 30s
2017/09/06 04:16:05.881915 beat.go:437: INFO Beat UUID: a1c35d76-0446-4282-93b8-4c9f8d17750f
2017/09/06 04:16:05.881941 beat.go:192: INFO Setup Beat: auditbeat; Version: 6.0.0-beta2
2017/09/06 04:16:05.882124 client.go:123: INFO Elasticsearch url: http://elasticsearch:9200
2017/09/06 04:16:05.882479 module.go:80: INFO Publisher name: 9faf05cdf86d
2017/09/06 04:16:05.882515 metricbeat.go:37: INFO Register [ModuleFactory:[], MetricSetFactory:[audit/file, audit/kernel]]
2017/09/06 04:16:05.882670 cfgwarn.go:23: WARN EXPERIMENTAL: The audit.kernel metricset is a beta feature
2017/09/06 04:16:05.883142 cfgwarn.go:23: WARN EXPERIMENTAL: The audit.file metricset is an experimental feature
2017/09/06 04:16:05.883347 beat.go:264: INFO auditbeat start running.
2017/09/06 04:16:13.740069 audit_linux.go:81: ERR [audit.kernel] failed to delete existing rules: operation not permitted
Very interesting. I feel like there's something I don't know about audit permissions. Any thoughts @andrewkroh?
For confirmed bugs, please report:
The metricbeat is writing data into the elasticsearch i can check that with http://localhost:9200/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open .monitoring-kibana-2-2017.04.26 MkrSWQjBRdGEaQf4csUuog 1 1 6 0 63.7kb 63.7kb
yellow open .kibana yEdBQl8JSNqNhVYMMgx77A 5 1 95 67 394.5kb 394.5kb
yellow open metricbeat-2017.04.26 ZH5tpPqZS52D-P9TQT5mIQ 5 1 2348 0 4.4mb 4.4mb
yellow open .monitoring-es-2-2017.04.26 SabclaYqQSmKOQ0KUCsKOQ 1 1 194 76 471.1kb 471.1kb
yellow open .monitoring-data-2 _6MtkjMeQvGNEs0diHJ1Lg 1 1 3 0 6.9kb 6.9kb
The index is created and has some fields in kibana.
But "Discover" reports "No results found".
When i try to add the index manually the logs give a 404
kibana | {"type":"response","@timestamp":"2017-04-26T06:54:58Z","tags":[],"pid":7,"method":"get","statusCode":404,"req":{"url":"/elasticsearch/metricbeat-*/_mapping/field/*?_=1493189698122&ignore_unavailable=false&allow_no_indices=false&include_defaults=true","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","kbn-version":"5.3.1","referer":"http://localhost:5601/app/kibana","connection":"keep-alive"},"remoteAddress":"192.168.160.1","userAgent":"192.168.160.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":404,"responseTime":11,"contentLength":9},"message":"GET /elasticsearch/metricbeat-*/_mapping/field/*?_=1493189698122&ignore_unavailable=false&allow_no_indices=false&include_defaults=true 404 11ms - 9.0B"}
docker-compose.yml
version: '2.1'
services:
metricbeat:
image: docker.elastic.co/beats/metricbeat:5.3.1
restart: unless-stopped
networks:
- beats
volumes:
- /proc:/hostfs/proc:ro
- /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
- /:/hostfs:ro
command: metricbeat -e -system.hostfs=/hostfs
environment:
- http_proxy
- https_proxy
- no_proxy=${no_proxy},elasticsearch
depends_on:
elasticsearch:
condition: service_healthy
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.3.1
container_name: elasticsearch
environment:
- "http.host=0.0.0.0"
- "transport.host=127.0.0.1"
ports:
- '127.0.0.1:9200:9200'
networks:
- beats
healthcheck:
test: ['CMD', 'curl', '-f', '-u', 'elastic:changeme', 'http://localhost:9200']
kibana:
image: docker.elastic.co/kibana/kibana:5.3.1
container_name: kibana
ports:
- '127.0.0.1:5601:5601'
networks:
- beats
depends_on:
elasticsearch:
condition: service_healthy
import_dashboards:
image: docker.elastic.co/beats/metricbeat:5.3.1
# Using -beat "" causes it to install all dashboards in the zip and not just the metricbeat ones.
networks:
- beats
command: >-
/usr/share/metricbeat/scripts/import_dashboards
-file /usr/share/metricbeat/beats-dashboards-5.3.1.zip
-es http://elasticsearch:9200
-user elastic
-pass changeme
depends_on:
elasticsearch:
condition: service_healthy
set_default_index_pattern:
image: docker.elastic.co/beats/metricbeat:5.3.1
networks:
- beats
command: >-
curl -XPUT http://elastic:changeme@elasticsearch:9200/.kibana/config/5.3.1
-d '{"defaultIndex" : "metricbeat-*"}'
depends_on:
elasticsearch:
condition: service_healthy
networks:
beats:
For confirmed bugs, please report:
set the config file: /xxxx/filebeat/config/filebeat.yml like this:
logging.level: info
logging.to_files: true
logging.files:
path: /usr/share/filebeat/logs
name: filebeat
keepfiles: 7
permissions: 0664
and then run docker container:
docker run -d --restart=always --name=filebeat -v /xxxx/filebeat/logs/:/usr/share/filebeat/logs -v /xxxx/filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml docker.elastic.co/beats/filebeat:6.2.3
but there is no log file in "/xxxx/filebeat/logs/" on VM,also no file in "/usr/share/filebeat/logs" in the container.
Your new official filebeat docker image runs at 245MB.
A simple build based on Alpine goes up to less than 20MB.
Please move your beat images to Alpine.
I'm adding the Dockerfile I currently use for filebeat below.
(adapted from https://github.com/primait/docker-filebeat)
FROM alpine:3.5
ENV FILEBEAT_VERSION=5.3.0 \
FILEBEAT_SHA1=c6f56d1a938889ec9f5db7caea266597f625fcc1
RUN mkdir /etc/filebeat && \
apk add --no-cache wget && \
wget -q https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz --no-check-certificate -O filebeat.tar.gz && \
echo "${FILEBEAT_SHA1} filebeat.tar.gz" | sha1sum -c - && \
tar xzvf filebeat.tar.gz && \
cp filebeat-*/filebeat /usr/local/bin && \
rm -rf filebeat*
WORKDIR /etc/filebeat
COPY filebeat.yml /etc/filebeat/
COPY filebeat.template.json /etc/filebeat/
COPY filebeat.template-es2x.json /etc/filebeat/
COPY inject.sh /usr/local/bin
ENTRYPOINT []
CMD [ "filebeat", "-e" ]
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.