graphite-project / docker-graphite-statsd Goto Github PK
View Code? Open in Web Editor NEWOfficial Docker image for Graphite
License: MIT License
Official Docker image for Graphite
License: MIT License
Hi,
how can I disable the docker nginx and use my own installed nginx as proxy?
Regards,
Hi all,
When I was searching for official release of graphite image I followed the dockerhub, entered graphite
and then chose 'official' in the drop down menu there on dockerhub, and no image was shown if I did that.
It took some time to find and insure that graphiteapp/graphite-statsd image is official one.
I would recommend you to label that image as official so people can choose this image if they want an official image.
Thank you for having official image anyway.
Hi, this is for information rather than an issue, you can close this if it's irrelevant;
I'm very new to all theses system, i followed your example (start docker and generate example data) and i ended up with this graph :
I modified the nc
command to show what it was sending and it looks nothing like it it was juste random numbers with no particular spike.
Here's an exert of the graph's json
[{"datapoints": [[45.1, 1538162030], [0.0, 1538162040], [49.1, 1538162050], [47.7, 1538162060], [0.0, 1538162070], [55.0, 1538162080], [57.0, 1538162090], [0.0, 1538162100], [42.3, 1538162110], [44.4, 1538162120], [0.0, 1538162130], [53.1, 1538162140], [57.8, 1538162150], [0.0, 1538162160], [45.1, 1538162170], [32.5, 1538162180], [0.0, 1538162190], [50.9, 1538162200], [60.5, 1538162210], [0.0, 1538162220], [39.0, 1538162230], [50.9, 1538162240], [0.0, 1538162250], [53.3, 1538162260], [53.0, 1538162270], [0.0, 1538162280], [40.3, 1538162290], [52.1, 1538162300], [0.0, 1538162310], [null, 1538162320]], "target": "stats.example", "tags": {"name": "stats.example"}}]
there are clear 0s in there...
Is this normal? am i not understanding something properly? do you have links to understand what is happening?
Good afternoon,
Logrotate seems still not to work even after the changes in 1.1.5-4.
Wrote a detailed report under the old post:
#81 (comment)
See issue #81
Greetings,
Stephan
Hi,
I'm currently using docker-compose managing a grafana and hopsoft/graphite-statsd deployment, where a host volume is attached to the hopsoft/graphite-statsd container's /opt/graphite/storage. Everything is working fine, except hopsoft/graphite-statsd is still at 0.91 that a few functions I need is missing. So, when i'm trying to migrate my hopsoft/graphite-statsd to graphiteapp/graphite-statsd:1.0.2-3. I thought it's just simply replacing the image name. Looks like i'm totally wrong. I noticed that graphiteapp/graphite-statsd:1.0.2-3 can't initialize itself with the same host volume. Anyone can give me some advice how to do this migration smoothly? I would like to preserve all the collected data that currently stored on my local volume by hopsoft/graphite-statsd, and I would like Grafana to function normally after migration. Anyway to achieve this?
my_graphite:
image: graphiteapp/graphite-statsd:1.0.2-3 #hopsoft/graphite-statsd #graphiteapp/graphite-statsd:1.0.2-3
volumes:
- my_data/graphite/storage:/opt/graphite/storage
/etc/service/carbon-relay/run runs unconditionally carbon-relay service.
There is a typo in file. The test has no effect
[[ -n ${RELAY} ]] || exit 1
it should be
[[ -n "${RELAY}" ]] || exit 1
Sometimes using monolithic docker image for all components is not convenient. We need to create separate images for carbon
and graphite-web
.
I'm running this https://github.com/graphite-project/docker-graphite-statsd docker image. And sometimes I get this error, which causes carbon to crash. Could the reason be, that I take a backup of the whisper folder while running (maybe corrupting the data?)
EDIT: looks like a cache.py related issue?
01/12/2018 18:49:20 :: [console] Unhandled Error
Traceback (most recent call last):
File “/opt/graphite/lib/python3.6/site-packages/twisted/python/threadpool.py”, line 250, in inContext
result = inContext.theWork()
File “/opt/graphite/lib/python3.6/site-packages/twisted/python/threadpool.py”, line 266, in <lambda>
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File “/opt/graphite/lib/python3.6/site-packages/twisted/python/context.py”, line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File “/opt/graphite/lib/python3.6/site-packages/twisted/python/context.py”, line 85, in callWithContext
return func(*args,**kw)
--- <exception caught here> ---
File “/opt/graphite/lib/carbon/writer.py”, line 189, in writeForever
writeCachedDataPoints()
File “/opt/graphite/lib/carbon/writer.py”, line 98, in writeCachedDataPoints
(metric, datapoints) = cache.drain_metric()
File “/opt/graphite/lib/carbon/cache.py”, line 187, in drain_metric
metric = self.strategy.choose_item()
File “/opt/graphite/lib/carbon/cache.py”, line 116, in choose_item
return next(self.queue)
builtins.StopIteration:
I have done a little experimentation and it appears it is possible to reduce the size of the Docker image by 50%.
The size reduction is mostly achieved by,
The key thing to note with Docker is that once a RUN is complete nothing can be deleted from the resulting layer. Clean-up/deletion needs to happen in the RUN(layer) to be effective.
The Dockerfile below is less user friendly to read than the current published one. With the majority of the install work being done in one RUN. The size reduction might warrant that loss of readability.
I have only tested the resulting container briefly. I did nothing more than run the example events generator bash snippet and observe stats on a chart in graphite-web.
Might Docker Image size reduction following something similar to Dockerfile changes below be of interest to the project?
FROM phusion/baseimage:0.10.2
MAINTAINER XXXX YYYY <[email protected]>
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update --fix-missing && \
apt-get install --yes --no-install-recommends \
nginx \
python-flup \
python-pip \
python-ldap \
expect \
memcached \
sqlite3 \
libcairo2 \
python-cairo \
python-rrdtool \
nodejs && \
apt-get clean && \
apt-get autoremove --yes && \
rm -rf /var/lib/apt/lists/*
# choose a timezone at build-time
# use `--build-arg CONTAINER_TIMEZONE=Europe/Brussels` in `docker build`
ARG CONTAINER_TIMEZONE
RUN if [ ! -z "${CONTAINER_TIMEZONE}" ]; \
then ln -sf /usr/share/zoneinfo/$CONTAINER_TIMEZONE /etc/localtime && \
dpkg-reconfigure -f noninteractive tzdata; \
fi
ARG version=1.1.4
ARG whisper_version=${version}
ARG carbon_version=${version}
ARG graphite_version=${version}
ARG whisper_repo=https://github.com/graphite-project/whisper.git
ARG carbon_repo=https://github.com/graphite-project/carbon.git
ARG graphite_repo=https://github.com/graphite-project/graphite-web.git
ARG statsd_version=v0.8.0
ARG statsd_repo=https://github.com/etsy/statsd.git
# Install compiler and dev libraries.
# Clone sources from statsd, whisper, carbon and graphite github repositories.
# Build and install whisper, carbon and graphite.
# Remove compiler and dev libraries.
# Delete source code directories.
# Clear apt cache.
RUN apt-get update --fix-missing && \
apt-get install --yes --no-install-recommends gcc python-dev libffi-dev libcairo2-dev pkg-config git && \
python -m pip install --no-cache-dir --upgrade pip setuptools && \
pip install --no-cache-dir django==1.11.15 fadvise msgpack-python && \
git clone -b ${statsd_version} ${statsd_repo} /opt/statsd && \
git clone -b ${whisper_version} --depth 1 ${whisper_repo} /usr/local/src/whisper && \
git clone -b ${carbon_version} --depth 1 ${carbon_repo} /usr/local/src/carbon && \
git clone -b ${graphite_version} --depth 1 ${graphite_repo} /usr/local/src/graphite-web && \
cd /usr/local/src/whisper && \
python ./setup.py install && \
cd /usr/local/src/carbon && \
pip install --no-cache-dir --requirement requirements.txt && \
python ./setup.py install && \
cd /usr/local/src/graphite-web && \
pip install --no-cache-dir --requirement requirements.txt && \
python ./setup.py install && \
apt-get purge --yes --auto-remove cpp gcc python-dev libffi-dev libcairo2-dev pkg-config git && \
apt-get install --yes --no-install-recommends libpython2.7 && \
apt-get clean && \
apt-get autoremove --yes && \
rm -rf /var/lib/apt/lists/* && \
cd ${HOME} && \
rm -rf /usr/local/src/whisper /usr/local/src/carbon /usr/local/src/graphite-web /tmp/* /var/tmp/* /usr/share/python-wheels/*
# config graphite
ADD conf/opt/graphite/conf/*.conf /opt/graphite/conf/
ADD conf/opt/graphite/webapp/graphite/local_settings.py /opt/graphite/webapp/graphite/local_settings.py
# ADD conf/opt/graphite/webapp/graphite/app_settings.py /opt/graphite/webapp/graphite/app_settings.py
WORKDIR /opt/graphite/webapp
RUN mkdir -p /var/log/graphite && \
PYTHONPATH=/opt/graphite/webapp django-admin.py collectstatic --noinput --settings=graphite.settings
# config statsd
ADD conf/opt/statsd/config_*.js /opt/statsd/
# config nginx
RUN rm /etc/nginx/sites-enabled/default
ADD conf/etc/nginx/nginx.conf /etc/nginx/nginx.conf
ADD conf/etc/nginx/sites-enabled/graphite-statsd.conf /etc/nginx/sites-enabled/graphite-statsd.conf
# init django admin
ADD conf/usr/local/bin/django_admin_init.exp /usr/local/bin/django_admin_init.exp
ADD conf/usr/local/bin/manage.sh /usr/local/bin/manage.sh
RUN chmod +x /usr/local/bin/manage.sh && /usr/local/bin/django_admin_init.exp
# logging support
RUN mkdir -p /var/log/carbon /var/log/graphite /var/log/nginx
ADD conf/etc/logrotate.d/graphite-statsd /etc/logrotate.d/graphite-statsd
# daemons
ADD conf/etc/service/carbon/run /etc/service/carbon/run
ADD conf/etc/service/carbon-aggregator/run /etc/service/carbon-aggregator/run
ADD conf/etc/service/graphite/run /etc/service/graphite/run
ADD conf/etc/service/statsd/run /etc/service/statsd/run
ADD conf/etc/service/nginx/run /etc/service/nginx/run
# default conf setup
ADD conf /etc/graphite-statsd/conf
ADD conf/etc/my_init.d/01_conf_init.sh /etc/my_init.d/01_conf_init.sh
# defaults
EXPOSE 80 2003-2004 2023-2024 8080 8125 8125/udp 8126
VOLUME ["/opt/graphite/conf", "/opt/graphite/storage", "/opt/graphite/webapp/graphite/functions/custom", "/etc/nginx", "/opt/statsd", "/etc/logrotate.d", "/var/log"]
WORKDIR /
ENV HOME /root
ENV STATSD_INTERFACE udp
CMD ["/sbin/my_init"]
Error Log
prod_statsd.1@instance-1 | E: Unable to locate package git
prod_statsd.1@instance-1 | *** /etc/my_init.d/01_conf_init.sh failed with status 100
prod_statsd.1@instance-1 | *** Killing all processes...
how to open cors in this docker container?
I did that in separate branch, but Etsy statsd
is dead on bionic.
Probably we should migrate to https://github.com/bitly/statsdaemon
Tried to use this image on Kubernetes.
There was an issue with persistent storage caused by lack of GRAPHITE_STORAGE_DIR
environment variable.
Variable GRAPHITE_STORAGE_DIR
should be set to /opt/graphite/storage
by default
An example is available at Full Example Configuration | NGINX
This is a follow up of #61
After that one was fixed, rdd´s showed as folders but without any content.
Trying to open an rrd folder in the browser throws this in /var/log/graphite/exception.log:
Traceback (most recent call last):
File "/opt/graphite/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/graphite/webapp/graphite/metrics/views.py", line 128, in find_view
leaves_only=leaves_only,
File "/opt/graphite/webapp/graphite/storage.py", line 264, in find
for match in self._find(query):
File "/opt/graphite/webapp/graphite/storage.py", line 295, in _find
for node in result or []:
File "/opt/graphite/webapp/graphite/finders/standard.py", line 101, in find_nodes
for datasource_name in RRDReader.get_datasources(absolute_path):
File "/opt/graphite/webapp/graphite/readers/rrd.py", line 61, in get_datasources
#info = rrdtool.info(RRDReader._convert_fs_path(fs_path))
As far as I dug. it seems to be a bug in Graphite + Python3.
I created this workaround in the meantime by patching the image:
FROM graphiteapp/graphite-statsd
# At the moment, this does not work without this patch
RUN sed \
-i \
's/return os.path.realpath(fs_path)/return os.path.realpath(fs_path).decode("utf-8")/' \
/opt/graphite/webapp/graphite/readers/rrd.py
That allowed me to read the rrd files.
Do you have any suggestions on the best way to make modifications to settings in graphite-web's local_settings.py? I thought about mapping the volume /opt/graphite/webapp/graphite/ but looking in the container there is a lot more in it than what is in github, so not sure how to get those locally first.
Any reason this directory isn't treated the same way as /opt/graphite/conf, where I can map to an empty volume and get the config files on the host that way?
We're just trying to change the FETCH_TIMEOUT value to something a bit higher because we have a few queries which return a lot of metrics.
Using the latest image: 1.1.0-pre2
Here's the logs:
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_conf_init.sh...
*** Running /etc/rc.local...
*** Booting runit daemon...
*** Runit started as PID 14
Nov 29 05:26:01 4d5ab037c9ef syslog-ng[36]: syslog-ng starting up; version='3.5.6'
Traceback (most recent call last):
File "/opt/graphite/bin/carbon-cache.py", line 32, in <module>
run_twistd_plugin(__file__)
File "/opt/graphite/lib/carbon/util.py", line 111, in run_twistd_plugin
config.parseOptions(twistd_options)
File "/usr/local/lib/python2.7/dist-packages/twisted/application/app.py", line 624, in parseOptions
usage.Options.parseOptions(self, options)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/usage.py", line 277, in parseOptions
self.postOptions()
File "/usr/local/lib/python2.7/dist-packages/twisted/scripts/_twistd_unix.py", line 77, in postOptions
app.ServerOptions.postOptions(self)
File "/usr/local/lib/python2.7/dist-packages/twisted/application/app.py", line 635, in postOptions
% (self['logger'], e))
twisted.python.usage.UsageError: Logger 'None' could not be imported: No module named 'None'
Traceback (most recent call last):
File "/opt/graphite/bin/carbon-aggregator.py", line 32, in <module>
run_twistd_plugin(__file__)
File "/opt/graphite/lib/carbon/util.py", line 111, in run_twistd_plugin
config.parseOptions(twistd_options)
File "/usr/local/lib/python2.7/dist-packages/twisted/application/app.py", line 624, in parseOptions
usage.Options.parseOptions(self, options)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/usage.py", line 277, in parseOptions
self.postOptions()
File "/usr/local/lib/python2.7/dist-packages/twisted/scripts/_twistd_unix.py", line 77, in postOptions
app.ServerOptions.postOptions(self)
File "/usr/local/lib/python2.7/dist-packages/twisted/application/app.py", line 635, in postOptions
% (self['logger'], e))
twisted.python.usage.UsageError: Logger 'None' could not be imported: No module named 'None'
...
By default the nginx configuration shipped in the docker container returns "*" for theAccess-Control-Allowed-Origin header. This is a problem when using using include credentials as part of AJAX request since a specific host must be returned. The net effect is that authenticated cross-origin requests from grafana to graphite fail.
It looks like graphite already exposes this via ALLOWED_HOSTS (https://graphite.readthedocs.io/en/latest/config-local-settings.html#general-settings) but this isn't easy to access without doing a lot of work on the container itself.
I was able to work around this since I was using another nginx instance as we're running grafana/graphite on the same box, but I had to install nginx-extras and use more_set_headers to override the headers being returned to the correct value, but just based off what I saw from doing Google searches a lot of folks people will be prone to just echoing back the origin url and giving themselves XSS vulnerabilities .
If there is an easy way to do this would love a pointer to the docs :)
echo counters | nc localhost 8126
shows the random data is being sent and I can see the example.wsp file, but the dashboard is empty and there's no metrics in the main page. What am I missing?
Hi,
I'm trying to add a p90 "feature" under graphite-web/graphite/render/functions.py movingWindow function, and then make it a docker image (in the Dockerfile by only changing the graphite git-repo to point to my fork, where i only added following lamda p90 funciton below).
aggFuncs = {
'average': safeAvg,
'median': safeMedian,
'sum': safeSum,
'min': safeMin,
'max': safeMax,
'diff': safeDiff,
'stddev': safeStdDev,
'count': safeLen,
'range': lambda row: safeSubtract(safeMax(row), safeMin(row)),
'multiply': lambda row: safeMul(*row),
'p90': lambda row: _getPercentile(row, 90), <<<<<<------------ my added line
'last': safeLast,
}
I then checked the mentioned functions.py in the docker but didn't see my p90 function.
So, am i doing this right? Or, I"m being too naive here? What's the right way to add a p90 function?
Thanks!
Sending build context to Docker daemon 349.2kB
Step 1/53 : FROM phusion/baseimage:0.10.1
---> 2391dfad8777
Step 2/53 : MAINTAINER Denys Zhdanov <[email protected]>
---> Using cache
---> 3152d73b6dce
Step 3/53 : RUN apt-get -y update && apt-get -y upgrade && apt-get -y install vim nginx python-dev python-flup python-pip python-ldap expect git memcached sqlite3 libffi-dev libcairo2 libcairo2-dev python-cairo python-rrdtool pkg-config nodejs && rm -rf /var/lib/apt/lists/*
---> Using cache
---> 868784acd4e8
Step 4/53 : ARG CONTAINER_TIMEZONE
---> Using cache
---> 47bd1063221a
Step 5/53 : ENV DEBIAN_FRONTEND noninteractive
---> Using cache
---> bf45329bfea3
Step 6/53 : RUN if [ ! -z "${CONTAINER_TIMEZONE}" ]; then ln -sf /usr/share/zoneinfo/$CONTAINER_TIMEZONE /etc/localtime && dpkg-reconfigure -f noninteractive tzdata; fi
---> Using cache
---> b10718a0ccc0
Step 7/53 : RUN pip install --upgrade pip && pip install django==1.11.12
---> Running in fc3fc2bc70a2
Collecting pip
Downloading https://files.pythonhosted.org/packages/62/a1/0d452b6901b0157a0134fd27ba89bf95a857fbda64ba52e1ca2cf61d8412/pip-10.0.0-py2.py3-none-any.whl (1.3MB)
Installing collected packages: pip
Found existing installation: pip 8.1.1
Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-10.0.0
Traceback (most recent call last):
File "/usr/bin/pip", line 9, in <module>
from pip import main
ImportError: cannot import name main
The command '/bin/sh -c pip install --upgrade pip && pip install django==1.11.12' returned a non-zero code: 1
This is the raw docker file returning this error.
Given:
{ "graphiteHost": "127.0.0.1", "graphitePort": 2003, "port": 8125, "flushInterval": 10000, "backends": [ "./backends/graphite", "./backends/repeater" ], "servers": [ { server: "./servers/udp", address: "0.0.0.0", port: 8125 } ], "repeater": [ { host: "127.0.0.1", port: 9125 }] }
When:
docker run \ -d \ --name graphite \ --restart=always \ --network=graphite \ -p 80:80 \ -p 2003-2004:2003-2004 \ -p 2023-2024:2023-2024 \ -p 8125:8125/udp \ -p 8126:8126 \ -v /etc/graphite/conf:/opt/graphite/conf \ -v /data/graphite:/opt/graphite/storage \ -v /etc/statsd/config/config_udp.js:/opt/statsd/config_udp.js \ -v /var/log/graphite:/var/log \ graphiteapp/graphite-statsd
Then:
Alpine-based images are typically much smaller than Debian-based ones.
following the doc here, i setup graphite in docker with the latest tag
I tried to acces Django with the default root/root account on the mentioned linked (http://localhost/account/login). However it fails, with the following message:
Authentication attempt failed, please make sure you entered your login and password correctly
I found a few similar thread on launchpads but none of them were addressed. manage.py doesn't seem to exit on the container.
Is the doc wrong ? should i create a user myself ?
Since graphite runs in /opt/graphite virtualenv, it is necessary to install rrdtool pyhton package in there. In the current state (109aba0) , the situation is the same as not having any rrd tool installed and Graphite ends up ignoring rrd files.
These lines in the dockerfile would suffice as far as I checked:
RUN apt-get update && apt-get install -y librrd-dev
RUN . /opt/graphite/bin/activate && pip install rrdtool
both are quite useful
Hi,
*** I think I found the problem, the persistent volume is created with a lost+found dir, this cause:
this variable:
graphite_storage_dir_contents=$(find /opt/graphite/storage -mindepth 1 -print -quit)
in 01_conf_init.sh to not be empty, because of this django_admin_init.exp is not invoked:
if [[ -z $graphite_storage_dir_contents ]]; then
/usr/local/bin/django_admin_init.exp
fi
original:
I'm trying to run the docker image on kubernetes when I try to enter the portal I get this error:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/graphite/webapp/graphite/composer/views.py", line 35, in composer
profile = getProfile(request)
File "/opt/graphite/webapp/graphite/user_util.py", line 25, in getProfile
return default_profile()
File "/opt/graphite/webapp/graphite/user_util.py", line 41, in default_profile
'password': '!'})
File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 464, in get_or_create
return self.get(**lookup), False
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 374, in get
num = len(clone)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 232, in len
self._fetch_all()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 53, in iter
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 900, in execute_sql
raise original_exception
OperationalError: no such table: auth_user
this is the configuration I'm using:
---
apiVersion: v1
kind: Service
metadata:
name: graphite-svc
labels:
app: graphite
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 80
selector:
app: graphite
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: graphite-persistent-storage
spec:
accessModes:
- ReadWriteOnce
storageClassName: fast
resources:
requests:
storage: 300Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: graphite-deploy
labels:
app: graphite
spec:
replicas: 1
selector:
matchLabels:
app: graphite
template:
metadata:
labels:
app: graphite
spec:
containers:
- name: graphite
image: graphiteapp/graphite-statsd
ports:
- containerPort: 80
- containerPort: 2003
protocol: UDP
volumeMounts:
- name: graphite-data
mountPath: /opt/graphite/storage
env:
- name: GRAPHITE_STORAGE_DIR
value: /opt/graphite/storage
volumes:
- name: graphite-data
persistentVolumeClaim:
claimName: graphite-persistent-storage
when running without persistent volume the image works as expected. I don't really know where to start looking. the only other different thing I see is that there is a lost+found dir in /opt/graphite/storage
Thanks.
Coming from 1.0.2-2 and upgrading to 1.1.2-1, I see several errors related to tags:
curl -s "http://graphite/tags?pretty=1"
{
"error": "no such table: tags_tag"
}
curl -X POST "http://graphite/tags/tagSeries" \
--data-urlencode 'path=disk.used;rack=a1;datacenter=dc1;server=web01'
{"error": "no such table: tags_series"}
These are the tables I see in graphite.db. Since I did no configuring on my end I'm assuming this includes the local tags db.
account_mygraph dashboard_dashboard_owners
account_profile dashboard_template
account_variable dashboard_template_owners
account_view django_admin_log
account_window django_content_type
auth_group django_migrations
auth_group_permissions django_session
auth_permission events_event
auth_user tagging_tag
auth_user_groups tagging_taggeditem
auth_user_user_permissions url_shortener_link
dashboard_dashboard
Is there a migration guide / am I missing something?
I mapped the web port of the container to 8080 on the host (-p 8080:80
), which breaks login.
When trying to login from the homepage
http://host:8080/account/login
302
-redirects to http://host/
http://host:8080/
now, I am logged in.http://host/
which is wrong.This might bite people who want to Proxy via https, too. (Assuming host:80 --redirect--> host:443 --proxy--> container:80)
The release notes for https://github.com/graphite-project/docker-graphite-statsd/releases/tag/1.1.5-2 should also note that due to the switch to Alpine the user that the Nginx process runs as has changed.
For anyone upgrading from a previous version and using their previous nginx.conf file, this will cause Nginx to not run and cause problems. The code change is here 8d74450#diff-91e8cbe5e36ea5d88807739f99b79f71.
Hi, I'm looking for a way to set the timezone so graphite displays the data relative to the server's location. It looks like local_settings.py is not mapped to a place where it is persisted in your build. And, it would be really nice if the startup read the TZ environment variable and patched the local_settyings.py. Unless you have a better idea, of course :)
I could fix this in my own build, but it seems that it would be worthwhile for everybody using your build.
To build latest release and upload to Docker Hub.
Is there a way to change the startup of gunicorn? Such that I can change the number of workers or add different settings. Is there a script within the container that I am not aware of?
This is question is probably more indicative of my lack of knowledge more than anything.
In our build script we clone this repo and run the docker file on Ubuntu 16.04.
The command 'bin/sh -c pip install --upgrade pip returns a non-zero code:1
Should the way pip is called be modified??
Is there another way around this error other than the following steps??
Manually clone repo
Update DockerFile to hardcode older pip version
RUN pip install --upgrade pip==9.0.3
Tags in graphite-web are broken after 1.1.4-4, because TAGDB = '' by default, and it should be 'graphite.tags.localdatabase.LocalDatabaseTagDB' or absent.
I wonder why there is no carbon-relay daemon included?
For some tests I wanted to use docker for a simple graphite cluster and thought it would work out of the box if I modified the specific configuration files - but nope.
It would be neat if at least a docker environment variable could be set to start the carbon relay daemon.
Found that running this image, that it can take up to 2 minutes before dummy stats sent into the running container before querying for the metric returned usable results.
Run the image with:
docker run --rm -d \
--name graphite-statsd \
-p 8080:80 \
-p 2003-2004:2003-2004 \
-p 2023-2024:2023-2024 \
-p 8125:8125/udp \
-p 127.0.0.1:8126:8126 \
graphiteapp/graphite-statsd:latest
This unfortunately makes it very difficult to determine when the services in the container are ready for use.
Some enquiries in #graphite on Freenode IRC suggested that it takes some time for a whisper file to be created as once about 90 seconds has past after the initial stat was submitted, any subsequent entries to the same stats are picked up almost immediately.
Is this the source of the delay? Is there anything here someone using the image could do?
If there is nothing to be done, seems like the delay from the initial stat being emitted and showing up in graphite should be mentioned in the documentation around verifying it is working to avoid any confusion.
Got this error today when trying this container, fresh, no volumes defined, ZFS as backing storage for docker. I remember getting this same years and years ago with my own install in a corporate setting (no docker) when the docker user wasn't able to write to the storage folder.
Jan 06 15:50:54 lanfear docker[25322]: Traceback (most recent call last):
Jan 06 15:50:54 lanfear docker[25322]: File "/opt/graphite/lib/python3.6/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
Jan 06 15:50:54 lanfear docker[25322]: inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
Jan 06 15:50:54 lanfear docker[25322]: File "/opt/graphite/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
Jan 06 15:50:54 lanfear docker[25322]: return self.currentContext().callWithContext(ctx, func, *args, **kw)
Jan 06 15:50:54 lanfear docker[25322]: File "/opt/graphite/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
Jan 06 15:50:54 lanfear docker[25322]: return func(*args,**kw)
Jan 06 15:50:54 lanfear docker[25322]: File "/opt/graphite/lib/carbon/writer.py", line 189, in writeForever
Jan 06 15:50:54 lanfear docker[25322]: writeCachedDataPoints()
Jan 06 15:50:54 lanfear docker[25322]: --- <exception caught here> ---
Jan 06 15:50:54 lanfear docker[25322]: File "/opt/graphite/lib/carbon/writer.py", line 142, in writeCachedDataPoints
Jan 06 15:50:54 lanfear docker[25322]: state.database.create(metric, archiveConfig, xFilesFactor, aggregationMethod)
Jan 06 15:50:54 lanfear docker[25322]: File "/opt/graphite/lib/carbon/database.py", line 145, in create
Jan 06 15:50:54 lanfear docker[25322]: self.sparse_create, self.fallocate_create)
Jan 06 15:50:54 lanfear docker[25322]: File "/opt/graphite/lib/python3.6/site-packages/whisper.py", line 528, in create
Jan 06 15:50:54 lanfear docker[25322]: fallocate(fh, headerSize, remaining)
Jan 06 15:50:54 lanfear docker[25322]: File "/opt/graphite/lib/python3.6/site-packages/whisper.py", line 93, in _py_fallocate
Jan 06 15:50:54 lanfear docker[25322]: raise IOError(res, 'fallocate')
Jan 06 15:50:54 lanfear docker[25322]: builtins.OSError: [Errno 95] fallocate
Jan 06 15:50:54 lanfear docker[25322]: 06/01/2019 06:50:54 :: [console] Error creating stats.statsd.graphiteStats.flush_time: [Errno 95] fallocate
Jan 06 15:50:54 lanfear docker[25322]: 06/01/2019 06:50:54 :: [console] Unhandled Error
Running this as
/usr/bin/docker run --name graphite -p 8080:80 -p 2003:2003 graphiteapp/docker-graphite-statsd
I did a peek inside the actual container and looks like it's running as root inside, and folders do exists inside. Bit stumped why it wouldn't be able to create the .wsp files. It can create the folders in the storage/
Repro steps:
docker run -d --name graphite --restart=always -p 80:80 -p 2003-2004:2003-2004 -p 2023-2024:2023-2024 -p 8125:8125/udp -p 8126:8126 graphiteapp/docker-graphite-statsd
As per the README.md, "The container will initialize properly if you mount empty volumes at /opt/graphite/conf, /opt/graphite/storage, or /opt/statsd/config."
However, with empty /opt/graphite/conf directory, carbon continually prints this message to the logs:
Error: missing required config '/opt/graphite/conf/carbon.conf'
Graphite web starts, but has no data. No files are created in /opt/graphite/conf volume/directory
Current version of hopsoft/graphite-statsd does work with empty /opt/graphite/conf directory, and populates it with the following files:
aggregation-rules.conf
blacklist.conf
carbon.amqp.conf
carbon.conf
dashboard.conf
graphTemplates.conf
relay-rules.conf
rewrite-rules.conf
storage-aggregation.conf
storage-schemas.conf
whitelist.conf
Could you please add the section 'TAGDB' with comments to the Graphite webapp sample configuration
to make it more clear how to configure it and how it works for the new users?
Dear,
I have been following the readme to startup a instance, but got error when i access localhost: :
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 132, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/graphite/webapp/graphite/composer/views.py", line 35, in composer
profile = getProfile(request)
File "/opt/graphite/webapp/graphite/user_util.py", line 25, in getProfile
return default_profile()
File "/opt/graphite/webapp/graphite/user_util.py", line 41, in default_profile
'password': '!'})
File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py", line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 405, in get_or_create
return self.get(**lookup), False
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 328, in get
num = len(clone)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 144, in len
self._fetch_all()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 965, in _fetch_all
self._result_cache = list(self.iterator())
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 238, in iterator
results = compiler.execute_sql()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 98, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/sqlite3/base.py", line 318, in execute
return Database.Cursor.execute(self, query, params)
OperationalError: no such table: auth_user
My configuration is:
statsd:
container_name: graphite
image: graphiteapp/graphite-statsd
ports:
- "8088:80"
- "2003-2004:2003-2004"
- "2023-2024:2023-2024"
- "8125:8125/udp"
- "8126:8126"
volumes:
- "/Users/kevin/bigdata/graphite/conf:/opt/graphite/conf"
- "/Users/kevin/bigdata/graphite/storage:/opt/graphite/storage"
- "/Users/kevin/bigdata/graphite/log:/var/log"
- "/Users/kevin/bigdata/graphite/log:/var/graphite-log"
- "/Users/kevin/bigdata/graphite/log:/opt/statsd"
- "/Users/kevin/bigdata/graphite/logrotate.d:/etc/logrotate.d"
And i found "graphite.db" file under graphite/storage is zerobyte, seems the database initialized failure.
Currently the init script conf/etc/my_init.d/01_conf_init.sh
performs this action:
if [[ -z $statsd_dir_contents ]]; then
git clone -b v0.7.2 https://github.com/etsy/statsd.git /opt/statsd
cp $conf_dir/opt/statsd/config.js /opt/statsd/config.js
fi
However, there is no such file in $conf_dir/opt/statsd/
, there are two - config_tcp.js
and config_udp.js
.
I guess the solution would be to only copy one of them (I don't know statsd
well enough, but I assume it requires exactly one config file).
Workaround is not mounting the volume there, but this is a bug.
Add docker-compose file for starting dockerized version of statsd
. Can somebody please take a look ?
NAME READY STATUS RESTARTS AGE
graphite-649647f8d6-pn49t 0/1 Init:CrashLoopBackOff 6 7m
Jan 07 07:51:20 ip-172-20-60-166 kubelet[1249]: I0107 07:51:20.740085 1249 kuberuntime_manager.go:757] checking backoff for container "create-syncdb" in pod "graphite-649647f8d6-pn49t_graphite(63be982e-124f-11e9-9860-02b6de547408)"
Jan 07 07:51:20 ip-172-20-60-166 kubelet[1249]: I0107 07:51:20.740166 1249 kuberuntime_manager.go:767] Back-off 5m0s restarting failed container=create-syncdb pod=graphite-649647f8d6-pn49t_graphite(63be982e-124f-11e9-9860-02b6de547408)
Jan 07 07:51:20 ip-172-20-60-166 kubelet[1249]: E0107 07:51:20.740190 1249 pod_workers.go:186] Error syncing pod 63be982e-124f-11e9-9860-02b6de547408 ("graphite-649647f8d6-pn49t_graphite(63be982e-124f-11e9-9860-02b6de547408)"), skipping: failed to "StartContainer" for "create-syncdb" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=create-syncdb pod=graphite-649647f8d6-pn49t_graphite(63be982e-124f-11e9-9860-02b6de547408)"
Jan 07 07:51:33 ip-172-20-60-166 kubelet[1249]: I0107 07:51:33.705362 1249 kuberuntime_manager.go:757] checking backoff for container "create-syncdb" in pod "graphite-649647f8d6-pn49t_graphite(63be982e-124f-11e9-9860-02b6de547408)"
I am trying to install with helm and not able to do the deployment. When I dig it saw the error for sync_db not able to create.
Good morning,
logrotate seems not to work with the Docker Image 1.1.5-3. I even can not start the logrotate manually because I cannot find the service.
Under /etc/logrotate.d there is a logrotate file present.
The instructions mention that you can mount /opt/graphite/storage and /opt/graphite/conf to keep data and config outside the container. They also mention that it is ok to mount empty directories as they will be correctly populated.
On a first attempt this did not work.
I started the container like this:
docker run -d --name graphite --restart=always -p 8080:80 -p 2003-2004:2003-2004 -p 2023-2024:2023-2024 -p 8125:8125/udp -p 8126:8126 -v /opt/graphite/storage:/opt/graphite/storage -v /opt/graphite/conf:/opt/graphite/conf graphiteapp/graphite-statsd
I expected to see /opt/graphite/conf populated, but the directory was empty.
I connected a shell to the running container and notice there that from within the container it was impossible to write to the mounted volumes. The culprit was selinux.
The solution was to start the container like this:
docker run -d --name graphite --restart=always -p 8080:80 -p 2003-2004:2003-2004 -p 2023-2024:2023-2024 -p 8125:8125/udp -p 8126:8126 -v /opt/graphite/storage:/opt/graphite/storage:Z -v /opt/graphite/conf:/opt/graphite/conf:Z graphiteapp/graphite-statsd
Adding the "Z" option to the mount assures correct interaction with selinux, and now the processes in the container could write to the conf and storage dirs.
It might be useful to mention this in the docs.
But this was not sufficient to have everything working. Carbon did collect metrics, and files started to apprear in /opt/graphite/storage/whisper, but in the graphite browser the metrics remained empty.
On a hunch I just stopped and started the container again. And the metrics now appeared.
Could it be that when starting with an empty storage dir (no whisper subdir) the graphite browser doesn't scan this directory?
With a number of images (node, nginx, etc) now being runnable on multiple architectures can we add support for ARM (specifically aarch64) in this image?
I found that
while true; do echo -n "example:$((RANDOM % 100))|c" | nc -w 1 -u 127.0.0.1 8125; done
script from README doesn't work on my Mac OS laptop.
The problem is that nc
doesn't respect -w
option and hang up forever.
I was able to generate example stats using socat
instead:
while true
do
echo -n "example:$((RANDOM % 100))|c" | socat -t 0 - UDP:192.168.99.100:8125
sleep 1
done
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.