c99koder / personal-influxdb Goto Github PK

Hey there! Great work on writing a bunch of these things out. I have a cat who was recently diagnosed as diabetic, and had a freestyle libre2 attached to him to monitor his blood sugar levels constantly. I work in DevOps, and so I immediately googled to see if there was an API available I could query to get his data, and came across your repo.

I took the liberty (hope you don't mind of taking the librelinkup and config python files, writing a requirements.txt, creating a docker file, and running it in Kubernetes.

The docker image:
https://hub.docker.com/repository/docker/evanrich/freelibre2influx/general

requirements.txt:

influxdb==5.3.1
requests==2.25.1
colorlog==6.7.0
pytz==2022.1

Here's the dockerfile:

FROM python:3.11.4-alpine3.18

LABEL maintainer="Evan Richardson (evanrich81[at]gmail.com)"

# Set the working directory to /app
WORKDIR /app

# Copy the requirements.txt file into the container at /app
COPY requirements.txt /app/

# Install the Python dependencies specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Copy the link.py and config.py files to the root directory
COPY librelinkup.py /
COPY config.py /

ENV INFLUXDB_HOST="default_host"
ENV INFLUXDB_PORT="default_port"
ENV INFLUXDB_USER="default_username"
ENV INFLUXDB_PASSWORD="default_password"
ENV LIBRELINKUP_DATABASE="database"
ENV LIBRELINKUP_USERNAME="librelinkup_username"
ENV LIBRELINKUP_PASSWORD="librelinkup_password"

# Run the Python script link.py
CMD ["python", "/librelinkup.py"]

I run this in a cronjob on kubernetes with the following code:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: freelibre-cron
  namespace: default
spec:
  schedule: "0 */1 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            app: freelibre-job
        spec:
          containers:
          - name: freelibre-upload-cron
            image: evanrich/freelibre2influx:latest
            imagePullPolicy: IfNotPresent
            envFrom:
            - secretRef:
                name: freestyle-secrets
          restartPolicy: OnFailure

All of the environmental variables (passwords, usernames, etc) are stored in DOPPLER, a free tier external secrets platform. I have external-secrets running in my home cluster and it automatically pulls in the credentials it needs with the following:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: freestyle-secrets
  namespace: default
spec:
  secretStoreRef:
    kind: ClusterSecretStore
    name: doppler
  target:
    name: freestyle-secrets
  data:
    - secretKey: INFLUXDB_HOST
      remoteRef:
        key: INFLUXDB_HOST
    - secretKey: INFLUXDB_PORT
      remoteRef:
        key: INFLUXDB_PORT
    - secretKey: INFLUXDB_USERNAME
      remoteRef:
        key: INFLUXDB_ADMIN_USER
    - secretKey: INFLUXDB_PASSWORD
      remoteRef:
        key: INFLUXDB_ADMIN_PASSWORD
    - secretKey: LIBRELINKUP_USERNAME
      remoteRef:
        key: LIBRELINKUP_USERNAME
    - secretKey: LIBRELINKUP_PASSWORD
      remoteRef:
        key: LIBRELINKUP_PASSWORD
    - secretKey: LIBRELINKUP_DATABASE
      remoteRef:
        key: LIBRELINKUP_DATABASE

This all then gets visualized in Grafana using a single panel dashboard.

I didn't fork your repo or anything, but, @c99koder if you're interested, I will and throw my files in so you can add them for others to use. The container/etc can easily be expanded upon for the other python scripts, I just had no use for them.

Thanks again!

Not ready to PR (if you're interested), but I've made some updates for this running in Docker. https://github.com/psyciknz/personal-influxdb.git
I've made a docker file and converted a bunch of the config.py code (specifically around the influx server and trakt-tv) to using environment vars.

When Trakt-tv comes back up I'll be able to test it properly. As because they went down, I started looking at a local backup.

Interesting concept.

PSN achievements are displayed all lowercase with - instead of spaces

Hello,

I'm having trouble using your scripts. I'm not sure if I have done something wrong or if the scripts are maybe outdated or something?

The steam script throws the following error:

File "D:\python influx\steam.py", line 79, in <module> for app in fetch_recents(): File "D:\python influx\steam.py", line 55, in fetch_recents json = response.json() File "C:\Users\arian\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\models.py", line 900, in json return complexjson.loads(self.text, **kwargs) File "C:\Users\arian\AppData\Local\Programs\Python\Python39\lib\json\__init__.py", line 346, in loads return _default_decoder.decode(s) File "C:\Users\arian\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Users\arian\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The fitbit Script throws me this error, maybe because I didn't fill in FITBIT_ACCESS_TOKEN and FITBIT_INITIAL_CODE but honestly I just didn't find out how I could get these.

Traceback (most recent call last): File "/chronos/scripts/fitbit-to-influx/fitbit-to-influx.py", line 223, in <module> response.raise_for_status(); File "/chronos/scripts/fitbit-to-influx/.venv/lib/python3.7/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.fitbit.com/oauth2/token

I cannot get data from trakt.tv since 2021-03-08, but I realised this few days ago :(
I got this error message:
Traceback (most recent call last):
File "./trakt-tv.py", line 78, in
for item in Trakt['sync/history'].get(pagination=True, per_page=100, start_at=datetime(date.today().year, date.today().month, 1), extended='full'):
TypeError: 'NoneType' object is not iterable

I tried to recreate my json file. It didn't work but now after I restored the original (it was from december):
Traceback (most recent call last):
File "./trakt-tv.py", line 76, in
Trakt.configuration.defaults.oauth.from_response(auth)
File "/usr/local/lib/python3.7/site-packages/trakt/core/configuration.py", line 157, in from_response
raise ValueError('Invalid "response" parameter provided to oauth.from_response()')
ValueError: Invalid "response" parameter provided to oauth.from_response()

I'm totally lost what happened. Can you help me?

Hi

Exist changed from API v1 to v2. Does this reflect the change?

Either way it doesn’t seem to be pulling exist data for me.

Could you let me know the app settings on the side of Exist? (Auth type, URL etc) perhaps the issue lies there? Do we need to run any scripts to get a key after making the app on Exist, as described in their developer docs?

thank you!

I haven't had a chance to produce a full pull request but new version details are required again for the Switch integration. Would it be worth moving these to a config file to make them easier to update when needed?

'x-moon-app-internal-version': '303',
'x-moon-app-display-version': '1.15.0',
'x-moon-os-version': '14.5.1',
'user-agent': 'moon_ios/1.15.0 (com.nintendo.znma; build:303; iOS 14.5.1) Alamofire/4.8.2',

Hi Sam!

I'm trying to ingest my flight data from FsHub into InfluxDB but am getting the following error when running the script (fshub.py):

Got flights 100 from FsHub
Traceback (most recent call last):
File "./fshub.py", line 143, in
cursor = fetch(100,cursor)
File "./fshub.py", line 44, in fetch
if flight['departure']['icao'] != None and flight['arrival']['icao'] != None:
TypeError: 'NoneType' object is not subscriptable

Have you seen this before?

My profile (if it helps) can be found here: https://fshub.io/pilot/2/profile

Thanks for developing this awesome connector!

Hi there,

first of all: great project you have created here!
But I have one issue: I'm already running on influxdb v2 - are there any plans in supporting it also?
Writing to influxdb v2 is possible with some preparation which needs to be done on the database ... except one call: create_database(db) does not work:

Error message: not implemented: CREATE DATABASE

Sorry for new issue but you closed the old one and I cannot reopen.

I still cannot connect to steam:

2023-08-23 03:11:17,165 Starting new HTTPS connection (1): steamcommunity.com:443
2023-08-23 03:11:17,580 https://steamcommunity.com:443 "GET /id/76561197996904310/games/?tab=all HTTP/1.1" 200 4961
2023-08-23 03:11:17,592 Uncaught exception:
Traceback (most recent call last):
File "/root/steam.py", line 85, in
recents = scrape_recents()
File "/root/steam.py", line 79, in scrape_recents
data = soup.find('script', string=re.compile('var rgGames = [{')).string
AttributeError: 'NoneType' object has no attribute 'string'

steamfinder is showing fine

Trying https://steamcommunity.com/id/76561197996904310/games/?tab=all. gives me profile not found?

Sorry if I ask stupid questions. I want to have this deployed and can barely keep my eyes open :)

Hello,

does Steam & Instagram support multiple accounts? Or do I have to run the same script multiple times with different parameters?

Thanks in advance,

Scarrty

Hey :)

I have installed influxv1 and running steam.py I get:

2023-08-23 02:50:52,360 Connecting to localhost:8086
2023-08-23 02:50:52,361 Starting new HTTP connection (1): localhost:8086
2023-08-23 02:50:52,363 http://localhost:8086 "POST /query?q=CREATE+DATABASE+%22gaming%22 HTTP/1.1" 200 None
2023-08-23 02:50:52,364 Connected to localhost:8086
2023-08-23 02:50:52,365 http://localhost:8086 "GET /query?q=SELECT+last%28%22total%22%29+AS+%22total%22+FROM+%22time%22+WHERE+%22platform%22+%3D+%27Steam%27+AND+%22total%22+%3E+0+AND+%22player_id%22+%3D+%2790175600927979536%27+GROUP+BY+%22application_id%22+ORDER+BY+%22time%22+DESC&db=gaming HTTP/1.1" 200 None
2023-08-23 02:50:52,367 Starting new HTTPS connection (1): steamcommunity.com:443
2023-08-23 02:50:52,792 https://steamcommunity.com:443 "GET /id//games/?tab=all HTTP/1.1" 200 6311
2023-08-23 02:50:52,803 Uncaught exception:
Traceback (most recent call last):
File "/root/steam.py", line 85, in
recents = scrape_recents()
File "/root/steam.py", line 79, in scrape_recents
data = soup.find('script', string=re.compile('var rgGames = [{')).string
AttributeError: 'NoneType' object has no attribute 'string'

Any idea?

Thanks!

Hi and sorry for new issue.

I have added psn, influx receives the data. Yet the gaming dashboard has no data at all. I assume this is the correct dashboard for it?

Example

Thanks!

As the title says the fitbit script doesn't seem to import my heartrate data - I have used your provided fitbit grafana template and the heartrate panel stays empty. I tried to edit it and look if a typo has made it's way in but couldn't find any problem there. Sadly my python knowledge is not extensive enough to debug myself.

Hello :)
Unfortunately the psn.py does not take all data.

I also am not sure if it is an API max history issue as I have missing data between existing data and last data missing as well.

Could we check this somehow?
I have started the script several times, each time same data gets pulled.

Thanks!

Hello,

I have another problem. After your help I got the fitbit script to work. But after about a day (don't know for sure, logs got lost) I get the error message "HTTP request failed: 401 Client Error: Unauthorized for url:"

I can fix this by generating a new access token but I don't want to do this manually every day. There is an option in the fitbit api workflow page to set the duration for the token in ms and I've already set it to ridicously high. Do you have a recommendation?

A Dockerfile would be nice.

Is there any way you could include playstation stats?

Hello,

I have tried trakt as well. It gets 16 watched episodes in 2 days. Seems it does not pull correctly.

Thanks

Hello

the rescue time dashboard is only displaying ${_cell2} (not exact) in the dashboard from the JSON in the repo. Is there a fix for this? Happy to share screenshots!

thanks again!!

Hello, upon running the pip command I am getting this error:

python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [19 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 14, in
File "/usr/lib/python3/dist-packages/setuptools/init.py", line 16, in
import setuptools.version
File "/usr/lib/python3/dist-packages/setuptools/version.py", line 1, in
import pkg_resources
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 77, in
import('pkg_resources.extern.packaging.specifiers')
File "/usr/lib/python3/dist-packages/pkg_resources/_vendor/packaging/specifiers.py", line 24, in
from .utils import canonicalize_version
File "/usr/lib/python3/dist-packages/pkg_resources/_vendor/packaging/utils.py", line 8, in
from .tags import Tag, parse_tag
File "/usr/lib/python3/dist-packages/pkg_resources/vendor/packaging/tags.py", line 5, in
import logging
File "/tmp/pip-install-nmi5pbb/logging_593e45d1e049402eafcf6c9c41b75056/logging/init.py", line 618
raise NotImplementedError, 'emit must be implemented '
^
SyntaxError: invalid syntax
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

Thanks!

Hi there, I'm in the process of setting up a test FitBit > Grafana, however, I've fallen at the final hurdle.

I've set up a "Authorization Code Flow" flow type, added all the details to the python script, but when I run it I get the following message.

python .\fitbit.py
Got devices from Fitbit
Got sleep sessions from Fitbit
Got steps from Fitbit
Got distance from Fitbit
HTTP request failed: 400 Client Error: Bad Request for url: https://api.fitbit.com/1/user/-/activities/floors/date/today/1d.json

If I run the CURL command from the tutorial it shows the results okay, it's just whenever I try to run the python script it fails. I'm not good with python or scripting so I'd appriciated any help.

Thanks,
Tom