stuianna / cmclogger

A Python daemon for fetching, querying and storing crypto-currency data using the coinmarketcap free API.

CMCLogger - Coin Market Cap Cryptocurreny Data Logger

A python API and script for requesting, parsing and storing the latest cryptocurrency data availalbe using the Coin Market Cap Free API. Data entries are stored in a SQLite3 database, with CLI features for querying data and logger status.

TLDR - Just Get Some Data

# Install the module and script
pip install CMCLogger

# Initialise the required configuration, supplying your API key. (Made above)
CMCLogger -a 'your-api-key' -g

# Start the logger. Use 'nohup CMCLogger' to start detached, 'CMCLogger &' to start in background.
CMCLogger 

# Query the latest stored price information for ticker 'BTC'
CMCLogger -q BTC
'BTC: $11676.55 (3.53%)'

# Same as above, but more detailed.
CMCLogger -dq BTC
'BTC: $11676.55 1H: -0.22% 1D: 3.53% 7D: 4.77% 24h Volume: 66.65 Billion'

# Get the logger status. Health is the moving average success rate of the last 30 calls.
CMCLogger -s
'Last successful call 1 minutes ago, health 100.0%.'

# Stop the logger
CMCLogger -k

# Copy the SQLite database to an excel file
CMCLogger -x

Stored Data

When the logger is initialised, it creates a directory structure inside $XDG_CONFIG_HOME. This could be ~/.local/share/CMCLogger or ~/.config/CMCLogger, use echo $XDG_CONFIG_HOME to find the location on your system.

The directory structure is as follows:

CMCLogger
├── config.ini
└── data
    ├── cryptoData.db
    ├── log
    └── status.ini

config.ini

Contains configuration parameters used. Change any of these settings and restart the logger to apply them. Values shown are the default.

[CMC_API]
api_private_key = your-private-key-here
conversion_currency = AUD
curreny_symbol = $
rank_start_index = 1
rank_end_index = 200
request_interval = 5

[General]
status_file_format = ini

cryptoData.db

A SQLite3 database containing all data retreived by the logger. The database contains a separate table, named afeter the cryptocurrency symbol for each individual currecy collected. SQLitebrower, is a good tool for browsing databases, or use CMCLogger -x to convert the database to an Excel file for viewing.

status.ini

Contains information on the status of the logger. This information is queried and returned when using CMCLogger -s or CMCLogger -ds

[Last Successful Call]
timestamp = 2020-04-23 23:10:55.275000+03:00
error_code = 0
error_message = None
elapsed = 17
credit_count = 1

[Last Failed Call]
timestamp = 2020-04-23 22:39:26.646297+03:00
error_code = 255
error_message = No network connections available.
elapsed = 0
credit_count = 0

[Current Session]
health = 100.0
successful_calls = 35644
failed_calls = 587
success_rate = 98.38

[All Time]
successful_calls = 35746
failed_calls = 589
success_rate = 98.38

log

Runtime logs stored by the Python logging module.

Polybar / I3WM Integration

I3WM

Add to .i3/config to start the logger on login.

exec --no-startup-id CMCLogger

Polybar

Add a polybar module for the target bar:

[module/crypto]
type = custom/script
exec = ~/bin/crypto
tail = true
interval = 300

The target script simply cycles through a set of symbols, a blank string is returned if the last entry was more than 10 minutes ago.

#!/bin/bash

queryArray=("BTC" "LTC" "ETH")
TEMPFILE="/tmp/tmp.CMCLOGGER"
source $TEMPFILE 2> /dev/null

function getNextSymbol {
	position=0
	for symbol in ${queryArray[@]}; do
		((position++))
		if [[ "$symbol" == "$LAST_SYMBOL" ]]; then
			size=${#queryArray[@]}
			position=$((position%size))
			CURRENT_SYMBOL=${queryArray[$position]}
			break
		fi
		CURRENT_SYMBOL=${queryArray[0]}
	done
	echo "LAST_SYMBOL=$CURRENT_SYMBOL" > $TEMPFILE
}

function checkStatus {
	jsonStatus=$(CMCLogger -js 2> /dev/null)
	lastCall=$(echo "$jsonStatus" | jq -r '.last_call')

	if (( $lastCall > 10 )); then
		echo ''
		exit
	fi
}

getNextSymbol
checkStatus
CMCLogger -q $CURRENT_SYMBOL

cmclogger's Issues

Configuration value are overwritten

Sometimes, when the logger is restarted, the configuration values .config/CMCLogger/config.ini are overwritten with the default values.

Un-handled API exception.

Unrecoverable run-time error. Out-of-date project structure, but still relevent.

Traceback (most recent call last):
  File "...CMCLogger/modules/cmcapi_wrapper.py", line 139, in __parseSuccessfulGetLatestRequest
    jsonFields = json.loads(response.text)
  File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "...", line 76, in <module>
    cmcLogger.fetchAndStoreData(credentials[currentCred]['api_key'])
  File "...CMCLogger/CMCLogger.py", line 67, in fetchAndStoreData
    goodResponse = self.__api.getLatest(apiKey)
  File ".../CMCLogger/modules/cmcapi_wrapper.py", line 90, in getLatest
    return self.__parseSuccessfulGetLatestRequest(response)
  File ".../CMCLogger/modules/cmcapi_wrapper.py", line 141, in __parseSuccessfulGetLatestRequest
    print(response.text)
OSError: [Errno 5] Input/output error