alpstable / gidari Goto Github PK

View Code? Open in Web Editor NEW

29.0 29.0 29.0 32.58 MB

Transport web data to local/remote storage using Gidari

License: Apache License 2.0

Makefile 0.79% Go 97.34% Shell 1.87%

api-wrapper csv data-science go http mongodb storage

gidari's People

Contributors

Stargazers

Watchers

gidari's Issues

investigate removing nlreturn from .golangci.

add linting action for ci

add read support to postgres storage

support maps in `MakeRecordsRequest`

investigate how to remove gosec, i.e. fixing weak crypto package imports

move repository and proto out of pkg and into the root

add polygon to transport.TestUpsert

add size of data to the storage response

rename alpine-hodler/drivers to alpine-hodler/sherpa

enable github actions

add a transport configuration for naming the collection/table

Add more granular rate limiting to web client

investigate moving the repl script [here](https://github.com/alpine-hodler/gidari/pull/51/files#diff-a2e30a67f616f365cdfd2c581bc3e6280f5f6ec46008405c2de3b8475287365bR33) to the script directory.

move internal/storage package to pkg/storage; deprecate internal/storage.go

add storage upsert and tx support for postgres

investigate breaking up transport.go file

add tx support to storage and repository (mongo)

Correct misteps in deprecating pkg package

Make repoJobCh in transport.Upsert a buffered channel.

investigate removing funlen from .golangci , this probably requires an insane refactor

add truncate logic to the transport upsert. i.e. if truncate: true in the config, then truncate the tables as they are populated.

rename storage.docker-compose.yaml to docker-compose.yaml

add start/end timestamps to transport web worker log

Add replica set support for mono in docker-compose.yml

transactionExceededLifetimeLImitSeconds bug

When running a long-standing query on mongodb, you get an error like this:

FATA[0061] error upserting data: (TransactionExceededLifetimeLimitSeconds) operation was interrupted because the transaction exceeded the configured 'transactionLifetimeLimitSeconds

Here is an example configuration:

url: https://api-public.sandbox.exchange.coinbase.com
authentication:
  apiKey:
    passphrase: ***
    key:  ***
    secret: ***
dnsList:
  - mongodb://mongo1:27017/coinbasepro
rateLimit:
  burst: 5
  period: 1
truncate: true
requests:
  - endpoint: /accounts
    method: GET
  - endpoint: /currencies
    method: GET
  - endpoint: /products
    method: GET
  - endpoint: /products/BTC-USD/stats
    method: GET
    table: btc_stats
  - endpoint: /products/BTC-USD/candles
    method: GET
    timeseries:
        startName: start
        endName: end
        period: 18000
        layout: "2006-01-02T15:04:05Z"
    query:
      granularity: "60"
      start: "2018-11-09T17:11:08Z"
      end: "2019-01-12T18:11:08Z"