panifie / pingpong.jl Goto Github PK

Cryptocurrency trading bot, and backtesting framework in julia

Home Page: https://panifie.github.io/PingPong.jl/

License: GNU General Public License v3.0

Julia 99.58% Python 0.06% Emacs Lisp 0.05% Dockerfile 0.20% Shell 0.11%

finance backtesting-trading-strategies backtesting ccxt cryptocurrency cryptocurrency-exchanges trading quantitative-trading cryptocurrency-trading-bot trading-bot

pingpong.jl's People

Contributors

Stargazers

Watchers

Forkers

charles-achilefu x-erich-x bradley39e mrchaos femtotrader trading-k capita-7 cybito janckerchen

pingpong.jl's Issues

Packages still missing precompiles

Once again, missing keyword arguments in the function signature.

julia> e = getexchange!(:okx)
Exchange: OKX | 1406 markets | 14 timeframes
julia> exs.check(e, type=:basic)
ERROR: MethodError: no method matching check(::ExchangeTypes.CcxtExchange{ExchangeID{:okx}}; type::Symbol)

Closest candidates are:
  check(::Exchange, ::Any) got unsupported keyword argument "type"
   @ Exchanges /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:693
  check(::Exchange) got unsupported keyword argument "type"
   @ Exchanges /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:693

Stacktrace:
 [1] top-level scope
   @ REPL[13]:1

[PingPongInteractive] ERROR: LoadError: UndefVarError: `za` not defined

precompiled error as using PingPongInteractive , submit a pull request

❯ julia --project=./PingPongInteractive
julia> ]
(PingPongInteractive) pkg> instantiate

julia> using PingPongInteractive 
Precompiling PingPongInteractive

ERROR: LoadError: UndefVarError: `za` not defined
Stacktrace:
 [1] include(mod::Module, _path::String)
   @ Base ./Base.jl:495
 [2] include(x::String)
   @ Optimization /opt/opensource/jl/PingPong.jl/Optimization/src/Optimization.jl:1
 [3] top-level scope
   @ /opt/opensource/jl/PingPong.jl/Optimization/src/Optimization.jl:9

fix

Optimization/src/module.jl:6
- using .Instances.Data: zinstance, za
+ using .Instances.Data: zinstance, Zarr as za

Simulators

Spread: always enabled
Slippage: for large trades
Market impact: for very large trades

[Help] Python: AuthenticationError: binance {"code":-2015,"msg":"Invalid API-key, IP, or permissions for action."}

I have searched online discussions about this issue and have done the following work:

Confirmed that it is not the US version of Binance.
the IP can access Binance because the website is working.
Created an API key after opening the futures account (I am able to trade futures in the mock trading).
Confirmed that all permissions are granted, as shown in the screenshot.
No IP restrictions have been set.
Confirmed that the apiKey and secret have no typo.

However, I still encounter an apiKey error. Any other aspects to consider?

┌ Error: exchange: auth error
│   resp = Python: AuthenticationError: binance {"code":-2015,"msg":"Invalid API-key, IP, or permissions for action."}
└ @ Exchanges /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:473

[Help] How to debug(stepinto) in vscode?

I tried debugging the code in VS Code, but I can't step into the code either by setting breakpoints or using the @enter macro. What configuration is needed?
(The environment variables in .envrc can be inherited by launching VS Code from the terminal.)

using PingPong
using Data: Data as data
using SimMode: SimMode as sm

@environment!
s = st.strategy(:SimpleStrategy, exchange=:binance) 

# step into failed
@enter fetch_ohlcv(s, from=-1000) 
# step into failed
@enter load_ohlcv(s)
# step into failed
@enter sm.start!(s)

optionalize dependencies

Split up dependencies for:

plotting
indicators/explorations

add support for python strategies

Although not really recommended, embedding python strategies is a very easy task.

Several errors in Performance Metrics

follow https://panifie.github.io/PingPong.jl/stats/#Performance-Metrics

Stats.sharpe(strategy_instance, tf"1d", rfr=0.01)
Stats.sortino(strategy_instance, tf"1d", rfr=0.01)
Stats.calmar(strategy_instance, tf"1d")
Stats.expectancy(strategy_instance, tf"1d")

output

julia> balloons(s) # works great
julia> Stats.sharpe(s, tf"1d", rfr=0.01) #<----error
ERROR: MethodError: no method matching sharpe(::Strategies.SimStrategy{:SimpleStrategy, ExchangeID{:binance}, NoMargin, :USDT}, ::TimeFrames.Day; rfr::Float64)

Closest candidates are:
  sharpe(::Strategies.Strategy, ::Any, ::Any) got unsupported keyword argument "rfr"
   @ Stats /opt/opensource/jl/PingPong.jl/Stats/src/metrics.jl:77
  sharpe(::Strategies.Strategy, ::Any) got unsupported keyword argument "rfr"
   @ Stats /opt/opensource/jl/PingPong.jl/Stats/src/metrics.jl:77
  sharpe(::Strategies.Strategy) got unsupported keyword argument "rfr"
   @ Stats /opt/opensource/jl/PingPong.jl/Stats/src/metrics.jl:77

julia> Stats.sortino(s, tf"1d", rfr=0.01)  #<----error
ERROR: MethodError: no method matching sortino(::Strategies.SimStrategy{:SimpleStrategy, ExchangeID{:binance}, NoMargin, :USDT}, ::TimeFrames.Day; rfr::Float64)

Closest candidates are:
  sortino(::Strategies.Strategy, ::Any, ::Any) got unsupported keyword argument "rfr"
   @ Stats /opt/opensource/jl/PingPong.jl/Stats/src/metrics.jl:107
  sortino(::Strategies.Strategy, ::Any) got unsupported keyword argument "rfr"
   @ Stats /opt/opensource/jl/PingPong.jl/Stats/src/metrics.jl:107
  sortino(::Strategies.Strategy) got unsupported keyword argument "rfr"
   @ Stats /opt/opensource/jl/PingPong.jl/Stats/src/metrics.jl:107

julia> Stats.calmar(s, tf"1d") # works great
-0.3158866997026408

julia> Stats.expectancy(s, tf"1d")  #<----error
ERROR: UndefVarError: `returns` not defined
Stacktrace:
 [1] expectancy(s::Strategies.SimStrategy{:SimpleStrategy, ExchangeID{:binance}, NoMargin, :USDT}, tf::TimeFrames.Day)
   @ Stats /opt/opensource/jl/PingPong.jl/Stats/src/metrics.jl:206
 [2] top-level scope
   @ /opt/opensource/jl/PingPong.jl/TimeTicks/src/module.jl:61

Consider using other exchange classes

OctoBot (ccxt based): https://github.com/Drakkar-Software/OctoBot-Tentacles/tree/master/Trading/Exchange
Humming bot connectors (non ccxt based): https://github.com/hummingbot/hummingbot/tree/master/hummingbot/connector/exchange

I am more inclined on using octo-bot ones since ccxt based, and ccxt added websockets recently anyway

Project status

v1

v1 is tagged
The test suite is mostly passing, although there are still 1 or 2 flapping tests.
The pre built images are on the docker registry

v1 will only receive bug fixes and new adhoc functions for better exchange support.

v2

v2 is not currently being worked on, this is a list of things that need to happen for a v2 release and other things that would be nice to have:

required

Ccxt gets transpilation support for julia. The main goal of v2 would be to make python optional and not an hard dependency, so all the python ccxt handling would need to be removed
Complete wrapping of the interal exchange api into a dedicated ExchangeApi module. Currently it is mostly split into the Exchanges and LiveMode modules. It is fine to model it over the ccxt api but this needs to happen to make ccxt swappable.
Dependencies trimming: excluding plotting and optimization which are inevitably dependency heavy we could avoid pulling other packages like DataStructures and instead vendor the used container types. Also telegram support should be made optional.
Reduce number of modules: some modules could be merged, like Lang,Misc,TimeTicks Data,Processing SimMode,Executors PaperMode,LiveMode
With python made optional, it is worthwhile to perform an analysis for type instabilities using JET.jl

optional

With the exchange api wrapped we can (and should) add the hummingbot gateway as addition exchange support
Better test suite. Ccxt lacks a self hostable exchange api server for testing purposes which makes it neccessary to test against live exchange testnets this is annoying because of api keys, rate limits, and testnets going down every once in a while, all things causing tests to flap. Also julia mocking seems somewhat buggy, and although we mock some responses in a few tests, mocking can't be used everywhere because we also need to test async behaviour.
Overhaul of AssetInstance trades history from a vector of trades to a dataframe, for reasoning see here
simplify timeframes package to always and only use Millisecond as time unit
reconstructing the orders/trades history from the exchange with leverage and margin. (Currently exchanges don't have apis for position history)
rename: (abstract)asset -> (abstract)instrument, assetinstance -> market

Failed to install manually according to the official website documentation

Installed manually according to the official documentation, but finally failed when installing.

❯ cd  /PingPong.jl
(ok)
❯ mkdir .conda
(ok)
❯ direnv allow
direnv: loading /PingPong.jl/.envrc
direnv: export +JULIA_CONDAPKG_ENV +JULIA_CPU_TARGET +JULIA_FULL_PRECOMP +JULIA_LOAD_PATH +JULIA_NOPRECOMP +JULIA_NUM_THREADS +JULIA_PRECOMP +JULIA_PROJECT +PINGPONG_LIQUIDATION_BUFFER
(ok)
❯ julia --project ./PingPong
(ok)
(PingPong) pkg> instantiate
(failed)
   Installed Blosc ───────────────── v0.7.3
   Installed TimeFrames ──────────── v0.2.0
   Installed MicroMamba ──────────── v0.1.14
   Installed EzXML ───────────────── v1.2.0
   Installed Blosc_jll ───────────── v1.21.5+0
   Installed LRUCache ────────────── v1.6.1
   Installed LMDB ────────────────── v1.0.0
   Installed Zstd_jll ────────────── v1.5.6+0
   Installed ArrowTypes ──────────── v2.3.0
   Installed FunctionalCollections ─ v0.5.0
   Installed Unitful ─────────────── v1.20.0
   Installed Retry ───────────────── v0.4.1
   Installed PythonCall ──────────── v0.9.20
   Installed XMLDict ─────────────── v0.4.1
   Installed LazyJSON ────────────── v0.2.2
   Installed micromamba_jll ──────── v1.5.8+0
   Installed IniFile ─────────────── v0.5.1
   Installed TableMetadataTools ──── v0.1.0
   Installed AWSS3 ───────────────── v0.10.4
   Installed LMDB_jll ────────────── v0.9.27+0
   Installed Lz4_jll ─────────────── v1.9.4+0
   Installed SymDict ─────────────── v0.3.0
   Installed Chain ───────────────── v0.6.0
   Installed DataFramesMeta ──────── v0.15.2
   Installed UnsafePointers ──────── v1.0.0
   Installed Telegram ────────────── v1.1.5
   Installed PropertyDicts ───────── v0.1.2
   Installed Term ────────────────── v2.0.6
   Installed CondaPkg ────────────── v0.2.22
   Installed DiskArrays ──────────── v0.4.2
   Installed ConcurrentCollections ─ v0.1.1
   Installed AWS ─────────────────── v1.90.4
  Downloaded artifact: Blosc
  Downloaded artifact: Zstd
  Downloaded artifact: Lz4
Precompiling project...
  59 dependencies successfully precompiled in 52 seconds. 53 already precompiled.
  1 dependency had output during precompilation:
┌ micromamba_jll
│   Downloading artifact: micromamba
│
│  [pid 14641] waiting for IO to finish:
│   Handle type        uv_handle_t->data
│   timer              0x60000006c480->0x109c7e3e0
│  This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
│
│  [pid 14641] waiting for IO to finish:
│   Handle type        uv_handle_t->data
│   timer              0x60000006c480->0x109c7e3e0
│  This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
└
  2 dependencies errored.
  For a report of the errors see `julia> err`. To retry use `pkg> precompile`

things that I am not particularly happy about

The split of buy/sell orders into long/short, which doubles every downstream type into long/short. This is necessary to support hedged position management, but I don't think I will ever implement it since I use isolated, so I might have just chosen to not support it and simplify the types
The sign handling for long/short amounts. Initially It was all positive numbers, but then I thought turning the sign negative for shorts would reduce the amount of errors, maybe it did (?) but now a short sell gains "negative cash" and a short buy reduces negative cash to zero 😕
The fully "expanded" orders and trades structures, that could be made more array friendly. This is covered here
The exchange api creeping into many packages, now ccxt is kind of an hard dep
Pythonisms creeping into the exchange api. If ever ccxt gets julia support, the switch to native julia will not be so trivial
this issue being a glass half empty kind of issue

Missing functionalities

Isolated margin

Funding rate estimation (currently we just use higher trading fees)
Updating margin of open positions
Updating leverage of open positions (and also with pending orders)

Margin

borrow/repay

Execution

Take profit and stop loss attached to orders

Avoid ccxt candles emulation

Using watchers, when fetching candles, we call fetchOHLCV et similar, but in case candles are emulated we shouldn't call it, and instead process from trades directly since we already do it in other cases. So we have to ensure exc.has["fetchOHCLV"] is not emulated.

Apis wrappers

cryptocompare
alphavantage
coincodex

Data management

Currently we only consider the executor under the pingpong model. That is that the strategy has to deal with its own data feeds.

If the pingpong model also accounted for data updates, a pong! call within the strategy could branch into a Watcher instance in live mode and a prefetched dataframe in sim mode.

However since a strategy has always carte blanche, the benefits of managing data for the strategy are very small (we can't manage every piece of data, so managing only a few doesn't improve the status quo).

Moreover the main ping! function works with a timestamp as primary input, so there is never any ambiguity on the time context of execution and whatever data the strategy uses, should already be time context aware because of this.

Performance tweaks

backtest

There are many repeated function calls like openat,closeat,etc... that could be memoized
Consider either removing getproperty for asset instances and strategies or replace all dot accessors within SimMode with some getfield wrapper function. Because the getproperty accessor is slower than getfield if it is customized
an alternative implementation for orders/trades would use arrays instead of structs to be more allocation friendly

watchers

StructArrays or TypedTables could be used when fetching (streaming) ohlcv data feeds
OHLCV watcher implementations parse the whole python dicts into named tuples (CcxtTrade, CcxtTicker), in cases where full parsing is not required, the python data should be parsed more lazily

other

Scour the codebase for lazy use of @view/s on assignments, that should be replaced with copyto! (starting from Data)
search/replace Period with FixedPeriod to see if union splitting improves performance
Full specialization: many functions signatures are defined over generic types (non fully parametrized), benchmarks are needed to assert if full specialization is worth the extra compilation time / bin size

The behavior of bn.binancedownload(freq=:daily) is strange.

The behavior of freq=:daily is strange in three ways:

It overwrites existing data instead of appending to it.
It downloads the oldest data instead of the most recent data.
It can only download just over a year of data, rather than the entire history.

expect to do

# Download monthly data up until the previous month.
bn.binancedownload("eth", market=:data, freq=:monthly, kind=:klines)

# Download daily data to complete the data for this month.
bn.binancedownload("eth", market=:data, freq=:daily, kind=:klines)

# After the two steps, we should get the most complete historical data possible.

actually happen

freq==:monthly  # Mission accomplished successfully,  from 2017-08-17 to 2024-05-31
freq==:daily  #overwrite above with  data from 2017-08-17 to 2018-12-29

btw, the parameter freq=:monthly is unnecessary for bn.binanceload, because data from both parameter are saved to tf_1m.

ERROR: LoadError: InitError: Evaluation into the closed module `PingPong` breaks incremental compilation because the side effects will not be permanent.

pull last commit and test SimpleStrategy failed.

julia> @environment!

julia> s = st.strategy(:SimpleStrategy, exchange=:binance)

Precompiling SimpleStrategy
        Info Given SimpleStrategy was explicitly requested, output will be shown live
ERROR: LoadError: InitError: Evaluation into the closed module `PingPong` breaks incremental compilation because the side effects will not be permanent. This is likely due to some other module mutating `PingPong` with `eval` during precompilation - don't do this.
Stacktrace:
  [1] eval
    @ ./boot.jl:385 [inlined]
  [2] __init__()
    @ PingPong /opt/opensource/jl/PingPong.jl/PingPong/src/PingPong.jl:6
  [3] run_module_init(mod::Module, i::Int64)
    @ Base ./loading.jl:1134
  [4] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1122
  [5] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1067
  [6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)

Profitability calculator

Some strategies work based on past profitability, so they require a way to calc the profits of previous trades. Because ping pong does not make assumptions about execution of trades (buys and sells are not necessarily interleaved) a module for ensuring that the strategy executes only 1 trade at a time is required. And the method to track profitability should be implemented in such module

any documents on how to backtest a strategy?

how to use the package, how to get information, like profit, sharp ratio, and other stuffs in backtrader in python

LoadError: Creating a new global in closed module `Main` (`resp`) breaks incremental compilation

According to the library documentation, the validation procedure was running successfully yesterday, but today a new issue has arisen.

using PingPong
using Data: Data as data
using SimMode: SimMode as sm
@environment!
s = st.strategy(:SimpleStrategy, exchange=:okx)  <--- error, but pased yestoday

output

ERROR: LoadError: Creating a new global in closed module `Main` (`resp`) breaks incremental compilation because the side effects will not be permanent.
Stacktrace:
  [1] setproperty!
    @ ./Base.jl:51 [inlined]
  [2] setproperty!
    @ ./Base.jl:49 [inlined]
  [3] authenticate!(exc::ExchangeTypes.CcxtExchange{ExchangeTypes.ExchangeID{:binance}}, tries::Int64)
    @ Exchanges /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:473
  [4] authenticate!
    @ /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:463 [inlined]
  [5] exckeys!(exc::ExchangeTypes.CcxtExchange{ExchangeTypes.ExchangeID{:binance}}, key::String, secret::String, pass::String, wa::String, pk::String)
    @ Exchanges /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:496
  [6] exckeys!(exc::ExchangeTypes.CcxtExchange{ExchangeTypes.ExchangeID{:binance}}; sandbox::Bool)
    @ Exchanges /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:514
  [7] exckeys!
    @ /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:504 [inlined]
  [8] setexchange!(::ExchangeTypes.CcxtExchange{ExchangeTypes.ExchangeID{:binance}}; markets::Symbol, kwargs::@Kwargs{})
    @ Exchanges /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:228
  [9] macro expansion
    @ /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:187 [inlined]
 [10] macro expansion
    @ ./missing.jl:466 [inlined]
 [11] macro expansion
    @ /opt/opensource/jl/PingPong.jl/Lang/src/module.jl:73 [inlined]
 [12] getexchange!(x::Symbol, params::PythonCall.Wrap.PyDict{PythonCall.Core.Py, PythonCall.Core.Py}; sandbox::Bool, markets::Symbol, kwargs::@Kwargs{})
    @ Exchanges /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:178
 [13] getexchange!
    @ /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:174 [inlined]
 [14] bare_load(mod::Module, t::Type, config::Misc.Config{Float64})
    @ Strategies /opt/opensource/jl/PingPong.jl/Strategies/src/load.jl:154
 [15] strategy!(mod::Module, cfg::Misc.Config{Float64})
    @ Strategies /opt/opensource/jl/PingPong.jl/Strategies/src/load.jl:143
 [16] strategy!(src::Symbol, cfg::Misc.Config{Float64})
    @ Strategies /opt/opensource/jl/PingPong.jl/Strategies/src/load.jl:220
....

Pluto notebook

Write a pluto notebook to showcase features

bn.binancedownload(pairs) raises ERROR: ArgumentError: No symbols found matching SubString{String}["BTC/USDT", "ETH/USDT", "SOL/USDT"]

Hello,

Currently following documentation but

julia> using PingPong

julia> @environment!

julia> s = st.strategy(:Example)
Name: Example (Sim)
Config: No Margin, 10.0(USDT)(Base Size), 100.0(USDT)(Initial Cash)
Universe: 3 instances, 1 exchanges
Trades: 0 ~ 0(longs) ~ 0(shorts) ~ 0(liquidations)
Holdings: 0
Pending buys: 0
Pending sells: 0
USDT: 100.0 (on phemex) (Cash)
USDT: 100.0 (Total)

julia> pairs = im.raw.(s.universe.data.asset)
3-element Vector{SubString{String}}:
 "BTC/USDT"
 "ETH/USDT"
 "SOL/USDT"

julia> using Scrapers: Scrapers as scr

julia> const bn = scr.BinanceData
Scrapers.BinanceData

julia> bn.binancedownload(pairs)
ERROR: ArgumentError: No symbols found matching SubString{String}["BTC/USDT", "ETH/USDT", "SOL/USDT"]
Stacktrace:
 [1] binancedownload(syms::Vector{…}; zi::Data.ZarrInstance{…}, quote_currency::String, reset::Bool, kwargs::@Kwargs{})
   @ Scrapers.BinanceData /pingpong/Scrapers/src/binance.jl:239
 [2] binancedownload(syms::Vector{SubString{String}})
   @ Scrapers.BinanceData /pingpong/Scrapers/src/binance.jl:233
 [3] top-level scope
   @ REPL[7]:1
Some type information was truncated. Use `show(err)` to see complete types.

add support for zarr subgroups

currently all arrays are stored in a flat structure in the ZarrInstance main group, whereas they should be saved according to their path in a subgroup

Convert some Trality examples

Hello,

Trality, a Python crypto trading bot will stop. See https://www.trality.com/blog/trality-winding-down-product
I'm considering moving to Julia and an open source trading environment.
Converting some Trality examples to PingPong.jl could help to attract new users.
Here are some examples https://gist.github.com/c0indev3l/855381f07865dfca079e8252e736ae35
and documentation https://docs.trality.com/

Kind regards

transpiling ccxt to jl

ccxt/ccxt#15852

There is https://github.com/JuliaCN/Py2Jl which could be used to bootstrap the julia base classes from python base classes.

Looking at the regex at https://github.com/ccxt/ccxt/blob/master/build/transpile.js seems like most of the translations are doable, translating the isinstance and is None ones from python, and the common ones from PHP, because julia lacks dotcall syntax

POC with all modules merged

A test should be done to compare loading times between a repository version where everything is imported from one main modules and compare it to the current multi project version

Replace progressbar

Replace the current progressbar with Term.jl

Attempting to build a docker image under osx fails.

I tried docker pull image and found no version for arm64.

❯ docker pull docker.io/panifie/pingpong-precomp
Using default tag: latest
latest: Pulling from panifie/pingpong-precomp

no matching manifest for linux/arm64/v8 in the manifest list entries

I tried to clone the repositories and execute the build script, and got the following error

❯ ./scripts/build.sh
[+] Building 10.3s (27/27) FINISHED                                                                                   docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                  0.0s
 => => transferring dockerfile: 3.50kB                                                                                                0.0s
 => [internal] load metadata for docker.io/library/julia:1.10                                                                        10.1s
 => [internal] load .dockerignore                                                                                                     0.0s
 => => transferring context: 96B                                                                                                      0.0s
 => [base 1/3] FROM docker.io/library/julia:1.10@sha256:b6148b3d26f9727010112b3920ec093b126dc029b2d6290b8c7341189e0567c8              0.0s
 => [internal] load build context                                                                                                     0.0s
 => => transferring context: 47.16kB                                                                                                  0.0s
 => CACHED [base 2/3] RUN mkdir /pingpong     && apt-get update     && apt-get -y install sudo direnv git     && useradd -u 1000 -G   0.0s
 => CACHED [base 3/3] WORKDIR /pingpong                                                                                               0.0s
 => CACHED [python1 1/5] COPY --chown=ppuser:ppuser ./Lang/ /pingpong/Lang/                                                           0.0s
 => CACHED [python1 2/5] COPY --chown=ppuser:ppuser ./Python/*.toml /pingpong/Python/                                                 0.0s
 => CACHED [python1 3/5] RUN /usr/local/julia/bin/julia -C generic --project=/pingpong/Python -e "import Pkg; Pkg.instantiate()"      0.0s
 => CACHED [python1 4/5] COPY --chown=ppuser:ppuser ./Python /pingpong/Python                                                         0.0s
 => CACHED [python1 5/5] RUN /usr/local/julia/bin/julia -C generic --project=/pingpong/Python -e "using Python"                       0.0s
 => CACHED [precompile1 1/2] COPY --chown=ppuser:ppuser ./PingPong/*.toml /pingpong/PingPong/                                         0.0s
 => CACHED [precompile1 2/2] RUN /usr/local/julia/bin/julia -C generic --project=/pingpong/PingPong -e "import Pkg; Pkg.instantiate(  0.0s
 => CACHED [precompile2 1/1] RUN JULIA_PROJECT= /usr/local/julia/bin/julia -C generic -e "import Pkg; Pkg.add(["DataFrames", "CSV",   0.0s
 => CACHED [precompile3 1/2] COPY --chown=ppuser:ppuser ./ /pingpong/                                                                 0.0s
 => CACHED [precompile3 2/2] RUN git submodule update --init                                                                          0.0s
 => CACHED [precomp-base 1/1] WORKDIR /pingpong                                                                                       0.0s
 => CACHED [pingpong-precomp 1/3] RUN /usr/local/julia/bin/julia -C generic -e "import Pkg; Pkg.instantiate()"                        0.0s
 => CACHED [pingpong-precomp 2/3] RUN /usr/local/julia/bin/julia -C generic -e "using PingPong; using Stats"                          0.0s
 => CACHED [pingpong-precomp 3/3] RUN /usr/local/julia/bin/julia -C generic -e "using Stats"                                          0.0s
 => CACHED [pingpong-precomp-interactive 1/3] RUN JULIA_PROJECT= /usr/local/julia/bin/julia -C generic -e "import Pkg; Pkg.add(["Mak  0.0s
 => CACHED [pingpong-precomp-interactive 2/3] RUN /usr/local/julia/bin/julia -C generic -e "import Pkg; Pkg.instantiate()"            0.0s
 => CACHED [pingpong-precomp-interactive 3/3] RUN /usr/local/julia/bin/julia -C generic -e "using PingPongInteractive"                0.0s
 => CACHED [pingpong-sysimage-interactive 1/3] RUN apt-get install -y gcc g++                                                         0.0s
 => CACHED [pingpong-sysimage-interactive 2/3] RUN scripts/docker_compile.sh;     su ppuser -c "cd /pingpong;     . .envrc;     cat   0.0s
 => ERROR [pingpong-sysimage-interactive 3/3] RUN /usr/local/julia/bin/julia -C generic --sysimage "/pingpong/PingPong.so" -e "using  0.1s
------
 > [pingpong-sysimage-interactive 3/3] RUN /usr/local/julia/bin/julia -C generic --sysimage "/pingpong/PingPong.so" -e "using PingPongInteractive":
0.091 ERROR: could not load library "/pingpong/PingPong.so"
0.091 /pingpong/PingPong.so: cannot open shared object file: No such file or directory
------
Dockerfile:100
--------------------
  98 |     USER ppuser
  99 |     # Resets condapkg env
 100 | >>> RUN $JULIA_CMD --sysimage "/pingpong/PingPong.so" -e "using PingPongInteractive"
 101 |     CMD $JULIA_CMD --sysimage PingPong.so
 102 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c $JULIA_CMD --sysimage \"/pingpong/PingPong.so\" -e \"using PingPongInteractive\"" did not complete successfully: exit code: 1

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/qty1j4wummojfvfriog4u1yjf

backtest interactivity

The backtest could be made more interactive by either:

(terminal) use a progress bar (which we already have used for other things like scrapers)
(graphical) update a makie chart in realtime as the backtest progresses

[Help] questions about Quickstart examples.

While studying the example code in the library documentation, I have a few questions.

fetch_ohlcv and load_ohlcv come from Engine, but I didn't see these methods exported. Why not use the egn. prefix, like with st.?
The implementation of load_ohlcv(s) is puzzling. The return value of Data.load_ohlcv(...) isn't used, and it seems to serve no useful purpose here.

using PingPong

@environment!
s = st.strategy(:SimpleStrategy, exchange=:binance) 

fetch_ohlcv(s, from=-1000)   #<---why not egn.fetch_ohlcv()
load_ohlcv(s)

sm.start!(s)

function load_ohlcv(
    s::Strategy; tf=s.config.min_timeframe, pairs=(raw(a) for a in assets(s))
)
    ...
    Data.load_ohlcv(exc, pairs_str, tf_str)  #<---The return value isn't used
    fill!(s)
end

Assets types

Symbols are currently parsed through some regex. ~~We could instead query ccxt.market["info"]~~
Ccxt also provides the feeCurrency but is not necessarily useful for backtesting as much as for live trading

Docker container

Make dockerfile with prebuilt packages, consider static compilation

Use decimals for instead of floats?

Could use https://github.com/JuliaMath/DecFP.jl
but it states 50x slowdown...would need a PoC with some benchmarks to see how much a backtest would be affected

bn.binancedownload() failed.

The fix(#46) has introduced new issues. The previously working download function has stopped working.

julia> using Scrapers: Scrapers as scr, BinanceData as bn

julia> bn.binancedownload("eth", market=:data, freq=:monthly, kind=:klines)

┌ Warning: XMLError: Tag nav invalid from HTML parser (code: 801, line: 8)
└ @ EzXML ~/.julia/packages/EzXML/DL8na/src/error.jl:97
┌ Warning: XMLError: Tag svg invalid from HTML parser (code: 801, line: 17)
└ @ EzXML ~/.julia/packages/EzXML/DL8na/src/error.jl:97
┌ Warning: XMLError: Tag path invalid from HTML parser (code: 801, line: 18)
└ @ EzXML ~/.julia/packages/EzXML/DL8na/src/error.jl:97

┌ Warning: binance: download failed (or up to date)
│   sym = "ETHUSDT"
│   last_file = nothing

load_ohlcv(s) : ERROR: UndefVarError: `compactun` not defined

Follow the package documentation for validation.

...
s = st.strategy(:SimpleStrategy, exchange=:okx) 
fetch_ohlcv(s, from=-1000) 
load_ohlcv(s) # <---------------error
...

output

julia> s = st.strategy(:SimpleStrategy, exchange=:okx)
Importing SimpleStrategy [9faf1dd7-6398-46c6-90c9-24e0234666dd].
Name: SimpleStrategy (Sim)
Config: nomargin, 10.0(USDT)(Base Size), 100.0(USDT)(Initial Cash)
Universe: 1 instances, 1 exchanges
Trades: 0 ~ 0
Holdings: 0
Pending buys: 0
Pending sells: 0
USDT: 100.0 (on okx) (Cash)
USDT: 100.0 (Total)
julia> fetch_ohlcv(s, from=-1000)
┌ Warning: fetch: (okx) likely ignores `since` argument
│   since_date = 2021-09-23T00:00:00
│   dt(since_ts) = 2021-09-23T15:29:34.417
│   dt(from) = 2021-09-23T15:29:34.417
│   pair = "BTC/USDT"
└ @ Fetch /opt/opensource/jl/PingPong.jl/Fetch/src/impl.jl:221
Dict{String, Data.PairData} with 1 entry:
  "BTC/USDT" => PairData("BTC/USDT", "1d", 299×6 DataFrame…

julia> load_ohlcv(s)
Error showing value of type DataFrames.DataFrame:
ERROR: UndefVarError: `compactun` not defined
Stacktrace:
  [1] print(io::IOContext{IOBuffer}, ai::AssetInstance{Asset, ExchangeID{:okx}, NoMargin})
    @ Instances /opt/opensource/jl/PingPong.jl/Instances/src/module.jl:715
  [2] sprint(f::Function, args::AssetInstance{Asset, ExchangeID{:okx}, NoMargin}; context::IOContext{Base.TTY}, sizehint::Int64)
    @ Base ./strings/io.jl:112
  [3] _text_render_cell(::Val{…}, io::IOContext, v::AssetInstance{…}; compact_printing::Bool, isstring::Bool, limit_printing::Bool, linebreaks::Bool)
    @ PrettyTables ~/.julia/packages/PrettyTables/E8rPJ/src/backends/text/render.jl:47
  [4] _text_render_cell

bn.binanceload() load nothing

The download progress bar continued for a while and downloaded a lot of data, but it couldn't load. Additionally, where are the downloaded files stored? Are there any temporary files that need to be cleaned up?

julia> bn.binancedownload("eth", market=:data, freq=:monthly, kind=:klines)
(download progressbar showup for a while)

julia> bn.binanceload("eth", market=:data, freq=:monthly, kind=:klines)
Data.PairData("ETHUSDT_data_klines", "1m", 0×6 DataFrame
 Row │ timestamp  open     high     low      close    volume
     │ DateTime   Float64  Float64  Float64  Float64  Float64
─────┴────────────────────────────────────────────────────────, ZArray{Float64} of size 0 x 6)

If I change the parameter :data to :um,, the data loads successfully. It appears that the :data downloaded is saved in the :um directory.

julia> bn.binanceload("eth", market=:um, freq=:monthly, kind=:klines)
Data.PairData("ETHUSDT_um_klines", "1m", 3570959×6 DataFrame
     Row │ timestamp            open     high     low      close    volume
         │ DateTime             Float64  Float64  Float64  Float64  Float64
─────────┼─────────────────────────────────────────────────────────────────────
       1 │ 2017-08-17T04:01:00   301.13   301.13   301.13   301.13     2.75787
       2 │ 2017-08-17T04:02:00   300.0    300.0    300.0    300.0      0.0993
       3 │ 2017-08-17T04:03:00   300.0    300.0    300.0    300.0      0.31389
       4 │ 2017-08-17T04:04:00   301.13   301.13   301.13   301.13     0.23202
       5 │ 2017-08-17T04:05:00   300.0    301.13   300.0    301.13     0.75705
       6 │ 2017-08-17T04:06:00   300.1    300.1    300.1    300.1      0.90018
       7 │ 2017-08-17T04:07:00   300.1    300.1    300.1    300.1      0.0
       8 │ 2017-08-17T04:08:00   300.1    300.1    298.0    298.0      0.31493
       9 │ 2017-08-17T04:09:00   298.0    298.0    298.0    298.0      0.0
      10 │ 2017-08-17T04:10:00   298.0    298.0    298.0    298.0      0.0
      11 │ 2017-08-17T04:11:00   298.0    298.0    298.0    298.0      0.0
      12 │ 2017-08-17T04:12:00   298.0    298.0    298.0    298.0      0.0
      13 │ 2017-08-17T04:13:00   298.0    298.0    298.0    298.0      0.0
    ⋮    │          ⋮              ⋮        ⋮        ⋮        ⋮         ⋮

watchers as abstract types

make the Watcher an abstract subtype of the Rocket actor
refactor watcher implementations as concrete subtypes of the Watcher abstract type

GPUStrategy

The main goal of this would be to allow to backtesting runs on a GPU, and in particular allow to run backtests using online jupyter instances that offer GPU computing like colab or kaggle.
Since we mostly target online notebooks, and they are all nvidia, we would be using CUDA.jl

Making the backtest GPU compatible mostly means building structure adaptors for these types:

Strategy
Exchange
AssetInstance
DataFrame
Context

The Exchange type is a wrapper for a ccxt exchange, it is already discouraged to call python from a strategy, and the backtest itself never calls any python code, so this is not an issue per se, however it should be made abudantly clear in the documentation that a GPUStrategy cannot use python code.

This is however not enough:
To achieve parallelizability we have to run 1 backtest per GPU core, which means that the strategy has to be implemented as a kernel, in particular the main ping! function and whatever the function calls, needs to be a kernel. Of course we should provide example implementations and warn about the complexities of gpu programming.

UndefVarError: `scr` not defined

Hello,

According doc

https://github.com/panifie/PingPong.jl/blame/3ef7dd4cf48a9ff956ec53d3b4cf732fd8e18675/docs/src/index.md#L86

But

julia> const bn = scr.BinanceData
ERROR: UndefVarError: `scr` not defined

Any idea how to fix this?

strictier function signatures

Basically all functions with arguments any combination of Strategy AssetInstance Order Trade etc... have "loose" signatures. Meaning for example that you could call them with arguments where the strategy refers to an exchange, and an asset refers to another...
They would have to be rewritten for example like:

function position!(
    s::IsolatedStrategy{Sim}, ai::MarginInstance, t::PositionTrade{P};
) where {P<:PositionSide}

function position!(
    s::Strategy{X,N,E,Isolated,C}, ai::AssetInstance{A,E,Isolated}, t::Trade{O,A,E,P};
) where {X,N,E<:ExchangeID,C,A<:AbstractAsset,O,P<:PositionSide}

[Help] How do I choose which project to execute the script in?

In PingPong project, the following script(from the package doc) failed. Or should I Pkg.add these modules in the root of the source code? TimeTicks, Exchanges, and Fetch are all separate modules and don't seem to be visible to any single project simultaneously.

using TimeTicks
using Exchanges
using Fetch: Fetch as fe

exc = getexchange!(:kucoin)
timeframe = tf"1m"
pairs = ("BTC/USDT", "ETH/USDT")
# Will fetch the last 1000 candles, `to` can also be passed to download a specific range
fe.fetch_candles(exc, timeframe, pairs; from=-1000)

output

julia> using PingPong  # make sure PingPong module is loaded

julia> using TimeTicks
ERROR: ArgumentError: Package TimeTicks not found in current path.
- Run `import Pkg; Pkg.add("TimeTicks")` to install the TimeTicks package.
Stacktrace:
 [1] macro expansion
   @ ./loading.jl:1772 [inlined]
 [2] macro expansion
   @ ./lock.jl:267 [inlined]
 [3] __require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1753
 [4] #invoke_in_world#3
   @ ./essentials.jl:926 [inlined]
 [5] invoke_in_world
   @ ./essentials.jl:923 [inlined]
 [6] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1746

[Stats.multi] ERROR: ArgumentError: column name :cum_balance not found in the data frame

julia> using Stats

julia> sharpe(s)
0.3847495799061426

julia> sortino(s)
0.4138890526166903

julia> calmar(s)
-0.3158866997026408

julia> multi(s,:sharpe,:sortino,:calmar)
ERROR: ArgumentError: column name :cum_balance not found in the data frame
Stacktrace:
 [1] lookupname
   @ ~/.julia/packages/DataFrames/58MUJ/src/other/index.jl:413 [inlined]
 [2] getindex
   @ ~/.julia/packages/DataFrames/58MUJ/src/other/index.jl:422 [inlined]
 [3] getindex(df::DataFrames.DataFrame, ::typeof(!), col_ind::Symbol)
   @ DataFrames ~/.julia/packages/DataFrames/58MUJ/src/dataframe/dataframe.jl:557
 [4] getproperty(df::DataFrames.DataFrame, col_ind::Symbol)
   @ DataFrames ~/.julia/packages/DataFrames/58MUJ/src/abstractdataframe/abstractdataframe.jl:431
 [5] multi(::Strategies.SimStrategy{…}, ::Symbol, ::Vararg{…}; tf::TimeFrames.Day, normalize::Bool, norm_max::@NamedTuple{})
   @ Stats /opt/opensource/jl/PingPong.jl/Stats/src/metrics.jl:241
 [6] multi(::Strategies.SimStrategy{:SimpleStrategy, ExchangeID{:binance}, NoMargin, :USDT}, ::Symbol, ::Vararg{Symbol})
   @ Stats /opt/opensource/jl/PingPong.jl/Stats/src/metrics.jl:235
 [7] top-level scope
   @ REPL[45]:1
Some type information was truncated. Use `show(err)` to see complete types.

deprecate PairData

Remove the PairData type, and instead rely on DataFrames metadata for info.

panifie / pingpong.jl Goto Github PK

pingpong.jl's People

Contributors

Stargazers

Watchers

Forkers

pingpong.jl's Issues

v1

v2

required

optional

Isolated margin

Margin

Execution

backtest

watchers

other

Recommend Projects

Recommend Topics

Recommend Org

Jobs