panifie / pingpong.jl Goto Github PK
View Code? Open in Web Editor NEWCryptocurrency trading bot, and backtesting framework in julia
Home Page: https://panifie.github.io/PingPong.jl/
License: GNU General Public License v3.0
Cryptocurrency trading bot, and backtesting framework in julia
Home Page: https://panifie.github.io/PingPong.jl/
License: GNU General Public License v3.0
julia> e = getexchange!(:okx)
Exchange: OKX | 1406 markets | 14 timeframes
julia> exs.check(e, type=:basic)
ERROR: MethodError: no method matching check(::ExchangeTypes.CcxtExchange{ExchangeID{:okx}}; type::Symbol)
Closest candidates are:
check(::Exchange, ::Any) got unsupported keyword argument "type"
@ Exchanges /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:693
check(::Exchange) got unsupported keyword argument "type"
@ Exchanges /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:693
Stacktrace:
[1] top-level scope
@ REPL[13]:1
precompiled error as using PingPongInteractive , submit a pull request
❯ julia --project=./PingPongInteractive
julia> ]
(PingPongInteractive) pkg> instantiate
julia> using PingPongInteractive
Precompiling PingPongInteractive
ERROR: LoadError: UndefVarError: `za` not defined
Stacktrace:
[1] include(mod::Module, _path::String)
@ Base ./Base.jl:495
[2] include(x::String)
@ Optimization /opt/opensource/jl/PingPong.jl/Optimization/src/Optimization.jl:1
[3] top-level scope
@ /opt/opensource/jl/PingPong.jl/Optimization/src/Optimization.jl:9
fix
Optimization/src/module.jl:6
- using .Instances.Data: zinstance, za
+ using .Instances.Data: zinstance, Zarr as za
I have searched online discussions about this issue and have done the following work:
However, I still encounter an apiKey error. Any other aspects to consider?
┌ Error: exchange: auth error
│ resp = Python: AuthenticationError: binance {"code":-2015,"msg":"Invalid API-key, IP, or permissions for action."}
└ @ Exchanges /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:473
I tried debugging the code in VS Code, but I can't step into the code either by setting breakpoints or using the @enter
macro. What configuration is needed?
(The environment variables in .envrc
can be inherited by launching VS Code from the terminal.)
using PingPong
using Data: Data as data
using SimMode: SimMode as sm
@environment!
s = st.strategy(:SimpleStrategy, exchange=:binance)
# step into failed
@enter fetch_ohlcv(s, from=-1000)
# step into failed
@enter load_ohlcv(s)
# step into failed
@enter sm.start!(s)
Split up dependencies for:
Although not really recommended, embedding python strategies is a very easy task.
follow https://panifie.github.io/PingPong.jl/stats/#Performance-Metrics
Stats.sharpe(strategy_instance, tf"1d", rfr=0.01)
Stats.sortino(strategy_instance, tf"1d", rfr=0.01)
Stats.calmar(strategy_instance, tf"1d")
Stats.expectancy(strategy_instance, tf"1d")
output
julia> balloons(s) # works great
julia> Stats.sharpe(s, tf"1d", rfr=0.01) #<----error
ERROR: MethodError: no method matching sharpe(::Strategies.SimStrategy{:SimpleStrategy, ExchangeID{:binance}, NoMargin, :USDT}, ::TimeFrames.Day; rfr::Float64)
Closest candidates are:
sharpe(::Strategies.Strategy, ::Any, ::Any) got unsupported keyword argument "rfr"
@ Stats /opt/opensource/jl/PingPong.jl/Stats/src/metrics.jl:77
sharpe(::Strategies.Strategy, ::Any) got unsupported keyword argument "rfr"
@ Stats /opt/opensource/jl/PingPong.jl/Stats/src/metrics.jl:77
sharpe(::Strategies.Strategy) got unsupported keyword argument "rfr"
@ Stats /opt/opensource/jl/PingPong.jl/Stats/src/metrics.jl:77
julia> Stats.sortino(s, tf"1d", rfr=0.01) #<----error
ERROR: MethodError: no method matching sortino(::Strategies.SimStrategy{:SimpleStrategy, ExchangeID{:binance}, NoMargin, :USDT}, ::TimeFrames.Day; rfr::Float64)
Closest candidates are:
sortino(::Strategies.Strategy, ::Any, ::Any) got unsupported keyword argument "rfr"
@ Stats /opt/opensource/jl/PingPong.jl/Stats/src/metrics.jl:107
sortino(::Strategies.Strategy, ::Any) got unsupported keyword argument "rfr"
@ Stats /opt/opensource/jl/PingPong.jl/Stats/src/metrics.jl:107
sortino(::Strategies.Strategy) got unsupported keyword argument "rfr"
@ Stats /opt/opensource/jl/PingPong.jl/Stats/src/metrics.jl:107
julia> Stats.calmar(s, tf"1d") # works great
-0.3158866997026408
julia> Stats.expectancy(s, tf"1d") #<----error
ERROR: UndefVarError: `returns` not defined
Stacktrace:
[1] expectancy(s::Strategies.SimStrategy{:SimpleStrategy, ExchangeID{:binance}, NoMargin, :USDT}, tf::TimeFrames.Day)
@ Stats /opt/opensource/jl/PingPong.jl/Stats/src/metrics.jl:206
[2] top-level scope
@ /opt/opensource/jl/PingPong.jl/TimeTicks/src/module.jl:61
OctoBot (ccxt based): https://github.com/Drakkar-Software/OctoBot-Tentacles/tree/master/Trading/Exchange
Humming bot connectors (non ccxt based): https://github.com/hummingbot/hummingbot/tree/master/hummingbot/connector/exchange
I am more inclined on using octo-bot ones since ccxt based, and ccxt added websockets recently anyway
v1 is tagged
The test suite is mostly passing, although there are still 1 or 2 flapping tests.
The pre built images are on the docker registry
v1 will only receive bug fixes and new adhoc functions for better exchange support.
v2 is not currently being worked on, this is a list of things that need to happen for a v2 release and other things that would be nice to have:
ExchangeApi
module. Currently it is mostly split into the Exchanges and LiveMode modules. It is fine to model it over the ccxt api but this needs to happen to make ccxt swappable.Millisecond
as time unitInstalled manually according to the official documentation, but finally failed when installing.
❯ cd /PingPong.jl
(ok)
❯ mkdir .conda
(ok)
❯ direnv allow
direnv: loading /PingPong.jl/.envrc
direnv: export +JULIA_CONDAPKG_ENV +JULIA_CPU_TARGET +JULIA_FULL_PRECOMP +JULIA_LOAD_PATH +JULIA_NOPRECOMP +JULIA_NUM_THREADS +JULIA_PRECOMP +JULIA_PROJECT +PINGPONG_LIQUIDATION_BUFFER
(ok)
❯ julia --project ./PingPong
(ok)
(PingPong) pkg> instantiate
(failed)
Installed Blosc ───────────────── v0.7.3
Installed TimeFrames ──────────── v0.2.0
Installed MicroMamba ──────────── v0.1.14
Installed EzXML ───────────────── v1.2.0
Installed Blosc_jll ───────────── v1.21.5+0
Installed LRUCache ────────────── v1.6.1
Installed LMDB ────────────────── v1.0.0
Installed Zstd_jll ────────────── v1.5.6+0
Installed ArrowTypes ──────────── v2.3.0
Installed FunctionalCollections ─ v0.5.0
Installed Unitful ─────────────── v1.20.0
Installed Retry ───────────────── v0.4.1
Installed PythonCall ──────────── v0.9.20
Installed XMLDict ─────────────── v0.4.1
Installed LazyJSON ────────────── v0.2.2
Installed micromamba_jll ──────── v1.5.8+0
Installed IniFile ─────────────── v0.5.1
Installed TableMetadataTools ──── v0.1.0
Installed AWSS3 ───────────────── v0.10.4
Installed LMDB_jll ────────────── v0.9.27+0
Installed Lz4_jll ─────────────── v1.9.4+0
Installed SymDict ─────────────── v0.3.0
Installed Chain ───────────────── v0.6.0
Installed DataFramesMeta ──────── v0.15.2
Installed UnsafePointers ──────── v1.0.0
Installed Telegram ────────────── v1.1.5
Installed PropertyDicts ───────── v0.1.2
Installed Term ────────────────── v2.0.6
Installed CondaPkg ────────────── v0.2.22
Installed DiskArrays ──────────── v0.4.2
Installed ConcurrentCollections ─ v0.1.1
Installed AWS ─────────────────── v1.90.4
Downloaded artifact: Blosc
Downloaded artifact: Zstd
Downloaded artifact: Lz4
Precompiling project...
59 dependencies successfully precompiled in 52 seconds. 53 already precompiled.
1 dependency had output during precompilation:
┌ micromamba_jll
│ Downloading artifact: micromamba
│
│ [pid 14641] waiting for IO to finish:
│ Handle type uv_handle_t->data
│ timer 0x60000006c480->0x109c7e3e0
│ This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
│
│ [pid 14641] waiting for IO to finish:
│ Handle type uv_handle_t->data
│ timer 0x60000006c480->0x109c7e3e0
│ This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
└
2 dependencies errored.
For a report of the errors see `julia> err`. To retry use `pkg> precompile`
Using watchers, when fetching candles, we call fetchOHLCV
et similar, but in case candles are emulated we shouldn't call it, and instead process from trades directly since we already do it in other cases. So we have to ensure exc.has["fetchOHCLV"]
is not emulated
.
Currently we only consider the executor under the pingpong model. That is that the strategy has to deal with its own data feeds.
If the pingpong model also accounted for data updates, a pong!
call within the strategy could branch into a Watcher
instance in live mode and a prefetched dataframe in sim mode.
However since a strategy has always carte blanche, the benefits of managing data for the strategy are very small (we can't manage every piece of data, so managing only a few doesn't improve the status quo).
Moreover the main ping!
function works with a timestamp as primary input, so there is never any ambiguity on the time context of execution and whatever data the strategy uses, should already be time context aware because of this.
openat,closeat,etc...
that could be memoizedgetproperty
for asset instances and strategies or replace all dot accessors within SimMode with some getfield
wrapper function. Because the getproperty accessor is slower than getfield if it is customizedCcxtTrade
, CcxtTicker
), in cases where full parsing is not required, the python data should be parsed more lazilycopyto!
(starting from Data)Period
with FixedPeriod
to see if union splitting improves performanceThe behavior of freq=:daily is strange in three ways:
expect to do
# Download monthly data up until the previous month.
bn.binancedownload("eth", market=:data, freq=:monthly, kind=:klines)
# Download daily data to complete the data for this month.
bn.binancedownload("eth", market=:data, freq=:daily, kind=:klines)
# After the two steps, we should get the most complete historical data possible.
actually happen
freq==:monthly # Mission accomplished successfully, from 2017-08-17 to 2024-05-31
freq==:daily #overwrite above with data from 2017-08-17 to 2018-12-29
btw, the parameter freq=:monthly
is unnecessary for bn.binanceload
, because data from both parameter are saved to tf_1m
.
pull last commit and test SimpleStrategy failed.
julia> @environment!
julia> s = st.strategy(:SimpleStrategy, exchange=:binance)
Precompiling SimpleStrategy
Info Given SimpleStrategy was explicitly requested, output will be shown live
ERROR: LoadError: InitError: Evaluation into the closed module `PingPong` breaks incremental compilation because the side effects will not be permanent. This is likely due to some other module mutating `PingPong` with `eval` during precompilation - don't do this.
Stacktrace:
[1] eval
@ ./boot.jl:385 [inlined]
[2] __init__()
@ PingPong /opt/opensource/jl/PingPong.jl/PingPong/src/PingPong.jl:6
[3] run_module_init(mod::Module, i::Int64)
@ Base ./loading.jl:1134
[4] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
@ Base ./loading.jl:1122
[5] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
@ Base ./loading.jl:1067
[6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
Some strategies work based on past profitability, so they require a way to calc the profits of previous trades. Because ping pong does not make assumptions about execution of trades (buys and sells are not necessarily interleaved) a module for ensuring that the strategy executes only 1 trade at a time is required. And the method to track profitability should be implemented in such module
how to use the package, how to get information, like profit, sharp ratio, and other stuffs in backtrader in python
According to the library documentation, the validation procedure was running successfully yesterday, but today a new issue has arisen.
using PingPong
using Data: Data as data
using SimMode: SimMode as sm
@environment!
s = st.strategy(:SimpleStrategy, exchange=:okx) <--- error, but pased yestoday
output
ERROR: LoadError: Creating a new global in closed module `Main` (`resp`) breaks incremental compilation because the side effects will not be permanent.
Stacktrace:
[1] setproperty!
@ ./Base.jl:51 [inlined]
[2] setproperty!
@ ./Base.jl:49 [inlined]
[3] authenticate!(exc::ExchangeTypes.CcxtExchange{ExchangeTypes.ExchangeID{:binance}}, tries::Int64)
@ Exchanges /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:473
[4] authenticate!
@ /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:463 [inlined]
[5] exckeys!(exc::ExchangeTypes.CcxtExchange{ExchangeTypes.ExchangeID{:binance}}, key::String, secret::String, pass::String, wa::String, pk::String)
@ Exchanges /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:496
[6] exckeys!(exc::ExchangeTypes.CcxtExchange{ExchangeTypes.ExchangeID{:binance}}; sandbox::Bool)
@ Exchanges /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:514
[7] exckeys!
@ /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:504 [inlined]
[8] setexchange!(::ExchangeTypes.CcxtExchange{ExchangeTypes.ExchangeID{:binance}}; markets::Symbol, kwargs::@Kwargs{})
@ Exchanges /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:228
[9] macro expansion
@ /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:187 [inlined]
[10] macro expansion
@ ./missing.jl:466 [inlined]
[11] macro expansion
@ /opt/opensource/jl/PingPong.jl/Lang/src/module.jl:73 [inlined]
[12] getexchange!(x::Symbol, params::PythonCall.Wrap.PyDict{PythonCall.Core.Py, PythonCall.Core.Py}; sandbox::Bool, markets::Symbol, kwargs::@Kwargs{})
@ Exchanges /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:178
[13] getexchange!
@ /opt/opensource/jl/PingPong.jl/Exchanges/src/constructors.jl:174 [inlined]
[14] bare_load(mod::Module, t::Type, config::Misc.Config{Float64})
@ Strategies /opt/opensource/jl/PingPong.jl/Strategies/src/load.jl:154
[15] strategy!(mod::Module, cfg::Misc.Config{Float64})
@ Strategies /opt/opensource/jl/PingPong.jl/Strategies/src/load.jl:143
[16] strategy!(src::Symbol, cfg::Misc.Config{Float64})
@ Strategies /opt/opensource/jl/PingPong.jl/Strategies/src/load.jl:220
....
Write a pluto notebook to showcase features
Hello,
Currently following documentation but
julia> using PingPong
julia> @environment!
julia> s = st.strategy(:Example)
Name: Example (Sim)
Config: No Margin, 10.0(USDT)(Base Size), 100.0(USDT)(Initial Cash)
Universe: 3 instances, 1 exchanges
Trades: 0 ~ 0(longs) ~ 0(shorts) ~ 0(liquidations)
Holdings: 0
Pending buys: 0
Pending sells: 0
USDT: 100.0 (on phemex) (Cash)
USDT: 100.0 (Total)
julia> pairs = im.raw.(s.universe.data.asset)
3-element Vector{SubString{String}}:
"BTC/USDT"
"ETH/USDT"
"SOL/USDT"
julia> using Scrapers: Scrapers as scr
julia> const bn = scr.BinanceData
Scrapers.BinanceData
julia> bn.binancedownload(pairs)
ERROR: ArgumentError: No symbols found matching SubString{String}["BTC/USDT", "ETH/USDT", "SOL/USDT"]
Stacktrace:
[1] binancedownload(syms::Vector{…}; zi::Data.ZarrInstance{…}, quote_currency::String, reset::Bool, kwargs::@Kwargs{})
@ Scrapers.BinanceData /pingpong/Scrapers/src/binance.jl:239
[2] binancedownload(syms::Vector{SubString{String}})
@ Scrapers.BinanceData /pingpong/Scrapers/src/binance.jl:233
[3] top-level scope
@ REPL[7]:1
Some type information was truncated. Use `show(err)` to see complete types.
currently all arrays are stored in a flat structure in the ZarrInstance
main group, whereas they should be saved according to their path
in a subgroup
Hello,
Trality, a Python crypto trading bot will stop. See https://www.trality.com/blog/trality-winding-down-product
I'm considering moving to Julia and an open source trading environment.
Converting some Trality examples to PingPong.jl could help to attract new users.
Here are some examples https://gist.github.com/c0indev3l/855381f07865dfca079e8252e736ae35
and documentation https://docs.trality.com/
Kind regards
There is https://github.com/JuliaCN/Py2Jl which could be used to bootstrap the julia base classes from python base classes.
Looking at the regex at https://github.com/ccxt/ccxt/blob/master/build/transpile.js seems like most of the translations are doable, translating the isinstance
and is None
ones from python, and the common ones from PHP, because julia lacks dotcall syntax
A test should be done to compare loading times between a repository version where everything is imported from one main modules and compare it to the current multi project version
Replace the current progressbar with Term.jl
I tried docker pull image and found no version for arm64.
❯ docker pull docker.io/panifie/pingpong-precomp
Using default tag: latest
latest: Pulling from panifie/pingpong-precomp
no matching manifest for linux/arm64/v8 in the manifest list entries
I tried to clone the repositories and execute the build script, and got the following error
❯ ./scripts/build.sh
[+] Building 10.3s (27/27) FINISHED docker:desktop-linux
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 3.50kB 0.0s
=> [internal] load metadata for docker.io/library/julia:1.10 10.1s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 96B 0.0s
=> [base 1/3] FROM docker.io/library/julia:1.10@sha256:b6148b3d26f9727010112b3920ec093b126dc029b2d6290b8c7341189e0567c8 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 47.16kB 0.0s
=> CACHED [base 2/3] RUN mkdir /pingpong && apt-get update && apt-get -y install sudo direnv git && useradd -u 1000 -G 0.0s
=> CACHED [base 3/3] WORKDIR /pingpong 0.0s
=> CACHED [python1 1/5] COPY --chown=ppuser:ppuser ./Lang/ /pingpong/Lang/ 0.0s
=> CACHED [python1 2/5] COPY --chown=ppuser:ppuser ./Python/*.toml /pingpong/Python/ 0.0s
=> CACHED [python1 3/5] RUN /usr/local/julia/bin/julia -C generic --project=/pingpong/Python -e "import Pkg; Pkg.instantiate()" 0.0s
=> CACHED [python1 4/5] COPY --chown=ppuser:ppuser ./Python /pingpong/Python 0.0s
=> CACHED [python1 5/5] RUN /usr/local/julia/bin/julia -C generic --project=/pingpong/Python -e "using Python" 0.0s
=> CACHED [precompile1 1/2] COPY --chown=ppuser:ppuser ./PingPong/*.toml /pingpong/PingPong/ 0.0s
=> CACHED [precompile1 2/2] RUN /usr/local/julia/bin/julia -C generic --project=/pingpong/PingPong -e "import Pkg; Pkg.instantiate( 0.0s
=> CACHED [precompile2 1/1] RUN JULIA_PROJECT= /usr/local/julia/bin/julia -C generic -e "import Pkg; Pkg.add(["DataFrames", "CSV", 0.0s
=> CACHED [precompile3 1/2] COPY --chown=ppuser:ppuser ./ /pingpong/ 0.0s
=> CACHED [precompile3 2/2] RUN git submodule update --init 0.0s
=> CACHED [precomp-base 1/1] WORKDIR /pingpong 0.0s
=> CACHED [pingpong-precomp 1/3] RUN /usr/local/julia/bin/julia -C generic -e "import Pkg; Pkg.instantiate()" 0.0s
=> CACHED [pingpong-precomp 2/3] RUN /usr/local/julia/bin/julia -C generic -e "using PingPong; using Stats" 0.0s
=> CACHED [pingpong-precomp 3/3] RUN /usr/local/julia/bin/julia -C generic -e "using Stats" 0.0s
=> CACHED [pingpong-precomp-interactive 1/3] RUN JULIA_PROJECT= /usr/local/julia/bin/julia -C generic -e "import Pkg; Pkg.add(["Mak 0.0s
=> CACHED [pingpong-precomp-interactive 2/3] RUN /usr/local/julia/bin/julia -C generic -e "import Pkg; Pkg.instantiate()" 0.0s
=> CACHED [pingpong-precomp-interactive 3/3] RUN /usr/local/julia/bin/julia -C generic -e "using PingPongInteractive" 0.0s
=> CACHED [pingpong-sysimage-interactive 1/3] RUN apt-get install -y gcc g++ 0.0s
=> CACHED [pingpong-sysimage-interactive 2/3] RUN scripts/docker_compile.sh; su ppuser -c "cd /pingpong; . .envrc; cat 0.0s
=> ERROR [pingpong-sysimage-interactive 3/3] RUN /usr/local/julia/bin/julia -C generic --sysimage "/pingpong/PingPong.so" -e "using 0.1s
------
> [pingpong-sysimage-interactive 3/3] RUN /usr/local/julia/bin/julia -C generic --sysimage "/pingpong/PingPong.so" -e "using PingPongInteractive":
0.091 ERROR: could not load library "/pingpong/PingPong.so"
0.091 /pingpong/PingPong.so: cannot open shared object file: No such file or directory
------
Dockerfile:100
--------------------
98 | USER ppuser
99 | # Resets condapkg env
100 | >>> RUN $JULIA_CMD --sysimage "/pingpong/PingPong.so" -e "using PingPongInteractive"
101 | CMD $JULIA_CMD --sysimage PingPong.so
102 |
--------------------
ERROR: failed to solve: process "/bin/sh -c $JULIA_CMD --sysimage \"/pingpong/PingPong.so\" -e \"using PingPongInteractive\"" did not complete successfully: exit code: 1
View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/qty1j4wummojfvfriog4u1yjf
The backtest could be made more interactive by either:
While studying the example code in the library documentation, I have a few questions.
fetch_ohlcv
and load_ohlcv
come from Engine, but I didn't see these methods exported. Why not use the egn.
prefix, like with st.
?load_ohlcv(s)
is puzzling. The return value of Data.load_ohlcv(...)
isn't used, and it seems to serve no useful purpose here.using PingPong
@environment!
s = st.strategy(:SimpleStrategy, exchange=:binance)
fetch_ohlcv(s, from=-1000) #<---why not egn.fetch_ohlcv()
load_ohlcv(s)
sm.start!(s)
function load_ohlcv(
s::Strategy; tf=s.config.min_timeframe, pairs=(raw(a) for a in assets(s))
)
...
Data.load_ohlcv(exc, pairs_str, tf_str) #<---The return value isn't used
fill!(s)
end
Symbols are currently parsed through some regex. We could instead query ccxt.market["info"]
Ccxt also provides the feeCurrency
but is not necessarily useful for backtesting as much as for live trading
Make dockerfile with prebuilt packages, consider static compilation
Could use https://github.com/JuliaMath/DecFP.jl
but it states 50x slowdown...would need a PoC with some benchmarks to see how much a backtest would be affected
The fix(#46) has introduced new issues. The previously working download function has stopped working.
julia> using Scrapers: Scrapers as scr, BinanceData as bn
julia> bn.binancedownload("eth", market=:data, freq=:monthly, kind=:klines)
┌ Warning: XMLError: Tag nav invalid from HTML parser (code: 801, line: 8)
└ @ EzXML ~/.julia/packages/EzXML/DL8na/src/error.jl:97
┌ Warning: XMLError: Tag svg invalid from HTML parser (code: 801, line: 17)
└ @ EzXML ~/.julia/packages/EzXML/DL8na/src/error.jl:97
┌ Warning: XMLError: Tag path invalid from HTML parser (code: 801, line: 18)
└ @ EzXML ~/.julia/packages/EzXML/DL8na/src/error.jl:97
┌ Warning: binance: download failed (or up to date)
│ sym = "ETHUSDT"
│ last_file = nothing
Follow the package documentation for validation.
...
s = st.strategy(:SimpleStrategy, exchange=:okx)
fetch_ohlcv(s, from=-1000)
load_ohlcv(s) # <---------------error
...
output
julia> s = st.strategy(:SimpleStrategy, exchange=:okx)
Importing SimpleStrategy [9faf1dd7-6398-46c6-90c9-24e0234666dd].
Name: SimpleStrategy (Sim)
Config: nomargin, 10.0(USDT)(Base Size), 100.0(USDT)(Initial Cash)
Universe: 1 instances, 1 exchanges
Trades: 0 ~ 0
Holdings: 0
Pending buys: 0
Pending sells: 0
USDT: 100.0 (on okx) (Cash)
USDT: 100.0 (Total)
julia> fetch_ohlcv(s, from=-1000)
┌ Warning: fetch: (okx) likely ignores `since` argument
│ since_date = 2021-09-23T00:00:00
│ dt(since_ts) = 2021-09-23T15:29:34.417
│ dt(from) = 2021-09-23T15:29:34.417
│ pair = "BTC/USDT"
└ @ Fetch /opt/opensource/jl/PingPong.jl/Fetch/src/impl.jl:221
Dict{String, Data.PairData} with 1 entry:
"BTC/USDT" => PairData("BTC/USDT", "1d", 299×6 DataFrame…
julia> load_ohlcv(s)
Error showing value of type DataFrames.DataFrame:
ERROR: UndefVarError: `compactun` not defined
Stacktrace:
[1] print(io::IOContext{IOBuffer}, ai::AssetInstance{Asset, ExchangeID{:okx}, NoMargin})
@ Instances /opt/opensource/jl/PingPong.jl/Instances/src/module.jl:715
[2] sprint(f::Function, args::AssetInstance{Asset, ExchangeID{:okx}, NoMargin}; context::IOContext{Base.TTY}, sizehint::Int64)
@ Base ./strings/io.jl:112
[3] _text_render_cell(::Val{…}, io::IOContext, v::AssetInstance{…}; compact_printing::Bool, isstring::Bool, limit_printing::Bool, linebreaks::Bool)
@ PrettyTables ~/.julia/packages/PrettyTables/E8rPJ/src/backends/text/render.jl:47
[4] _text_render_cell
The download progress bar continued for a while and downloaded a lot of data, but it couldn't load. Additionally, where are the downloaded files stored? Are there any temporary files that need to be cleaned up?
julia> bn.binancedownload("eth", market=:data, freq=:monthly, kind=:klines)
(download progressbar showup for a while)
julia> bn.binanceload("eth", market=:data, freq=:monthly, kind=:klines)
Data.PairData("ETHUSDT_data_klines", "1m", 0×6 DataFrame
Row │ timestamp open high low close volume
│ DateTime Float64 Float64 Float64 Float64 Float64
─────┴────────────────────────────────────────────────────────, ZArray{Float64} of size 0 x 6)
If I change the parameter :data to :um,, the data loads successfully. It appears that the :data
downloaded is saved in the :um
directory.
julia> bn.binanceload("eth", market=:um, freq=:monthly, kind=:klines)
Data.PairData("ETHUSDT_um_klines", "1m", 3570959×6 DataFrame
Row │ timestamp open high low close volume
│ DateTime Float64 Float64 Float64 Float64 Float64
─────────┼─────────────────────────────────────────────────────────────────────
1 │ 2017-08-17T04:01:00 301.13 301.13 301.13 301.13 2.75787
2 │ 2017-08-17T04:02:00 300.0 300.0 300.0 300.0 0.0993
3 │ 2017-08-17T04:03:00 300.0 300.0 300.0 300.0 0.31389
4 │ 2017-08-17T04:04:00 301.13 301.13 301.13 301.13 0.23202
5 │ 2017-08-17T04:05:00 300.0 301.13 300.0 301.13 0.75705
6 │ 2017-08-17T04:06:00 300.1 300.1 300.1 300.1 0.90018
7 │ 2017-08-17T04:07:00 300.1 300.1 300.1 300.1 0.0
8 │ 2017-08-17T04:08:00 300.1 300.1 298.0 298.0 0.31493
9 │ 2017-08-17T04:09:00 298.0 298.0 298.0 298.0 0.0
10 │ 2017-08-17T04:10:00 298.0 298.0 298.0 298.0 0.0
11 │ 2017-08-17T04:11:00 298.0 298.0 298.0 298.0 0.0
12 │ 2017-08-17T04:12:00 298.0 298.0 298.0 298.0 0.0
13 │ 2017-08-17T04:13:00 298.0 298.0 298.0 298.0 0.0
⋮ │ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮
Watcher
an abstract subtype of the Rocket
actorWatcher
abstract typeThe main goal of this would be to allow to backtesting runs on a GPU, and in particular allow to run backtests using online jupyter instances that offer GPU computing like colab or kaggle.
Since we mostly target online notebooks, and they are all nvidia, we would be using CUDA.jl
Making the backtest GPU compatible mostly means building structure adaptors for these types:
The Exchange
type is a wrapper for a ccxt exchange, it is already discouraged to call python from a strategy, and the backtest itself never calls any python code, so this is not an issue per se, however it should be made abudantly clear in the documentation that a GPUStrategy cannot use python code.
This is however not enough:
To achieve parallelizability we have to run 1 backtest per GPU core, which means that the strategy has to be implemented as a kernel, in particular the main ping!
function and whatever the function calls, needs to be a kernel. Of course we should provide example implementations and warn about the complexities of gpu programming.
Hello,
According doc
But
julia> const bn = scr.BinanceData
ERROR: UndefVarError: `scr` not defined
Any idea how to fix this?
Basically all functions with arguments any combination of Strategy
AssetInstance
Order
Trade
etc... have "loose" signatures. Meaning for example that you could call them with arguments where the strategy refers to an exchange, and an asset refers to another...
They would have to be rewritten for example like:
function position!(
s::IsolatedStrategy{Sim}, ai::MarginInstance, t::PositionTrade{P};
) where {P<:PositionSide}
to
function position!(
s::Strategy{X,N,E,Isolated,C}, ai::AssetInstance{A,E,Isolated}, t::Trade{O,A,E,P};
) where {X,N,E<:ExchangeID,C,A<:AbstractAsset,O,P<:PositionSide}
In PingPong project, the following script(from the package doc) failed. Or should I Pkg.add these modules in the root of the source code? TimeTicks, Exchanges, and Fetch are all separate modules and don't seem to be visible to any single project simultaneously.
using TimeTicks
using Exchanges
using Fetch: Fetch as fe
exc = getexchange!(:kucoin)
timeframe = tf"1m"
pairs = ("BTC/USDT", "ETH/USDT")
# Will fetch the last 1000 candles, `to` can also be passed to download a specific range
fe.fetch_candles(exc, timeframe, pairs; from=-1000)
output
julia> using PingPong # make sure PingPong module is loaded
julia> using TimeTicks
ERROR: ArgumentError: Package TimeTicks not found in current path.
- Run `import Pkg; Pkg.add("TimeTicks")` to install the TimeTicks package.
Stacktrace:
[1] macro expansion
@ ./loading.jl:1772 [inlined]
[2] macro expansion
@ ./lock.jl:267 [inlined]
[3] __require(into::Module, mod::Symbol)
@ Base ./loading.jl:1753
[4] #invoke_in_world#3
@ ./essentials.jl:926 [inlined]
[5] invoke_in_world
@ ./essentials.jl:923 [inlined]
[6] require(into::Module, mod::Symbol)
@ Base ./loading.jl:1746
julia> using Stats
julia> sharpe(s)
0.3847495799061426
julia> sortino(s)
0.4138890526166903
julia> calmar(s)
-0.3158866997026408
julia> multi(s,:sharpe,:sortino,:calmar)
ERROR: ArgumentError: column name :cum_balance not found in the data frame
Stacktrace:
[1] lookupname
@ ~/.julia/packages/DataFrames/58MUJ/src/other/index.jl:413 [inlined]
[2] getindex
@ ~/.julia/packages/DataFrames/58MUJ/src/other/index.jl:422 [inlined]
[3] getindex(df::DataFrames.DataFrame, ::typeof(!), col_ind::Symbol)
@ DataFrames ~/.julia/packages/DataFrames/58MUJ/src/dataframe/dataframe.jl:557
[4] getproperty(df::DataFrames.DataFrame, col_ind::Symbol)
@ DataFrames ~/.julia/packages/DataFrames/58MUJ/src/abstractdataframe/abstractdataframe.jl:431
[5] multi(::Strategies.SimStrategy{…}, ::Symbol, ::Vararg{…}; tf::TimeFrames.Day, normalize::Bool, norm_max::@NamedTuple{})
@ Stats /opt/opensource/jl/PingPong.jl/Stats/src/metrics.jl:241
[6] multi(::Strategies.SimStrategy{:SimpleStrategy, ExchangeID{:binance}, NoMargin, :USDT}, ::Symbol, ::Vararg{Symbol})
@ Stats /opt/opensource/jl/PingPong.jl/Stats/src/metrics.jl:235
[7] top-level scope
@ REPL[45]:1
Some type information was truncated. Use `show(err)` to see complete types.
Remove the PairData
type, and instead rely on DataFrames
metadata for info.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.