Comments (34)
Also in a future push, many different ExecutionClient
s can be registered with the common ExecutionEngine
so that multiple venues can be traded from a single TradingNode
.
Once domain objects are sent into the system we're limited by the GIL. However asyncio usage in the clients will definitely be necessary to avoid any network IO bounds for these new features.
from nautilus_trader.
Update
The asyncio implementation has largely been figured out, and work is currently in progress and making some good headway.
My approach is to first get the unified API integrated, data is working well with order book capability pending but at the top of the backlog. Order management is getting there.
Currently developing live on Binance and the BitMEX testnet.
Later on there could be scope for allowing more advanced per exchange functionality by detecting which exchange the ccxtpro client is for, and utilizing the params
for requests and info
for responses.
from nautilus_trader.
Even with optimized adapters there's still a good place for the CCXT Pro integration as it would be near impossible to write individual adapters for every possible exchange or even keep pace with the efforts there. That project is continually adding exchanges and capability, with that it leaves the door open for more arb strategies taking data from exchanges one isn't as latency sensitive to.
from nautilus_trader.
Hi @yohplala
Thank you for your interest! I'm aware of cryptofeed and have had several discussions with the author Bryant. It took some modifications to be able to integrate it into NautilusTrader as our architecture is designed so that the live system runs on a single event loop - cryptofeed was previously running on its own internal event loop.
Also at the time, not many private endpoints had been implemented - for a full NautilusTrader integration we need access to the private account and trade streams at the very least. I know cryptofeed has since implemented some of these but I haven't had a look in detail yet.
However I do like how they handle the feeds by abstracting over both http2 and websockets and plan to introduce something similar into the network
subpackage.
Expect some contributions from several sources soon including @ian-wazowski and @limx0 and we'll combine and refactor to something solid we can build integrations on top of.
from nautilus_trader.
The architecture for the execution stack has been finalized for now. ExecutionEngine
sits on top of ExecutionCache
which sits on top of ExecutionDatabase
with a seam there for any specific database adapter. A LiveExecutionEngine
can inherit ExecutionEngine
which will be one call site for the uvloop event loop to utilise coroutines.
Across on the data side refactorings are about to begin to end up with the same patterns, with another uvloop call site.
The exact interaction with the clients and uvloop remains to be determined, will update during first client implementation.
from nautilus_trader.
This is a great question.
The decision to integrate with CCXT Pro was to get a number of crypto exchanges running as quickly as possible, it's by no means an optimal solution though.
The vision is to eventually start including exchange specific adapters which have all of the exchange features implemented, and can be optimized as much as possible (including the use of Cython and efficient libraries).
from nautilus_trader.
By the way. There is old commit that cjseller implemented about cryptofeed integration. I hope you can get help there
Hi, any link toward that? I am not sure to understand, a PR in cryptofeed, or a PR in nautilus?
Sorry for the question, I am a bit confused :).
PR in nautilus_trader. I remember about 3months ago commits there was a working implementation(but deleted)
Give me a secs
from nautilus_trader.
Sorry I can not find the commits with my mobile phone :(
No worries, I will have a look shortly :)
Thanks for your help, really appreciated!
from nautilus_trader.
Sorry I can not find the commits with my mobile phone :(
No worries, I will have a look shortly :)
Thanks for your help, really appreciated!
This is working example.
and the cryptofeeder.py is
from cryptofeed import FeedHandler
from cryptofeed.callback import BookCallback, TradeCallback
from cryptofeed.defines import BID, ASK, L2_BOOK, TRADES
from cryptofeed.backends.zmq import BookZMQ, TradeZMQ
from cryptofeed.exchanges import Upbit, Binance
from multiprocessing import Process
from yapic import json
def receiver(port):
import zmq
addr = 'tcp://127.0.0.1:{}'.format(port)
ctx = zmq.Context.instance()
s = ctx.socket(zmq.SUB)
# empty subscription for all data, could be book for just book data, etc
s.setsockopt(zmq.SUBSCRIBE, b'')
s.bind(addr)
while True:
data = s.recv_string()
key, msg = data.split(" ", 1)
print(key)
print(json.loads(msg))
def main():
# p = Process(target=receiver, args=(5678,))
# p.start()
symbols = [symbol for symbol in Upbit.info()['symbols'] if symbol.split("-")[1] == "KRW"]
# print(Binance.info()['symbols'])
from pprint import pprint
pprint(symbols)
f = FeedHandler()
#f.add_feed(Upbit(symbols=symbols, channels=[TRADES], callbacks={TRADES: TradeCallback(trade)}))
#f.add_feed(Upbit(symbols=symbols, channels=[L2_BOOK], callbacks={L2_BOOK: BookCallback(book)}))
f.add_feed(Upbit(symbols=symbols, channels=[TRADES, L2_BOOK], callbacks={TRADES: TradeZMQ(port=5678), L2_BOOK: BookZMQ(port=5678)}))
f.run()
if __name__ == '__main__':
main()
from nautilus_trader.
Sounds good to me.
from nautilus_trader.
So in an initial refactoring DataClient
has been separated from a new common DataEngine
. This allows many different DataClient
implementations to be registered with the engine. BacktestDataEngine
overrides some methods for now to keep the current behaviour. TradingNode
still hard codes a LiveDataEngine
and LiveExecEngine
. The former will be an implementation for Nautilus Enterprise (C# distributed backend), and will likely be moved into adapters/enterprise
for example.
For live trading my current intention is to allow as many DataClients
as the user wishes to be specified in the config.json
. These will then be instantiated and registered with the DataEngine
as the system spools up.
from nautilus_trader.
Just a small FYI that streaming in CCXT is only supported in the paid pro version: https://ccxt.pro
That said, I agree that CCXT is the right library here. Well organized and provides an insane amount of flexibility. Esp in crypto, cross-exchange stat arb and mean-reversion strategies are major categories.
from nautilus_trader.
In order to use asyncio efficiently in python, the code has to run in entirely in async. Scheduling async coroutines from a non-async context doesn't work well in my experience. However, in this case async coroutines will be called from cython, which I don't know yet how will it work because I didn't use it in this way before.
from nautilus_trader.
We discussed about using CCXT in a separate thread and also mentioned CCXT PRO there. The PRO version is nice, but it needs more love in order to be production ready. For streaming I suggest having our own implementation based on asyncio.
from nautilus_trader.
Interesting. I've got no skin in the game on CCXT PRO. I've used CCXT in the past (and a handrolled Bitmex streamer), but in this case I'm actually trading equities (so just a spectator on the crypto stuff).
Amazingly, seems like there is some prior art for async in Cython: https://github.com/MagicStack/uvloop/blob/d6c67e7a7283f433e314e96a3bae2085e1d72e8f/uvloop/server.pyx#L51-L75
I'd imagine that TradingStrategy is the most likely place that a user would want to drop new coroutines onto the loop, so would just need to make sure the loop is accessible there. Fingers crossed that Cython doesn't complicate the ability to pass a coroutine to a create_task() call.
from nautilus_trader.
I think that as long as your top level event handling and orchestration is async you're good. Not much of a problem if you call down into the meat of the Cython code that's sync. Assuming you don't snag on any network or disk calls.
from nautilus_trader.
(sorry for sidetracking the CCXT issue. Guess streaming/async is really its own general animal)
from nautilus_trader.
I'm not sure how the design will be, it depends if the exec client can be called from a non-async context and be able to successfully fire async request using ccxt. In this case there will be no need to have async coroutines in the strategy. The flow that I see here is the following, e.g. order submission
- Start listening on the private websocket stream to order/position/balance topics (in exec client?)
- Submit an order from the strategy
- The order submission request is is passed to the exec client, which will fire a http request (using async if possible) and will return immediately
- The order update is received upon succesful http request in the websocket stream and an on_event is triggered in the strategy
- Optionally, the position is maintained independently in the trader as opposed to using the position updates from the exchange. This has to be discussed
from nautilus_trader.
As an inspiration I recommend having a look at https://github.com/CoinAlpha/hummingbot/. It looks like they managed to call async coroutines from cython. e.g. https://github.com/CoinAlpha/hummingbot/blob/03125f2435a8076af342a4648e8cdc41da080e43/hummingbot/connector/exchange/kucoin/kucoin_market.pyx
from nautilus_trader.
Cython:
Interesting project, uvloop. They claim to be 2-4x faster than the asyncio stock loop. This is good to know. Thanks.
from nautilus_trader.
I think we're settled that the CCXT integration is a good idea, whilst leaving open the option for the PRO streaming also. Regardless having our own stand alone efficient streaming solution is needed.
That general control flow you mention above sounds like the right sort of thing. Note however that currently the platform runs backtests single threaded with messages being handled in sequence. LiveExecutionEngine
sits on top of the common ExecutionEngine
implementation, its simple less than 100 lines, handler methods for commands and events place them on a queue - with a processing thread pulling them off and handling them one at a time.
I'll keep pushing the platform towards something similar for the DataEngine
with some kind of a live version utilizing a thread. Its then up to the clients to make their async requests and pass objects back into the handler methods as fast as possible... something like that.
A different architecture may emerge as I iterate towards the above. Remember the DataEngine
holds shared memory which strategies may want to call at any time (granted there is some GIL safety). So any updating of the data really has to be single threaded (blazingly fast with Cython anyway). Lets keep the async stuff as far to the periphery as possible probably just in the clients and their caller. Also note Redis is single threaded (but again, burning C level fast) which is where the ExecutionDatabase
holds the state for the engine (in the live case).
uvloop does look cool. It would also be in keeping with one of the points of difference for this platform being the mostly Cython production code, and consequently fast performance.
from nautilus_trader.
I won't have time to really read into it, but is there an off-chance that the hummingbot classes are generic/decoupled enough that they could be used themselves for broker communication (and tied into the Nautilus exec model once it's set)?
Just asking because integrations can be a major time suck, but also can be make-or-break for adoption. Shortcuts that don't compromise elegance would help with speed to market. Worst case they're replaced with something custom later as cycles allow. Of course if the interface doesn't match well then custom may be the only rout.
Guess this is more a question for @scoriiu since you're familiar with hummingbot interface design.
from nautilus_trader.
It's a good proposition, I had the same dilemma some time back when I was researching for robust webosocket streams and came across hummingbot. Back then the code was not as separated as it is nowadays, so I had to write my own implementation being inspired by it. I think now it would be fairly easy to take the code and rework it a bit. They still lack the connector for Bitmex, so that one we probably have to write ourselves. Will have to have a deeper look in their code since it changed quite a lot since last time I checked.
from nautilus_trader.
It's a good point @mLewisLogic, I've spent weeks before getting FIX integrations right. I'll have a deeper dive into hummingbot to see whats what.
Further to the above I also need to read the asyncio docs.
from nautilus_trader.
Update
The asyncio implementation has largely been figured out, and work is currently in progress and making some good headway.
My approach is to first get the unified API integrated, data is working well with order book capability pending but at the top of the backlog. Order management is getting there.
Currently developing live on Binance and the BitMEX testnet.
Later on there could be scope for allowing more advanced per exchange functionality by detecting which exchange the ccxtpro client is for, and utilizing the
params
for requests andinfo
for responses.
Just out of interest, How do you think of various network latency by different underlying http lib(i.e. curl, aiohttp, requests).
I do not think that ccxtpro did not design for latency-sensitive.
There are also significant latency differences between Python network libraries.
from nautilus_trader.
Even with optimized adapters there's still a good place for the CCXT Pro integration as it would be near impossible to write individual adapters for every possible exchange or even keep pace with the efforts there. That project is continually adding exchanges and capability, with that it leaves the door open for more arb strategies taking data from exchanges one isn't as latency sensitive to.
Agree, Your vision is great.
from nautilus_trader.
The current state of play here is that CCXT will continued to be supported, in particular for those users with a CCXT Pro license this is an extremely useful feature.
What needs to be tested now is that the adapter in its current state can be used with the 27 exchanges marked with the ccxt pro
badge in the CCXT README.md
.
There are currently specific adapters for Binance and BitMEX which require CCXT Pro.
Going forward a new network
module will be added with some generic WebSocket and HTTP client base classes. These can then be leveraged to write adapters which don't depend on CCXT, starting with those exchanges.
from nautilus_trader.
Going forward a new
network
module will be added with some generic WebSocket and HTTP client base classes. These can then be leveraged to write adapters which don't depend on CCXT, starting with those exchanges.
Hi there, just discovered Nautilus.
I read your comment. Only for information, did you already stumbled on cryptofeed?
I am a user of this (high level user, really not expert / not low level). I believe the work done to manage in a uniform way both websocket or rest APIs is terrific.
Maybe this can be something interesting for you?
Bests
from nautilus_trader.
Going forward a new
network
module will be added with some generic WebSocket and HTTP client base classes. These can then be leveraged to write adapters which don't depend on CCXT, starting with those exchanges.Hi there, just discovered Nautilus.
I read your comment. Only for information, did you already stumbled on cryptofeed?
I am a user of this (high level user, really not expert / not low level). I believe the work done to manage in a uniform way both websocket or rest APIs is terrific.
Maybe this can be something interesting for you?
Bests
Hi
I implemented integration with cryptofeed using zeromqCallback(seperated two processor(like Microservice architecture), maintaining low latency as possible.). and this code is under live testing on upbit exchange.
I hope the this part will be done and merged within this weekend.
from nautilus_trader.
I implemented integration with cryptofeed using zeromqCallback(seperated two processor(like Microservice architecture), maintaining low latency as possible.). and this code is under live testing on upbit exchange.
Hey, terrific! Is there any PR that can be reviewed? (just out of curiosity)
from nautilus_trader.
Yet there is no pr under reviewing.
PR will be pushed within 6 hours? Maybe...
By the way. There is old commit that cjseller implemented about cryptofeed integration. I hope you can get help there
from nautilus_trader.
By the way. There is old commit that cjseller implemented about cryptofeed integration. I hope you can get help there
Hi, any link toward that? I am not sure to understand, a PR in cryptofeed, or a PR in nautilus?
Sorry for the question, I am a bit confused :).
from nautilus_trader.
By the way. There is old commit that cjseller implemented about cryptofeed integration. I hope you can get help there
Hi, any link toward that? I am not sure to understand, a PR in cryptofeed, or a PR in nautilus?
Sorry for the question, I am a bit confused :).
Sorry I can not find the commits with my mobile phone :(
from nautilus_trader.
Lets direct future discussion around this to #231
from nautilus_trader.
Related Issues (20)
- Upgrade hyper dependency to 1.1.0 HOT 1
- Are there any tutorials for stock HOT 1
- KeyError in Binance Spot Instrument Provider HOT 1
- Catalog.query() leaks memory for QuoteTicks and TradeTicks HOT 3
- The example strategy in "Quickstart" document is not working HOT 3
- TWAP Execution broken with Streaming/Persistence HOT 1
- Logging V2 - optimizations and performance profiling
- Nautilus CFD Subscriber
- Binance Order Status EXPIRED_IN_MATCH HOT 3
- Issue serializing IB orders HOT 1
- TWAP Execution fails when number of shares is less than intervals HOT 2
- ParquetDataCatalog write_data takes a very long time HOT 1
- Logger Output Not Printing to Console After Commit 120bc0e HOT 3
- Is there any feature that I can have restful api to request the open orders of the account? HOT 1
- Error when self.subscribe_order_book_snapshots(instrument_id=self.instrument_id,depth=20) and def on_order_book(self, order_book: OrderBook) -> None: HOT 4
- No lattency between ts_init and ts_event for quote_tick HOT 5
- AttributeError: 'nautilus_trader.model.orders.limit.LimitOrder' object has no attribute 'order_side'. HOT 1
- StreamingFeatherWriter doesn't honour include_types filter HOT 1
- Flush logging when terminating process to avoid truncating logs
- Support for multiple trading nodes on IB client HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nautilus_trader.