bartosh / backtrader Goto Github PK

Prior starting live trading session, most strategies that use indicators will need some level of historical data backfill to fire up those indicators.
For exchanges that doesn't support OHLCV using 'backfill_start' is not an option.
backfill_start is also limited to the exchange ability to provide long historical data records.

therefore I propose implementing 'backfill_from', which will give the user the ability to load the historical data from a file instead of using a 'backfill_start'.

Notes:

• example for 'backfill_from' can be found on feed\ibdata.py
• if exchange doesn't support OHLCV data will be collected as ticks. question is how to stitch with history data in minute resolution?

This discussion has been open to gather ideas for future development

Hi,

i am running a process that one of its instances appears on netstat command with CLOSE_WAIT status and another instance in ESTABLISHED status
Reading About the status Here
it says

CLOSE_WAIT
Indicates that the server has received the first FIN signal from the client and the connection is in the process of being closed

So this essentially means that his is a state where socket is waiting for the application to execute close()

A socket can be in CLOSE_WAIT state indefinitely until the application closes it.
Faulty scenarios would be like filedescriptor leak, server not being execute close() on socket leading to pile up of close_wait sockets

So where should we pin point this - CCXT exchange level ? or could this be a feed issue?

Got the following exception raised when no data could be found in DTN for given symbol:

File "w:\backtrader\backtrader\stores\iqfeedstore.py", line 283, in get_history
ascend=True)
File "C:\Python36\lib\site-packages\pyiqfeed\conn.py", line 2161, in request_bars_in_period
raise NoDataError(err_msg)
pyiqfeed.exceptions.NoDataError: 'Request: HIT,SHOP,60,20080101 050000,20081231 50000,,000000,235959,1,H_0000000000,100,s,0\r\n, Error: !NO_DATA!'

I wonder if raising the NoDataError exception is a good solution here. Why not just return False from the IQFeed._load , which will indicate the end of bars?

In addition the following exception was raised during the error formatting:

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "scripts\store_iqfeed_to_influxdb.py", line 157, in
run_cerebro(symbol.rstrip(), args)
File "scripts\store_iqfeed_to_influxdb.py", line 137, in run_cerebro
cerebro.run(exactbars=args.exactbars)
File "w:\backtrader\backtrader\cerebro.py", line 1172, in run
runstrat = self.runstrategies(iterstrat)
File "w:\backtrader\backtrader\cerebro.py", line 1255, in runstrategies
data._start()
File "w:\backtrader\backtrader\feed.py", line 203, in _start
self.start()
File "w:\backtrader\backtrader\feeds\iqfeed.py", line 143, in start
self._st_start()
File "w:\backtrader\backtrader\feeds\iqfeed.py", line 160, in _st_start
self._timeframe, self._compression)
File "w:\backtrader\backtrader\stores\iqfeedstore.py", line 296, in get_history
error = "Unable to get historical data. Error:" % err
TypeError: not all arguments converted during string formatting

Hi, I'm trying to get IQFeed working with multiple US symbols, but got 2 major questions which I'd like to have your input.

1. As I know pyiqfeed requires a DTN password, login, and developer account ID which are inputs to pyiqfeed/service.py FeedService. I am wonder when itnergration is done with backtrader, where should the password.py be placed?

2. I've some initial success but realize there's seems to be a huge delay when the number of stocks exceeds n>15. Want to have you inputs in terms of how to get this working.

When I tried to run live using Bitmex exchange I got some errors:

data = bt.feeds.CCXT(exchange='bitmex', symbol='BTC/USD', timeframe=bt.TimeFrame.Ticks, compression=1)

Error (1):

***** DATA NOTIF: LIVE
Traceback (most recent call last):
  File "run_backtrader_ccxt.py", line 43, in <module>
    cerebro.run()
  File "/Users/dedeco/Envs/backenv2/lib/python3.6/site-packages/backtrader/cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
  File "/Users/dedeco/Envs/backenv2/lib/python3.6/site-packages/backtrader/cerebro.py", line 1298, in runstrategies
    self._runnext(runstrats)
  File "/Users/dedeco/Envs/backenv2/lib/python3.6/site-packages/backtrader/cerebro.py", line 1542, in _runnext
    drets.append(d.next(ticks=False))
  File "/Users/dedeco/Envs/backenv2/lib/python3.6/site-packages/backtrader/feed.py", line 404, in next
    ret = self.load()
  File "/Users/dedeco/Envs/backenv2/lib/python3.6/site-packages/backtrader/feed.py", line 476, in load
    _loadret = self._load()
  File "/Users/dedeco/Envs/backenv2/lib/python3.6/site-packages/backtrader/feeds/ccxt.py", line 89, in _load
    return self._load_ticks()
  File "/Users/dedeco/Envs/backenv2/lib/python3.6/site-packages/backtrader/feeds/ccxt.py", line 146, in _load_ticks
    trade_id = int(trade['id'])
ValueError: invalid literal for int() with base 10: 'e6476f26-f09a-6972-bf1e-bf123e00e160'

The id from bitmex is not int, so I fixed the problem the lines 145 - 154 from ccxt.py (feeds) changing to int and add some comparation by timestamp:

        for trade in trades:
            trade_id = trade['id']

            ts  = datetime.strptime(trade['info']['timestamp'], '%Y-%m-%dT%H:%M:%S.%fZ').timestamp()

            if ts > self._last_ts:
            #if trade_id > self._last_id:
                trade_time = datetime.strptime(trade['datetime'], '%Y-%m-%dT%H:%M:%S.%fZ')
                self._data.append((trade_time, float(trade['price']), float(trade['amount'])))
                self._last_id = trade_id

Hi,
I decided to get last vesion of Backtrader@ccxt and install it again with
pip install git+https://github.com/bartosh/backtrader.git@ccxt

First I am surprised to see :
Successfully installed backtrader-1.9.65.122
and not version 1.9.66.122
Edit : ok this is normal.

Then, I tried to run a previous script and get this error quickly:

  File "D:\Dev2\Anaconda3\lib\site-packages\backtrader\brokers\ccxtbroker.py", line 43, in __init__
    self.store = CCXTStore.get_store(exchange, config, retries)

  File "D:\Dev2\Anaconda3\lib\site-packages\backtrader\stores\ccxtstore.py", line 81, in get_store
    if not set(config.items()).issubset(set(store_conf.items())):

TypeError: unhashable type: 'dict'

My settings are :

    feedconf = {'apiKey': '***', 'secret': ''}

    data = bt.feeds.CCXT(exchange="bitmex", symbol="BTC/USD",fromdate=datetime.utcnow()+timedelta(minutes=-4),
                         timeframe=bt.TimeFrame.Minutes,compression=1, config=feedconf,backfill_start=True)

    broker_config = {'apiKey': '***',
                         'secret': '***',
                         'urls': {'api': 'https://testnet.bitmex.com'},
                         'enableRateLimit': True,
                         'rateLimit': 15000,
                         }
    broker = bt.brokers.CCXTBroker(exchange='bitmex',
                                       currency='Xbt',
                                       config=broker_config)
    cerebro.setbroker(broker)
    cerebro.adddata(data)

Hi,

Is it normal I only get 1 notification from "notify_order" function ?

Only "Created" status is sent few seconds later.

All other order status are missed.

Hi,
Using Bitmex feed and Bitmex broker on BTC/USD production. (same before and after 0ce07e5)

I have a strange position object.
First self.getposition().size does not exist :

  File "D:\Dev2\Anaconda3\lib\site-packages\backtrader\strategy.py", line 325, in _next
    super(Strategy, self)._next()

  File "D:\Dev2\Anaconda3\lib\site-packages\backtrader\lineiterator.py", line 266, in _next
    self.next()

  File "D:\Dev2\Dev\JimTrading\BTStrategies.py", line 150, in next
    if self.getposition().size>0:

AttributeError: 'float' object has no attribute 'size'

And if I run :

    def __init__(self):
        print(self.getposition())

I get a float :
0.01
which seems to be the account value and not my position on future perpetual contract XBTUSD.

Could it be CCXT error ?

Followed by the discussion in this post, I believe that there are a few design changes the backtrader-ccxt bridge need to take place.

At the current design, live requests are made over the network on the main thread, via the strategies and backtrader internals. That leave the software vulnerable to Hangs of the main thread.

As suggested in the post some/all request need to be managed on threads that keep updating values or do some other work in a way that doesn't delay the excursion of the main thread.

This is a vital refactor to make it possible to really use this project on production

Has anyone experienced slower performance when using CCXT feed with a broker vs using CCXT feed without a broker?

Can anyone explain the difference?

Every crypto exchange have a limit for how many request the user is allowed to do in a given time.
this can be controled in ccxt project under the 'rateLimit' param.

The problem is that we instantiate the exchange in at least 2 places:
Feed and Broker (and we can create multiple feeds as well)

the solution is to warp ccxt under a singleton class per exchange
(for the case we want to work with feeds/brokers from multiple brokers)

note:
when a broker instance is needed the feed's ccxt instance will probably need to get replaces so private and public method can be accessed

In China, we have to use shadowsocks as a proxy to access foreign country's websites. In quandl.py there is proxy. Can you add proxy in exchange.py for Chinese users?