tradytics / eiten Goto Github PK

Running this command from the examples:

python portfolio_manager.py --is_test 0 --future_bars 30 --data_granularity_minutes 60 --history_to_use all --apply_noise_filtering 1 --market_index QQQ --only_long 1 --eigen_portfolio_number 3 --stocks_file_path stocks/stocks.txt

returns:
Traceback (most recent call last):
File "portfolio_manager.py", line 37, in
main()
File "portfolio_manager.py", line 33, in main
eiten.run_strategies()
File "Work/eiten/eiten.py", line 107, in run_strategies
historical_price_info, future_prices, symbol_names, predicted_return_vectors, returns_matrix, returns_matrix_percentages = self.load_data()
File "Work/eiten/eiten.py", line 88, in load_data
self.data_dictionary = self.dataEngine.collect_data_for_all_tickers()
File "Work/eiten/data_loader.py", line 153, in collect_data_for_all_tickers
historical_price, future_price, symbol_names)
File "Work/eiten/data_loader.py", line 171, in remove_bad_data
most_common_length = length_dictionary[0]
IndexError: list index out of range

Any thoughts on what I'm doing wrong?

When I run this from the examples:

python portfolio_manager.py --is_test 1 --future_bars 90 --data_granularity_minutes 3600 --history_to_use all --apply_noise_filtering 1 --market_index QQQ --only_long 1 --eigen_portfolio_number 3 --stocks_file_path stocks/stocks.txt

I get:
Traceback (most recent call last):
File "portfolio_manager.py", line 37, in
main()
File "portfolio_manager.py", line 33, in main
eiten.run_strategies()
File "/Users/matt/Desktop/eiten-master/eiten.py", line 118, in run_strategies
returns_matrix)
File "/Users/matt/Desktop/eiten-master/strategy_manager.py", line 59, in random_matrix_theory_based_cov
filtered_covariance_matrix = self.strategyHelperFunctions.random_matrix_theory_based_cov(returns_matrix)
File "/Users/matt/Desktop/eiten-master/strategies/strategy_helper_functions.py", line 18, in random_matrix_theory_based_cov
variances = np.diag(np.cov(returns_matrix))
File "<array_function internals>", line 6, in diag
File "/Users/matt/opt/anaconda3/lib/python3.7/site-packages/numpy/lib/twodim_base.py", line 283, in diag
raise ValueError("Input must be 1- or 2-d.")
ValueError: Input must be 1- or 2-d.

I used this just yesterday to do an analysis with no problem. Also, I looked at the other issue in the list that said maybe there's some issue with a stock picked in stocks.txt, so picked two stocks (AAPL and BA) to check if that could be it and it still causes an error.

Does the portfolio backtesting backtest the current portfolios or portfolios constructed at the time of backtesting?

Thanks!

Thanks for making this! Looks really cool and I am excited to try it out.

I would like to use this to backtest a portfolio's asset allocation and compare it to an index.

Example:

Ticker,% of Allocation
AAPL,0.5
TSLA,0.25
MSFT,0.20
AMZN,0.05

Given this fixed asset allocation of the tickers above, how do I backtest this portfolio and compare it against an index?

The key thing to note here is that the allocation is fixed in terms of % of portfolio.

This is all on Google Colab.

I tried showing plot but I get this:
<Figure size 1200x600 with 1 Axes>

So I tried doing:

%matplotlib inline
!python portfolio_manager.py --is_test 1 --future_bars 90 --data_granularity_minutes 3600 --history_to_use all --apply_noise_filtering 1 --market_index QQQ --only_long 1 --eigen_portfolio_number 3 --stocks_file_path stocks/stocks.txt

But still no output.

Any ideas why?

Fresh install and run the README command ╰─ python portfolio_manager.py --is_test 1 --future_bars 90 --data_granularity_minutes 3600 --history_to_use all --apply_noise_filtering 1 --market_index QQQ --only_long 1 --eigen_portfolio_number 3 --stocks_file_path stocks/stocks.txt

Then got the following error, is this because yfinance is having issue?

Loading data for all stocks...
0%| | 0/6 [00:00<?, ?it/s]Exception in thread Thread-2:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/multitasking/init.py", line 102, in _run_via_pool
return callee(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/yfinance/multi.py", line 166, in _download_one_threaded
data = _download_one(ticker, start, end, auto_adjust, back_adjust,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/yfinance/multi.py", line 178, in _download_one
return Ticker(ticker).history(period=period, interval=interval,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/yfinance/base.py", line 155, in history
data = data.json()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/models.py", line 898, in json
return complexjson.loads(self.text, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/init.py", line 357, in loads
return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Hello,

I was wondering how to save this to a JSON file for algo-trading purposes

future_prices is initialized to None, https://github.com/tradytics/eiten/blob/master/data_loader.py#L51

If is_test is not 1, future_prices doesn't change, https://github.com/tradytics/eiten/blob/master/data_loader.py#L99

future_prices as None is called with .values, https://github.com/tradytics/eiten/blob/master/data_loader.py#L111

With the most recent code, I run the following command:

python3 portfolio_manager.py --is_test 1 --future_bars 90 --data_granularity_minutes 3600 --history_to_use all --apply_noise_filtering 1 --market_index QQQ --only_long 1 --eigen_portfolio_number 3 --stocks_file_path stocks/stocks.txt

Previously, this worked. Now, I get an error:

Traceback (most recent call last):
  File "portfolio_manager.py", line 37, in <module>
    main()
  File "portfolio_manager.py", line 33, in main
    eiten.run_strategies()
  File "/Users/jmartinezlago/eiten/eiten.py", line 107, in run_strategies
    historical_price_info, future_prices, symbol_names, predicted_return_vectors, returns_matrix, returns_matrix_percentages = self.load_data()
  File "/Users/jmartinezlago/eiten/eiten.py", line 95, in load_data
    self.data_dictionary[symbol]["historical_prices"])
KeyError: 'historical_prices'

The random_matrix_theory_based_cov seems to be vanishing with the covariance values. Empirical evidence shows this:

Experiment one:
How Eiten works in the current push, without matrix normalization:

This is a regular matrix without the noise cleaning method

After transformation the matrix becomes this:

We can see a shadow of red in the diagonal so I though I would plot out of scale. This is the result:

This leads me to question whether there is an implementation issue or if the method is really reliable.

Hi,
I'm building portfolios for my home country's stock market.
However there is limited options for shorting stocks.
Is it possible to run Eiten without allowing negative weight (shorting)?

When i use the integer(for example 100) as value for for --history_to_use, error happens.

File "C:\Users\hello\Repo\eiten\argchecker.py", line 8, in __init__
    self.check_arguments(args)
  File "C:\Users\hello\Repo\eiten\argchecker.py", line 20, in check_arguments
    assert not(args.history_to_use != "all" and int(args.history_to_use_int) <
AttributeError: 'Namespace' object has no attribute 'history_to_use_int'

args.history_to_use_int should be the reason, there isn't history_to_use_int argument parameter

Command to run:

python3 portfolio_manager.py --is_test 1 --future_bars 90 --data_granularity_minutes 3600 --history_to_use all --apply_noise_filtering 1 --market_index QQQ --only_long 1 --eigen_portfolio_number 3 --stocks_file_path stocks/stocks.txt

When loading data for all stocks:

Exception No objects to concatenate

Then, I see only info about 4 stocks, even though there's 15 in my stock file.

I went through the Genetic Algorithm Strategy code and it seems to me to be actually random. The purpose of selecting the top K is that we want to preserve their qualities and use them in crossover. However, that preservation does not occur since their are straight crossed. Further, there is no probability involved into mutation.

Point being: The qualities of top k are most likely lost because the code does not preserve them, and does not prioritize them into crossover (we want to reproduce its qualities).

Hi how does one load in this https://finance.yahoo.com/quote/ICNYB.PA?p=ICNYB.PA ?

Tried different combo's of ticker but throws error or ignored (if use ICNYB.PA) ...

Txs

First of all thank you! This tool was so easy to get up and running, despite the obvious underlying complexity.

After running portfolio_manager.py on a few different stock lists I am left with the following simple questions:

• How should I interpret "portfolio weights" to actually construct a portfolio? Does the tool assume a single weighted simultaneous purchase of all the symbols in the list and hold them indefinitely? Does it buy over time? It would be good if we could put in an amount for the initial principal and have the tool show the portfolio weights in terms of Day 0 dollar value.
• The tool has 4 main output charts, what is the difference between "Future Test Results" and "Simulated Future Returns"? The README only explains the former, I am having trouble interpreting the latter. They seem to be wildly different!

On the subject.

Additionally, would one be able to integrate realtime data, plus auto buy/sell?

@tradytics

Firstly, thank you for putting this together. I've been working on portfolio optimization through backtesting and this is definitely a simple and great setup. As I was testing your code, I wanted to call out a few things,

1. The only_long setting does not work. Irrespective of whether this is set to 0 or 1, the resulting weights are long_short lying between (-1, 1). This is the exact command i used as per your description:
   python portfolio_manager.py --is_test 1 --future_bars 90 --data_granularity_minutes 3600 --history_to_use all --apply_noise_filtering 1 --market_index SPY --only_long 0 --eigen_portfolio_number 3 --stocks_file_path stocks/stocks.txt
   I only changed only_long to 1 or 0 in my tests and it doesn't alter the weights. Let me know if I'm doing something wrong.

2. I then used the weights that were returned by your algorithm and plotted it on the efficient frontier created with over 150,000 portfolios (from random weights) on the same ohlc dataset.
   Below are the weights from your algorithm,

-------- Weights for Eigen Portfolio --------
Symbol: AAPL, Weight: 0.3492
Symbol: AMD, Weight: -0.7297
Symbol: AMZN, Weight: 0.3650
Symbol: FB, Weight: 0.4094
Symbol: MSFT, Weight: 0.3314
Symbol: NFLX, Weight: 0.7475
Symbol: NVDA, Weight: 1.0672
Symbol: TSLA, Weight: -1.5400

-------- Weights for Maximum Sharpe Portfolio (MSR) --------
Symbol: AAPL, Weight: 0.6987
Symbol: AMD, Weight: -0.1833
Symbol: AMZN, Weight: -0.9613
Symbol: FB, Weight: 0.7741
Symbol: MSFT, Weight: 0.7640
Symbol: NFLX, Weight: -0.2281
Symbol: NVDA, Weight: -0.1171
Symbol: TSLA, Weight: 0.2530

-------- Weights for Genetic Algo (GA) --------
Symbol: AAPL, Weight: -1.2315
Symbol: AMD, Weight: 0.9703
Symbol: AMZN, Weight: 1.3787
Symbol: FB, Weight: -1.1198
Symbol: MSFT, Weight: 1.9202
Symbol: NFLX, Weight: 0.2728
Symbol: NVDA, Weight: 0.9967
Symbol: TSLA, Weight: 0.0455

This is what the efficient frontier looks like for the above weights with the backtest data:

For the portfolios returned by your max sharpe (Eiten max sharpe) and Eigen (Eiten Eigen PF) the sharpe ratios are 0.07 and 0.15 respectively. As you can see, the genetic algorithm weights returned a sharpe ratio of 1.78 and the starred max_sharpe is 1.63 (close, but definitely more room for optimization). The starred max_sharpe is just the best performing random weight portfolio. We can nearly draw a straight line from the risk free return to the GA point cutting through the starred max_sharpe points. I guess the GA has only cranked up the risk for higher returns.

Similary the EF for the future data is as shown below and the max_sharpe in this case is 3.78 (from random weights):

Let me know if i'm doing anything wrong running this as the performance of max sharpe and eigen portfolio from your algorithm does not correspond to the best performing portfolio even in the back testing case. There is still room to reduce risk and increase returns here.

Hi,
I'm building portfolios for my home country's stock market.
However there is limited options for shorting stocks.
Is it possible to run Eiten without allowing negative weight (shorting)?

There is probably an error in the choice of stocks because the choice is made on stocks of which we know the good behavior
for the period 2015/01 2020/01 the proposed portfolio (AAPL, AMD, AMZN, FB, MSFT, NFLX, NVDA, TSLA) with equal weights and buy and hold has a performance of approximately 1000% !!.
Eiten GA returns 450%. Not so good.
If you randomly choose stocks from 2015, the result is not good.

There is a serious issue for the portfolio weights. When run for long only, there is not code do divide sum(weights). Total weights is almost round 2.5 to 4. So the result scaled 2.5 to 4. Because the project use the bull market data. So we got a pretty beautiful cumsum curve.