firmai / deltapy

DeltaPy - Tabular Data Augmentation (by @firmai)

Hi there 👋

I'm Derek, a professor at NYU teaching Machine Learning in Financial Engineering

• My interests are in synthetic data generation, agent-based simulators, and asset management using machine learning.
• I have worked on projects at HSBC, G-Research, Alan Turing Insitute, Oxford-MAN Institute and other large quantitative funds.

Jan 09 - Quant Letter: January 2024, Week-2 Jan 03 - Quant Letter: January 2024, Week-1 Dec 20 - Quant Letter: December 2023, Week-3

👥 Researcher at the @alan-turing-institute 📓 Associate member @oxford-man-institute 🧭 Founder of the @firmai open-source project 👥 Curator for ML-Quant resources 🦌 GitAwards top 100 for Jupyter Notebooks

Since joining Github, I pushed 7931 commits, opened 198 issues, received 20665 stars across 67 personal projects and contributed to 32 public repositories.

Packages

Everything listed here is available under Unlicense

My research has been used by large institutional banks and quantitative hedge funds (see SSRN).

• DeltaPy (12,409 Downloads) — First tabular data augmentation package in Python (market data) [code][report]
• PandaPy (21,163 Downloads) — Pandas alternative that mimics ‘Structs’ in the C Language (market data) [code][report]
• AtsPy (53,147 Downloads) — First automated time series package in Python (alternative data) [code][report]
• DataGene (4,246 Downloads) — First package assessing dataset similarity (market data) [code][report]
• MLAM (35,409 Downloads) — First repository for machine learning in asset management (market data) [code][report]

Other packages under this license include MTSS-GAN [code][report], the first multivariate conditional time series generator, FairPut [code][report], a FAIR package using LightGBM, and PandasVault [code], an advanced Pandas repository.

.. .-- .-. .. - . .- - - .... . .--. .- .-. .-.. --- ..- .-.

deltapy's Issues

ValueError: x contains a constant. Adding a constant with trend='c' is not allowed.

Hi,

I am getting the following error with this function:

df_out = interact.autoregression(df.copy()); df_out.head()

D:\Anaconda3\envs\tipjar\lib\site-packages\statsmodels\tsa\ar_model.py:691: FutureWarning: 
statsmodels.tsa.AR has been deprecated in favor of statsmodels.tsa.AutoReg and
statsmodels.tsa.SARIMAX.
AutoReg adds the ability to specify exogenous variables, include time trends,
and add seasonal dummies. The AutoReg API differs from AR since the model is
treated as immutable, and so the entire specification including the lag
length must be specified when creating the model. This change is too
substantial to incorporate into the existing AR api. The function
ar_select_order performs lag length selection for AutoReg models.
AutoReg only estimates parameters using conditional MLE (OLS). Use SARIMAX to
estimate ARX and related models using full MLE via the Kalman Filter.
To silence this warning and continue using AR until it is removed, use:
import warnings
warnings.filterwarnings('ignore', 'statsmodels.tsa.ar_model.AR', FutureWarning)
  warnings.warn(AR_DEPRECATION_WARN, FutureWarning)
Traceback (most recent call last):
  File "D:\Anaconda3\envs\tipjar\lib\site-packages\IPython\core\interactiveshell.py", line 3417, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-82-60579f16e2a8>", line 1, in <module>
    df_out = dp.interact.autoregression(x.copy())
  File "D:\Anaconda3\envs\tipjar\lib\site-packages\deltapy\interact.py", line 61, in autoregression
    fitted_model = AR(df.values[i, :]).fit(autoreg_lag)
  File "D:\Anaconda3\envs\tipjar\lib\site-packages\statsmodels\tsa\ar_model.py", line 1223, in fit
    X = self._stackX(k_ar, trend)  # sets self.k_trend
  File "D:\Anaconda3\envs\tipjar\lib\site-packages\statsmodels\tsa\ar_model.py", line 1044, in _stackX
    X = add_trend(X, prepend=True, trend=trend, has_constant="raise")
  File "D:\Anaconda3\envs\tipjar\lib\site-packages\statsmodels\tsa\tsatools.py", line 121, in add_trend
    raise ValueError(msg)
ValueError: x contains a constant. Adding a constant with trend='c' is not allowed

Some functions have look ahead bias

Hi firmai,

Thanks for creating deltapy! I just wanted to point out that not all functions are look ahead safe: their values change if new data is appended to the dataframe, making them hard to use in real life. For example:

a = transform.instantaneous_phases(df, ["close"]).iloc[
    -2
]
b = transform.instantaneous_phases(df.iloc[:-1], ["close"]).iloc[-1]
pd.testing.assert_series_equal(a,b) # -> this fails

I havent tested all, but this applies atleast to the transform.bkb and transform.instantaneous_phases functions.

transform.modify error

Hi,

This one does not find the function magnify. I think it may be something to do with the tsaug package:

df_out = transform.modify(df.copy(),["Close"]); df_out.head()