cmdty / curves Goto Github PK

Update build.cake to create Python Conda package during CI, and publish to conda-forge on release. Also updated Python testing task in build.cake to perform tests in a Conda environment.

Keep encountering multiple errors when trying to install the package.

" note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure."

I have the correct packages installed (versions greater than these).
pythonnet==2.5.2
setuptools==40.8.0
wheel==0.33.1
pandas>=1.0.3

Tried manually installing to the python Lib directory but I'm still not able to import the curves module (in either the python sheel, or Conda).

@cmdty
Hello,
Thanks a lot for your work.
In your tutorial, you mention Spline Interpolation used for input contracts that have gaps. But in you example (see below) there is no gap.
However, if I change the first contract with date(2019, 5, 15), 34.875) instead of date(2019, 5, 31), 34.875), I do create a gap. And the result of that seems completly wrong (see the remaining days of may: price going up before decreasing).
Could you have a look at that?

from curves import max_smooth_interp
from curves import contract_period as cp

contracts = [
(date(2019, 5, 31), 34.875),
(date(2019, 6, 1), date(2019, 6, 2), 32.87),
((date(2019, 6, 3), date(2019, 6, 9)), 32.14),
(pd.Period(year=2019, month=6, freq='M'), 31.08),
(cp.month(2019, 7), 29.95),
(cp.q_3(2019), 30.18),
(cp.q_4(2019), 37.64),
(cp.winter(2019), 38.05),
(cp.summer(2020), 32.39),
(cp.winter(2020), 37.84),
(cp.gas_year(2020), 35.12)
]

pc_for_spline, bc_for_spline = bootstrap_contracts(contracts, freq='D')
smooth_curve = max_smooth_interp(bc_for_spline, freq='D')

%matplotlib inline
pc_for_spline.plot(legend=True)
ax = smooth_curve.plot(legend=True)
ax.legend(["Piecewise Daily Curve", "Smooth Daily Curve"])

Linear system (18) of the maximum smoothness spline document involves a block matrix whose sub-matrices are sparse. This structure potentially lends itself to a much more efficient (in terms of both memory and CPU usage) solution strategy than the naive approach in the current implementation. A more efficient approach would be something like:

1. Formulate the inversion of the block matrix in (18) as a 2 x 2 block matrix with sub-matrices a function of the sub-matrices and inverses of these sub-matrices of the original matrix. The fact that the bottom right sub-matrix is a zero matrix should hopefully simplify things.
2. Use the block-diagonal sparseness of the sub-matrices to implement efficient inversions, multiplications, subtractions and additions to evaluate the matrix inverse formulated in the previous step.

Hello,

I have tried to build a quarterly peak curve using the input of one Cal value and two quarters. This unfortunately brings out NaN values in the output. See simplified code below.

Basic imports and European peak weighting (Monday-Friday) definitions:

from curves import bootstrap_contracts, max_smooth_interp, adjustments, weighting, contract_period

def pk_weight(period):
    if period.dayofweek < 5:
        return 1.0
    else:
        return 0.0

Defining peak contract for Q1-2022, Q3-2022 and Cal-2022:

pk_cont = [(datetime.date(2022, 1, 1), datetime.date(2022, 3, 31), 75.71),
 (datetime.date(2022, 7, 1), datetime.date(2022, 9, 30), 47.56),
 (datetime.date(2022, 1, 1), datetime.date(2022, 12, 31), 60.65)]

Running bootstrapper to extrapolate Q2 and Q4 values:

q_peak_piecewise_curve, q_peak_bc_daily = bootstrap_contracts(pk_cont, freq='Q', average_weight=pk_weight)

The output is missing a Q4 value, which prevents me from running max_smooth_interp with a suitable mult_season_adjust to sort the shaping out on the final quarterly curve:

2022Q1 75.71
2022Q2 58.68
2022Q3 47.56
2022Q4 NaN
Freq: Q-DEC, dtype: float64

My expectation is that it should generate two identical values for Q2 and Q4 (adjusted for peak hours in that contract).

My initial suspicion was that the issue lies within the fact that 2022-12-31 is a Saturday and as such - despite the weighting function - the bootstrapper thinks it's an incomplete Q4 contract and therefore excludes it.

To test, I tried using the following contract dates instead to no avail:

pk_cont = [(datetime.date(2022, 1, 3), datetime.date(2022, 3, 31), 75.71),
 (datetime.date(2022, 7, 1), datetime.date(2022, 9, 30), 47.56),
 (datetime.date(2022, 1, 3), datetime.date(2022, 12, 30), 60.65)]

Do you have any ideas how to work around this?

Regards

Thank you for this nice library. I started to play with it and I came up with this exception that seems incorrect:

DoubleCurve<Day> curve = new MaxSmoothnessSplineCurveBuilder<Day>()
                .AddContract(new Month(2021, 1), 43.76)
                .AddContract(new Month(2021, 2), 44.28)
                .AddContract(new Quarter(2021, 1), 42.85)
                .BuildCurve();

I am starting with a simple exercise of using the mult_season_adjust logic in max_smooth_interp to shape quarters within a year.

Works

def wd_weight(period):
    start = period.asfreq('B', 's')
    end = period.asfreq('B', 'e')
    return len(pd.period_range(start=start, end=end, freq='B'))

pk_cont = [(datetime.date(2022, 1, 3), datetime.date(2022, 3, 31), 71.81),
           (datetime.date(2022, 1, 3), datetime.date(2022, 12, 30), 58.0)]

q_peak_piecewise_curve, q_peak_bc_daily = bootstrap_contracts(pk_cont, freq='Q', average_weight=wd_weight)

pk_ratio_config = {1: 1.07015, 2: 0.79521, 3: 0.9248, 4: 1.20984}
def pk_mult_adjust(period):
    return pk_ratio_config[period.quarter]
q_peak_smooth_curve = max_smooth_interp(q_peak_bc_daily, average_weight=wd_weight, freq='Q', mult_season_adjust=pk_mult_adjust)

q_peak_piecewise_curve keeps Q1 and has a flat price structure in Q2, Q3, Q4 that meets zero-arbitrage conditions with the provided Cal value.
q_peak_smooth_curve has later adjusted the ratios based on the config to reflect the summer/winter structure.

Fails

However, if I use the below set of inputs (Q1, Q3 and Cal), mult_season_adjust does no longer do the trick:

pk_cont = [(datetime.date(2022, 1, 3), datetime.date(2022, 3, 31), 71.81),
           (datetime.date(2022, 7, 1), datetime.date(2022, 9, 30), 45.73),
           (datetime.date(2022, 1, 3), datetime.date(2022, 12, 30), 58.0)]

q_peak_piecewise_curve keeps Q1 and Q3 values and has a flat price structure in Q2 and Q4 that meets zero-arbitrage conditions with the provided Cal value.
q_peak_smooth_curve no longer manages to cope with any ratio adjustments, meaning that smooth_curve = piecewise_curve. By specifying some contracts we by definition break the pk_ratio_config ratios, however, we should be able to preserve some and leave the 'last' (to be specified somehow) contract to 'break' those ratios.

The next steps for me are to then go from quarterly to monthly, to weekly, to daily etc. Therefore, critical to get it right from the beginning.

Please let me know your thoughts.

First of all, thank you very much for creating this library. It's a very nifty tool! Great job!
I'm seeing a weird behaviour when bootstrapping contracts that include days from multiple months:

`test_contracts = [

(datetime.date(2020, 8, 24), datetime.date(2020, 8, 30), 10.0), 
(datetime.date(2020, 8, 31), datetime.date(2020, 9, 6), 10.0),
(datetime.date(2020, 9, 1), datetime.date(2020, 9, 30), 10.0)

]`

piecewise_curve, bootstrapped_contracts = curves.bootstrap_contracts(test_contracts, freq='D')

The generated bootstrapped contracts are the following, notice the very low value for 2020-08-31.

[Contract(start=Period('2020-08-24', 'D'), end=Period('2020-08-30', 'D'), price=10.000000000000002), Contract(start=Period('2020-08-31', 'D'), end=Period('2020-08-31', 'D'), price=1.724137931034483), Contract(start=Period('2020-09-01', 'D'), end=Period('2020-09-06', 'D'), price=11.379310344827585), Contract(start=Period('2020-09-07', 'D'), end=Period('2020-09-30', 'D'), price=9.655172413793101)]

The behaviour I would expect is for the contracts to all be equal to 10.0. Do you agree? Any ideas why the above is generated instead?

Please disregard if I misunderstood your documentation, as I believed 3.9 was supported as per below comment:

The curves package is compatible with the Python interpreter up to version 3.9.

pythonnet-2.5.2 is not Python 3.9 compatible, only the pre-release is for now.

Error:

note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pythonnet
  Running setup.py clean for pythonnet
Failed to build pythonnet
Installing collected packages: pythonnet, curves
  Running setup.py install for pythonnet: started
  Running setup.py install for pythonnet: finished with status 'error'
  error: subprocess-exited-with-error

  Running setup.py install for pythonnet did not run successfully.
  exit code: 1

  [6 lines of output]
  usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: setup.py --help [cmd1 cmd2 ...]
     or: setup.py --help-commands
     or: setup.py cmd --help

  error: option --single-version-externally-managed not recognized
  [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

Encountered error while trying to install package.

pythonnet

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

The python dependencies for build/ci/test is a bit of a mess, being found in a requirements.txt, build.cake and azure-pipelines.yml. This should be consolidated and tidied up.

Hi,

First of all, thanks for your great piece of work!

I'm trying to understand two things about the spline methodology, while reading Benth (2008).
It seems that the library here does not utilize equation 9 (1st deriative of t_n =0). Can you some shed some light on this? I'd assume that it would be part of the constraint matrix.

Moreover, Benth describes the knots to support overlapping periods, which is quite key in power markets with monthly/quarterly/yearly contracts overlapping. I'm trying to understand the amendments you've made (I can see that the dimensions of the B vector along with the A constraintmatrix differ), and I'd like to contribute to adding this element.
In order to obtain the dimensions specified in Benth (A = 3n + (m-2) x 5n), wouldn't it require 2 knots when we go from one contract to another (whereas it is one point in this implementation if I understood correctly), i.e. T_1^e != T_2^s

I am running into a problem when including a contract which has a period that overlaps a month change. In this example this would be Wk18-2020.

 # Day contracts
 (datetime.date(2020, 4, 3), datetime.date(2020, 4, 3), 18.02),
 (datetime.date(2020, 4, 6), datetime.date(2020, 4, 6), 12.73),
 # Week contracts
 (datetime.date(2020, 4, 6), datetime.date(2020, 4, 12), 16.0),
 (datetime.date(2020, 4, 13), datetime.date(2020, 4, 19), 16.78),
 (datetime.date(2020, 4, 20), datetime.date(2020, 4, 26), 20.82),
 (datetime.date(2020, 4, 27), datetime.date(2020, 5, 3), 17.97), # This is what is causing issues
 # Month contracts
 (datetime.date(2020, 5, 1), datetime.date(2020, 5, 31), 22.21),
 (datetime.date(2020, 6, 1), datetime.date(2020, 6, 30), 27.3)

While output prices are correct for the average of the peak week, the distribution within the week is less than ideal. See below:

27/04/2020 | 13.65808
28/04/2020 | 13.65808
29/04/2020 | 13.65808
30/04/2020 | 13.65808
01/05/2020 | 35.21769

Is there any way to force these shapes when already using an average_weight function for peaks?

Attaching a notebook for your benefit.

Many thanks

Month overlap.ipynb.zip

Could solve the problem of instability caused by oscillation along the curve.

In a notebook an example model of building a high granulary (half-hourly or quarter-hourly) power forward curve.

I am trying to use the package to generate a power forward curve. How can I define a baseload (24 hours per day) and a peakload (12 hours per day from 8 to 20) contract with the same start and end date to pass to the bootstrap function? Explicitely setting the hour in the start and end date would not work if the peakload contract is longer than a day. E.g. a Peak Jan-20 contract would have all the 8 to 20 hours for Mon-Fri days in January 2020. To generate the forward curve I need to mix baseload and peakload contracts in the list of contracts passed to the bootstrap

First of all, thank you very much for publishing this library, it is a great piece of work!
However, I am currently running into an 'out of memory' error (7 GB RAM available on machine) when executing the max_smooth_interp function for hourly data over 3 years. Is there any way this can be fixed without decreasing the length of the period or the granularity?

Hi, it's me again with yet another bootstrap problem...

Currently using the mult_season_adjust in the max_smooth_interp function to generate sensible shapes for quarters relative to the year, months/qtr, days/month and I am running into an error.

I start with contracts of the lowest granularity (years and qtr) to generate a strip of quarters with ratios that i like based on my calibration/historical analysis. However, when I try to use these quarterly values together with any monthly values I have, bootstrap_contracts throws the following error:

ArgumentException: sum of weighting evaluated to non-positive number for the following contract: Start: 2020-05, End: 2020-05, Price: 24.77

I am using a peak weighting in the example that fails. The offpeak weighting computes properly, which leads me to believe that the optimiser does not like the combination of (start_dt, end_dt, price) and (pd.Period(), price) contract types.

Attaching a notebook for your benefit.
weighting issues.ipynb.zip

The curve PyPI package is dependent on an old Pandas version and is using now deprecated features. Update to newer Pandas version and remove deprecated feature usage.