Extension to hourly components? about prophet

[loneharoon]

It would be really great if you can extend it to work on lower (hourly, minutely) temporal granularity.

[bletham]

This is done in the v0.2 branch, for Python. Thanks to @hellais for kicking it off.

If you have sub-daily data, try it out! Example here:
https://github.com/facebookincubator/prophet/blob/v0.2/notebooks/non-daily_data.ipynb

[kikizxd]

Hi,
I am currently trying to use Prophet to predict flows in a network. The data like this:

ds	y
2017-07-28 09:30:00	5012
2017-07-28 09:30:01	3582
2017-07-28 09:30:02	3205
2017-07-28 09:30:03	2680
... ...	... ...
2017-07-28 10:30:00	4510
2017-08-28 09:30:00	6831
2017-08-28 09:30:01	4370
2017-08-28 09:30:02	3196
2017-08-28 09:30:03	2642
... ...	... ...
2017-08-28 10:30:00	1862
2017-09-28 09:30:00	4382
2017-09-28 09:30:01	5542
2017-09-28 09:30:02	2572
2017-09-28 09:30:03	2576
... ...	... ...
2017-09-28 10:30:00	2770

I want to predict the next period data--- '2017-10-28 9:30:00 10:30:00'. But I saw the Prophet can’t predict secondly data. I have tried to adapt 'forecaster.py' ,but it can't work.

I need your help,please.
@bletham

[bletham]

The R version uses the zoo Date type for the ds column, which only has day resolution. Our focus has so far been on daily data and so that was the most straightforward approach, but we are definitely interested in making this work for hour data.

Besides updating the R datatype to be a datetime (Python already is pd.datetime), there are two other parts of the code that assume daily data: holidays and seasonality.

Holidays essentially join on date to identify the rows in the history/future that should get the feature for that holiday. We have to decide how we want to think about holidays with hourly data.

Seasonality features are done by converting the dates to number of days since Unix epoch. The way this is done right now provides only differences in the (integer) number of days; for hourly data that needs to be changed to get the fractional component.

[bletham]

Fitting sub-daily data is now finished in R in the v0.2 branch. A big thanks to @qwng who went through the painful process of converting all of the date types to POSIXct. If you have sub-daily data try it out!

devtools::install_github('facebookincubator/prophet', subdir='R', ref='v0.2')

[bletham]

@CarlosVaquero this is coming in #101, within the next week.

[Harti]

@bletham just in time to be included in my master thesis (printing on Saturday), good job hahaha

[oscarlinux]

Hi @bletham , is there any estimated date when v0.2 is going to be released?

Thanks,
Oscar

[mlape]

There is definitely still a need for adding hourly seasonality to python's fbprophet!

[eliorerz]

Hi, @kikizxd
I've also been trying to use Prophet to detect anomalies in secondly data and have not been able to.
Have you had any progress with this?
Thanks.

[bletham]

@azriele If you have v0.2 it should be working with secondly data, what is the particular issue you're running into?

[bletham]

Yep!

[jmarrietar]

When is this extension available for R users?.

Thank you very much.

[hellais]

I took a stab at extending prophet to support daily seasonality by looking at hourly components as well: https://github.com/hellais/prophet/tree/feature/daily-seasonality.

I am not entirely sure if 3 is the correct number of components for the fourier series.

[benoitkoen]

Are hourly and half-hourly predictions supported in the end?

[ofir887]

hourly predictions is coming soon ?

[hellais]

@benoitkoen @ofir887 I opened PR #135 with support for hourly predictions for the python library. It's missing R support before it can be merged into master as I have limited R knowledge someone else would have to write that.

[mucun1988]

@bletham great work to handle non-daily data. Just want to make sure. How is holiday handled for hourly data? How is holiday handled for monthly data?

[bletham]

@mucun1988 this was something we debated a bit. With daily data, holidays provide a constant shift on the holiday date. With sub-daily data, this behavior has currently not been changed, meaning, there is a constant shift across the entire holiday date. This is clearly not going to work all the time, you can imagine that you might want a holiday effect that peaks during the day and returns to baseline at the end of the day. But I think that this will be a reasonable approximation for now.

You can imagine fitting a holiday-specific daily seasonality term, basically by replacing the current holiday regressor [0, 0, 1, 0, ...] with something that looks like the daily seasonality regressors, but is zero off-holiday. #101 will soon allow for entirely custom external regressors, which could be used to do this manually. We could build functionality in the future for constructing these automatically, but I can think of a few ways to do this (individual daily curves for each holiday? daily curve for all holidays with a multiplier specific to each holiday?) and I think it would be problem-specific which is appropriate.

Monthly data are treated like daily data (the model is continuous-time), so intra-month holidays would best be specified as being on whatever day of the month the past/future dates are labeled with.

[mucun1988]

@bletham Monthly data are treated like daily data (the model is continuous-time), so intra-month holidays would best be specified as being on whatever day of the month the past/future dates are labeled with.

So suppose my data in monthly and I would like model thanksgiving holiday, then it does not matter I specify thanksgiving as Nov 1 or Nov 20 (as long as some date in Nov). Do I understand correctly?

[bletham]

You would want to specify it on whatever day in your history has the effect from Thanksgiving. So if the input data are monthly like:

ds, y
2016-10-01, 20
2016-11-01, 30
2016-12-01, 40

and the row with 2016-11-01 represents the entire month of Nov, then you'd want to put the holiday on 2016-11-01, and on 11-01 moving forward.

Then when making the dataframe of dates for future predictions, just make sure it is also the first of the month, like

ds
2017-10-01
2017-11-01
2017-12-01

Does that make sense?

[mucun1988]

@bletham that makes sense and many thanks!

[vly]

Awesome, thank you! Have been awaiting this feature for some time.

[CarlosVaquero]

Hi @bletham and @seanjtaylor, Thanks for sharing this fantastic contribution. If I understand correctly, the current holiday regressor (in make_holiday_features) is returning a binary vector [0, 0, 1, 0, ...] only because scale_ratio = (self.holidays_prior_scale = 10) / (self.seasonality_prior_scale = 10)

Is it a problem if the 'expanded_holidays' would contain instead non integer negative values?

And, Would it be sensible to extend the ‘expanded_holidays’ DataFrame with other features than holidays? That is, having in seasonal_features an arbitrary number of columns to be appended relating to other variables than 'holidays'.

[bletham]

This is done in v0.2 that is now available on CRAN and pypi. Check out the documentation here:
https://facebookincubator.github.io/prophet/docs/non-daily_data.html

And if you run into any issues feel free to reopen here or make a new task.

[eliorerz]

@bletham Great!
For some reason I had v 0.1.1 and I just upgraded it to 0.2
So as far as I can understand I can use secondary data and weekly seasonality?
Thanks.

[bbrewington]

@bletham The link to "Non Daily Data" above isn't working b/c facebookincubator.github.io change to facebook.github.io

Here's a working link - https://facebook.github.io/prophet/docs/non-daily_data.html

[bletham]

Thanks, that's a shame the URL forwarding doesn't include subdirectories.