mbok / elasticsearch-linear-regression Goto Github PK

Hi mate, thank you for your great plugin! It's really nice! Are you thinking to extend compatibility to elasticsearch 6.x.x? Now we can't use elasticsearch-linear-regression beacuse it is not compatible with last elasticsearch version.

Breakouts (e.g. documents with a response variable value outside of the upper and lower hyper-plane spaced by a specified number of standard deviations above and below the middle linear regression hyper-plane) in time series data may indicate anomaly.
A concrete concept has still to be defined.
Real world use cases are e.g stock markets, see https://www.dailyfx.com/forex/education/trading_tips/daily_trading_lesson/2014/10/24/Trend-Following-with-Regression-Channels.html.

@TechReport{pebay2008formulas,
title={Formulas for robust, one-pass parallel computation of covariances and arbitrary-order statistical moments.},
author={Pebay, Philippe Pierre},
year={2008},
institution={Sandia National Laboratories}
}

Hello,
probably not the best place to ask for the comment about the mentioned use case, but I'd give it a try.
I am trying to come up with optimal way to forecast the purchase time for Product P and User U.
We currently index these events in ES (pushed by e-commerce system)

orderId,user,product,quantity,time,days
"order1","U","P",1,"2017-01-01",17167
"order2","U","P",2,"2017-01-29",17195
"order3","U","P",3,"2017-04-02",17258
"order4","U","P",1,"2017-07-06",17353
"order5","U","P",2,"2017-08-03",17381

where days is just a integer showing number of days since 1.1.1970 for the event time.

What I want is to predict is the next time of purchase and the quantity.
quantity is last purchased quantity in this case 2
and the forecast time should be somewhere in October.

I've played with this plugin:
and it works well if I have additional calculated field "lag" for each event which denotes the time period until NEXT purchase, so then the data above should look like:

orderId,user,product,quantity,time,days,lag
"order1","U","P",1,"2017-01-01",17167,28
"order2","U","P",2,"2017-01-29",17195,63
"order3","U","P",3,"2017-04-02",17258,95
"order4","U","P",1,"2017-07-06",17353,29
"order5","U","P",2,"2017-08-04",17382,?
GET /buying_habit/_search?size=0 { "query": { "match_all": {} }, "aggs": { "demand_p": { "linreg_predict": { "fields": ["quantity", "lag"], "inputs": [2] } } } }

Of course in this index I will have 10K of different products and 1M different Users.
My first question is how to update this field for the LAST event when new event comes in?
Is it possible to do it in index time?
Does this make sense at all or there is a better way?
Btw in case that there is only one purchase, I'd use the default lifecycle of the product in days (comes from e-commerce as well). But for cases where there is a buying pattern (at least 2 events) I 'd need to use user specific data.
I plan to run forecast query for each User/Product pair every hour to calculate the next forecast time (effectively when user SHOULD run out of supply).
What would be the way to optimize that (avoid doing this one by one)?

Thanks very much in advance,
Milan

Dear @mbok
In document you have add a wiki link to download training data with *.csv format. But this link is not work now (with me). Do you help me upload *.csv file to source in github or send to me by email [email protected].
Now I am create an homework assignments need use the your product. Thank you very much.