Provide the user with some sense of progress while extract_features is calculating about tsfresh

[MaxBenChrist]

Good point. If you extract features for different kind of time series this should be easily implemented.

[earthgecko]

Can it at least be a log friendly progress indicator and not a some spinnings forward slashes, pipes and backslashes or some incrementally filling up progress bar which are visually pleasing but can be hell to logs.

Although we must think about the humans sitting there wondering how long it will take and if they have time to read a comic, we must also think of the machines that are conducting their work and calculating programmatically as well. A progress indicator may not be of interest to them as they are not wondering about comics, but just trying to do the job assigned to them and calculate some features.

Hopefully it is possible to find a happy middle ground here that provides appropriate info to all executing parties, whether they be people or machines.

[earthgecko]

Just try clarify, this point.

Universe Sandbox2 just presented me with an example of a progress indicator implementation that could be classified and classed as person_friendly and also not_great_for_a_machine_or_log. It seemed relevant to share and I ironically choose Universe Sandbox2 over having to further contemplate the 50 / 3 Python paradox/dilemma (with which I have recently been confronted with) after further discussing and explaining it in detail to with my wife and dog on our evening walk. She got it and agrees, our dog Dude has not weighed in his opinion yet, but I am not waiting on it.

an example of a not so machine friendly progress implementation

by Steam (although it is not meant to be consumed programmatically), this is in no way a criticism of Steam's progress indicator, I find it quite informative and visually detailed and I like that, but I am person.

[ClimbsRocks]

@earthgecko : Yeah, I've had my logs wrecked by some of those terrible progress indicators you're talking about. They also seem like unnecessary complexity at this stage. I'm just hoping for something simple, that's also reassuring for first-time users as they sit wondering whether things are working on their dataset or not :)

[nils-braun]

I could take care of this, but I would wait until the parallelization is implemented

[MaxBenChrist]

@nils-braun what about: https://pypi.python.org/pypi/tqdm

I am tinkering with it around right now.

[MaxBenChrist]

@nils-braun or do you already have a solution in mind?

[nils-braun]

The question it comes down is: what should be the "minimal step" in a progress? Should each calculator function be one step in the progress? Or each kind/simple (depending on the parallelization)?

[MaxBenChrist]

I think a progress for the chunks is sufficient.

With chunks I refer to the squares (sample x kind) out of the original data frame.

[nils-braun]

so based on the parallelisation - ok. This would however mean that if you have one single time series with one kind, you only have one step in the progress bar (just to make this clear).

[MaxBenChrist]

lets say

df =
id val1 val2
1 4 4
2 3 4
3 4 1
4 1 4

and you extract the features in parallel for those 4 "chunks"

id val1
1 4
2 3
id val2
1 4
2 4
id val1
3 4
4 1
id val2
3 1
4 4

after the completion of any one of those "chunks" the progress bar should increase by 1/4.

[nils-braun]

... actually we do not have any parallelization mode for this, aren't we? We do have parallelisation per kind (which would be 2 in this example) or per sample (with would be 4 in this case, with the ids 1, 2, 3, 4 for both kinds). Or did I miss something?

[nils-braun]

Ah sorry, I got it. Do not bother with my last comment...

[MaxBenChrist]

Actually, the "per_sample" parallelization is parallelizing per kind and per sample

[nils-braun]

The progress bar for future extraction is now includes. I will include another one for feature selection soon.

[MaxBenChrist]

@moritzgelb implemented this in #152