Unable to pip install tsfresh on a Mac with M1 chip about tsfresh

[kempa-liehr]

Duplicate of #937

@giova86, thanks for your workaround.

@bfdykstra, @seanlaw, @vanbenschoten: The matrixprofile project certainly is interesting and I had approved the original pull request. On the other hand, I have not seen any of the matrix profiles features as statistically significant features in one of my projects. Therefore, I am going to remove the dependency on matrix profiles.

[kempa-liehr]

Fixed in https://github.com/blue-yonder/tsfresh/tree/v0.20.0.
Many thanks for your patience and perseverance.

Merry Christmas and a Happy New Year!

[JoachimSchaeffer]

I'm using an M1 chip and had the same issues when trying to install it under python 3.9.
However, with Python 3.9.7 pip install worked.
You might need to
brew install llvm

[seanlaw]

@bfdykstra I don't know anything regarding the matrixprofile API but if you'd like to submit a PR to this repo with the stumpy replacement parts then I can help review it.

Alternatively, would it make sense to simply remove the matrixprofile parts instead of replacing it given the lack of activity in that package? My concern is that we may be forcing stumpy to support matrixprofile features that are absent in stumpy and so we should be open to removing the matrixprofile bits if necessary.

[seanlaw]

My preferred solution is to refactor the matrix profile features into a separate module, e.g. tsfresh-matrixprofiles, which can be installed on top of tsfresh.

@kempa-liehr that's probably a good idea. Having said that, it appears that users are unable to install tsfresh right now and the solution provided above addresses this issue while preserving some level of backwards compatibility. If the goal is to split it out then I think we should simply spend the time to remove the matrixprofile package (which doesn't seem to be supported anymore) parts altogether for this PR, which would also resolve the installation issues. My suggestion would be to spend this time fixing the current issue, which is an installation problem. What do you think?

[giova86]

I tried to use Rosetta and it seems to work. I followed these steps:

1. open terminal with Rosetta
2. create and start venv (I'm using Python 3.9.13)
3. install tsfresh (pip install tsfresh)
4. since there is a bug with statsmodels 0.13.0 I have downgraded it to 0.12.1
5. use if name == "main"

I checked with this very simple code:

from tsfresh import extract_relevant_features
from tsfresh.examples.robot_execution_failures import load_robot_execution_failures

timeseries, y = load_robot_execution_failures()

if __name__ == "__main__":
    features = extract_relevant_features(timeseries, y, column_id="id", column_sort="time")
    print(features)

[jarhed]

I'm no expert with all this stuff. Just saw a youtube thing on matrix profiles. It looks like it's using /usr/bin/clang instead of the brew-installed /usr/local/bin/gcc which has the -fopenmp flag working. Is there any way to force it to use gcc?

[adamcatto]

I'm running into the same issue. brew install llvm did not help on Python 3.10.6

[giova86]

I have the same problem...brew install llvm did not help. I'm using Python 3.9.13

[bfdykstra]

I used what @JoachimSchaeffer did and set my python version to exactly 3.9.7 and pip install worked.

I started taking a crack at doing what the matrixprofile author said and replacing matrix profile with stumpy. However stumpy does not provide an implementation of maximum_subsequence, but we could probably roll our own implementation of that based on what matrix profile already has.

I'm also not entirely sure that there is a 1-1 mapping between the APIs of the two packages:
https://matrixprofile.docs.matrixprofile.org/api.html#matrixprofile-compute
https://stumpy.readthedocs.io/en/latest/api.html#stumpy.stump

[bfdykstra]

@seanlaw Thank you weighing in!
Not sure how critical the matrix profile stats are as features, maybe @nils-braun could provide some insight as to what would be lost if we removed them?

From the documentation of tsfresh it says:

Calculates the 1-D Matrix Profile[1] and returns Tukey’s Five Number Set plus the mean of that Matrix Profile.

Matrix profile's api documentation says it computes the exact or approximate matrix profile based on sample percent.
I see with stumpy that we can use either stumpy.stump or stumpy.scrump for an exact or approximate matrix profile.

@seanlaw Does stumpy support calculations based off of a sample percent?

Looking at the tests, might shed some light on the desired usage:

def test_matrix_profile_window(self):
        # Test matrix profile output with specified window
        np.random.seed(9999)
        ts = np.random.uniform(size=2 ** 10)
        w = 2 ** 5
        subq = ts[0:w]
        ts[0:w] = subq
        ts[w + 100 : w + 100 + w] = subq
        param = [
            {"threshold": 0.98, "windows": 36, "feature": "min"},
            {"threshold": 0.98, "windows": 36, "feature": "max"},
            {"threshold": 0.98, "windows": 36, "feature": "mean"},
            {"threshold": 0.98, "windows": 36, "feature": "median"},
            {"threshold": 0.98, "windows": 36, "feature": "25"},
            {"threshold": 0.98, "windows": 36, "feature": "75"},
        ]

        self.assertAlmostEqual(matrix_profile(ts, param=param)[0][1], 2.825786727580335)

[seanlaw]

Not sure how critical the matrix profile stats are as features, maybe @nils-braun could provide some insight as to what would be lost if we removed them?

Objectively, I don't think these stats features are useful at all (save for the min of the matrix profile) as they aren't necessarily informative as an ML feature. Hence, why I suggested removing this part completely.

@seanlaw Does stumpy support calculations based off of a sample percent?

I am not too familiar with what is happening in matrixprofile but I believe that you can use stumpy.scrump and specify percentage=1.0 to obtain the exact matrix profile (see API docs).

Perhaps something like this would be equivalent for allowing the test to pass:

def test_matrix_profile_window(self):
    # Test matrix profile output with specified window
    np.random.seed(9999)
    ts = np.random.uniform(size=2 ** 10)
    w = 2 ** 5
    subq = ts[0:w]
    ts[0:w] = subq
    ts[w + 100 : w + 100 + w] = subq

    # mp = stumpy.stump(ts, m=36)[:, 0]
    mp = stumpy.scrump(ts, m=36, percentage=1.0)[:, 0]

    self.assertAlmostEqual(mp.min(), 2.825786727580335)

Then, to obtain the other numbers, you'd simply do:

mp.max()
mp.mean()
np.median(mp)
np.percentile(mp, 25)
np.percentile(mp, 75)

I think this is the easiest way forward without attempting to force a one-to-one match between the packages. We'll need to remove the threshold parameters or simply ignore them when passed in. Please feel free to ask more questions if I've been to vague. Does that help?

[kempa-liehr]

My preferred solution is to refactor the matrix profile features into a separate module, e.g. tsfresh-matrixprofiles, which can be installed on top of tsfresh.

[myersau3]

I'm using an M1 chip and had the same issues when trying to install it under python 3.9. However, with Python 3.9.7 pip install worked. You might need to brew install llvm

Unfortunately this didn't work for me, but it sounds like the matrixprofile dependency is going to be removed so I will wait.

[arresejo]

The solution I've found was to downgrade the Python version to 3.10.8 using pyenv in order to easily manage multiple Python versions (useful resource: https://opensource.com/article/19/5/python-3-default-mac).