dtcooper / python-fitparse

Python library to parse ANT/Garmin .FIT files

python-fitparse

⚠️ NOTE: I have limited to no time to work on this package these days!

I am looking for a maintainer to help with issues and updating/releasing the package. Please reach out via email at [email protected] if you have interest in helping.

If you're having trouble using this package for whatever reason, might we suggest using an alternative library: fitdecode by polyvertex.

Cheers,

David

Here's a Python library to parse ANT/Garmin .FIT files.

Install from :

pip install fitparse

FIT files

• FIT files contain data stored in a binary file format.
• The FIT (Flexible and Interoperable Data Transfer) file protocol is specified by ANT.
• The SDK, code examples, and detailed documentation can be found in the ANT FIT SDK.

Usage

A simple example of printing records from a fit file:

import fitparse

# Load the FIT file
fitfile = fitparse.FitFile("my_activity.fit")

# Iterate over all messages of type "record"
# (other types include "device_info", "file_creator", "event", etc)
for record in fitfile.get_messages("record"):

    # Records can contain multiple pieces of data (ex: timestamp, latitude, longitude, etc)
    for data in record:

        # Print the name and value of the data (and the units if it has any)
        if data.units:
            print(" * {}: {} ({})".format(data.name, data.value, data.units))
        else:
            print(" * {}: {}".format(data.name, data.value))

    print("---")

The library also provides a fitdump script for command line usage:

$ fitdump --help
usage: fitdump [-h] [-v] [-o OUTPUT] [-t {readable,json}] [-n NAME] [--ignore-crc] FITFILE

Dump .FIT files to various formats

positional arguments:
  FITFILE               Input .FIT file (Use - for stdin)

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose
  -o OUTPUT, --output OUTPUT
                        File to output data into (defaults to stdout)
  -t {readable,json}, --type {readable,json}
                        File type to output. (DEFAULT: readable)
  -n NAME, --name NAME  Message name (or number) to filter
  --ignore-crc          Some devices can write invalid crc's, ignore these.

See the documentation for more: http://dtcooper.github.io/python-fitparse

Major Changes From Original Version

After a few years of laying dormant we are back to active development! The old version is archived as v1-archive.

• New, hopefully cleaner public API with a clear division between accessible and internal parts. (Still unstable and partially complete.)

• Proper documentation! Available here.

• Unit tests and example programs.

• (WIP) Command line tools (eg a .FIT to .CSV converter).

• Component fields and compressed timestamp headers now supported and not just an afterthought. Closes issues #6 and #7.

• FIT file parsing is generic enough to support all types. Going to have specific FitFile subclasses for more popular file types like activities.

• (WIP) Converting field types to normalized values (for example, bool, date_time, etc) done in a consistent way, that's easy to customize by subclassing the converter class. I'm going to use something like the Django form-style convert_<field name> idiom on this class.

• The FIT profile is its own complete python module, rather than using profile.def.

  • Bonus! The profile generation script is less ugly (but still an atrocity) and supports every ANT FIT SDK from version 1.00 up to 5.10.

• A working setup.py module. Closes issue #2, finally! I'll upload the package to PyPI when it's done.

• Support for parsing one record at a time. This can be done using <FitFile>.parse_one() for now, but I'm not sure of the exact implementation yet.

Updating to new FIT SDK versions

• Download the latest ANT FIT SDK.
• Update the profile:

python3 scripts/generate_profile.py /path/to/fit_sdk.zip fitparse/profile.py

License

This project is licensed under the MIT License - see the LICENSE file for details.

python-fitparse's Issues

Timestamp_16 support

Currently in FIT we have timestamps which are 32bit and timestamp_16 which are 16bit difference from previous timestamp and are currently not read into nice values.

I wrote generator function based on ParseVivosmartHR. Here seems to be offical function how to do it (in Java):

mesgTimestamp += ( timestamp_16 - ( mesgTimestamp & 0xFFFF ) ) & 0xFFFF; 

It works but it definitely isn't written as it should, but I don't know fitparse enough to improve it.

#Input is what we get in FitFile.get_messages
def fix_times(messages):                                                        
    timestamp=None                                                              
    timestamp_16=None                                                           
    real_time=0                                                                 
    UTC_REFERENCE = 631065600  # timestamp for UTC 00:00 Dec 31 1989            
    for message in messages:                                                    
        #print (message)                                                        
        #message["full_timestamp"]="TIMESTAMP"                                  
        field = message.get("timestamp")                                        
        if field is not None:                                                   
            timestamp = field.raw_value                                         
            #I'm not sure if this is correct to set this to None. Without it records with just timestamp doesn't have same new timestamp anymore which could be a bug or it should be that way                                          
            timestamp_16 = None                                                 
        tm_16 = message.get("timestamp_16")                                     
        if tm_16 is not None:                                                   
            timestamp_16 = tm_16.raw_value                                      
        #print(timestamp, timestamp_16)                                         
        if timestamp_16 is None:                                                
            ts_value = timestamp                                                
        else:                                                                   
            new_time = int( timestamp/2**16) * 2**16 + timestamp_16             
            if new_time >= real_time:                                           
                real_time = new_time                                            
            else:                                                               
                real_time = new_time + 2**16                                    
            ts_value = real_time                                                
        if ts_value is not None and ts_value >= 0x10000000:                     
            value = datetime.datetime.utcfromtimestamp(UTC_REFERENCE            
                    + ts_value)                                                 
        else:                                                                   
            value = None                                                        
        message.fields.append(                                                  
                FieldData(                                                      
                    field_def=None,                                             
                    field=FIELD_TYPE_TIMESTAMP_1,                               
                    parent_field=None,                                          
                    value=value,                                                
                    raw_value=ts_value                                          
                    ))                                                          
        yield message

Refactoring and optimization by decorators

@dtcooper @pR0Ps
I would like to start discussion about general change of the FitFile architecture, already started in #61 and #57. The problem is speed, code simplicity vs. feature richness.

IMO is may be solved by the decorator pattern, e.g. see this example on python.org. (It's not exactly same as Python decorators, but the Python decorators may be used to implement that.) I think the main method is FitFile.get_messages() which should yield message by message. The other decorators has to implements get_messages() and may add further functionality.

Example: MessageCacheDecorator. I have proposed in #61 to remove message caching from the FitFile. Then a new class MessageCacheDecorator may be created. So, users, which would like to keep the old functionality would do just:

fit = FitFile(...)
fit = MessageCacheDecorator(fit)
m1 = fit.get_messages(...)
m2 = fit.messages

Similarly to the data processors. Each conversion may be a decorator, too. So, it does not need to be in the FitFile core. And, if you really need to keep BC, the FitFile may be split to: FitFileDecoder and FitFile just as a decorator (or subclass) to FitFileDecoder.

Saving fit files?

Not actually an issue but a feature request: I can't seem to find a way to make (write) a FIT file from a list of records.
Typical usage scenario: data merge from some GPX/TCX source.
I can open a FIT file using python-parse and I could even add records with the missing information but I can't find a method to write a FIT file out of the updated list of records.
Thanks in advance.

FIT File with RunScribe Plus

I recently purchased a RunScribe Plus, which adds running dynamics (but not Garmin Running Dynamics) data to the the .fit file via a ConnectIQ data field (example file here: file )

I am no longer able parse the file using fitparse. I get the following error

`~\AppData\Local\Continuum\anaconda3\envs\pfda\lib\site-packages\fitparse\records.py in iter(self)
152 def iter(self):
153 # Sort by whether this is a known field, then its name
--> 154 return iter(sorted(self.fields, key=lambda fd: (int(fd.field is None), fd.name)))
155
156 def repr(self):

TypeError: '<' not supported between instances of 'NoneType' and 'str'`

when trying to run this code:

for record in fitfile.get_messages('record'): for record_data in record: pass

Any ideas?

Documentation out of date

The documentation at https://pythonhosted.org/fitparse/ is out of date. The documentation is for version 0.0.1 but the latest version is 1.1.0.

One of the main issues (and the one that made me notice that it is outdated) is that the docs tell you to install with pip install python-fitparse...

I do not have an overview of how up to date the documentation is so cannot assess whether it is better to update the docs or to temporarily remove the reference in the readme.

Avoid DeprecationWarning in python 3.7 causesd by "from collections import Iterable"

It just needs to be replaced with

from collections.abc import Iterable

Raw acceleration values

Is it possible to extract the raw accelerometer values from a fit file?

I tested the following example from the docs but it doesn't contain any acceleration fields:

from fitparse import FitFile

fitfile = FitFile('C:\\Users\\Simon\\Desktop\\2018-05-27-13-44-59.fit')

# Get all data messages that are of type record
for record in fitfile.get_messages('record'):

    # Go through all the data entries in this record
    for record_data in record:

        # Print the records name and value (and units if it has any)
        if record_data.units:
            print(" * %s: %s %s" % (
                record_data.name, record_data.value, record_data.units,
            ))
        else:
            print(" * %s: %s" % (record_data.name, record_data.value))

New uncovered (edge-?)case : parse_string ValueError

Hello everyone,

I wanted to report a small bug - or rather an uncovered case - in the get_messages() function, and see if anyone else sees the same thing.

We have a system in production extracting ~800 fit files per day and since 15 days we have seen ~ 50 (increasing by the day) fit files with an exception :

File "/xxxx/garmin.py", line 224, in build_sessions
    for message in fit.get_messages(as_dict=True):
  File "/usr/local/lib/python3.5/dist-packages/fitparse/base.py", line 450, in get_messages
    message = self._parse_message()
  File "/usr/local/lib/python3.5/dist-packages/fitparse/base.py", line 160, in _parse_message
    message = self._parse_data_message(header)
  File "/usr/local/lib/python3.5/dist-packages/fitparse/base.py", line 320, in _parse_data_message
    raw_values = self._parse_raw_values_from_data_message(def_mesg)
  File "/usr/local/lib/python3.5/dist-packages/fitparse/base.py", line 271, in _parse_raw_values_from_data_message
    raw_value = base_type.parse(raw_value)
  File "/usr/local/lib/python3.5/dist-packages/fitparse/records.py", line 336, in parse_string
    end = string.index(0x00)
ValueError: substring not found

I'm not a specialist but I've tried to debug this and started by outputting the raw_value for these files, here are some of them :

b'IM G 33 NTAP'
b'S0E2 - 1h + 4*10'
b'CAP Fartleck VMA'
b'Basse Intensit\xc3\xa9'
b'RF5 or Cross-Tra'

I think it's user inputs, and they are confusing the parse_string() function because for some reason don't contain the 0x00 substring.
It must be a new functionality from Garmin that's incorporated in the fit files for the people who have updated their hardware ?
Furthemore the mesg_type.name == 'developer_data_id' (not sure what this means though).

I'm not sure how to handle these messages but if I do that in base.py:70 :

    # Otherwise, just scrub the singular value
    try:
        raw_value = base_type.parse(raw_value)
    except:
        print('Substring "0x00" not found, skipping value')
        print(raw_value)

It works just fine (for us), but I don't know what the consequences downstream could be if we just append the raw_value without parsing it...

I would love to make a PR to incorporate the handling of this upstream but I'm not comfortable enough without having more information and feedback.

Does anyone want to chime in on this ?
Thanks

Updating Profile based on custom messages/fields in config.csv, types.csv and messages.csv

You can add new messages or add new fields to a message using by updating the config.csv, types.csv and messages.csv.
It will be great if the profile can be updated based the new information on these files.

Python 2.7.6 doesn't have itertools.zip_longest

Hi guys.

I try to install fitparse from pypi and got error.
Did I need future module?

Downloading/unpacking fitparse
  Downloading fitparse-1.0.0.tar.gz
  Storing download in cache at /Users/key/.pip_cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Ff%2Ffitparse%2Ffitparse-1.0.0.tar.gz
  Running setup.py (path:/Users/key/Documents/virtualenv/gpslogparser/build/fitparse/setup.py) egg_info for package fitparse
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/Users/key/Documents/virtualenv/gpslogparser/build/fitparse/setup.py", line 4, in <module>
        import fitparse
      File "fitparse/__init__.py", line 1, in <module>
        from fitparse.base import FitFile, FitParseError
      File "fitparse/base.py", line 9, in <module>
        from fitparse.profile import FIELD_TYPE_TIMESTAMP, MESSAGE_TYPES
      File "fitparse/profile.py", line 7, in <module>
        from fitparse.records import (
      File "fitparse/records.py", line 3, in <module>
        from itertools import zip_longest
    ImportError: cannot import name zip_longest
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/Users/key/Documents/virtualenv/gpslogparser/build/fitparse/setup.py", line 4, in <module>

    import fitparse

  File "fitparse/__init__.py", line 1, in <module>

    from fitparse.base import FitFile, FitParseError

  File "fitparse/base.py", line 9, in <module>

    from fitparse.profile import FIELD_TYPE_TIMESTAMP, MESSAGE_TYPES

  File "fitparse/profile.py", line 7, in <module>

    from fitparse.records import (

  File "fitparse/records.py", line 3, in <module>

    from itertools import zip_longest

ImportError: cannot import name zip_longest

Change Bitcoin donation address?

Might want to rethink one starting with ******.

To be clear, I have no issue at all with adding a donation address and personally I couldn't care less what it says. However, in my opinion it's just not very professional. Plus, the internet hate machine can be vindictive, no matter the reason. Seems safer to just generate a new one and sidestep the whole issue.

add a setup.py file to master branch

Until the ng branch has matured (it seems unstable at the moment, as I cannot parse my FIT files), I would like to use the master branch of the project (preferably using pip/distutils to install the module from github). I guess that the same, or a very similar, setup.py file that is used on the ng branch could be used, so the effort involved should be minimal.

RecordBase.__init__ slows down

fitparse uses many many objects and many of them are of RecordBase, The RecordBase.__init__ is a bit slow. E.g. I've tried to optimize FieldData.__init__:

     def __init__(self, field_def = None, field= None, parent_field= None, value= None, raw_value= None, units= None):
        self.field_def = field_def
        self.field = field
        self.parent_field = parent_field
        self.value = value
        self.raw_value = raw_value
        self.units = units
        ...

And parsing of event_timestamp.fit has sped up by 0.8 sec. Generally, the getattr, hasattr and setattr slows down the process when used so extensively.

I may create PR to change all RecordBase and descendants init. But there's a remark in the records.py:

    # TODO: switch back to namedtuple, and don't use default arguments as None
    #       and see if that gives us any performance improvements

Note: Also, the processor.py uses getattr, extensivelly.

Generating fitparse/profile.py for FIT 2.0

Hi,
I've added support for FIT 2.0 locally including supporting developer data fields, but am curious how to generate fitparse/profile.py. The file header says it's autogenerated.

Can you describe that process?

Thanks!

Problem on timestamp extraction

Hi,

I work on a tool to extract data from Swimming Strap HRM from fit files and when I try to display data as chart, I see big difference between real chart (from Garmin Connect) and data extracted from fit-parse.

After investigation, I found a difference in the fitparse timestamp calculation compared to same data extracted from the Garmin FitCSVTool.

First 4 lines of Strap HRM data extracted with Garmin FitCSVTool (format: filtered_bpm, event_timestamp) :

71|71|71|71|71|71|71|71,2224910.171875|2224911.158203125|2224912.220703125|2224913.0419921875|2224913.7802734375|2224914.5126953125|2224915.34765625|2224915.7939453125
71|71|71|71|71|71|71|71,2224916.271484375|2224916.982421875|2224917.2734375|2224919.5859375|2224920.556640625|2224921.078125|2224921.45703125|2224921.990234375
71|71|72|72|72|73|73|74,2224924.7451171875|2224925.24609375|2224925.7421875|2224926.2412109375|2224927.1728515625|2224927.73828125|2224928.2197265625|2224928.6328125
74|74|75|77|83|84|87|89,2224929.126953125|2224929.619140625|2224930.1142578125|2224930.62890625|2224931.1552734375|2224931.67578125|2224932.1982421875|2224932.72265625

I use this code to extract and format data :

from fitparse import FitFile
fitfile = FitFile('lagny_fim.fit')
for _, msg in enumerate(fitfile.get_messages('hr')):

    if msg.header.local_mesg_num == 12:
        startat = round(msg.get('event_timestamp').value)
        continue

    timestamps = []
    for field in msg.fields:
        if field.name == 'event_timestamp':
            timestamps.append(startat + float(field.value))

    print('%s,%s' % (
        '|'.join(str(bpm) for bpm in msg.get('filtered_bpm').value),
        '|'.join(str(t) for t in timestamps)
    ))

Data extracted with this script :

71|71|71|71|71|71|71|71,2224912.17188|2224913.1582|2224914.2207|2224915.04199|2224915.78027|2224916.5127|2224917.34766|2224917.79395|2224918.0|2224918.0                                                                        
71|71|71|71|71|71|71|71,2224918.27148|2224918.98242|2224919.27344|2224921.58594|2224922.55664|2224923.07812|2224923.45703|2224923.99023|2224926.0|2224926.0                                                                     
71|71|72|72|72|73|73|74,2224926.74512|2224927.24609|2224927.74219|2224928.24121|2224929.17285|2224929.73828|2224930.21973|2224930.63281|2224934.0|2224934.0                                                                     
74|74|75|77|83|84|87|89,2224935.12695|2224935.61914|2224936.11426|2224936.62891|2224937.15527|2224937.67578|2224938.19824|2224938.72266|2224942.0|2224942.0   

As you ca see, there are an error of +2 seconds on calculated timestamp and maybe other errors that occur in the extraction that would explain the overall errors in extracting heart rate data.

Do you have an idea of what can happen?
Am I making a mistake in extracting data?

I can provide the original fit file for investigation.

Thank you.

Do not cache messages in FitParse._messages

When parsing a large FIT file (2MG, 75.000 data messages), the memory consumption is about 90 MB. I have tried to remove FitParse._messages and the memory consumption has dropped to 13MB. (And a few percent faster.)

I think the FitParse._messages and the property FitParse.messages should be removed - caching is the task of the user code.

If you want to keep it, then there should be some parameter like cache_result (default False).

max_speed && speed problem

fit file:
2019-2-13-10-36-52-12791.fit.zip

fitparse version 1.0.1
max_speed : 14.201

fitparse version 1.1.0
max_speed: 14201

and speed encounter the same problem.

Question: extracting duration

Hi, thanks for this useful library. Just having a play with some of the usage examples. Appreciate anyone who could tell me if this is a reliable way to calculate duration (and if there isn't an easier way)...

import fitparse

ff = fitparse.FitFile('…')

records = list(ff.get_messages(name='record'))

start_time = None
end_time = None

for record in records:
    timestamp = record.get_value('timestamp')
    if not start_time:
        start_time = timestamp
    end_time = timestamp

print("Started:", start_time)
print("Duration:", end_time - start_time)

I'm assuming:

• list(ff.get_messages(name='record')) will always be in chronological order
• there isn't a 'duration' field I could pull out instead

Can anyone confirm this?

Export activities to json?

Is there a way to export records to json?

`test_int_log` Failing

Hey @pR0Ps -- this test is failing with our new generated profile, but I'm trying to figure out what it tested in the first place, and if it's a bug or the test. Thanks!

python-fitparse/tests/test.py

Line 395 in 99ea845

def test_int_long(self):

compressed_speed_distance handling is broken

Hi,

compressed_speed_distance is pretty broken, at least on my FR70; the speed is decoded wrong, and the distance is not properly unwrapped. Please take a look at the included patch; it might not be perfect, but it seems to fix the problems okay enough for me.

commit 4757d3bbb2c01616c8457d64aadc56403e2a0cf4
Author: Steinar H. Gunderson <[email protected]>
Date:   Fri Aug 31 19:46:11 2012 +0200

    Fix handling of compressed_speed_distance.

    First of all, the bit decoding of the speed field was missing a shift,
    leading to messed up values. I corrected it so it's in line with decode.c
    in FIT SDK 4.00. There was also seemingly some attempt at converting from
    km/h to m/s that didn't match anything reasonable (maybe it was an attempt
    at fudging the speed to it would be correct despite the missing shift?),
    so I just deleted it. That function now returns raw values.

    Second, I added decoupling of the two fields into speed and distance;
    the user shouldn't need to care how the field was originally stored.
    compressed_speed_distance is still available as raw values if the user
    needs it for whatever reason (mostly since I didn't bother finding out
    how to delete it), but again, it returns raw values now. (The speed
    and distance fields return the expected decoded m/s and m units.)

    Finally, the code now de-wraps the distance field so it doesn't wrap
    around every 256 meters. Again, decode.c is used as a reference here,
    since I couldn't immediately find the spec for this in the PDFs.

diff --git a/fitparse/base.py b/fitparse/base.py
index 6e11afd..164d902 100644
--- a/fitparse/base.py
+++ b/fitparse/base.py
@@ -58,6 +58,7 @@ class FitFile(object):

         self._last_offset = 0
         self._reference_timestamp = None
+        self._accumulated_distance = 0
         self._global_messages = {}
         self.definitions = []
         self.records = []
@@ -177,6 +178,8 @@ class FitFile(object):

         fields = []
         dynamic_fields = {}
+        raw_speed = None
+        raw_distance = None

         for i, (field, f_size) in enumerate(definition.fields):
             f_raw_data, = self._struct_read(field.type.get_struct_fmt(f_size), definition.arch)
@@ -186,6 +189,9 @@ class FitFile(object):
             if field.name == r.COMPRESSED_TIMESTAMP_FIELD_NAME and \
                field.type.name == r.COMPRESSED_TIMESTAMP_TYPE_NAME:
                 self._reference_timestamp = f_raw_data
+            elif field.name == r.COMPRESSED_SPEED_DISTANCE_FIELD_NAME and \
+                 field.type.name == r.COMPRESSED_SPEED_DISTANCE_TYPE_NAME:
+                raw_speed, raw_distance = bound_field.data

             fields.append(bound_field)

@@ -220,8 +226,16 @@ class FitFile(object):
                 fields.append(r.BoundField(timestamp, ts_field))
                 self._last_offset = header.seconds_offset

-        # XXX -- do compressed speed distance decoding here, similar to compressed ts
-        # ie, inject the fields iff they're in definition.type.fields
+            # XXX -- the conversion back and forth to raw values here is a bit icky.
+            speed_field = definition.type.fields.get(r.SPEED_FIELD_DEF_NUM)
+            if speed_field and raw_speed is not None:
+                fields.append(r.BoundField(raw_speed * 1000. / 100., speed_field))
+
+            if raw_distance is not None:
+                self._accumulated_distance += (raw_distance - self._accumulated_distance) & 0xFFF;
+                distance_field = definition.type.fields.get(r.DISTANCE_FIELD_DEF_NUM)
+                if distance_field:
+                    fields.append(r.BoundField(self._accumulated_distance * 100. / 16., distance_field))

         data = r.DataRecord(header, definition, fields)

diff --git a/fitparse/records.py b/fitparse/records.py
index a0c27d5..5e9beb6 100644
--- a/fitparse/records.py
+++ b/fitparse/records.py
@@ -35,9 +35,13 @@ MESSAGE_DATA = 0
 LITTLE_ENDIAN = 0
 BIG_ENDIAN = 1

+SPEED_FIELD_DEF_NUM = 6
+DISTANCE_FIELD_DEF_NUM = 5
 TIMESTAMP_FIELD_DEF_NUM = 253
 COMPRESSED_TIMESTAMP_FIELD_NAME = 'timestamp'
 COMPRESSED_TIMESTAMP_TYPE_NAME = 'date_time'
+COMPRESSED_SPEED_DISTANCE_FIELD_NAME = 'compressed_speed_distance'
+COMPRESSED_SPEED_DISTANCE_TYPE_NAME = 'record-compressed_speed_distance'

 UNKNOWN_FIELD_NAME = 'unknown'

@@ -299,13 +303,12 @@ _convert_local_date_time = lambda x: datetime.datetime.fromtimestamp(631065600 +
 _convert_bool = lambda x: bool(x)


-# XXX -- untested
 # see FitSDK1_2.zip:c/examples/decode/decode.c lines 121-150 for an example
 def _convert_record_compressed_speed_distance(raw_data):
     first, second, third = (ord(b) for b in raw_data)
-    speed = first + (second & 0b1111)
+    speed = first + ((second & 0b1111) << 8)
     distance = (third << 4) + ((second & 0b11110000) >> 4)
-    return speed / 100. / 1000. * 60. * 60., distance / 16.
+    return speed, distance


 class MessageIndexValue(int):
'''

FitFile Parse error on Activity File with Developer data

r = FitFile('C:/Downloads/1484672807-GIR.fit')

r.parse()

FitParseErrorTraceback (most recent call last)
in ()
----> 1 r.parse()

C:\Users\e408191\AppData\Local\Continuum\Anaconda2\lib\site-packages\fitparse\base.pyc in parse(self)
424
425 def parse(self):
--> 426 while self._parse_message():
427 pass
428

C:\Users\e408191\AppData\Local\Continuum\Anaconda2\lib\site-packages\fitparse\base.pyc in _parse_message(self)
130 message = self._parse_definition_message(header)
131 else:
--> 132 message = self._parse_data_message(header)
133
134 self._messages.append(message)

C:\Users\e408191\AppData\Local\Continuum\Anaconda2\lib\site-packages\fitparse\base.pyc in _parse_data_message(self, header)
265 if not def_mesg:
266 raise FitParseError('Got data message with invalid local message type %d' % (
--> 267 header.local_mesg_num))
268
269 raw_values = self._parse_raw_values_from_data_message(def_mesg)

FitParseError: Got data message with invalid local message type 4

1484672807-GIR.zip

FIT containing multiple activities?

More of a question than an issue. Can you let me know how to determine which records align with which activity in instances where multiple activities have been included in one FIT file (e.g. triathlon with swim, transition, bike, transition, run)?

The records pull in fine but I'm trying to segregate the records into distinct sets for each activity.

Thanks in advance for any suggestions.

Can't open FIT file

Is that project still alive?
I tried this below on a file from a forerunner

from fitparse import FitFile
fitfile = FitFile("76NE5832.FIT")

That result as

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/fitparse/base.py", line 38, in __init__
    self._parse_file_header()
  File "/usr/local/lib/python2.7/dist-packages/fitparse/base.py", line 123, in _parse_file_header
    self._read_and_assert_crc(allow_zero=True)
  File "/usr/local/lib/python2.7/dist-packages/fitparse/base.py", line 83, in _read_and_assert_crc
    crc_expected, crc_actual = self._crc, self._read_struct('H')
  File "/usr/local/lib/python2.7/dist-packages/fitparse/base.py", line 75, in _read_struct
    raise FitEOFError("Tried to read %d bytes from .FIT file but got %d" % (size, len(data)))
fitparse.utils.FitEOFError: Tried to read 2 bytes from .FIT file but got 0

fitparse using python3

Will it work with python3 if yes could guide me how

Feature request: read FIT data from a string instead of a file

This could of course be solved by reading the data to a temporary file and then loading that, but that seems a bit unnecessary. I'd like this support this a bit more cleanly in my FIT to TCX script -- make it read from standard input and write to standard output.

TypeError basestring type cannot be instantiated

After upgrading fitparse to the latest (master branch), one of my standard test files fails to parse. This is a file produced by a NK SpeedCoach GPS 2 device. The file parses well on SportTracks, Strava, etc.

C:\Users\e408191\AppData\Local\Continuum\Anaconda2\lib\site-packages\fitparse\base.pyc in _parse_raw_values_from_data_message(self, def_mesg)
244 is_byte = base_type.name == 'byte'
245 # Struct to read n base types (field def size / base type size)
--> 246 struct_fmt = str(int(field_def.size / base_type.size)) + base_type.fmt
247
248 # Extract the raw value, ask for a tuple if it's a byte type

TypeError: The basestring type cannot be instantiated

davidfit.zip

FitHeaderError: Invalid .FIT File Header

Hi
When I an parsing fit files which are stored in the database I got this error FitHeaderError: Invalid.FIT File Header only for one fit file, I am able to parse the remaining fit files

[WIP] travis ci for project

I'd like to add travis ci scripts for python-fitparse

CSV output support

I added simple CSV output support with python csv library. It is in my branch. If this would be something you would like I'll clean it up and add it properly.

Currently messages are read and saved in list. Then if csv output is requested message list is lopped over and field names are saved of all fields that have non None value. Then message list is read again and DictWriter is used with previously saved fields list to save fields as dictionaries with only keys that were previously found.

If verbose is used. Message list is read again and printed.

I'm not sure if it is better to save messages to list or read whole FIT file 3 times and if this is the best way to choose which field types do we want to save to CSV. This could also be configurable since I found that out of 10+ fields only 4 make sense in my monitor fit files.

Error when updating profile version

When trying to update the sdk version to 20.66.00, I get the following error:

> python scripts/generate_profile.py /path/to/fit-sdk.zip fitparse/profile.py
Traceback (most recent call last):
  File "scripts/generate_profile.py", line 548, in <module>
    main(xls, profile)
  File "scripts/generate_profile.py", line 506, in main
    xls_file, profile_version = get_xls_and_version_from_zip(input_xls_or_zip)
  File "scripts/generate_profile.py", line 486, in get_xls_and_version_from_zip
    archive.open('c/fit.h').read().decode(),
  File "/Users/pokey/.pyenv/versions/3.6.5/lib/python3.6/zipfile.py", line 1352, in open
    zinfo = self.getinfo(name)
  File "/Users/pokey/.pyenv/versions/3.6.5/lib/python3.6/zipfile.py", line 1281, in getinfo
    'There is no item named %r in the archive' % name)
KeyError: "There is no item named 'c/fit.h' in the archive"

I believe this is because the desired path is actually not at the root of the zip, but rather just below the top-level FitSDKRelease_20.66.00 directory

Ubuntu 12.04.5 LTS error

I'm not quite sure if this is a real issue, but I had no problem to run a script on macOs localhost,
but when running in production on a Ubuntu 12.04.5 LTS server, I got this error:

Traceback (most recent call last):
  File "/path/to/myscript.py", line 10, in <module>
    for record in fitfile.get_messages('record'):
  File "/usr/local/lib/python2.7/dist-packages/fitparse/base.py", line 450, in get_messages
    message = self._parse_message()
  File "/usr/local/lib/python2.7/dist-packages/fitparse/base.py", line 160, in _parse_message
    message = self._parse_data_message(header)
  File "/usr/local/lib/python2.7/dist-packages/fitparse/base.py", line 318, in _parse_data_message
    header.local_mesg_num))
fitparse.utils.FitParseError: Got data message with invalid local message type 11

I'm asking for some help if you got any idea about this? Many thanks.

exception in parse_string

FitParseError invalid field size 227 for type 'sint32'

After upgrading fitparse to the latest (master branch), one of my standard test files fails to parse. This is a file produced by a NK SpeedCoach GPS 2 device. The file parses well on SportTracks, Strava, etc.

C:\Users\e408191\AppData\Local\Continuum\Anaconda2\lib\site-packages\fitparse\base.pyc in _parse_definition_message(self, header)
196 # examples in the wild. For now, just throw an exception
197 raise FitParseError("Invalid field size %d for type '%s' (expected a multiple of %d)" % (
--> 198 field_size, base_type.name, base_type.size))
199
200 # If the field has components that are accumulators

FitParseError: Invalid field size 227 for type 'sint32' (expected a multiple of 4)

3x250m.zip

Fails parse of `developer_data_id` record without `application_id`

When trying to parse a fit file I'm running into trouble. This is what I get back running fitdump on master:

Traceback (most recent call last):
  File "./scripts/fitdump", line 92, in <module>
    main()
  File "./scripts/fitdump", line 86, in main
    for n, message in enumerate(messages, 1):
  File "/media/BigHDD/Home stuff/Dropbox/Dropbox/Personal/Programming/python-fitparse/fitparse/base.py", line 450, in get_messages
    message = self._parse_message()
  File "/media/BigHDD/Home stuff/Dropbox/Dropbox/Personal/Programming/python-fitparse/fitparse/base.py", line 163, in _parse_message
    add_dev_data_id(message)
  File "/media/BigHDD/Home stuff/Dropbox/Dropbox/Personal/Programming/python-fitparse/fitparse/records.py", line 367, in add_dev_data_id
    application_id = message.get('application_id').raw_value
AttributeError: 'NoneType' object has no attribute 'raw_value'

I believe this is due to fitparse trying to read a nonexisting field in a developer_data_id record. This file has only the developer_data_index and not application_id which fitparse seems to always want to read.

New release?

Hi, the version on pypi is a bit old and doesn't contain the fix to #15 for example. Would it be possible to release a new version there soon?

Timestamps don't have timezone information

I don't know if FIT-files keep timezone information or if one is implied somehow. If that is the case it would be nice to have timezone information in the datetime objects created by fitparse.

Coordinates units?

Sorry to bother you guys, but in my .fit file the output of fitparse for laps and records are like:

"start_position_lat": 299017398,
"start_position_long": 1451139338,

What kind of unit is this?
For information, desired output is:

"start_position_lat" : 25.06334876641631,
"start_position_long":: 121.6330941952765,

I get them for another file, they should be the same - but I'm currently unable to do the conversion between each other.
Any help is greatly appreciated.

Smart Mode Logic

I have one more question (not an issue) as we've been unable to establish the exact logic Garmin uses for smart recording mode. I'm curious if you know how it works or have any documents that detail it?

We have a function to expand a stream that has been recorded with smart mode on. Right now we propagate values back in time instead of pulling the last value forward.

For example:

time[0] = 1; watts[0] = 100
time[1] = 2; watts[1] = 110
time[2] = 4; watts[2] = 115

yields [push back]:

watts[0] = 110
watts[1] = 110
watts[2] = 115
watts[3] = 115

instead of [pull forward]:
watts[0] = 110
watts[1] = 110
watts[2] = 110
watts[3] = 115

Do you know which approach is proper?

Thanks in advance for any pointers.

compressed timestamps

I think line 123 of base.py should be:

seconds_offset = header_data & 0b11111 # bits 0-4

currently it takes the last four instead of last five bits.

Tim

time stamp calculation error with compressed header

I am dowloading fit files from FR70 using garmin-forerunner-610-extractor.
I have found one issue concerning the time stamp. for hr data, records ares spaced by 5 seconds but I got variable steps using fitparse.

Looking at the specifications, I have found some mistakes and I am proposing the following solution:

def _parse_record_header(self):
    header_data, = self._struct_read(FitFile.RECORD_HEADER_FMT)

    header_type = self._get_bit(header_data, 7)

    if header_type == r.RECORD_HEADER_NORMAL:
        message_type = self._get_bit(header_data, 6)
        local_message_type = header_data & 0b11111  # Bits 0-4
        # TODO: Should we set time_offset to 0?
        return r.RecordHeader(
            header_type, message_type, local_message_type, None,
        )
    else:
        # Compressed timestamp
        local_message_type = (header_data >> 5) & 0b11  # bits 5-6
        #old
        #seconds_offset = header_data & 0b1111  # bits 0-3
        #new
        seconds_offset = header_data & 0b11111  # bits 0-4
        return r.RecordHeader(
            header_type, r.MESSAGE_DATA, local_message_type, seconds_offset)

in base.py,_parse_data_record

    if header.type == r.RECORD_HEADER_COMPRESSED_TS:
        ts_field = definition.type.fields.get(r.TIMESTAMP_FIELD_DEF_NUM)
        if ts_field:
        #new (ref: D00001275 Flexible and Interoperable Data Transfer (FIT) Protocol Rev 1.3.pd, page 17f)
        if header.seconds_offset >=  (self._last_timestamp & 0x0000001F):
               timestamp = (self._last_timestamp & 0xFFFFFFE0)  + header.seconds_offset
            else:
               timestamp = (self._last_timestamp & 0xFFFFFFE0)  + header.seconds_offset + 0x20
            #old
            #timestamp = self._last_timestamp + header.seconds_offset
            fields.append(r.BoundField(timestamp, ts_field))
            self._last_timestamp = timestamp

With these, I can recover my 5 seconds step and proper activity duration. I still have some concern about the initial time stamp to use because I found discrepancies between garmin connect and garmin-extractor/fitparse.

It's a piece of software you've done, thanks

Philippe

compressed / plugin framework HR data with event_timestamps not in datetime

Thanks for this awesome library first of all! I just have some questions on this file that I couldn't seem to wrap my head around.

When I parse the HR data in the attached file I found some event_timestamps that are represented with a 32 bit integer:

[<FieldData: filtered_bpm: (91, 90, 90, 90, 90, 90, 90, 90) [bpm], def num: 6, type: uint8 (uint8), raw value: (91, 90, 90, 90, 90, 90, 90, 90)>,
<FieldData: event_timestamp: 103767638410559546792952933162482 [s], def num: 9, type: uint32 (uint32), raw value: 103767638410559546792952933162482>,
<FieldData: event_timestamp: 103770559888591208310651742719474 [s], def num: 9, type: uint32 (uint32), raw value: 103770559888591208310651742719474>,
<FieldData: event_timestamp: 103773481366622869828350552276466 [s], def num: 9, type: uint32 (uint32), raw value: 103773481366622869828350552276466>,
<FieldData: event_timestamp: 103776402844654531346049361833458 [s], def num: 9, type: uint32 (uint32), raw value: 103776402844654531346049361833458>,
<FieldData: event_timestamp: 103779324322686192863748171390450 [s], def num: 9, type: uint32 (uint32), raw value: 103779324322686192863748171390450>,
<FieldData: event_timestamp: 103782245800717854381446980947442 [s], def num: 9, type: uint32 (uint32), raw value: 103782245800717854381446980947442>,
<FieldData: event_timestamp: 103785167278749515899145790504434 [s], def num: 9, type: uint32 (uint32), raw value: 103785167278749515899145790504434>,
<FieldData: event_timestamp: 103788088756781177416844600061426 [s], def num: 9, type: uint32 (uint32), raw value: 103788088756781177416844600061426>,
<FieldData: event_timestamp: 103791010234812838934543409618418 [s], def num: 9, type: uint32 (uint32), raw value: 103791010234812838934543409618418>,
<FieldData: event_timestamp: 103793931712844500452242219175410 [s], def num: 9, type: uint32 (uint32), raw value: 103793931712844500452242219175410>,
<FieldData: event_timestamp_12: (242, 125, 9, 151, 112, 9, 151, 112, 9, 151, 112, 9), def num: 10, type: byte (byte), raw value: (242, 125, 9, 151, 112, 9, 151, 112, 9, 151, 112, 9)>]

Any ideas on how I can convert this back to a ISO Date time / unix timestamp / datetime object.

pluginHR.zip

Protocol and profile versions are not float

The FitFile.protocol_version, profile_version are just strings "major.minor" (i.e. "int dot int" format), not floats. Floats are inaccurate representations.
I propose to create a custom class FitVersion with fields major, minor and __str__ func with proper formatting. And maybe magic functions for comparison (__eq__, __gt__, ...). Can make a PR for that.

Feature request: creating FIT file from any readable format (CSV, XML, JSON or any other)

I can't use java converter tool from FIT dev pack because I have to use a little modified fit files. And I can't fix this issue because of lack java knowledge. If there is some possibility of creating such a tool on python?

Project status

Is this project staled?

Installing from pip produces error message

Trying to install python-fitparse from pip produces the following error:

pip install python-fitparse
Collecting python-fitparse
Could not find a version that satisfies the requirement python-fitparse (from versions: )
No matching distribution found for python-fitparse

Python: 3.7.0
pip: 18.1
OS: OS X 10.13.4

Can't parse FIT file when ride on in-door bike trainer

Hi,

I rode bike on indoor trainer using Garmin Edge800.
python-fitparse can't parse this file.

Traceback (most recent call last):
File "test.py", line 13, in
activity.parse()
File "/Users/mitsukuni/tmp/python-fitparse/fitparse/activity.py", line 29, in parse
return_value = super(Activity, self).parse(_args, *_kwargs)
File "/Users/mitsukuni/tmp/python-fitparse/fitparse/base.py", line 82, in parse
self._parse_file_header()
File "/Users/mitsukuni/tmp/python-fitparse/fitparse/base.py", line 288, in _parse_file_header
throw_exception("Invalid header size")
File "/Users/mitsukuni/tmp/python-fitparse/fitparse/base.py", line 278, in throw_exception
raise FitParseError("Bad .FIT file header: %s" % error)
fitparse.exceptions.FitParseError: Bad .FIT file header: Invalid header size

Exception in add_dev_field_description at line 395

Recently started to see an exception when parsing FIT files. I believe it may be related to them originating from a wahoo element device.

The error is a key error for the BASE_TYPES dict. The key is 143 which in hex is 0x8F. Some of the other variables include:

<DataMessage: field_description (#206) -- local mesg: #0, fields: [developer_data_index: 3, field_definition_number: 0, fit_base_type_id: uint64, field_name: time_ms, units: ms]>

My guess is this is related to 64bit data types coming in the messages.

Can this be solved by updating the profile to a later SDK?

Thanks for any guidance...

Came across a FIT file that cannot be parsed

2017-10-04-20-03-29.fit.zip
I came across a fit file that cannot be successfully parsed but it appears that the file can be imported by other libs like Golden Cheetah so I thought I would bring it to your attention.

Here are the logs from the error when i try to parse the file.
Traceback (most recent call last): File "fitparseTest.py", line 7, in <module> for record in fitfile.get_messages('record'): File "/Users/wingchiuhau/.virtualenvs/testfit/lib/python2.7/site-packages/fitparse/base.py", line 450, in get_messages message = self._parse_message() File "/Users/wingchiuhau/.virtualenvs/testfit/lib/python2.7/site-packages/fitparse/base.py", line 160, in _parse_message message = self._parse_data_message(header) File "/Users/wingchiuhau/.virtualenvs/testfit/lib/python2.7/site-packages/fitparse/base.py", line 340, in _parse_data_message cmp_raw_value, accumulator[component.def_num], component.bits, File "/Users/wingchiuhau/.virtualenvs/testfit/lib/python2.7/site-packages/fitparse/base.py", line 307, in _apply_compressed_accumulation base_value = raw_value + (accumulation & ~max_mask) TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

MessageType records field time_from_course has wrong name

The MessageType(name='record') has field 11 named 'event, but it's time_from_course in the Profile.xls. I report it as an issue of the genprofile.py probably?