tfgm / sbedecoder Goto Github PK

Hi, I've been trying to get your program to work, but I get a message -packages\dpkt\pcap.py", line 122, in init raise ValueError('invalid tcpdump header') ValueError: invalid tcpdump header

My xml file is from the CME (followed your link) and so is the data which is e-mini futures posted at http://www.cmegroup.com/market-data/datamine-historical-data.html

I was running 2.7 python (anaconda) and ran setup.py before running mdp_decoder.py

I ran your tests and they were fine, but it seems that you don't use real data in your test.

Any suggestions?

Thanks in advance.

Hi. I'm liking the prospects of using your library. Just ran into this and thought you might want to fix it:

    from sbedecoder.message import SBEMessage, SBERepeatingGroupIterator
ImportError: cannot import name SBERepeatingGroupIterator

It's in the tmpl file and so in the generated code.

Currently only supports Python 2.7

The classes generated with generator do not support subgroups.

If you are using the non-generated SbeSchema.parse you'll be able to parse messages that have groups with subgroups.

https://github.com/real-logic/simple-binary-encoding/wiki/Message-Versioning

Hi, this is not really an issue, but I'm not sure where to place the question.

Could you post some data or a a http link to sbe data that you know works with your program? I would like to step through it, but the data I have, as you pointed out is not in sbe format, although it is from the CME.

Just a meg or two would be nice.

Thanks in advance.

Hi,

I know this is a duplicate of a previous issue. However, its unclear to me if it was successfully resolved. So my question is if the software supports the MDP 3.0 market depth messages found from: https://www.cmegroup.com/confluence/display/EPICSANDBOX/Market+Depth.

If not, could you guide how to contribute to the project. If yes, where can one find the template?

Thanks! :)

Hi
I'm looking for something to decode CME Market Depth data. I found your programs. I found one closed issue about invalid tcpdump header. Do you have plan to include parser for CME market depth data? If you do I would like to help.
Best

Are any plans to update the project to support the recent changes to the sbe xml spec? Specifically support for ref tag in composite elements.
https://github.com/real-logic/simple-binary-encoding/wiki/FIX-SBE-XML-Primer#composite-element

Thanks in advance!

https://github.com/FIXTradingCommunity/fix-simple-binary-encoding

suggestion:

paddy:sbedecoder dev$ git diff
diff --git a/setup.py b/setup.py
index df311f4..bfd10f8 100644
--- a/setup.py
+++ b/setup.py
@@ -9,7 +9,7 @@ setup(
     license="MIT",
     keywords="sbe mdp3 orderbook",
     url="https://github.com/tfgm/sbedecoder",
-    packages=['sbedecoder', 'mdp'],
+    packages=['sbedecoder', 'mdp', 'mdp.orderbook'],
     scripts=['scripts/mdp_decoder.py', 'scripts/mdp_book_builder.py'],
     long_description='see https://github.com/tfgm/sbedecoder/INSTALL.md',
     install_requires=['dpkt', 'lxml', 'nose', 'mako', 'autopep8'],

You can currently use pip install git+https://github.com/tfgm/sbedecoder to pip install the sbedecoder package, but it would be great if this could be uploaded to pypi.org. It would also help with project visibility as a lot of people can easily search through their web ui for this type of package.

Thank you!

It's still problematic to base is_string_type on the field's semantic_type.
Though #17 fixes single-character strings, there is an issue with integral
types having semantic type of String (e.g. SecurityID and OrderID).

The value property tries to split a raw integral value on its null bytes as
if it were a string.

I'll follow up with a pull request shortly.

What version are you using? There are error messages if I try to use 3.5.1, but not with 2.7. Also, dpkt will not install successfully under 3.5.1, but it will under 2.7

It seems like message header size is fixed at 10 byte's which is unreasonable assumption. I think it's more accurate to use the parsed header's actual size and pass that as the header size to use. For exampel look at this monkey patched version, since my header is 8 bytes instead of 10 (which is actually the sample given in most of SBE's docs):

def _fixed_construct_body(self, message, field_offset, endian):
    message_id = int(message['id'])
    message_type = self.get_message_type(message_id)
    self._add_fields(field_offset, message, message_type, endian, add_header_size=True, header_size=message_type.header_size)
    self._add_groups(message, message_type, endian)

def _fixed_add_fields(self, field_offset, entity, entity_type, endian, add_header_size=True, header_size=8):
    # Now run through the remaining types and update the fields
    for field_type in entity.get('fields', []):
        field = self._build_message_field(field_type, field_offset, header_size=header_size, endian=endian, add_header_size=add_header_size)
        field_offset += field.field_length
        entity_type.fields.append(field)
        # make it an attribute too
        setattr(entity_type, field.name, field)

SBESchema._construct_body = _fixed_construct_body
SBESchema._add_fields = _fixed_add_fields

So when using add_header_size the message_type.header_size is passed instead of 10. It can probably be merged into 1 parameter and determine if it's none vs not too. I can submit pull request if approved.