tfgm / sbedecoder Goto Github PK
View Code? Open in Web Editor NEWPython based Simple Binary Encoding (SBE) decoder
License: MIT License
Python based Simple Binary Encoding (SBE) decoder
License: MIT License
suggestion:
paddy:sbedecoder dev$ git diff
diff --git a/setup.py b/setup.py
index df311f4..bfd10f8 100644
--- a/setup.py
+++ b/setup.py
@@ -9,7 +9,7 @@ setup(
license="MIT",
keywords="sbe mdp3 orderbook",
url="https://github.com/tfgm/sbedecoder",
- packages=['sbedecoder', 'mdp'],
+ packages=['sbedecoder', 'mdp', 'mdp.orderbook'],
scripts=['scripts/mdp_decoder.py', 'scripts/mdp_book_builder.py'],
long_description='see https://github.com/tfgm/sbedecoder/INSTALL.md',
install_requires=['dpkt', 'lxml', 'nose', 'mako', 'autopep8'],
It's still problematic to base is_string_type
on the field's semantic_type
.
Though #17 fixes single-character strings, there is an issue with integral
types having semantic type of String
(e.g. SecurityID and OrderID).
The value
property tries to split a raw integral value on its null bytes as
if it were a string.
I'll follow up with a pull request shortly.
Hi. I'm liking the prospects of using your library. Just ran into this and thought you might want to fix it:
from sbedecoder.message import SBEMessage, SBERepeatingGroupIterator
ImportError: cannot import name SBERepeatingGroupIterator
It's in the tmpl
file and so in the generated code.
The classes generated with generator do not support subgroups.
If you are using the non-generated SbeSchema.parse
you'll be able to parse messages that have groups with subgroups.
Hi,
I know this is a duplicate of a previous issue. However, its unclear to me if it was successfully resolved. So my question is if the software supports the MDP 3.0 market depth messages found from: https://www.cmegroup.com/confluence/display/EPICSANDBOX/Market+Depth.
If not, could you guide how to contribute to the project. If yes, where can one find the template?
Thanks! :)
Hi, this is not really an issue, but I'm not sure where to place the question.
Could you post some data or a a http link to sbe data that you know works with your program? I would like to step through it, but the data I have, as you pointed out is not in sbe format, although it is from the CME.
Just a meg or two would be nice.
Thanks in advance.
You can currently use pip install git+https://github.com/tfgm/sbedecoder
to pip install the sbedecoder package, but it would be great if this could be uploaded to pypi.org. It would also help with project visibility as a lot of people can easily search through their web ui for this type of package.
Thank you!
It seems like message header size is fixed at 10 byte's which is unreasonable assumption. I think it's more accurate to use the parsed header's actual size and pass that as the header size to use. For exampel look at this monkey patched version, since my header is 8 bytes instead of 10 (which is actually the sample given in most of SBE's docs):
def _fixed_construct_body(self, message, field_offset, endian):
message_id = int(message['id'])
message_type = self.get_message_type(message_id)
self._add_fields(field_offset, message, message_type, endian, add_header_size=True, header_size=message_type.header_size)
self._add_groups(message, message_type, endian)
def _fixed_add_fields(self, field_offset, entity, entity_type, endian, add_header_size=True, header_size=8):
# Now run through the remaining types and update the fields
for field_type in entity.get('fields', []):
field = self._build_message_field(field_type, field_offset, header_size=header_size, endian=endian, add_header_size=add_header_size)
field_offset += field.field_length
entity_type.fields.append(field)
# make it an attribute too
setattr(entity_type, field.name, field)
SBESchema._construct_body = _fixed_construct_body
SBESchema._add_fields = _fixed_add_fields
So when using add_header_size
the message_type.header_size
is passed instead of 10. It can probably be merged into 1 parameter and determine if it's none vs not too. I can submit pull request if approved.
Hi
I'm looking for something to decode CME Market Depth data. I found your programs. I found one closed issue about invalid tcpdump header. Do you have plan to include parser for CME market depth data? If you do I would like to help.
Best
What version are you using? There are error messages if I try to use 3.5.1, but not with 2.7. Also, dpkt will not install successfully under 3.5.1, but it will under 2.7
Currently only supports Python 2.7
Hi, I've been trying to get your program to work, but I get a message -packages\dpkt\pcap.py", line 122, in init raise ValueError('invalid tcpdump header') ValueError: invalid tcpdump header
My xml file is from the CME (followed your link) and so is the data which is e-mini futures posted at http://www.cmegroup.com/market-data/datamine-historical-data.html
I was running 2.7 python (anaconda) and ran setup.py before running mdp_decoder.py
I ran your tests and they were fine, but it seems that you don't use real data in your test.
Any suggestions?
Thanks in advance.
Are any plans to update the project to support the recent changes to the sbe xml spec? Specifically support for ref tag in composite elements.
https://github.com/real-logic/simple-binary-encoding/wiki/FIX-SBE-XML-Primer#composite-element
Thanks in advance!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.