errors: invalid tcpdump header about sbedecoder

[tfgm-bud]

@rumcode Thanks for trying the SBE decoder. I took a look at those sample files at http://www.cmegroup.com/market-data/datamine-historical-data.html. While the files there are MDP 3.0 they aren't SBE encoded -- they are plan ASCII FIX files:

$ gunzip XCME_MD_ES_FUT_20160315.gz
$ cat XCME_MD_ES_FUT_20160315 | tr '\001' '\|' | head
1128=9|9=131|35=f|49=CME|34=5915937|52=20160314214500009131503|60=20160314214500001946065|75=20160315|5799=10000000|1151=ES|326=21|327=0|1174=4|10=075|
1128=9|9=181|35=X|49=CME|75=20160315|34=5915938|52=20160314214500437290133|60=20160314214500437207105|5799=10000100|268=1|279=0|269=1|48=49705|55=ESH6|83=2434488|270=201875.0|271=1|346=1|1023=1|10=153|
1128=9|9=182|35=X|49=CME|75=20160315|34=5915939|52=20160314214500964855160|60=20160314214500964728850|5799=10000100|268=1|279=0|269=1|48=49705|55=ESH6|83=2434489|270=202425.0|271=1|346=1|1023=10|10=228|
1128=9|9=181|35=X|49=CME|75=20160315|34=5915940|52=20160314214501236422156|60=20160314214501236279685|5799=10000100|268=1|279=1|269=1|48=6505|55=ESM6|83=4428027|270=201000.0|271=14|346=5|1023=2|10=146|
1128=9|9=181|35=X|49=CME|75=20160315|34=5915941|52=20160314214501260912493|60=20160314214501260788281|5799=10000100|268=1|279=1|269=1|48=49705|55=ESH6|83=2434490|270=201875.0|271=4|346=2|1023=1|10=164|
1128=9|9=183|35=X|49=CME|75=20160315|34=5915942|52=20160314214501469025445|60=20160314214501468880637|5799=10000100|268=1|279=1|269=0|48=49705|55=ESH6|83=2434491|270=201575.0|271=12|346=4|1023=10|10=016|
1128=9|9=181|35=X|49=CME|75=20160315|34=5915943|52=20160314214502284489462|60=20160314214502284402077|5799=10000100|268=1|279=0|269=1|48=49705|55=ESH6|83=2434492|270=201900.0|271=3|346=1|1023=2|10=160|
1128=9|9=181|35=X|49=CME|75=20160315|34=5915944|52=20160314214502900288541|60=20160314214502900214739|5799=10000100|268=1|279=0|269=1|48=49705|55=ESH6|83=2434493|270=201925.0|271=3|346=1|1023=3|10=161|
1128=9|9=181|35=X|49=CME|75=20160315|34=5915945|52=20160314214503380287759|60=20160314214503380201283|5799=10000100|268=1|279=0|269=1|48=49705|55=ESH6|83=2434494|270=201950.0|271=3|346=1|1023=4|10=168|
1128=9|9=181|35=X|49=CME|75=20160315|34=5915946|52=20160314214503973789379|60=20160314214503973709166|5799=10000100|268=1|279=0|269=1|48=49705|55=ESH6|83=2434495|270=202000.0|271=3|346=1|1023=6|10=193|

The upside for you is that you don't need to decode those files to use them. The downside is that you can't take advantage of sbe decoder's ability to print the tag=value pairs in a more human readable format or the sbe decoder's ability to assemble market data books. :-(

Since it is just FIX, it is pretty easy to work with though. A quick and dirty approach to get something more readable is to just use "split" to parse the lines into a list of key value pairs and then use a dictionary lookup to translate the keys (and values as necessary) to a textual form.

If you need book building, you probably could adapt our 'orderbook' pretty easily too:

https://github.com/tfgm/sbedecoder/tree/master/orderbook