wojciechzankowski / iextrading4j-hist

IEX Trading library to parse TOPS and DEEP multicast packets

Hi there, it is me 👋

Self-driven, passionate and committed software engineer specializing in market trading and asset management discipline. Flexible with languages, tools and everything that leads to good solution. Enjoying his life in the beautiful area of Zürich 🇨🇭. From time to time I commit to open source projects.

iextrading4j-hist's Issues

DeepSample program is throwing error org.pcap4j.core.PcapNativeException

data%2Ffeeds%2F20171104%2F20171104_IEXTP1_DEEP1.0.pcap.gz
Hi All,

Thanks for your support. I am looking for one small guidance/help.

I was trying to read Sample(PCAP format) file using below libraries and observed that it throws exception like below.
I was using api readDEEPsample() of DeepSample.java
iextrading4j-hist/iextrading4j-hist-samples/src/main/java/pl/zankowski/iextrading4j/hist/samples/DEEPSample.java

Exception:

org.pcap4j.core.PcapNativeException: pcap_open_offline_with_tstamp_precision is not supported by the pcap library installed in this environment.
at org.pcap4j.core.Pcaps.openOffline(Pcaps.java:234)

Libraries used :

iextrading4j-hist-api-1.0.0.jar
iextrading4j-hist-deep-1.0.0.jar
iextrading4j-hist-tops-1.0.0.jar
pcap4j-core-1.7.2.jar
pcap4j-packetfactory-static-1.7.2.jar

Sample pcap file :
data%2Ffeeds%2F20171104%2F20171104_IEXTP1_DEEP1.0.pcap.gz file which contains pcap file

Additional Info

Pcaps.libVersion WinPcap version 4.1.3 (packet.dll version 4.1.0.2980), based on libpcap version 1.0 branch 1_0_rel0b (20091008)

May I request you please to help me on this.

thanks
Vikas

Try to run tops sample

Hi, I tried to run your tops sample but I only get this output:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

I have installed npcap latest version and selected checkbox "Install with WinPcap API-compatible mode"

Can you support me?

Thanks a lot.

Issue : Some of Deeps/Tops History data is failed to read

Hi All,

Thanks for your support.

During parsing Deeps/Tops History data , There are mixed of a reading response. Some of data has been parsed and some of them got failed.
[](I have attached complete response of execution of Tops pcap file
TopsResponse.docx)
. I have highlighted error red color.

I am referring to DeepSample/TopsSample code from https://github.com/WojciechZankowski/iextrading4j-hist

I have tried this tops data(in gz format) attached here and got an issue.
data%2Ffeeds%2F20171104%2F20171104_IEXTP1_TOPS1.6.pcap.gz

May I request you please to help me on this. Please guide me how to handle such error.

Do i need to write our own wrapper class on open source library to tackle such error ?

thanks,
Vikas

packet.getPayload() returns null

import org.pcap4j.core.*;
import org.pcap4j.packet.Packet;
import pl.zankowski.iextrading4j.hist.api.message.IEXSegment;
import pl.zankowski.iextrading4j.hist.deep.IEXDEEPMessageBlock;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.GZIPInputStream;

public class IexDeepReader {
public static File unGzip(File infile, boolean deleteGzipfileOnSuccess) throws IOException {
GZIPInputStream gin = new GZIPInputStream(new FileInputStream(infile));
FileOutputStream fos = null;
try {
File folder = new File("temp");
if (!folder.isDirectory())
folder.mkdirs();

        File outFile = new File(folder, infile.getName().replaceAll("\\.gz$", ""));
        if (outFile.isFile())//already exists
            return outFile;

        fos = new FileOutputStream(outFile);
        byte[] buf = new byte[100000];
        int len;
        while ((len = gin.read(buf)) > 0) {
            fos.write(buf, 0, len);
        }

        fos.close();
        if (deleteGzipfileOnSuccess) {
            infile.delete();
        }
        return outFile;
    } finally {
        if (gin != null) {
            gin.close();
        }
        if (fos != null) {
            fos.close();
        }
    }
}

private void readDEEPsample() throws PcapNativeException, InterruptedException, NotOpenException, IOException {
    String filename = "data%2Ffeeds%2F20180628%2F20180628_IEXTP1_DEEP1.0.pcap.gz";
    PcapHandle handle = Pcaps.openOffline(unGzip(new File(filename), false).getPath(),
                                          PcapHandle.TimestampPrecision.NANO);

    handle.loop(-1, new PacketListener() {
        @Override
        public void gotPacket(Packet packet) {
            byte[] data = packet.getPayload().getPayload().getPayload().getRawData();
            IEXSegment block = IEXDEEPMessageBlock.createIEXSegment(data);
            System.out.println(block);
        }
    });

    handle.close();
}

public static void main(String[] args) {
    IexDeepReader reader = new IexDeepReader();
    try {
        reader.readDEEPsample();
    } catch (PcapNativeException e) {
        e.printStackTrace();
    } catch (InterruptedException e) {
        e.printStackTrace();
    } catch (NotOpenException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    }
}

}

Cannot parse IEXSaleConditionFlag

Describe the bug
Cannot parse IEXSaleConditionFlag.

To Reproduce
Steps to reproduce the behavior:

1. Try to parse TOPS data from 2018-06-22 using library
2. There is lots of erros saying that IEXSaleConditionFlag cannot be parse: -112

Expected behavior
Should parse whole file without erros

Additional context
TOPS data 2018-06-22

org.pcap4j.core.PcapNativeException: path_to_pcap: No such file or directory

Hi,
Thanks for the project. I try to get it run.

I just installed it and after some problems with the Npcap I got the output:
13:49:17.765 [main] INFO org.pcap4j.core.NativeMappings - Pcap4J successfully loaded a native pcap library: Npcap version 0.99-r7, based on libpcap version 1.8.1
Exception in thread "main" org.pcap4j.core.PcapNativeException: path_to_pcap: No such file or directory
at org.pcap4j.core.Pcaps.openOffline(Pcaps.java:241)
at pl.zankowski.iextrading4j.hist.samples.TOPSSample.readTOPSsample(TOPSSample.java:21)
at pl.zankowski.iextrading4j.hist.samples.TOPSSample.main(TOPSSample.java:17)

Then I configured the parameter to "C:\Program Files (x86)\Nmap" and then I got
Exception in thread "main" org.pcap4j.core.PcapNativeException: C:\Program Files (x86)\Nmap: Permission denied
at org.pcap4j.core.Pcaps.openOffline(Pcaps.java:241)
at pl.zankowski.iextrading4j.hist.samples.TOPSSample.readTOPSsample(TOPSSample.java:20)
at pl.zankowski.iextrading4j.hist.samples.TOPSSample.main(TOPSSample.java:16)

I set the Nmap directory to allow access, but this I get this exception.

Thanks for help

ERROR [org.pcap4j.core.PcapHandle] :: java.lang.OutOfMemoryError: GC overhead limit exceeded

Hi All,

https://api.iextrading.com/1.0/hist?date=20171117

During parsing PCAP data using open source pcap4j library, i am getting Error "java.lang.OutOfMemoryError: GC overhead limit exceeded".
I have attached complete Error Stack trace here as attachment.

My System configuration details :
RAM 8 GB
HardDisk 500 GB

There is no more process running in my system which consumes my hardware resources. but still i am getting same error.

I also restarted system , thinking that it might be intermittent issue but it keeps coming.

Kindly help me on this.

thanks
GCError.txt

IEX TOPS Message has to contain 42 bytes

Hi Team,

pl.zankowski.iextrading4j.hist.tops.IEXTOPSMessageBlock is throwing exception

Exception

[main] ERROR org.pcap4j.core.PcapHandle - The executor has thrown an exception.
java.lang.IllegalArgumentException: IEX TOPS Message has to contain 42 bytes

I observed that this error coming due to below condition
if (bytes.length != 42) {
throw new IllegalArgumentException("IEX TOPS Message has to contain 42 bytes");
}

May I know why this condition is added ?
Please advise me how to tackle such exception

thanks,
Vikas

org.pcap4j.core.PcapHandle - The executor has thrown an exception.