lorenzo-stoakes / janus Goto Github PK

Seeing warnings about 0 updates - is this appropriate? Is there an issue?

[2020-05-22 11:47:34.584] [info] Switched to log level INFO
[2020-05-23 11:58:27.489] [warning] Only 0 received from /home/lorenzo/data/json/market_stream/1590163202631.json - expected minimum 2, ignoring.
[2020-05-23 12:06:27.992] [warning] Only 0 received from /home/lorenzo/data/json/market_stream/1590163202631.json - expected minimum 2, ignoring.
[2020-05-23 12:06:48.008] [warning] Only 0 received from /home/lorenzo/data/json/market_stream/1590163202631.json - expected minimum 2, ignoring.

Seeing the below error whenever fast-forwarding to end of market - investigate.

Error in apply next index, aborting.

Stats are currently skewed incorrectly by market suspension not being taken into account.

Seeing issues with remote hanging up and us trying to use an invalid client. Handle correctly.

[2020-05-21 17:00:08.063] [info] Saving metadata to /home/lorenzo/data/json/meta/1590076801820.json...
[2020-05-21 17:00:08.067] [info] Creating stream output file /home/lorenzo/data/json/market_stream/1590076801820.json...
[2020-05-21 17:00:08.067] [info] Starting stream...
[2020-05-22 15:09:15.450] [error] tls:stream-api.betfair.com:443: Read: SSL - The operation timed out
[2020-05-22 15:09:15.450] [error] Attempt to use invalid client.
[2020-05-22 15:09:15.450] [error] Attempt to use invalid client.
[2020-05-22 15:09:15.450] [error] Attempt to use invalid client.
[2020-05-22 15:09:15.450] [error] Attempt to use invalid client.
[2020-05-22 15:09:15.450] [error] Attempt to use invalid client.
[2020-05-22 15:09:15.450] [error] Attempt to use invalid client.
[2020-05-22 15:09:15.450] [error] Attempt to use invalid client.
[2020-05-22 15:09:15.450] [error] Attempt to use invalid client.
[2020-05-22 15:09:15.450] [error] Attempt to use invalid client.
[2020-05-22 15:09:15.450] [critical] 10 errors, cap is 10, aborting...
[2020-05-22 15:09:15.453] [info] Logging out...
[2020-05-22 15:09:17.619] [info] jupiter early-130-g9c7bd31
```

1. When waiting to send data the cores busy-wait which is wasteful.
2. While loading all cores wait to start when they could immediately start working instead.

Often seeing 'subscribed to 220 markets' etc. which is clearly incorrect.

Rewinding currently reapplies the entire market up to the point rewound to, obviously this is massively inefficient!

'Markets moved under a new eventId - In certain circumstances, a market may move from one eventId to another due to actions performed by our Exchange Operations team. This will cause the Exchange Stream API to hold two copies of the market in its cache and the initial image of the market provided will therefore contain both copies of the market. In these circumstances further Stream API updates will only be sent for the latest version of the market. You can identify the latest version of the market using the "version" parameter returned in the initial image and should only store the market with the higher version number."

Strange to see such a variance, could maybe spread work more efficiently using a different measure, or perhaps there is a bug?

Occurred in catch1 initial run.

Mean is heavily impacted by outliers, add median interval statistics to avoid this.

Need to allocate state rather than put everything on the stack because some of these objects might be pretty huge!

Appears that it doesn't.

At this point it just feels arbitrary. Remove altogether.

Update apollo to allow it to retain per-market data in memory loaded locally for each node. Then allow strategies to be loaded by dynamically loading them rather than having to reload all data each time.

Perhaps this could be achieved more easily by storing binary data in tmpfs or similar?

This would require a significant rework of apollo and analyse.

Create stats dump, a tool for outputting stats per market like the dump tool.

Add timestamp of bet placement time to bets.

Sim currently assumes instant matching, which is unrealistic. Add a delay and make in configurable, should allow inplay too.

Hideous levels of nesting, the whole approach feels clunky, needs improvement.

When virtualising see entire blocks of unmatched coming in and out of existence - this doesn't seem right. Investigate and correct if necessary.

Despite the existence of the decimal7 implementation, decimal numbers are referenced as doubles throughout the codebase with the inherent issues around comparing doubles. This needs to be tamed.

Total first draft version at the moment, look at optimising with profiling data.