Orderbook Caching

Abstract

Add a Map cache for Bid and Ask orderbook Solana accounts. The cache will only be considered "valid" for 1200ms. To accomplish this, either include a timestamp with each Map entry, to see if it was recent enough to want it. Or use guava's time eviction based policies. Guava policies are preferred.

Use case

100 people are on the SOL/USDC order book. Normally this will add 100 RPC requests every 1200ms. By adding caching, each time they poll for data, it will first check if there was a cache update within the past 1200ms. If so, they will use the cache. Otherwise, they will poll and update the cache (and include new timestamp if necessary). After that is implemented, for 100 users, 99 will hit the cache and only 1 will experience a polling delay (every 1200ms).

A lot of code will return a null if the desired function/search/lookup returns no results. Replace these flows with the Optional keyword.

1. REST API at the path /api/serum/market/{marketId}/spread, takes in a marketId
2. Returns a response with at least a bid and ask value:

{
    "bid": 38.50,
    "ask": 38,60
}

Notes:
This will make two order book requests. Other tickets will be made for a "bid" only API, and an "ask" only API, to limit it to one RPC request. A website may just want the top bid to return as an active price for an asset, and the "bid" only API would suffice, and use half the load. In the future, market caching would slightly improve these response times as well.

Identify more large trading entities and add them to our order book reverse lookup.

Background:
We need the ability to see the TX that created a Serum order that was a taker in the event queue.

Feature Idea:
Search for the owner's recent transactions (10-20 if public RPC, switch to private). Best match versus a Serum instruction, or Serum inner instruction/CPI.

Additional Solution Notes:
SolanaJ may need an update to read instruction data of CPIs.
Linear: SRM-44

Background:
In the codebase, "Solana addresses" such as token mint, exist in both String format at the custom Pubkey class with a very inconsistent mix.

E.g. Most maps use String as the Key for <"Address", Value> mappings, presumably due to lack of confidence in Pubkey's hashCode implementation.

Feature Idea:
Change every KV map to use Pubkey class instead of String for the key, and any other similar String usage.

In combination with this ticket, create some tests in SolanaJ to test Pubkey hashing, if anything needs to be changed at the library level.

Additional Solution Notes:
Linear: SRM-43

Investigate and resolve. Maybe it's just from a bad marketId?

In the table header for each order book, or in a row below the header, display something like:
Total amount (Bids): $500,000
Total amount (Asks): $2,000,000

Background:
The order book table takes up the whole page.

Feature Idea:
Add a max height, which creates a scrollbar after that height.

Additional Solution Notes:
400-600px maybe. The goal is to place another row of data/tables beneath the scrollable order book.

Research how we can rank or quantify the age of a quote in the order book.

Some possibilities:

• Diff-ing the order book and recording timestamps.
• Investigate on-chain Slab data structure for a deterministic approach
• Correlate to TXID (somehow)

This metric would then be displayed to the user, quantitatively (data column), visually (color), or both, or something else creative.

Take periodic snapshots of the order book objects and store them in postgres every 12 hours. (optimal interval TBD)

Allow querying of the snapshots.

Display dataset in a table:

• Rows: Owner
• Columns: "Notional $ quoted on bid/ask" (aggregate for owner)
• Sorted by: agg. notional, descending

Background:
Currently the Trade History window shows price, quantity and taker.

Feature Idea:
Add a maker column to the trade history window.

Additional Solution Notes:
Useful to see which makers are actually getting filled.

Abstract

Add a column called $ Total on each orderbook and trade history table.

It will be calculated as follows: $ Total = Price * Quantity.

This will produce the total $ amount being quoted for the given quote.

Extrapolate price and quantity for concentrated liquidity on Orca, display on order book, labeled for Orca.

Background:
The app is susceptible to downtime if the RPC server goes down or falls behind slots.

Feature Idea:

• Fallback to a 2nd RPC server if the first one goes down, or falls behind slots

Additional Solution Notes:

• Anyone can pick this up, or it will be slightly lower in my backlog

Background:
The user needs an easy view/table/page for entering or choosing a market maker entity or pubkey, and seeing all current open orders.

Feature Idea:
Create a new interface for viewing all open Serum orders for a given pubkey, or entity name.

• Some sort of getProgramAccounts call, without market filters
• Table row for each market
• Chart showing pie chart of markets quoted

Additional Solution Notes:
Linear: SRM-42

A new page accessible from the nav bar.

Goal: Market browsing page - big table of data, sort asc/desc by different columns

Metrics: notional deposited, etc etc. Use as much of the initial Market struct that we can and share with the user, even things llike fees accrued, put it in the table.

I'm told that the state-of-the-art for getting lots of account data is https://github.com/rpcpool/solana-geyser-park.

They say it's greatly preferred over RPC.

See how we can integrate this.

Using Geyser (assuming it is validated), index the event queue account of every market on Serum simultaneously (depending on how Geyser works).

Diff the new event queue data to determine new trade entries.

Insert the trade entries into postgres.

Switch trade history on index page to use postgres on initialization.

Background:
We need to automatically generate a markdown file that can be viewed on GitHub which describes all exposed APIs. This should occur during the maven build process.

Feature Idea:
Do this during maven build with the right plugins.

Additional Solution Notes:

• https://github.com/asciidoctor/asciidoctor-maven-plugin
• https://github.com/springfox/springfox-demos/tree/master/boot-static-docs
• https://www.javainuse.com/spring/boot_swagger
• https://www.baeldung.com/spring-rest-openapi-documentation (Has "OpenAPI" standard too)

• Create a compact/small interface above the Order Book for filtering the values displayed in the "Owner" column,
• By default, all options will be checked (Unidentified wallets, Mango, Atrix, Raydium, Alameda).
• Rendering in the order book table will be based on the checkboxes/toggles for the above options.

Background:
We currently user HTTP polling to get data from our APIs every 1200ms for the order book.

Feature Idea:
Use websockets to connect to the Solana RPC server, and also host our own WSS endpoint to proxy the updates to the user's browser.

Additional Solution Notes:
Tracking: SRM-41

There is a really inefficient loop somewhere, I think around the token list that was hacked together, adds at least 1-2 seconds to page load times. Remove the delay.

Feature Idea:
Please add the ability to copy the full market public key, as well as potentially the bids/asks keys and event queue. Showing lot sizes somewhere could also be cool. Maybe an 'Advanced Info' kind of pane that doesn't need to be shown by default. Would find this very useful as a dev.

e.g. 16x16 or 32x32 icons for each company/project identified.

Like a 16x16 mango favicon next to "Mango Markets" on the order book.

Add a bar chart that shows where bids and asks are placed across the order book.
The dataset could be x: price, y: quantity, or an aggregate metric (bar chart builds more on each tick, aggregate quantity). TBD

We need a solution to view all requests coming through via HTTP, and their whole stack trace and all methods called with timestamps/details, etc.

I've done this with Splunk in a professional environment, but it is expensive/propreitary.

Research for a web-based interface for viewing Java traces from HTTP requests.

Abstract

Rank each trader in order book by "activity" (# of quotes in book, or event queue). This numerical value will be indicated on the GUI through either a combination of displaying the rank number, color, or more.

• Add scanning for USDT markets on startup
• Add USDT Token hard-coded references in places where we can make an O(1) operation, and avoid a search. e.g. Some

When searching for markets by token, we need to be able to sort them by $ in liquidity on bid/asks (combined), as a metric for which one the user should choose. (e.g. The more resting liquidity on the bid/ask -> Sorted higher).

• Get best bid for SOL/USDC (For a SOL reference price)
• After getProgramAccounts is called for SOL and USDC quoted pairs: For the top 100 tokens, aggregate all the bid/ask pubkeys on all their markets, and batch them into getMultipleAccounts. Sum up the liquidity for each marketId, and insert into a Map<String, Float> or Map<String, Map<String, Object>> (if using a "metadata" approach
• Key: marketId (String)
• Value: aggregateAmount (Float) or metadata (Map) containing the Float.

USDC quoted markets are already in USD value, so only addition is needed. The SOL quoted markets will be SOL * (SOL_Price), added together.

Add Spring metrics to our library so it reports to JMX.
Add custom metrics for JMX, to see marketIds hit (possibly). (Maybe just access to the cache map via JMX)
Or other creative metrics.

Background:
The people want dark mode. (https://twitter.com/PartiallySorted/status/15412190423457628170

Feature Idea:
Add dark mode.

Additional Solution Notes:

e.g. openserum.io/?market=9wFFyRfZBsuAha4YcuxcXLKwMxJR43S7fPfQLusDBzvT (or some kind of modern permalinking)

If the URL variable is present, set activeMarketId in javascript to it.

Abstract

As of now, 2 getProgramAccounts RPC requests are made to a Solana RPC server on startup, first to cache the USDC markets, then the SOL markets. They are very slow, and can timeout, causing startup to fail.

Fix

1. Add multi-threading to the 2 RPC calls, so they are made concurrently. CompleteableFuture may be preferred, or the most modern Java concurrency pattern that is elegant.
2. If RPC timeout is experienced during either of the threads, retry up to 3 times.