skynetcap / serum-data Goto Github PK
View Code? Open in Web Editor NEWOpen Book Market Data interface on Solana
Home Page: https://openserum.io
License: MIT License
Open Book Market Data interface on Solana
Home Page: https://openserum.io
License: MIT License
Add a Map
cache for Bid and Ask orderbook Solana accounts. The cache will only be considered "valid" for 1200ms. To accomplish this, either include a timestamp with each Map entry, to see if it was recent enough to want it. Or use guava's time eviction based policies. Guava policies are preferred.
100 people are on the SOL/USDC order book. Normally this will add 100 RPC requests every 1200ms. By adding caching, each time they poll for data, it will first check if there was a cache update within the past 1200ms. If so, they will use the cache. Otherwise, they will poll and update the cache (and include new timestamp if necessary). After that is implemented, for 100 users, 99 will hit the cache and only 1 will experience a polling delay (every 1200ms).
A lot of code will return a null if the desired function/search/lookup returns no results. Replace these flows with the Optional
keyword.
/api/serum/market/{marketId}/spread
, takes in a marketId
bid
and ask
value:{
"bid": 38.50,
"ask": 38,60
}
Notes:
This will make two order book requests. Other tickets will be made for a "bid" only API, and an "ask" only API, to limit it to one RPC request. A website may just want the top bid to return as an active price for an asset, and the "bid" only API would suffice, and use half the load. In the future, market caching would slightly improve these response times as well.
Identify more large trading entities and add them to our order book reverse lookup.
Background:
We need the ability to see the TX that created a Serum order that was a taker in the event queue.
Feature Idea:
Search for the owner's recent transactions (10-20 if public RPC, switch to private). Best match versus a Serum instruction, or Serum inner instruction/CPI.
Additional Solution Notes:
SolanaJ may need an update to read instruction data of CPIs.
Linear: SRM-44
Background:
In the codebase, "Solana addresses" such as token mint, exist in both String format at the custom Pubkey class with a very inconsistent mix.
E.g. Most maps use String as the Key for <"Address", Value> mappings, presumably due to lack of confidence in Pubkey's hashCode implementation.
Feature Idea:
Change every KV map to use Pubkey
class instead of String
for the key, and any other similar String
usage.
In combination with this ticket, create some tests in SolanaJ to test Pubkey hashing, if anything needs to be changed at the library level.
Additional Solution Notes:
Linear: SRM-43
Investigate and resolve. Maybe it's just from a bad marketId?
In the table header for each order book, or in a row below the header, display something like:
Total amount (Bids): $500,000
Total amount (Asks): $2,000,000
Background:
The order book table takes up the whole page.
Feature Idea:
Add a max height, which creates a scrollbar after that height.
Additional Solution Notes:
400-600px maybe. The goal is to place another row of data/tables beneath the scrollable order book.
Research how we can rank or quantify the age of a quote in the order book.
Some possibilities:
This metric would then be displayed to the user, quantitatively (data column), visually (color), or both, or something else creative.
Take periodic snapshots of the order book objects and store them in postgres every 12 hours. (optimal interval TBD)
Allow querying of the snapshots.
Display dataset in a table:
Background:
Currently the Trade History window shows price, quantity and taker.
Feature Idea:
Add a maker column to the trade history window.
Additional Solution Notes:
Useful to see which makers are actually getting filled.
Add a column called $ Total
on each orderbook and trade history table.
It will be calculated as follows: $ Total = Price * Quantity
.
This will produce the total $ amount being quoted for the given quote.
Extrapolate price and quantity for concentrated liquidity on Orca, display on order book, labeled for Orca.
Background:
The app is susceptible to downtime if the RPC server goes down or falls behind slots.
Feature Idea:
Additional Solution Notes:
Background:
The user needs an easy view/table/page for entering or choosing a market maker entity or pubkey, and seeing all current open orders.
Feature Idea:
Create a new interface for viewing all open Serum orders for a given pubkey, or entity name.
Additional Solution Notes:
Linear: SRM-42
A new page accessible from the nav bar.
Goal: Market browsing page - big table of data, sort asc/desc by different columns
Metrics: notional deposited, etc etc. Use as much of the initial Market struct that we can and share with the user, even things llike fees accrued, put it in the table.
I'm told that the state-of-the-art for getting lots of account data is https://github.com/rpcpool/solana-geyser-park.
They say it's greatly preferred over RPC.
See how we can integrate this.
Using Geyser (assuming it is validated), index the event queue account of every market on Serum simultaneously (depending on how Geyser works).
Diff the new event queue data to determine new trade entries.
Insert the trade entries into postgres.
Switch trade history on index page to use postgres on initialization.
Background:
We need to automatically generate a markdown file that can be viewed on GitHub which describes all exposed APIs. This should occur during the maven build process.
Feature Idea:
Do this during maven build with the right plugins.
Additional Solution Notes:
Background:
We currently user HTTP polling to get data from our APIs every 1200ms for the order book.
Feature Idea:
Use websockets to connect to the Solana RPC server, and also host our own WSS endpoint to proxy the updates to the user's browser.
Additional Solution Notes:
Tracking: SRM-41
There is a really inefficient loop somewhere, I think around the token list that was hacked together, adds at least 1-2 seconds to page load times. Remove the delay.
Feature Idea:
Please add the ability to copy the full market public key, as well as potentially the bids/asks keys and event queue. Showing lot sizes somewhere could also be cool. Maybe an 'Advanced Info' kind of pane that doesn't need to be shown by default. Would find this very useful as a dev.
e.g. 16x16 or 32x32 icons for each company/project identified.
Like a 16x16 mango favicon next to "Mango Markets" on the order book.
Add a bar chart that shows where bids and asks are placed across the order book.
The dataset could be x: price, y: quantity, or an aggregate metric (bar chart builds more on each tick, aggregate quantity). TBD
We need a solution to view all requests coming through via HTTP, and their whole stack trace and all methods called with timestamps/details, etc.
I've done this with Splunk in a professional environment, but it is expensive/propreitary.
Research for a web-based interface for viewing Java traces from HTTP requests.
Rank each trader in order book by "activity" (# of quotes in book, or event queue). This numerical value will be indicated on the GUI through either a combination of displaying the rank number, color, or more.
Token
hard-coded references in places where we can make an O(1) operation, and avoid a search. e.g. SomeWhen searching for markets by token, we need to be able to sort them by $ in liquidity on bid/asks (combined), as a metric for which one the user should choose. (e.g. The more resting liquidity on the bid/ask -> Sorted higher).
getProgramAccounts
is called for SOL and USDC quoted pairs: For the top 100 tokens, aggregate all the bid/ask pubkeys on all their markets, and batch them into getMultipleAccounts. Sum up the liquidity for each marketId, and insert into a Map<String, Float>
or Map<String, Map<String, Object>>
(if using a "metadata" approachUSDC quoted markets are already in USD value, so only addition is needed. The SOL quoted markets will be SOL * (SOL_Price), added together.
Add Spring metrics to our library so it reports to JMX.
Add custom metrics for JMX, to see marketIds hit (possibly). (Maybe just access to the cache map via JMX)
Or other creative metrics.
Background:
The people want dark mode. (https://twitter.com/PartiallySorted/status/15412190423457628170
Feature Idea:
Add dark mode.
Additional Solution Notes:
e.g. openserum.io/?market=9wFFyRfZBsuAha4YcuxcXLKwMxJR43S7fPfQLusDBzvT
(or some kind of modern permalinking)
If the URL variable is present, set activeMarketId
in javascript to it.
As of now, 2 getProgramAccounts
RPC requests are made to a Solana RPC server on startup, first to cache the USDC markets, then the SOL markets. They are very slow, and can timeout, causing startup to fail.
CompleteableFuture
may be preferred, or the most modern Java concurrency pattern that is elegant.A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.