stellar / dashboard

Dashboard

Dependencies

To build this project, you must have the following dependencies installed:

• node 10.16.3
• yarn

Installation

yarn

Developing

yarn start

If you wish to use backend server API, you need to have redis running locally on port 6379 (default for redis)

(If you do not have redis installed) If on a mac, install redis using homebrew

brew install redis

(Other install directions can be found here: https://redis.io/download)

Make sure it's running

brew services start redis

Once you have redis installed, start this command

yarn run start:backend

It will create a proxy to browser-sync server started by gulp at http://localhost:5000

Connecting to Big Query

Connecting to Big Query is not required for running the backend (if you run with UPDATE_DATA=false), but is required for things like catching up ledger data in redis.

This project is pulling from SDF's crypto-stellar public data set, so no special credentials are required. However you will need a Google Cloud Platform project with a service account to be able to access Big Query.

Directions for creating a service account can be found here.

Once you've created a service account, add the service account key json file to the gcloud folder under the name service-account.json. An example json file shows what the file structure should look like.

dashboard's Issues

link to other stat tools

in the footer I think we should say "Other network stats:"
and link to:
stellar.network
stellarchain.io

Add fee stats visualization

Details not displayed for some operations

The dashboard doesn't display details for manage_buy_offer or set_options in the recent operations list. At least for manage_buy_offer, I would expect it to display details analogous to what is displayed for manage_offer.

Page Redesign

I'd like to redesign the dashboard. I'm still pretty new to React, so I'd like to use this as a chance to design and build the dashboard. I have a few questions on some of the content and what some of the data represents. Maybe there are more features that people would find useful, that I can design and implement. Let me know if this sounds like something the group would like to do.

Title should link back to stellar.org

Announcements Banner

This banner can be used for any announcements that affects the stellar network.

Some sample use cases:

• notifying the community of upcoming protocol upgrades and the time when it is expected to trigger
• successful upgrades to the protocol version of the network

Here's a sample of a banner created by hacking the inline html to better convey the issue:

Feature Request: Improvements to the lumens API

What problem does your feature solve?

When we use /api/v2/lumens API it returns values in the float format. It would be very useful to have values of /api/v2/lumens API to be reported as a whole numbers, to achieve that we can multiply current number with 10^7. We can consider adding a new filter to the API to return values in the different format.

Currently, programming languages such as javascript (old version), python etc. truncate the fraction part of the number and that makes it very difficult to get exact totalSupply number. It'd be great to have totalSupply and other fields provided as whole numbers.

What would you like to see?

In addition to the whole number support, it'd also be useful to have totalSupply sanity check inbuilt in the API. API should return following two numbers:

1. lumensSupply = totalSupply - feePool
2. circulatingLumensSupply= upgradeReserve + circulatingSupply + sdfMandate

Above two numbers should be same. This will ensure that totalSupply will always be equal to sum of four categories upgradeReserve, circulatingSupply, sdfMandate and feePool.

(Nomenclature for the above two numbers can be different too.)

What alternatives are there?

None!

Network switch

Switch between mainnet and testnet.

toggle test vs public network

I think we should follow what we did for laboratory (with a toggle at the top) and only display (and download of course) testnet or pubnet instead of a giant combined page (default pubnet):

• this would make the page mobile friendly (right now it's so resource hungry that it doesn't display the right data over slow networks)
• it would mitigate to a certain extent the flickering caused by last transactions and validator health status (right now, because testnet is below pubnet, it jumps quite a bit)

setup steps

Hi,

I wanted to try out this sample code in my laptop but I didn't find any setup document which I could follow and run this.

Can you please provide some steps on how to run this and see the dashboard?

Update Node Version to LTS 16

Currently is version 10, which is end of life. Will involve updating dependencies as well

Local time in Scheduled Maintenance banner incorrect

What version are you using?

Latest deployed as of this issue creation time.

What did you do?

The local time in the Scheduled Maintenance banner is incorrect for me:

Scheduled Maintenance: Public Network Protocol 13 Upgrade Vote on Thursday, June 4th 2020, at 4:00pm UTC (4:00pm local time)

I'm in Poland and we're in UTC+2 timezone on June 4th: https://www.worldtimebuddy.com/?qm=1&lid=756135,100&h=756135&date=2020-6-4&sln=18-19.

`api/v2/lumens/` isn't built to be horizontally scalable

Currently, the endpoint api/v2/lumens/ relies on cached data that is stored in memory. This creates an issue when we have multiple instance in kubernetes running the app. Depending on which image you hit, you'll get a slightly different payload as they (likely) all started running at different times and are running on different schedules.

Thoughts on a solution:

• Do we want to save cached data in a single database and use that as our source of truth? (The current postgresdb in the repo has the above problem of multiple instances)

flickering page when number of transactions fluctuates

The section "Recent operations: Live network" changes size based on the number of transactions, this causes the page to jump up/down based on the number of transactions in the previous ledger.

I would recommend picking a size (let's say 20 transactions) and have a fixed area for transactions (displaying empty space in that area if there are not enough transactions).

Move to a new data viz library

React-d3-components is hardly maintained anymore. To implement a feature addition for #91, I had to fork and update the implementation.

I'd recommend trying out:

1. Nivo, i.e. State of JS charts
2. Victory, supported by Formidable
3. React-viz, supported by Uber

Update website configs and template to match other projects [8]

Use Create React App/webpack instead of gulp, match file structure what we have in other projects, use Stellar Design System, etc.

Feature Request: Improve usefulness of the ops over last 100 ledgers graph

What problem does your feature solve?

There's currently no way to see if recent ledgers have been full (and therefore subject to surge pricing) or not.

What would you like to see?

Change the "Successful tx & ops over the last 100 ledgers" graph to be "Successful & failed ops over the last 100 ledgers":

The current graph is somewhat nonsensical anyway, it's meaningless to have a bar graph that stacks ops on top of txns. The total height of the bar—ops count + txn count—is a useless value. There's already another graph showing the number of successful txns. Change this graph to show the total number of operations, with each bar split into successful & failed operations. This mirrors the other txn graph.

"Txs & ops in the last 30 days" graph has insufficient history

What version are you using?

https://dashboard.stellar.org/

What did you do?

Looked at "Txs & ops in the last 30 days" graph.

What did you expect to see?

30 days worth of history.

What did you see instead?

2 days worth of history, screenshot below:

Connect to many horizon servers

If one goes down Dashboard will still display valid information.

backend: aggregate and serve 1hr, and 24hr ledger data

from the new designs: https://www.figma.com/file/4z9lsHvbsEoVAtzfj3mNzy/Stellar-Dashboard?node-id=206%3A808

we have last 30 days aggregated and cached now, but will need to add 1hr and 24hr

Remove Nodes section and link to stellarbeat

Requested by @jedmccaleb.

REEDME mistake.

Wrong path to the executing app.js file in a README.md.

Change please 'DEV=true node ./backend/app.js' to 'DEV=true node ./app.js'.

add a link to the repo at the bottom of the page

Update "Fee stats" box with new Horizon response format

Horizon 0.25.0 will change the structure of /fee_stats endpoint (see "Scheduled Breaking Changes"): https://github.com/stellar/go/blob/release-horizon-v0.24.0/services/horizon/CHANGELOG.md#v0240 Dashboard should be updated to use the new fields.

TXS & OPS IN THE LAST 30 DAYS: LIVE NETWORK

I am getting error when the section - TXS & OPS IN THE LAST 30 DAYS: LIVE NETWORK is getting loaded. On checking details getting following :

in multiple places in code the url is just pointing to axios.get('/ledger/public') or axios.get ('/api/nodes') or axios.get('/api/lumens') But it is not found. How do i resolve this. Which component is missing. I have just followed the steps given in build instructions.

Basic blocks for all sections

Placeholders for all sections we need to have on the page, using a card component with common props.

Check if nodes participate in SCP for uptime stats

Currently we make a simple connection to the node to check if it's up. It means that even if something is broken but the node accepts incoming connections it's status will be saved as up.

The easiest idea I have is to:

1. Create a new monitoring stellar-core instance.
2. Generate the config file using nodes.js file, add all nodes to quorum set. Remove history entries (except local one that will obviously do nothing).
3. Check /quorum endpoint missing field. If a node is there it means it's down.
4. Regenerate a config file with new nodes every X hours. If the list has changed, restart core.

Questions:

• What will happen when len(missing)>fail_at? Does the node continue updating quorum information?
• Will it continue to work with, say, 200 nodes in quorum set? I tested this with 39 nodes we currently have in the Dashboard and it's been working fine (for a couple minutes so far).

CC: @MonsieurNicolas @vogel

Network status

I have created a private network of 5 nodes. When I am pointing the live URL to my horizon setup, the network status Live is initially showing up and running, after some time it goes to down. However I have have checked the difference in close time in last 200 (closed_at ) from the horizonurl/ledgers?order=desc&limit=200 is at max 5 sec.

Went through the code, could not find anything else.

Have checked my network it is running properly.

Relabel

After the bitcoin distribution is complete, we should change the the pie chart to be two sections each subdivided:

• To Distribute
  • direct signup program
  • partnership program
  • SBC
  • SDF discretionary
• Distributed
  • direct signup program
  • partnership program
  • bitcoin program
  • SBC
  • other

Use Redis for caching

• Replace the in memory storage with Redis - will solve #179

• Replace Postgres with Redis. Postgres is currently being used for caching anyways, and we don't foresee the schema getting more complicated, so using Redis only in this project makes sense.

remove node list from the dashboard

• The node list is incomplete
• It requires manual updates
• It's misleading as people might think that the availability metric relates to validation while in fact it's just overlay (#49)
• removing it allows the dashboard to be focused on one thing: ledger activity.
• stellarbeat.io does a better job. we can link to that

Show what ledger number it is when you mouse over the graphs

add failed transactions to graphs

stellar/go#756

Holo

What version are you using?

What did you do?

What did you expect to see?

What did you see instead?

Support Protocol 18 operations

Liquidity pool operations should work on the Dashboard.

how to do transactions+op per second

Im trying to add in dashboard a javascript to add transactions+op per second, its a good indicator about network .

for example 1 trasaction in 200 ledgers with average of 5 seconds per transacion in 200 ledgers. Will be average transations per second 1/(200*5)sec so the final data i want to see in dashboard will be like
0.001 tx/sec.

or divided per 60 seconds. Num transacions in one minute /60 .

i think will be a good indicator.

Feature Request: Add confetti to celebrate protocol upgrade

What problem does your feature solve?

It would be great to celebrate protocol upgrades. In NetworkStatus component we can monitor ledger version, if previous state ledger version is lower than current state we can trigger confetti for a few seconds.

What would you like to see?

Confetti, there's a nice package for this: https://www.kirilv.com/canvas-confetti/

What alternatives are there?

No confetti.

cluster validators by entity

As more validator nodes join, the list becomes cluttered.
I propose clustering the nodes by entities - like "ibm" and "sdf"
Then instead of showing a list of nodes, we will show a list of entities - each one summarizing all of the entity's nodes.

backend: serve DEX data

Instead of relying on stellar.expert for this data we should be able to get the data from Hubble.

Talked with @sydneynotthecity and the tables that should have the data we need are:
history_trades
history_operations
history_assets

from crypto_stellar_internal_2, which is updated every 30 minutes.

reduce number of calls to Horizon when updating caches

see if we can reduce the number of calls to Horizon. For example the v1 updateApiLumens method calls Horizon 40 times simultaneously every 10 mins.

commonLumens.totalSupply - 2
commonLumens.circulatingSupply - 22
commonLumens.directDevelopmentAll - 4
commonLumens.distributionEcosystemSupport - 5
commonLumens.distributionUseCaseInvestment - 2
commonLumens.distributionUserAcquisition - 5

Total horizon calls: 40

and some of these calls are repeats (eg. circulatingSupply calls totalSupply).

This large number of calls at once creates intermittent Horizon timeout errors, as well as the length of the tests (when Horizon responses are even slightly slower, because of the many calls it creates a large lag where you have to increase the time of the backend tests to be able to handle, eg).

some possible options to explore are:

• deprecate endpoints (or deprecate fields w/in endpoints)
• cache each granular horizon endpoint data, and then on the api endpoints compute the responses from the caches on the fly. This should prevent repeat-horizon calls.
• possibly use hubble for some/all of this data instead of calling Horizon. Hubble also has some limitations (cost/query, updates every 30 mins), that will need to be looked into though

New nodes acceptance process

We've been receiving quite a lot of PR adding new nodes but some of them are unmaintained after merging. I've been thinking about the process and I came up with something like this:

1. Validation - first we need to validate if the node is really run by the organization. We wil require organizations to add their node information to stellar.toml file. The validator's host should be a subdomain of the organization's root domain. If the data matches, we merge the PR and deploy.
2. Trial period - every new node is hidden by default but we start monitoring it's uptime. Trial period lasts 30 days. If the node is up for less than 97% (t as trial) of time we remove the node from the list and block the organization from adding a new node for the next 3 (b as block) months.
3. Removing unhealthy nodes - after the trial period a node is made visible. To make sure that organizations are monitoring and maintaining their nodes we check the uptime stats every quarter. If the uptime in the last 3 months is below 99% (q as quarter uptime) we remove the node and block adding it back for the next 3 (b) months.

I selected t to be 97% because it's around one day in 30 days period. I guess these people would still be learning how to run a node so it sounds fair to have a downtime that sums up to 24 hours. q is higher (99%) but it's also around one day in 90 days period. We should require high uptime for production nodes. b is 3 months as it should be high enough to make sure people care about their nodes' uptime but also don't discourage them too much from adding them back.

It's not the final process, just a starting proposal. Thoughts?

couple of suggestions

Great dashboard. I do have a few suggestions and will submit a couple of changes.

1. https would be good. Using a free cert is fine

2. Number of active nodes - ethereum's dashboard has this and I think it would be useful
   https://ethstats.net/

Show what ledger number it is when you mouse over the graphs

use horizon time

The delta between current time and last ledger close time uses the browser's clock. That's not always accurate. We should use a more accurate timestamp, perhaps from horizon.

Add TypeScript to backend

1. Add typescript to project
2. Refactor backend code to use

Excessive resting CPU usage

Chrome, MacOS.

I left a tab open at dashboard.stellar.org. About 2 hours later the laptop fan was going nuts, Chrome was using ~200% cpu & the tab hosting the dashboard was unable to render.

When I closed the dashboard tab, utilisation returned to normal.

backend: serve testnet ledger data

figma design for dashboard v2: https://www.figma.com/file/4z9lsHvbsEoVAtzfj3mNzy/Stellar-Dashboard?node-id=206%3A808

the testnet page has aggregated ledger info, which isn't currently served from the backend (right now it's just mainnet on the backend)

backend: Pull ledger data from Hubble

Right now ledger data is aggregated from Horizon. Instead we can pull it from Hubble. Benefits:

• Dashboard makes a lot of calls to Horizon, which creates some intermittent rate limit errors (eg. of an update with a lot of Horizon calls, and right now there's three of those)
• if the ledgers cache in redis ever gets lost/corrupted, the catchup using Horizon makes a lot of calls to Horizon that creates rate limit errors (took 8 pod restarts to finish today's catchup)
• dashboard v2 has some additional ledger queries (1HR ledgers, 24HR ledgers) that will be easier to pull from hubble using sql than aggregating like it is doing now

Downsides to consider:

• hubble has a cost/query (rough estimate is about 600 mb/query 1GB/query)
• hubble is updated every 30 minutes

all: add eslint

use @stellar/eslint-config so the code is aligned with other SDF projects.

eg. https://github.com/stellar/account-viewer-v2/blob/master/.eslintrc.js

Create plan for updating Dashboard backend

plan here: https://stellarorg.atlassian.net/wiki/spaces/PET/pages/4244078593/Dashboard+Backend+Upgrade+Plan

once agreed on will break down into tickets