GithubHelp home page GithubHelp logo

buidl-labs / zcash_service_status_dashboard Goto Github PK

View Code? Open in Web Editor NEW
3.0 4.0 4.0 82 KB

A health check dashboard for ZEC Communities and Exchanges

Home Page: https://zcashservicestatus.info/

Python 99.76% Shell 0.24%
zec zcash blockchain health-check

zcash_service_status_dashboard's Introduction

Zcash Service Status Dashboard

This readme deals with installation and setup instructions for building a health-check system for Zcash communities and exchanges. The dashboard is live here.

The health check system involves three tools: Prometheus, Blackbox Exporter and Grafana.

1. Install Prometheus

Use the prometheus.yml file included in this repository for the config.

2. Install Grafana

4. Install Dependencies

sudo apt-get install python3-pip
pip3 install prometheus-client==0.7.1

5. Now, run these commands in different screens to run all the health checks.

python3 blockchain_explorers_health_check.py
python3 communities_and_forums_response_time.py
python3 exchanges_health_check.py
python3 metrics.py

Note: This readme will be further updated with 'why' section and some FAQs. Please feel free to open any issue and provide us your valuable feedback.

zcash_service_status_dashboard's People

Contributors

harukama avatar kebab-mai-haddi avatar lord-of-codes avatar rex4539 avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

zcash_service_status_dashboard's Issues

Extending support for communities

Extending support for Communities

Bold represents new additions after proposal 00:

Websites being Monitored:

Parameters being Monitored:

  • Downtime by checking 200 status for both HTTP and HTTPS.
  • Response Time.

Dashboard Changes:

Communities & Forums Overview Dashboard

We are removing Not Ok/Ok Graphs for all communities and websites. The revamped UX would look like this:
https://www.figma.com/file/m0NhaRo5PFtCPKn86CfMTK/ZCash-Service-Status-Dashboard?node-id=84%3A10

^ On this dashboard, you will see the list of all websites we are monitoring.

Specific website dashboard

User will be shown this dashboard once the user clicks on any specific website:
https://www.figma.com/file/m0NhaRo5PFtCPKn86CfMTK/ZCash-Service-Status-Dashboard?node-id=84%3A123

Revamping UX of the dashboard

SQL Dashboard

SQL Dashboard

Since we are ingesting data for:

  • 31 exchanges
  • 16 trading pairs

The data becomes a sweet spot for analysis. We want to create a platform that would lower the barrier for analyzing historical price data for the researchers/analysts in the ZCash community.

Users will be able to perform queries for the following usecases:

  • Best arbitrage opportunity across a trading pair across all exchanges by checking % deviation.
  • Best arbitrage opportunity across multiple trading pairs across all exchanges.

System Architecture:

Users will be able to use a web based UI to execute SQL queries on the historical price data stored in S3.

  • Frontend: Airpal
  • DB: PrestoDB

DB Architecture

There will be two tables:

  • exchanges
  • communities

exchanges

  • Columns:
    • ts : Timestamp
    • exchange_name : Name of the exchange
    • trading_pair: Trading Pair, for eg, ZECUSD, ZECBTC, ZECETH, etc
    • spot_price: Spot price at a time instance ts
    • spot_price_deviation: Percentage deviation of the spot price for an exchange in a trading pair
    • transaction_volume: Transaction volume as seen in the last fetched transaction.

The table would look like the following, when populated with some data:

ts                  | exchange_name | trading_pair | spot_price | spot_price_percentage_deviation | transaction_volume
2019-08-21T11:39:3Z |  Exmo         | ZECUSD       | 110.34     | 2.3                             | 110.34

communities

  • Columns:
    • ts: Timestamp
    • website: Name of the forum/community
    • status: Site up or site down

This could also help one know the duration for which site was down, using SQL.

Examples of what you can do:

  • Get the maximum deviation in spot price across all exchanges for a trading pair.

    • SELECT * from exchanges where trading_pair='ZECUSD' ORDERBY spot_price_deviation DESC LIMIT 1;
  • Get the top ten spot price deviation across all exchanges and all trading pairs

    • SELECT * from exchanges ORDERBY spot_price_deviation DESC LIMIT 10;
  • In the following queries, spot_price_deviation is stored and queried as percentage.

    • SELECT * from exchanges where (spot_price_deviation >= 50 and pair = 'ZECUSD' and ts >= 2019-01-01T00:00:00Z)
  • Get the count when a community was down.

    • SELECT count(*) from communities where status is 'Not Ok'

Note: Since we are using free and public APIs, due to rate-limits of exchanges, our data (as of now) gets ingested once in two minutes.

Cost

  • Total: $69,980/year.
  • Indivisual Components:
    • Startbust Presto cluster with six nodes - $55,200/year.
    • Startbust License for six nodes - $14,400/year.
    • Hive Metastore - $1380/year.

Extending Exchanges Coverage

Extending Exchanges Coverage

We plan to extend coverage of exchanges and trading pairs so that the dashboard becomes the platform when it comes to analyzing trading activity happening in ZEC.

Total Coverage:

Bold represents new addition after proposal 00:

  • Exchanges getting monitored (Total 31):

    • AirTM
    • BigOne
    • Binance
    • Bitasset
    • Bitci
    • Bitfinex
    • Bitlish
    • Bitsdaq
    • Bittrex
    • BX Thailand
    • Cex
    • Changelly
    • Coinbase
    • Coinrail
    • CoinRoom
    • Coinsave
    • Coinut
    • Crex24
    • Darb Finance
    • DoveWallet
    • Exmo
    • Gemini
    • Gopax
    • Huobi US
    • Kraken
    • Litebit
    • OKCoin
    • Polenix
    • The Rock Trading
    • Upbit
    • VALR
  • Trading pairs being monitored (Total 16):

    • ZEC/BTC
    • ZEC/USD
    • ZEC/USDC
    • ZEC/USDT
    • ZEC/ETH
    • ZEC/BNB
    • ZEC/EUR
    • ZEC/PAX
    • ZEC/TRY
    • ZEC/XBT
    • ZEC/TRY
    • ZEC/RUB
    • ZEC/KRW
    • ZEC/JPY
    • ZEC/PAC

Parameters Monitored

  • For a specific trading pair:

    • Around Metrics:

      • Mean Price Across All Exchanges.
      • Median Price Across All Exchanges.
      • Avg Trading Volume Across All Exchanges.
      • Total Trading Volume Across All Exchanges.
      • Spot Price Across All Exchanges.
      • Spot Price Absolute % Deviation Across All Exchanges.
    • Around Sanity Checks:

      • Transaction Volume Check Across All Exchanges Over Time.
      • Spot Price Check Across All Exchanges Over Time.
  • For a specific exchange:

    • Around Metrics:

      • All Trading Pairs Metrics On This Exchange (over the time period chosen on the dashboard):
        • Min Price
        • Max Price
        • Avg Price
        • Current Price
      • All Trading Pairs Volume On This Exchange Over Time.
    • Around Sanity Checks:

      • All Trading Pairs Transaction Volume Check Across This Exchange Over Time
      • All Trading Pairs Spot Price Check Across This Exchange Over Time
  • Interval checks: at every 5-minutes.

Dashboard Changes Overview:

Since we are now dealing with 31 exchanges and 16 trading pairs, the previous dashboard experience would result in information overload. We decided to completely revamp the exchange dashboard experience.

Exchanges Overview Dashboard

https://www.figma.com/file/m0NhaRo5PFtCPKn86CfMTK/ZCash-Service-Status-Dashboard?node-id=56%3A0

^ This how the exchange overview dashboard would look.

Specifics:

Specific Trading Pair Dashboard:

When a user clicks on a specific trading pair, they will be directed to this view: https://www.figma.com/file/m0NhaRo5PFtCPKn86CfMTK/ZCash-Service-Status-Dashboard?node-id=56%3A109

Specific Exchange Dashboard:

When a user clicks on a specific exchange, they will be directed to this view:

https://www.figma.com/file/m0NhaRo5PFtCPKn86CfMTK/ZCash-Service-Status-Dashboard?node-id=59%3A27

Future Iterations

Future Iterations of ZCash SSD

Short Term:

SQL Dashboard

Since we are ingesting data for:

  • 31 exchanges
  • 16 trading pairs

The data becomes a sweet spot for analysis. We want to create a platform that would lower the barrier for analyzing historical price data for the researchers/analysts in the ZCash community.

Users will be able to perform queries for the following usecases:

  • Best arbitrage opportunity across a trading pair across all exchanges by checking % deviation.
  • Best arbitrage opportunity across multiple trading pairs across all exchanges.

We had created the spec sheet for the SQL dashboard. Upon reaching awareness around the tech infrastructure cost, we decided to discuss it with the community first and get their approval before moving forward.

Websocket support

For consuming live data from the dashboard.

Twitter Bot

To increase awareness about the dashboard + engagement, we can create a Twitter Bot which autotweets when a certain event occurs. For eg:

  • Live status reports:
    • ZCash Community Website:
    • Example Tweet:
      "Service status dashboard reports ZCash Community is down!" - which
  • Interesting alerts around exchanges data:




Long Term:

Enabling Community Participation

We want to make it easier for people to visualize the data that we are ingesting. Graphana provides accounts support and hence users will be able to create accounts on https://www.zcashservicestatus.info/.

  • Upon creating an account they will be logged in and see the main dashboard with read-only access.
  • They can create their own dashboards with the data.

The grand vision would be the ZCash community creating charts that they think are insightful and sharing with the community as a whole.

Monitoring Node for Validators

We are inspired by Jameson Lopp's work on https://statoshi.info/.

What is Statoshi?
Statoshi's objective is to protect Bitcoin by bringing transparency to the activity occurring on the node network. By making this data available, Bitcoin developers can learn more about node performance, gain operational insight about the network, and the community can be informed about attacks and aberrant behavior in a timely fashion.

From: https://github.com/jlopp/statoshi

Statoshi’s objective is to protect Bitcoin by bringing transparency to the activity occurring on the node network. By making this data available, Bitcoin developers can learn more about node performance, gain operational insight about the network, and the community can be informed about attacks and aberrant behavior in a timely fashion. Any DevOps engineer will tell you that the first step is to measure everything, after which you can build reporting and automation around the raw metrics in order to maintain and improve the system. Statoshi is this first step.

We want to extend the dashboard with the same vision for ZCash.

Ideas for extension:
1) Build a pool of Statoshi nodes that act together in order to monitor a larger portion of the node network and thus provide a more complete picture. I’m currently working on a node connection management daemon that would facilitate running a pool of nodes that maximizes their connectivity to the network mesh by reducing redundant peer connections across nodes in the pool.

2) Sending alerts to Bitcoin developers about aberrant behavior on the network, such as changes of more than one standard deviation.

3) Automated quality assurance tests / sanity checks. For example, adding alerts that would fire if the sum of bitcoins in existence ever changes by an amount other than the block reward in a single block.

4) Graphing the data collected by Statoshi nodes is but one of many possibilities. It could be useful for us to provide batch files of the raw data collected so that other engineers can analyze it, or we could build an API devoted to making it easy to query the data.

5) Automated defensive measures / spooling up honest nodes to combat certain types of attacks.

Metrics to visualize: https://blog.lopp.net/bitcoin-by-the-numbers-2018-recap/

Other Platforms like these:

https://blog.lightning.engineering/posts/2019/07/24/lndmon-v0.1.html

Reporting Structure

Reporting Structure

Weekly reports being submitted in #the-zcash-foundation channel. The community can keep us accountable by going through each step described in the timeline & double-check if these were implemented.

ZCash Specific Metrics

ZCash Specific Metrics

We will be graphing out the following metrics:

  • Difficulty over time.
  • Sprout value pool over time.
  • Sapling value pool over time.
  • Transparent (T) transactions over time.
  • Shielded (Z) transactions over time.

API Support

Overview:

  • API specs

    • Host: zcashservicestatus.info
    • Base Path: /api/v1
  • Endpoints List:

    • Exchanges Specific:

      • GET /exchanges/coverage
      • GET /exchanges/check/transactionvolume
      • GET /exchanges/check/spotprice
    • Community & Forums Websites Specific:

      • GET /websites/coverage
      • GET /websites/downtime
  • API

Exchanges Endpoints

1. Exchanges & Trading Pairs covered by the dashboard

Need:

This is a generic API endpoint which gives:

  • the list of exchanges and trading pairs that are being monitored by the service status dashboard.
  • the list of checks that are supported.

Request:

Path
GET /exchanges/coverage

Response Format:

{  
   "exchanges": [
      {
        "name": "<name of the exchange eg: Bittrex>",
        "key": "<key of the exchange. Used for querying data eg: bittrex",
        "url": "<url of the exchange>",
        "supported_trading_pairs": [
            {
                "name": "<name of the trading pair eg: ZECUSD>"
                "spot_price_check_support": "<true/false>"
                "transaction_volume_check_support": "<true/false>"
            }
            ...
        ]
      }
      ...
}

:::warning
💡 Note

spot_price_check_support & transaction_volume_check_support boolean value shows whether the dashboard is ingesting data for the check in question for the particular trading pair on that exchange. There are some exchanges which do not support a particular trading pair.

:::

Sample Request/Response:

Request:
GET /api/v1/exchanges/coverage
Response:
{  
   "exchanges": [
      {
        "name": "bittrex",
        "key": "bittrex",
        "url": "https://international.bittrex.com/",
        "supported_trading_pairs": [
            {
                "name": "ZECUSD"
                "spot_price_check_support": true
                "transaction_volume_check_support": true
            }
        ]
      },
      {
        "name": "exmo", 
        "url": "https://exmo.com/",
        "supported_trading_pairs": [
            {
                "name": "ZECUSD"
                "spot_price_check_support": true
                "transaction_volume_check_support": true
            },
            {
                "name": "ZECETH"
                "spot_price_check_support": true
                "transaction_volume_check_support": true
            },
            {
                "name": "ZECBTC"
                "spot_price_check_support": true
                "transaction_volume_check_support": true
            }
        ]
      }
      ...
    ]
}



2. Get Overall Failure Check Status

Need:
  • This helps user get a bird's eye view of transaction volume check and spot price check for all trading pairs over all trading pairs and exchanges.
  • With this API, user will be able to know which specific check failed and then use the rest of the API's provided in this document for specific debugging.

Request:

Path
GET /exchanges/check
Request Parameters:
Parameter Description
period To request days: [1-9]d
To request months: [1-12]m
To request year: 1y
start Start time of the timerange you want the data from. For specific ranges.
end End time of the timerange you want the data from. Entering now to get data till now For specific ranges.

Response Format:

{  
   "request_time_stamp": "<timestamp when request was made>",
   "time": {
       "start_time": "The time from the data is fetched in UTC" 
       "end_time": "The time till the data is fetched in UTC"
   },
   "exchanges": [
      {
        "key": "<exchange key>",
        "trading_pairs_sanity_checks_count": [
            {
                "name": "ZECUSD"
                "transaction_volume_status_zero_count": "<count of how many times transaction_volume == 0"
                "spot_price_status_zero_count": "<count of how many times spot price == 0 "
            }
        ]
      },
      ...
    ]
}

Sample Request/Response:

Request 1: Get failure count for the past 24 hours
Request:
GET /api/v1/exchanges/check?period=1d
Response:
{  
   "request_time_stamp":"01-01-2019T00:45:00Z",
   "time": {
       "start_time": "01-01-2019T00:00:00Z" 
       "end_time": "02-01-2019T00:00:00Z"
   },
   "request_time_stamp": "",
   "time": {
       "start_time": "",
       "end_time": ""
   },
   "exchanges": [
      {
        "key": "bittrex",
        "trading_pairs_sanity_checks_count": [
            {
                "name": "ZECUSD"
                "transaction_volume_status_zero_count": 10
                "spot_price_status_zero_count": 3
            }
        ]
      },
      {
        "key": "exmo", 
        "trading_pairs_sanity_checks_count": [
            {
                "name": "ZECUSD"
                "transaction_volume_status_zero_count": 0
                "spot_price_status_zero_count": 0
            },
            {
                "name": "ZECETH"
                "transaction_volume_status_zero_count": 3
                "spot_price_status_zero_count": 2
            },
            {
                "name": "ZECBTC"
                "transaction_volume_status_zero_count": 0
                "spot_price_status_zero_count": 2
            }
        ]
      }
      ...
    ]
}



3. Transaction Volume Check

Need:
  • This endpoint outputs the specific logs of transaction volume check.
  • Helps user to debug the time when transaction volume was problematic (transaction volume == 0) of a particular trading pair on a particular exchange.

Request:

Path:
GET /exchanges/check/transactionvolume 
Request Parameters:
Parameter Description
exchange_key The key of the exchange for which transaction volume check logs were requested eg: bittrex
pair The name of the trading pair
start Start time of the query in UTC. Optional
end End time of the query in UTC.
Note: To get current status, use now and don't set start
failedlogs Set this parameter to true and the API will only return the logs when the transaction volume

:::warning
💡 Note:

  • The duration end - start cannot be more than one year.
  • This is subject to vary as we progress with the development but we will try our best to provide the maximum range possible.
    :::

Response Format:

{  
   "request_time_stamp":"<timestamp when the request was made>",
   "time": {
       "start_time": "The time from the data is fetched in UTC" 
       "end_time": "The time till the data is fetched in UTC"
   },
   "exchange_key": "<queried exchange key eg: bittrex>",
   "pair": "<trading pair eg: ZECUSD>",
   "status":[  
      {  
         "start_time":"",
         "end_time":"",
         "status":"< 0/1 >"
      },
     
   ],
   
}

:::warning
💡 Note:

  • Status = 0 -> transaction volume == 0 (Problematic)
  • Status = 1 -> transaction volume ! = 0 (Not problematic)
    :::

Sample Request/Response

Example Request 1:
Assumption: 
* Suppose status of transaction volume for ZECUSD on Exmo was 0 between January 1, 2019 00:04 - January 1, 2019 00:06 hrs. 

User queries: 
* Time period querying for: 
    * From: January 1, 2019 00:00 hrs
    * To: January 2, 2019 00:00 hrs. 
* Request was made at: January 1, 2019 at 00:45 hrs. 
Request:
GET /exchanges/check/transactionvolume?exchange_key=exmo&pair=ZECUSD&start=01-01-2019T00:00:00Z&end=01-01-2019T00:10:00Z
Response:
{  
   "request_time_stamp":"01-01-2019T00:45:00Z",
   "time": {
       "start_time": "01-01-2019T00:00:00Z" 
       "end_time": "02-01-2019T00:00:00Z"
   },
   "exchange_key": "exmo",
   "pair": "ZECUSD",
   "status":[  
      {  
         "start_time":"01-01-2019T00:00:00Z",
         "end_time":"01-01-2019T00:03:59Z",
         "status": 1
      },
      {  
         "start_time":"01-01-2019T00:04:00Z",
         "end_time":"01-01-2019T00:06:00Z",
         "status": 0
      },
      {  
         "start_time":"01-01-2019T00:07:01Z",
         "end_time":"02-01-2019T00:00:00Z",
         "status": 1
      }
   ],
   
}

:::warning
💡 Note:

  • The API batches consecutive status together.
  • In the above example, for 3 consecutive minutes
    :::

Example Request 2: Querying the current status of transaction volume
Request:
GET /exchanges/check/transactionvolume?exchange_key=bittrex&pair=ZECBTC&end=now
Response:
{  
   "request_time_stamp":"01-01-2019T00:45:00Z",
   "exchange_key":"bittrex",
   "pair": "ZECBTC",
   "status": 1
}



Example Request 3: Querying the time period when transaction volume was zero.
Request:
GET /exchanges/check/transactionvolume?exchange_key=exmo&pair=ZECUSD&start=01-01-2019T00:00:00Z&end=01-01-2019T00:10:00Z&failurelogs=true
Response:
{  
   "request_time_stamp":"01-01-2019T00:45:00Z",
   "time": {
       "start_time": "01-01-2019T00:00:00Z" 
       "end_time": "02-01-2019T00:00:00Z"
   },
   "exchange_key": "exmo",
   "pair": "ZECUSD",
   "status":[  
      {  
         "start_time":"01-01-2019T00:04:00Z",
         "end_time":"01-01-2019T00:06:00Z",
         "status": 0
      }
   ],
   
}




4. Spot Price Check

Need:
  • This endpoint outputs the specific logs for spot price check.
  • Helps user to debug the time when spot price was problematic (spot price == 0) of a particular trading pair on a particular exchange.

Request:

Path:
GET /exchanges/check/spotprice 
Request Parameters:
Parameter Description
exchange_key The key of the exchange for which transaction volume check logs were requested eg: bittrex
pair The name of the trading pair
start Start time of the query in UTC. Optional
end End time of the query in UTC.
Note: To get current status, use now and don't set start
failedlogs Set this parameter to true and the API will only return the logs when the spot price

:::warning
💡 Note:

  • The duration end - start cannot be more than one year.
  • This is subject to vary as we progress with the development but we will try our best to provide the maximum range possible.
    :::

Response Format:

{  
   "request_time_stamp":"<timestamp when the request was made>",
   "time": {
       "start_time": "The time from the data is fetched in UTC" 
       "end_time": "The time till the data is fetched in UTC"
   },
   "exchange_key": "<queried exchange key eg: bittrex>",
   "pair": "<trading pair eg: ZECUSD>",
   "status":[  
      {  
         "start_time":"",
         "end_time":"",
         "status":"< 0/1 >"
      },
     
   ],
   
}

:::warning
💡 Note:

  • Status = 0 -> spot price == 0 (Problematic)
  • Status = 1 -> spot price ! = 0 (Not problematic)
    :::

Sample Request/Response

Example Request 1:
Assumption: 
* Suppose status of spot price for ZECUSD on Exmo was 0 between January 1, 2019 00:04 - January 1, 2019 00:06 hrs. 

User queries: 
* Time period querying for: 
    * From: January 1, 2019 00:00 hrs
    * To: January 2, 2019 00:00 hrs. 
* Request was made at: January 1, 2019 at 00:45 hrs. 
Request:
GET /exchanges/check/spotprice?exchange_key=exmo&pair=ZECUSD&start=01-01-2019T00:00:00Z&end=01-01-2019T00:10:00Z
Response:
{  
   "request_time_stamp":"01-01-2019T00:45:00Z",
   "time": {
       "start_time": "01-01-2019T00:00:00Z" 
       "end_time": "02-01-2019T00:00:00Z"
   },
   "exchange_key": "exmo",
   "pair": "ZECUSD",
   "status":[  
      {  
         "start_time":"01-01-2019T00:00:00Z",
         "end_time":"01-01-2019T00:03:59Z",
         "status": 1
      },
      {  
         "start_time":"01-01-2019T00:04:00Z",
         "end_time":"01-01-2019T00:06:00Z",
         "status": 0
      },
      {  
         "start_time":"01-01-2019T00:07:01Z",
         "end_time":"02-01-2019T00:00:00Z",
         "status": 1
      }
   ],
   
}

:::warning
💡 Note:

  • The API batches consecutive status together.
  • In the above example, for 2 consecutive minutes the spot price was 0.
    :::

Example Request 2: Querying the current status of spot price
Request:
GET /exchanges/check/spotprice?exchange_key=bittrex&pair=ZECBTC&end=now
Response:
{  
   "request_time_stamp":"01-01-2019T00:45:00Z",
   "exchange_key":"bittrex",
   "pair": "ZECBTC",
   "status": 0
}



Example Request 3: Querying the time period when transaction volume was zero.
Request:
GET /exchanges/check/spotprice?exchange_key=exmo&pair=ZECUSD&start=01-01-2019T00:00:00Z&end=01-01-2019T00:10:00Z&failurelogs=true
Response:
{  
   "request_time_stamp":"01-01-2019T00:45:00Z",
   "time": {
       "start_time": "01-01-2019T00:00:00Z" 
       "end_time": "02-01-2019T00:00:00Z"
   },
   "exchange_key": "exmo",
   "pair": "ZECUSD",
   "status":[  
      {  
         "start_time":"01-01-2019T00:04:00Z",
         "end_time":"01-01-2019T00:06:00Z",
         "status": 0
      }
   ],
   
}




5. Spot Price Deviation

Need
  • For a particular trading pair, returns the absolute % deviation of individual exchange's price from the median price of all exchanges.
  • Real Life Usecase: Useful for building arbitrage bots.

Request:

Path
GET /exchanges/spotpricedev
Request Parameters:
Parameter Description
pair The name of the trading pair: ZECUSD, ZECBTC, etc.
start Start time of the period for which deviation is to be queried, optional.
end End time of the period for which deviation is to be queried, optional
period Time instance for which percentage deviation is needed, optional. If unset, it will be treated as now.

Response:

{  
   "timestamp":"<time_stamp>",
   "trading_pair": "<trading_pair>"
   "median": "<numeric median price in quote currency of trading pair>",
   "exchanges": [
      {
        "exchange_name": "<exchange_name>", 
        "price": "<numeric price in quote currency of trading pair>", 
        "deviation": "<numeric absolute % difference from median price>"
      } 
    ]
}
  • The response parameter, timestamp will be included when the user requests for a particular timestamp ( or now ). It will be absent if the user requests for a time period using start and end.
  • When the user wants to query spot price deviation for a time period, we will calculate average spot price of each of the exchange and then calculate the median. With respect to this median, we will return the deviation of the average spot price of each exchange.

Sample Request/Response

Example Request 1: Get current spot price deviation data for ZECUSD
Request:
GET /exchanges/spotpricedev?pair=ZECUSD&period=now
Response:
{  
   "timestamp":"2019-09-09T00:43:00Z",
   "trading_pair":"ZECUSD",
   "median": 100.3
   "exchanges":[  
      {  
         "key":"exmo",
         "price":"103.76035",
         "deviation": 03.45
      },
      {  
         "key":"bittrex",
         "price": 113.79035,
         "deviation":13.45
      }
   ],
   
}
Example Request 2: Get spot price deviation data for ZECBTC between Jan 1 and Jan 2, 2019.
Request:
GET /exchanges/spotpricedev?pair=ZECBTC&start=01-01-2019T00:00:00Z&end=02-01-2019T00:59:59Z
Response:
{  
   "trading_pair":"ZECBTC",
   "deviation_logs":[  
      {  
         "time":{  
            "start":"",
            "end":""
         },
         "median":100.3,
         "exchanges":[  
            {  
               "exchange_name":"Exmo",
               "price":"0.76035",
               "deviation":"3.45"
            },
            {  
               "exchange_name":"Bittrex",
               "price":"0.79035",
               "deviation":"13.45"
            }
            ....
         ]
      }
      ...
   ]
}




Community & Forums Websites Specific EndPoints:

1. Websites covered by the dashboard

Need:

This is a generic API endpoint which gives the list of Community & Forums Websites that are being monitored by the service status dashboard.

Request:

Path
GET /websites/coverage

Response Format:

{  
   "websites": [
      {
        "name": "<name of the website eg: ZCash Forum>",
        "key": "<key of the website. Used for querying data eg: bittrex",
        "url": "<url of the website>",
       
      }
      ...
}

Sample Request/Response:

Request:
GET /api/v1/websites/coverage
Response:
{  
   "communities": [
      {
        "name": "Zcash Chat", 
        "key": "zcash_chat",
        "url": "https://chat.zcashcommunity.com"
      },
      {
        "name": "Zcash", 
        "key": "zcash",
        "url": "https://z.cash"
      },
      ...
    ]
}



2. Downtime logs for website

Need:
  • This endpoint outputs the logs for downtime of a particular website.
  • Helps user to debug the time when transaction volume was problematic (transaction volume == 0) of a particular trading pair on a particular exchange.
Request:
Path
GET /communities/downtime
Request Parameters:
Parameter Description
pair The name of the trading pair: ZECUSD, ZECBTC, etc.
start Start time of the period for which deviation is to be queried, optional.
end End time of the period for which deviation is to be queried, optional
period Time instance for which percentage deviation is needed, optional. If unset, it will be treated as now.

Sample Request/Response:

Request
GET /communities/downtime?community_key=<zcash>&period=1d 
Response:
{
    "community_key": "zcash",
    "community_name": "ZCash",
    "website_url": "https://z.cash/"
    "downtimes": [
        {  
             "start_time":"01-01-2019T00:00:00Z",
             "end_time":"01-01-2019T00:03:59Z"
        },
        {  
            "start_time":"01-01-2019T00:04:00Z",
             "end_time":"01-01-2019T00:07:00Z"
        }
}

End Notes:

  • There will be rate-limiting with 10 requests overall from an IP per minute. Note: This may change as we progress keeping in mind the server's capacity. Although, we will try our best to provide the maximum requests per minute possible.
  • Source data will be made available publicly on S3 with data dictionary provided on the dashboard itself. This is done so that users can use the data for analysis if in case the APIs are not enough for them.

Maintenance/Upkeep

Maintenance/Upkeep

Irrespective of the notion that we would be submitting followup proposals around extending the monitoring service dashboard, we would be:

  • Maintaining the system actively for the next 3 months. That means bug fixes if any with the specs mentioned in the current proposal.
  • For 1 year: support if the system goes down.
  • Since reality may be very different at the end of one year (websites archived, communities/exchanges API may change, new services being added into categories, new categories being added etc.), we would like to submit follow-up proposals to accommodate these changes, but we shouldn't be held accountable for it if we aren't able to do so.

Community Feedback on Proposal 00

Community Feedback on Proposal 00

Proposal 00 Link on ZF Grants

This document collates all the feedback we got on proposal 00 + the things we are doing in this proposal to resolve them.

Feeback is divided into 2 categories:

  • Technical
  • UX

Technical

Around feature requests on Proposal 00:

@tromer

Will the dashboard include just the current status, or also historical status (e.g., seeing past downtimes of various services)?

Dashboard will show both current and historical status as we are storing data permanently.

Combined with storing data + Grafana's ability to filter data on the frontend, you will be able to see at a glance what happened in the last 7 days, 6 months, 1 year.


User can use the filters on the top right corner of any dashboard to filter data

Examples:

Will the information (at least the current status) be publicly accessible by a programmatic API, in addition to rendered HTML?

In proposal 00, we are shipping API support for both current as well as historical status that the dashboard provides. Check API Support for more details.

Around storage cost:

@gtank

@prastut if you anticipate relatively sparse access on the deeper logs, you can save a lot of money by rotating into something like Google nearline https://cloud.google.com/storage/docs/storage-classes

@tromer

a technical matter (not disputing the proposal's merit): 12TB/yr sounds like you're keeping very detailed logs. Perhaps unnecessarily detailed raw data, where for most purposes it would suffice to keep just processed/aggregated information. For example, when I query what happened a year ago, probably I just care about the per-hour availability of each service. Not the HTTP response it gave me every time I queried it during that hour.

You guys were right. 12 TB/year was an overblown estimate. We now have baseline data to support our predictions:


^ This image shows the data we stored for the month of August.

Given that we would be increasing coverage of exchanges and trading pairs on the dashboard from:

  • 10 exchanges to 31 exchanges.
  • 2 trading pairs to 16 trading pairs.

We estimate a 3 - 4x increase in the data. We have updated the tech infrastructure cost accordingly.

Around new features:

@gtank

plus, if you really are monitoring price pairs, open-access historical data might be quite fun

To support this, we plan to create an SQL Dashboard to attach the power of SQL over the data we are ingesting, in followup proposals. What we need to discuss is given the cost of implementing the SQL dashboard, whether the foundation would be willing to fund the execution of this specific component.

@garethtdavies

Quickly checking https://www.zcashservicestatus.info/d/Pic_sbHZz/community-and-forums?orgId=1 and I was by chance visiting the forums today and they were offline (was getting a Discourse error about the forum not existing or some such). I don't see any downtime on your chart so I guess you are checking for a 200 status code for that URL? Perhaps you could additionally check for some Zcash specific info on the index page (e.g. some metadata) as I presume the hosting service in this instance will still serve 200s even if the forum is down. It's also possible the outage was less than whatever your polling frequency is and Discourse does indeed return a 404.

  • We are checking for 200 status code for the each website we monitor.
  • Since metadata wouldn't be standardized across websites, it would be pretty hard to check it robustly.

UX

@gtank

nice grafana! do you plan to write anything explaining what the metrics mean? https://www.zcashservicestatus.info/d/n-3yUXNWk/exchanges?orgId=1 seems cool but i don't really know what i'm confirming with all those OKs

@tromer

A thought: while the green "OK" boxes are very reassuring, they're a bit boring. If it's not too much work, it would be cool if each such green box also gave some vital stats (e.g., "last price $4729/ZEC, updated 10 seconds ago", or "572859 forum posts, updated 2 minutes ago").

@sonya

@aviral.srivastava how about displaying that info somewhere inside the green box? there's plenty of room in the UI, in my opinion

Mostly feedback revolved around Not Ok/Ok graphs as they are very hard for end user to understand what is being checked.

In the process of thinking of a better UX for the dashboard, I realized that a healthcheck dashboard is often looked up after something bad happens. For example consider this user story:

* Alice sees that the zfnd.org is down when she tried to access the website at 10:00 AM. 
* She refreshed once or twice (usually people think that there is some problem with their system/internet connection etc. ). 
    * From trying other websites, she realized that the problem lies with zfnd.org and not with her system. 
* Now it can happen that zfnd.org starting functioning fine. In that case, if she checks zcashservicestatus.info, the Not Ok/Ok graphs would be showing her Ok for at that instance the website is performing fine. 

Hence the purpose of service status dashboard is nullifed. Purpose from my opinion:

  • We usually check these dashboards for information of what happened in the past, rather of what could happen in the future.
  • Nobody will have the service status open dashboard before something bad happens.

Hence we are proposing to completely remove the Not Ok/Ok Graphs as they serve no purpose, rather confuse the end user about the value proposition of the dashboard.

The revamped UX contains the changes we have made across exchanges and communities and forums section.

Coverage of Blockchain Explorers

Coverage of Blockchain Explorers

Explorers Getting Monitored

Total: 6

Parameters getting Monitored:

  • Blockchain height across all explorers.
  • Transaction tally in the last block across all explorers.
  • Downtime check of Blockchain Explorers.

Methodology

  • Since we can't single out a blockhain explorer for checking the data against, we will setup a full node + of ZCash and then use the public APIs of the above explorers to compare them. Our full node's data will be the single source of truth and the explorers will be compared with that.
  • UX for blockchain explorers will be figured out while execution.
  • Tech Infrastructure:
    • Server specs: 8GB RAM, 4 CPU, Ubuntu Machine.
    • Cost:
      • $40/month
      • $480/year.

Appendix

API Links:

Documenting API links, since for most explorers, documentation is not available/poorly documented/or gives 404:

Explorers that we are unable to monitor:

  1. https://minergate.com/blockchain/zec/blocks. Reason: Discontinued API.
  2. https://zcash.plutomonkey.com/. Reason: Explorer is down.

Outcomes + Roadmap + Funding

Outcome + Roadmap + Funding

Outcomes:

We will be delivering all these components:

Funding Components:

Component 1: Value of 31 ZEC decreased by ~50% due to market volatility

  • We had requested $3500 for shipping proposal 00, at ZEC/USD exchange rate of ~$112.9.
    image
  • We got the final payout on August 10th, 2019
    image
    • Close Price on August 10th, 2019 for Binance ZEC/USDT: 56.71.
    • Price dump in % change = (56.71 - 112.9)/112.9*100 = -49.76% ~= -50%
    • 31 ZEC*56.71 USD = $1758.01
    • Delta between asked - got = $3500 - $1758.01 = $1,741.99
  • We request the foundation to fund us the delta of $1,741.99 created by the price drop in the execution of proposal 00.

Component 2: Tech Infrastructure Cost

  • In previous proposal, we missed adding funding for server and domain cost.
  • In this proposal, we also add the annual cost of running the full ZCash Node which will be used to verify the data reported on blockchain explorers.

Components:

  • ZCash Service Status Dashboard:

    • Server specs: Amazon t3a.large, 2 CPU, 8 GB RAM.

      • Cost:
        • $0.0752/hr.
        • ~ $55/month
        • $660/year.
    • Storage specs: Amazon EBS

      • Data estimate: 10 GB/month.
      • Annual estimate: 120 GB
      • Therefore we are provisioning a 80 GB storage which reduces the hassle of upgrading each month.
      • Cost:
        • $0.1 per GB-month
        • $8/month
        • $96/year
    • August, 2019 Bill Link.

  • ZCash Full Node:

    • Server specs: Vultr 4 CPU, 8 GB RAM
    • Cost:
      • $40/month
      • $480/year.
  • Domain:

Component Annual Cost
ZCash Service Status Dashboard Server $660
ZCash Service Status Dashboard Storage Cost $96
ZCash Full Node $480
Dashboard Domain $4
Total $1240

Component 3: Research + Blog:

Description Estimation
Research time which was spent in researching the UX of the dashboard + specs for this proposal 70 hrs
Time required for writing 2 blogs 20 hrs
Total 90 hrs
  • Time estimated for this component's execution: 90 hours.
  • Funding amount quote at $50/hour: $4,500

Component 4: Execution of this proposal

Timeline Description Estimation
Week 1 end Complete data ingestion for new exchanges & trading pairs.
Dockerise the entire setup.

Update documentation for developers.
40 hrs
Week 2 end Revamped UX of Exchanges & Communities will be finished and shared with the community.

Blockchain Explorers UX will be finalized.
40 hrs
Week 3 end Complete ingestion from full node.

Blockchain explorers dashboard goes live.
40 hrs
Week 4 end Finish implementation of API.
Swagger UI for the API goes live.

Blogs first draft is shared with the community.
40 hrs
Week 5 - Week 6 end Buffer time kept for development delays and getting feedback from the community.

Changes accordingly in case specifications don't match as mentioned in the proposal.
20 hrs
Total 180 hrs
  • Time estimated for this component's execution: 160 hours.
  • Funding amount quote at $50/hour: $8,000

Consolidated Funding Details:

^*^ Screenshot taken on September 12, 2019.

Consolidated Funding Spreadsheet.

Payouts

Pre-Context:

  • Previous proposal took us 3 weeks to execute and ZEC/USD price dumped by 50%.
  • Since this proposal will take us 6 weeks to execute, history can repeat itself and price may dump by 50% or more, I propose the following mechanism around token transfers:
    • Milestone 0:

      • 50% of the total proposal amount, be payed out immediately once the proposal get's funded. Let's us hedge 50% of the risk of market volatility on our end.
      • hence as a consequence, I am not adding any buffer to the payout.
      • the only risk is of the time gap between proposal being submitted and proposal getting funded. I put faith in the ZFnd team to come to a swift decision on the proposal.
    • Milestone 1:

      • 25% of the total proposal amount.
      • Buffer of 50% added to amount to account for currency exchange risk.
    • Milestone 2:

      • 25% of the total proposal amount.
      • Buffer of 50% added to amount to account for currency exchange risk.

^*^ Screenshot taken on September 12, 2019.

Milestone Payout Details Spreadsheet.

Educating the community

Educating the community

We will be creating the following 2 blogposts to help educate the community:

  • our experience of how we went about getting a grant funded by ZCash foundation. Will involve the processes we followed.
  • why we are building the service status dashboard and our long term vision of the project.

We can crosspost the blog to wherever the community wants us to post:

  • Medium

Extend data to cover historical periods

Hi, first of all, thanks for contributing https://zcashservicestatus.info!

I'd like to see metrics extend back through time. Specifically right now I'm interested in the size of the Sprout and Sapling shielded pools throughout the entire history of the Zcash blockchain.

Is it possible to extend the metrics back through time? At least for on-chain metrics?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.