brave-intl / bat-ledger

A BAT-based micropayments system for users and publishers

bat-ledger

BAT back-end servers (ledger, eyeshade, and balance)

Running locally with docker-compose

It is important to configure your .env file before attemptiing to bring up the services . See Prepare .env file below

First, install docker and docker compose.

Check out https://github.com/brave-intl/bat-ledger

You can add any environment variables that need to be set by creating a .env file at the top of the repo. Docker compose will automatically load from this file when launching services.

# To bring up all the services :
    docker-compose up

# Logs from all services presented interleaved, you can press ctrl-c to stop.
# Ledger listens on port 3001, eyeshade on 3002, and balance on 3003

# Note you can run any subset of services (e.g. only eyeshade)
docker-compose up eyeshade-web eyeshade-consumer

# You can also launch and run services in the background
docker-compose up -d eyeshade-web eyeshade-consumer

# And stop running background services with
docker-compose stop

Docker Compose Network Configuration

All containers running within the legders docker-compose context are running in the default network named "ledger". If you need to have other docker containers directly access other containers from within the network (i.e. from WITHIN the container itself, not on your local development context), you can configure your application (i.e. publishers) to join the external network "ledger" by adding the following to the network configuration of the appropriate application's docker compose file

networks:
  ledger:
      external: true

Configuration

Configuration variables are stored as environment preferences. See config.js for a list of these variables for ledger, eyeshade, and balance respectively.

StandardJS

For linting we use StandardJS. It's recommended that you install the necessary IDE plugin. Since this repo uses ES7 features, you'll need a global install of both the standard and babel-eslint packages.

Running tests

Please note: Some tests access live APIs and require auth tokens which are not stored in the repo.

Prepare .env file

1. Copy example over: cp .env.example .env.
2. Confirm .env vars match the contents of .github/workflows/ci.yaml section env.
3. Fill in the remaining {CHANGE_ME} .env vars appropriately; please consult your local BAT dev to find the answers.

Running Individial tests

bat-ledgers is executing tests using ava which can be executed via npm run ava and the requires args for any individual test can be passed to the command using npm scripts args syntax. See below:

npm run ava -- -v -s eyeshade/workers/referrals.test.js

Or, if invoking the container externally,

docker-compose run eyeshade-web npm run ava -- -v -s test/eyeshade/suggestions.integration.test.js

See the github issue where this ability was added

Build local servers

npm run docker-build

Start local servers

npm run docker-up

Postgres migrations

You can run all migrations to upgrade the schema to the latest version using:

npm run docker-migrate-up

You can reverse a particular migration by running:

npm run docker-migrate-down -- migrations/0001_transactions/down.sql

Run tests

best to do in another terminal

npm run docker-test

Testing contribution

If you are testing contributions locally (not with e2e automated tests) you need to add surveyors manually. You can do this by running bellow command when ledger service is running.

curl -X POST --header 'Authorization: Bearer foobarfoobar' --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{"adFree":{"fee":{"USD":5},"votes":50,"altcurrency":"BAT","probi":"27116311373482831368"}}' 'http://127.0.0.1:3001/v2/surveyor/contribution'

Debugging

docker-compose run --rm -p 9229:9229 eyeshade-web npm run start-eyeshade -- --inspect=0.0.0.0

gyp

You may have to install node-gyp if you do not already have it on your machine. Use this document to install: https://github.com/nodejs/node-gyp#installation. Catalina users may have a longer road.

npm install

you may also have to use npm without running the postinstall scripts. use the --ignore-scripts flag.

bat-ledger's Issues

contributions report missing owner field

retire github authentication in eyeshade

we should retire github auth in eyeshade and other services where possible, we're moving toward a publishers based admin panel and our github auth has known issues, e.g. it does not work across dynos:

if the first hit to v1/login (redirects the user to github) and the second (user is redirected back from github) happen on different dynos, the login does not succeed

apps we could remove it for currently:

• balance
• helper
• extractor

re-encrypt env file and create lock file with 5.8

remove setup scripts

setup scripts have been abstracted to tests. remove setup folder, update readme

add POST /v2/batch/surveyor/voting

for batching voting surveyor

cc: @evq @davidtemkin

Handle changes in domain registration

If you've registered potato.com and become a Brave publisher; and potato.com expires and Cate Camper registers it right away, contributions to potato.com should no longer flow to you.

eyeshade, as the canonical registration checker, should be the service which checks periodically.
it also needs communicate this info with publishers: https://github.com/brave-intl/creators-private-issues/issues/1211

call /v1/reports/publishers/contributions as part of e2e tests

expose endpoint to create a card of a given currency for owner

will require the cards:write scope

Make code more legible and testable

there are a variety of improvements that could be made to make the code more legible and testable

legible

• using imports instead of requires and filename patterns
• breaking apart monolithic files
• destructuring
• using default values

testable

• breaking apart monolithic files
• more (and exporting existing) pure functions (where available)

these items can be addressed in any order, as well as broken up by server / worker

Tests: Database should be cleaned between tests

To avoid unexpected interactions and allow use of fixtures, tests should clean some or all database data during setup or teardown.

publisher / owner balance calculation doesn't account for fees

Dedup npm scripts

use -- to pass flags to the following scripts
lint
docker-up

publisher statements are not generating

Publishers are reporting that their statements are not generating.

See: https://twitter.com/robek_world/status/981563504686784517

Handle reconnect error when invoking binance websocket

TypeError: reconnect is not a function

Add /v3/publisher endpoints to bat-balance, and have bat-client use that service instead of bat-ledger

the /v*/publisher/* endpoints are read-only for the browser, so it's a good thing to move over.

move source-of-truth for publisher rules to eyeshade

1. Deprecate these calls in the ledger:

   GET /v2/publisher/identity
   GET /v2/publisher/identity/verified

2. Move these calls from the ledger to the eyeshade server:

   POST /v2/publisher/ruleset
   PATCH /v2/publisher/rulesets
   PUT /v2/publisher/ruleset/{publisher}
   DELETE /v2/publisher/ruleset/{publisher}
   GET /v2/reports/publisher/rulesets

3. Add messages from eyeshade to ledger that provide the functionality in the five calls above.

implement swaggerstats equivalent in runtime-prometheus.js

cf., http://swaggerstats.io/apidoc.html#operation/%2Fswagger-stats%2Fmetrics

Name	Type	Labels	Description
api_all_request_total	counter	-	The total number of all API requests received
api_all_success_total	counter	-	The total number of all API requests with success response
api_all_errors_total	counter	-	The total number of all API requests with error response
api_all_client_error_total	counter	-	The total number of all API requests with client error response
api_all_server_error_total	counter	-	The total number of all API requests with server error response
api_all_request_in_processing_total	gauge	-	The total number of all API requests currently in processing (no response yet)
nodejs_process_memory_rss_bytes	gauge	-	Node.js process resident memory (RSS) bytes
nodejs_process_memory_heap_total_bytes	gauge	-	Node.js process memory heapTotal bytes
nodejs_process_memory_heap_used_bytes	gauge	-	Node.js process memory heapUsed bytes
nodejs_process_memory_external_bytes	gauge	-	Node.js process memory external bytes
nodejs_process_cpu_usage_percentage	gauge	-	Node.js process CPU usage percentage
api_request_total	counter	methodpathcode	The total number of all API requests
api_request_duration_milliseconds	histogram	methodpathle	API requests duration
api_request_size_bytes	histogram	methodpathle	API requests size
api_response_size_bytes	histogram	methodpathle	API response size

minimumReconcileTimestamp is expected to be ms since epoch not s since epoch

we were sending unix time (as is incorporated into grants), but the browser assumes javascript style ms since epoch. changed in the server promotion payload since that doesn't require a new release.

remove site verification from eyeshade

have payments always use the current contribution surveyor

see https://github.com/brave-intl/bat-ledger/blob/master/ledger/controllers/wallet.js#L244, we use the client provided surveyor. depending on client behavior perhaps we can reject payments where the surveyorId does not match that of the current surveyor

npm install fails if you don't have an ssh key

The root cause appears to be fixed in bitgojs
BitGo/BitGoJS@6b2296f
we should update when a new release is available

"staging-lite" docker compose environment

bring up all relevant interoperating services
ledger services, grant server, referral server and publishers

upgrade old surveyors to un-stick contributions

currently we have a number of ledger clients stuck using old surveyors instead of the current one. it seems most likely that this is caused by brave-intl/bat-client#35 in combination with the fact that some clients reconcile date was not pushed back on grant claim and grants were made unredeemable before their maturity date.

we will fix the client side - to go out in a browser-laptop release but as a quick fix we should unstick clients by upgrading old surveyors servers side on demand.

address all outstanding nsp check advisories

@evq @mrose17 we should revisit and come up with a game plan regarding when this should happen to reduce exposure

verify all channels sent to POST /v2/owners by publishers

currently only non-website channels are verified (all are sent by publishers), but with brave-intl/creators-private-issues#940 we will need this change to keep things in sync until we can remove eyeshade's copy of the channel/owner relationships

blacklist fraudulant publishers and do sanity check when creating report

blacklisting publishers currently happens manually on voting document basis
mark publisher's excluded boolean if they are registered in our blacklist
if excluded is true, do not include in payout information
take the publisher, check the blacklist, flip the bool if

create mongodb indeces for blacklist and query them
  eyeshade/workers/wallet.js:voting-report
    mark excluded
  eyeshade/workers/reports.js:report-publishers-contributions
    sanity check to make sure no publishers in report have

update npm deps to pass travis tests

Reduce Rate Limit for ledger-web:3001/v1/grants/{grantId}

isolate why this is happening and allow tests to turn off / reduce time during testing

during the script

npm run test-integration

or, if using the docker container

docker-compose run --rm -v $(pwd)/test:/usr/src/app/test ledger-web npm run test-integration

the following error is received. if the endpoint is hit too often

status: 429
http://ledger-web:3001/v1/grants/d6a9c116-5779-4442-a0dc-6237e295ae29
{ statusCode: 429,
  error: 'Too Many Requests',
  message: 'try again in 86400 seconds' }

eyeshade reports should be fetchable without github login for local dev

remove v1 legacy api support (bitcoin)

non-constant time comparison used for simple token auth

might be hard to pull off due to latencies, but I think there's a potential side channel attack
@diracdeltas fyi

First request to helper's /v1/rates results in a subset of values

when hitting /v1/rates immediately after the server starts up, a subset of rate conversions are sent back, instead of the full 20, which is denoted in the helper tests.

To reproduce, shut everything down, start up the servers, and run the helper integration test and see it fail the first time because only 8 values are sent back. Sometimes 9 are sent back, depending on if other tests are running in parallel.

Add bat-helper server

and have bat-ledger and bat-eyeshade talk to it instead of the exchange rate services directly

Fix invalid error check

there is an invalid error check that occurs on ledger/controllers/registrar.js:164

GET /v1/owners/{owner}/wallet returns only settlement info for one channel

To implement https://github.com/brave-intl/creators-private-issues/issues/960 publishers needs to know the total amount an owner earned during the last settlement.

It appears as though a settlement document only has the information for a single channel, and we are only returning one settlement per owner in this request.

bat-ledger/eyeshade/controllers/owners.js

Line 467 in 4f7562a

entries = await settlements.find({ owner: owner }, { sort: { timestamp: -1 }, limit: 1 })

cc @evq

BAT balance is gone

cf. brave/browser-laptop#13354

expose all possible uphold card currencies in the owner wallet endpoint

currently we expose the list of currencies that the user has a card for, we should add possible currencies such that the response includes both lists

Support additional uphold conversions to purchase BAT

e.g., DASH, BCH, BTG.

this requires adding the appropriate network address. we probably want to rework the algorithm so that an address generation request is sent to uphold only if the user selects a particular alt-currency (at present, a wallet creation gets addresses for BTC, ETH, and LTC).

fix prometheus support to report heroku dynos all at once

Bringup CI

create full lifecycle for grant server during tests

during the test for settling a grant, the tester should not have to manage the db before running the tests. manual management can cause the db to be in poor state and for tests to fail. instead, automate by making sure that during a test, a single grant is added to the db, it is used, and subsequently flushed from the redis db by the end of the test.

Proxy ledger publisher verification related endpoints to Publishers app

In the Brave browser, the payments tab has publisher verified green checkmarks. This green checkmark is queried from Ledger.

Instead of Ledger fetching this information from Eyeshade, we're going to fetch the verified_status for an owner from the publishers api.

We'll be disabling the data propagation in a separate task.

Update ledger documentation to reflect changes for BAT Mercury

Remove SiteChannel verification

To simplify Eyeshade and dedupe data, we decided to move the Site channel verification flow so it's entirely within Publishers (https://github.com/brave-intl/creators-private-issues/issues/940). The Publishers task and this one need to completed together.

As a result, Eyeshade would no longer have verification tokens or unverified channels.
(TODO: Check if unverified channels exist in Eyeshade).

Disable propagation of channel verification from eyeshade to ledger

@evq

Add Promotion creation to Tests

while it is implicit currently, the promotion creation under the npm script create-promotion should be a test that can be isolated and integrated with the rest of the testing flow

Owner accounts can't be easily deleted

Many publishers have requested that their accounts be deleted. We do not have an easy way to do this. Right now we must manually delete the records from both eyeshade and publishers. This is error prone and not scalable.

We should create some endpoint on eyeshade that publishers can hit that deletes all records associated with an owner (owner, their channels, token, etc). Something like this would work:

DELETE /v2/owners/{owner}

Related to https://github.com/brave-intl/creators-private-issues/issues/969

Replace satoshis with generalized denomination

• eyeshade-server @mrose17
• ledger-server @evq

Relation download does not exist

when starting up the extractor worker, the relation download errs in

npm run start-extractor-worker

saying that it does not exist.

eligible publisher owner data is inaccessible during payment aggregation