Show all transaction history rather than last statuses

Two related problems here, but only opening one issue:

1. The header used for determing the remote IP should be configurable. Not all providers use X-Forwarded-For (even if it is the defacto header) and chained proxies don't always result in a predictable result. Please change this from a hardcoded value to one having x-forwarded-for as a default, but can be overridden in the config.

2. The xff header is being split by the colon. This should be a comma.

• The first side-effect is that in chained proxies all of the IP addresses show up increasing the chances that a user can register many addresses if there is a load balancer.
• The second is that if the remote user is on IPv6, only the first 16 bits of the address are stored in the database, resulting in entire countries being limited to a single giveaway signup.

Use non-accumulative total. This will be totaled later in the balance / transaction history query.

Related #6 185ff5f

My local PC testing on a full database shows the following performance improvements in the balance query (transactions.js)...

The most common case, get the balance for 1 owner key.. This query runs every time a checkout or pricing occurs. When I run the balance query for 1 public key the new query runs in 32ms and the old query takes 1.7 sec..

If, however, I run the balance query on all rows the new query runs in 5 seconds and the old query runs in 8.3 seconds .. So, admin queries should be almost 2x faster.

I optimized by adding two columns: last_trx_event and last_pay_event in the blockchain_trx and account_pay tables. The migration will create these columns, create INSERT triggers to keep them up-to-date, and finally (and lastly) populate existing records with the data.

The migration runs on my machine where nothing else is running in less than 2 seconds.

== 20200711182630-last_event_column_optimization: migrating =======
== 20200711182630-last_event_column_optimization: migrated (1.688s)

I did not add the same design pattern to the account_pay table. Multiple attempts to pay for a domain should be unusual so I don't think that it will be worth it. So this pattern will remain unchanged:

join account_pay ap on ap.id = (
  select id from account_pay
  where account_id = a.id
  order by id desc limit 1
)

After the migration, it is possible to optimize any SQL with the following join as follows:

-    join blockchain_trx_event le on le.id = (
-      select max(le.id)
-      from blockchain_trx lt
-      join blockchain_trx_event le on le.blockchain_trx_id = lt.id
-      where lt.account_id = a.id
-    )
+    join blockchain_trx_event le on le.id = t.last_trx_event

And:

-        join account_pay_event ae on ae.id = (
-          select max(le.id)
-          from account_pay_event le
-          where le.account_pay_id = ap.id
-        )
+        join account_pay_event ae on ae.id = ap.last_pay_event

When trying to sync the database, I get errors for cyclic references in the model:

Unhandled rejection Error: Cyclic dependency found. account_pay_event is dependent of itself. Dependency chain: account_pay_event -> account_pay => account_pay_event

Unhandled rejection Error: Cyclic dependency found. blockchain_trx_event is dependent of itself. Dependency chain: blockchain_trx_event -> blockchain_trx => blockchain_trx_event

Adding constraints: false to the relationships in question gets me through, but with the side effects. Is this a known issue? This discussion is where I found my fix. I use Objection with MySQL so am not familiar with Sequelize.

Running with a fresh Postgres 13 Docker image (13-alpine).

Updating the database in a live system

First: always make the change in the sequelize JavaScirpt model definition so it will be available in the code as JavaScript objects (ex: registrations/server/db/models/wallet.js). This makes it much easier to work with the database.

Currently I make the change locally (in my database) then document the DDL (sql changes) in fio-registrations/UPGRADE.md. This works but complicates upgrades because it is manual.

In any event, include any changes in server/db/models/* in UPGRADE.md..

Changes how Coinbase events are processed. There is not enough historical information in the Coinbase events to match payments to the time-line (without the webhook), so instead of appending new events, wipe and re-apply confirmed and pending payments to the timeline (update it).

• Fixes how MULTIPLE payments are applied
• Fixes Admin refresh (data integrity) if webhooks are or were down
• Adjust balance query to account for adjustments by holding credits longer (until a registration success or cancel)

When API keys are used for interacting with the registration site, it would be helpful if a list of allowed IP addresses could be associated with a key, and if not on the list would respond with a HTTP 403 (denied) error.

It would useful to allow connecting to a db server using tls by adding a conditional which would add the following option when creating the connection at https://github.com/fioprotocol/fio-registrations/blob/master/server/db/models/index.js#L19

  dialectOptions: {
    ssl: true
  }

After a payment is confirmed, if blockchain registration tries to create the account and finds that a FIO address or domain is already registered, a review status will be used (instead of success) so we can gather more information about how this happened.

When you use registration site to renew a domain that has expired, it comes up as not registered instead and does not allow the renewal.
https://reg.fioprotocol.io/domain/renew/fio

This could be implemented as a new Admin search term "credit"

Abstract

This enhancement adds the following functionality:

• Adds CAPTCHA when price of FIO Address is 0
• Add limits on how many FIO Addresses can be registered on each domain
• Supports custom authorizations

Motivation

The Foundation has been allocated 125M FIO tokens at Mainnet to be spent on FIO Address registrations. In fact these tokens are locked and can only be spent on fees.

The Foundation has already funded the development of the Registration Site to enable integrating wallets to easily accommodate FIO Address registration.

The proposal recommends additional functionality be added to the Registration Site to enable Address Giveaways:

• CAPTCHA and limits will limit abuse.
• Custom authorizations are required to use existing Foundation Address Giveaway accounts as those accounts are controlled via msig and transfer is prohibited, and will also allow the Registration site to register FIO Addresses on private domains, if proper permissions are granted.

Specification

Add CAPTCHA

Add Google Invisible reCAPTCHA v2 to registration site when price of FIO Address is set to 0.

Add FIO Address registration limits

Each domain being added to wallet profile in Admin interface should have a limit of addresses which can be registered on that domain. Once that limit is reached, an error message should be shown to the user when checking availability of FIO Address: "FIO Address registrations no longer available for that domain."

It should be possible to specify unlimited, e.g. by leaving the field blank.

This limit only tracks registrations on that domain via the registration site and on the specific wallet profile. The same domain may exist on another wallet profile and may have different limits.

A counter of registrations completed for each domain via the specific wallet profile should be shown next to each domain.

Registrations via Admin, e.g. via Retry should not enforce these limits, but should be included in count.

Support custom authorizations

The wallet profile in Admin should be extended to optionally include:

• actor
• permission

If specified, it will be used in the authorization node (actor, permission) as well as data node (actor). Example:

{
	"account": "fio.address",
	"name": "regaddress",
	"authorization": [{
			"actor": "actor_from_wallet_profile",
			"permission": "permission_from_wallet_profile"
		}
	],
	"data": {
		"fio_address": "bob@domain",
		"owner_fio_public_key": "FIO69UqrQQg5BBneqj723i7esZT4fbp6sH2rabo6bRyHTpoyuDiSu",
		"max_fee": 40000000000,
		"tpid": "",
		"actor": "actor_from_wallet_profile"
	}
}

The request will still be signed by the master private key.

If custom authorizations are not specified, the actor/permission should be used as it is today.