diem / off-chain-reference

An off-chain reference implementation: Supports exchanging payment information, KYC data and attestation of KYC data between VASPs.

Note to readers: Silvergate Capital Corporation announced in January 2022 that it acquired intellectual property and other technology assets related to running a blockchain-based payment network from Diem, further investing in its platform and enhancing its existing stablecoin infrastructure.

---

Diem Core implements a decentralized, programmable distributed ledger which provides a financial infrastructure that can empower billions of people.

Note to Developers

• Diem Core is a prototype.
• The APIs are constantly evolving and designed to demonstrate types of functionality. Expect substantial changes before the release.
• We’ve launched a testnet that is a live demonstration of an early prototype of the Diem Blockchain software.

Quick Links

Learn About Diem

• Diem Protocol: Key Concepts
• Life of a Transaction
• JSON-RPC API Specification
• JSON-RPC SPEC

Getting Started - Try Diem Core

• My First Transaction
• Getting Started With Move

Technical Papers

• The Diem Blockchain
• Move: A Language With Programmable Resources
• State Machine Replication in the Diem Blockchain

Governance

• Diem Improvement Proposals

Blog

• Diem: The Path Forward

License

Diem Core is licensed as Apache 2.0.

off-chain-reference's Issues

Make hrp in all LibraAddress interfaces mandatory (no defaults)

Having default args with hrp=lbr leads to developers missing passing in appropriate hrp depending on network. Remove all defaults and make it mandatory to pass in all hrp values

How can `PaymentProcess` call back to Sample VASP

PaymentProcessor is an engine processing payment commands with the logic defined in payment_logic.py.

When a payment becomes settled, usually the sample VASP wants to do something, such as logging, checking on chain etc. The question is, how can PaymentProcessor notifies the sample VASP of this ?

We can potentially do what we want in has_settled in BusinessContext. However I think this is a more proper place, because we make sure the message is sent out at this point

Use latest versions of payment objects when appropriate

When calling callback, and in other places, we call payment_process with whichever version of an object was provided. However, we know that any updates to a non live object will fail. Therefore it is better to keep an index of the latest version of a payment object, and call those functions with that latest version.

Receiver signature request & provision should happen before ready_for_settlement is set

I re-read the spec today. It seems the assumption in the spec is that the receiver first provides a recipient signature, and then VASPs can move to ready_for_settlement (the finality barrier). This makes sense since it allow parties to abort the transaction if the signature is no good.

So we should change the model of payment processing / status updates to have this flow instead of doing recipient signature flow after both sides are ready_for_settlement.

Refactor the executor error handling logic

• Make all errors expressed as Exceptions instead of a mix of return codes and exceptions.
• Clean up the exception hierarchy and catch exceptions at the right level at the right place
• Ensure invariants hold, eg. even when exceptions happen sequence business commands when requested.
• Unify validation handling (parsing, checks, logic).
• Allow a context to be passed to the validation functions (this is for the business context).

Persist the origin field in ProtocolCommand(JSONSerializable)

We seem to not be persisting the origin field when we JSON serialize the ProtocolCommand for storage (not networking). So we should:

• Serialize and deserialize when we JSON_STORE the object
• Ensure we do not serialize de/serialize when the JSON_NET flag is set
• Have tests to prevent regressions (and catch future bugs like this).

Implement the Get Payment Data API

The V0 off-chain spec specifies a web API for requesting information about objects by payment ID: https://fb.quip.com/cAfoAIM9XV3c (See Get Payment Data).

For this we need to keep track of the latest active Payment object with a specific ID and return it when requested on the specific API endpoint.

Storage Efficiency

The current storage interface StorableFactory uses a key-value store, and implements 'by hand' dicts, lists, transactions, and crash recovery. Other storage systems may be better suited to what we need in off-chain in terms of:

• Better performance, particularly for iteration operations (right now we are forced to keep these to a minimum).
• Features, since they already implement key abstractions (list, dict), provide transactional semantics and crash-recovery.

This issue is about exploring:

• Cleaning up the interface to StorableFactory to allow others to reimplement it using different storage mechanisms.
• Experimenting with alternative storage mechanisms, preferably some of those present in the python standard library and with few/no dependencies beyond it for the open source wallet.

Implement reference signature / verification API

We assume that signing and verification of signatures and certificates is a VASP concern, and delivered via the business context. However, for the open source wallet, as well as reference for others, and for conformance testing we should implement a reference implementation, and use it for checking the validity of the signature and signature formats.

Names of send_request and retransmit are misleading ...

Via Lu:

VASPPairChannel.send_request (https://github.com/novifinancial/off-chain-api/blob/master/src/offchainapi/protocol.py#L241) is a bit misleading. It is said to a hood to send request to other VASP, but it actually only generates the message to send. Same for retransmit

Store signed messages to not lose non-repudiation property.

Right now, we only store unsigned messages and discard the signature -- losing the non-repudiation property.

The challenge with not storing only signed messages is that upon key rotating, we may lose the ability to recover the messages (as jswcrypto embeds the signature and the message into a single blob). Thus, we should store a tuple of (verification key, signed-message), in such a way that we can always extract the message.

Adapt KYC data and Actor structures to new decsions

The new decisions are:
(1) We do not sign KYCData objects separately but only as part of the overall Requests objects.
(2) We do not do much validation of what is contained within the KYCData objects, and instead just pass on the JSON like object to the business context for checks.

Potential re-designing storage APIs

Today we have storage API in elegant pythonic styles (__getitem__, __setitem__ etc). However the following functionality are not naively supported with this design:

1. allow users to shard data (no way to define sharding key)
2. allow async I/O to a. integrate with internal async APIs and b. improve I/O concurrency

Item 2 is related to data races and is related to this issue #104. More specifically, we want to keep the APIs elegant but also eliminates the possibilities of race conditions, potentially by using locks.

For Item 1, we discussed the options of sharding. It seems like we don't have a lot of wide selection of sharding keys: they need to be available in read path. Reference ID and storage key (the key we use in the k/v store) look like two good start points. We can also allow users to define the sharding algorithm/strategy.

A side note is this may be a good chance to rethink how the storage layer works.

Add proper Sphinx API docs

Add a full Sphinx based API documentation, tutorial etc.

Naming and documentation changes and cleanup

1. KYC
   "extended KYC" is just regular KYC, should be renamed.
   "additional KYC" is equivalent to extended KYC, naming additional KYC is appropriate.

2. Signature
   The comments in get_recipient_signature in business.py should say too return a string value of the signature, not bytes, since add_recipient_signature in payment.py expects a string argument. We should also indicate it is a hex value of the bytes.

3. Business Interface
   There are several business.py interfaces that are not used. We should remove them if we don't have plans to use them for easier implementation.

Potentially send changes to the counter-party upon finalizing the process of a payment

In payment_logic.py, we may need to communicate changes to the counter-party as commands in the finally block of payment_process .

Simplify the basic VASP-to-VASP protocol

Attempt to simplify the VASP-to-VASP base protocol to make compliant re-implementations easier:

• Remove speculative creation of objects, and therefore restrict concurrent commands to those that do not act on objects not previously committed.
• Separate driving the creation of new commands from processing request / replies, to prepare the ground for an async flow in business context.
• Explore implementing pipelining of commands through a special batch command, rather than allowing for a sequence with no confirmations as it is done now.

Unused argument `version_number` in `get_object` of `payment_logic`

In the function get_object(self, version_number, dependencies) of payment_logic, the argument version_number does not seem to be used. Is it intentional?

Make the interface between command processor and business context crash tolerant

Right now if we crash, we do not automatically replay any pending new payments to the context in order to resolve them and get new commands. We should persist enough information, and then upon recovery ensure we do replay them, and progress the protocols.

Clarify conflict resolution process

How are conflicts meant to be resolved? With only two participants we cannot tolerate errors.

Allow unhosted wallets limited access to the Off-chain API

On V1 we will have unhosted wallets and it may be beneficial to allow them to provide off-chain information. It has yet to be decided what level of access they should have, and for sure they should not be provided with personal information. They may also be completely byzantine.

Implement reference Stable ID API

How stable IDs and subaddresses are derived and managed is a VASP concern and delivered through the Business Context. However, to illustrate options we should provide a sample implementation for the open source wallet that is secure.

Truthiness of StorageDict and StorageList

I just realized this interesting issue:
A Python object is always considered truthy, unless it has defined either of these dunder functions:
__bool__ and __len__

object.bool(self) : Called to implement truth value testing and the built-in operation bool(); should return False or True. When this method is not defined, len() is called, if it is defined, and the object is considered true if its result is nonzero. If a class defines neither len() nor bool(), all its instances are considered true.

See: https://docs.python.org/3.8/reference/datamodel.html#object.__bool__

Both StorageDict and StorageList define __len__, which makes them falsy when their DB are empty (the __len__ returns 0).

This seems to sound right but I'm not sure it is what we expect. For example, in new_version, if param store is falsy (i.e. the db is empty), we do a deep copy. However, we may well expect the DB to contain that versioned object, which is a miss.

If we want to change the semantics, we can override the __bool__ function to always return True

Import / Export signature keys in PEM format (or other standard).

Importing and exporting from PEM should be easy as jswcrypto exposes two functions for that: export_to_pem and from_pem. However, the following bug in the stable version of jswcrypto (the one we are currently using) prevents us from importing keys from PEM.

In the function _import_pyca_pri_okp, the stable of jswcrypto updates internal parameters by calling crv=params['crv'] which results into a KeyError. A new version of jswcrypto (version='0.8.dev1, according to their new setup.py file) fixes the problem by replacing that call by: crv=self._okp_curve_from_pyca_key(key): https://github.com/latchset/jwcrypto/blob/544cdf23c4d62aba7589b78330235d18bb55752a/jwcrypto/jwk.py#L467

Connect to networking framework

How to close a channel?

I added a function close_channel to OffChainVASP that I use to close TLS connections. I guess it will be useful if the peer VASP decides to change their TLS certificate on chain. Should we use that function to purge some other state as well?

Add richer information on protocol failures

Add a richer context to protocol failures. For example, add the expected sequence number when we signal we reject a command request because we are missing the previous ones.

Async IO exploration

I think async io is kind of standard in modern backend, since it can separate latency and throughput. We all agree we need with this intention, and if we want to make it async, it's better to do it now. When we have more and more code in the future, it will be harder to convert it to async.

What prevent us to do this: @gdanezis think async io have problem for the crash recovery. Would you mind explain in detail about your concern here?

so I open this issue to track the effort of exploring async io.

Handling None responses from VASPPairChannel.

There is a case in VASPPairChannel, where handle_request returns None instead of generating a response to send back to the client:

# As a server we first wait for the status of all server
# requests to sequence any new client requests.
    if self.is_server() and self.num_pending_responses() > 0:
           self.pending_requests += [request]
           return None

This is the case where the server is placing the client's request into the waiting queue to wait for responses to its own requests first.

Currently, the server replies to the client with an error (asking it to wait) when this happens. Instead, it would be nice to make the client's network request interrupt processing until a response is ready (should not take too long), and defer the execution until a bit later.

Should we check timestamps format in PaymentAction?

The class PaymentAction redefines custom_update_checks; should we check that the timestamp of the payment is in the right format? If we do, which format?

Check that other's status is valid in payment logic

In payment_logic.py, check that other's status is valid upon checking a new payment (check_new_payment) or checking an update of a payment (check_new_update).

Implement retransmission into the networking stack.

Right now we do not automatically retrasmit? We should at some point integrate this, and test robustness.

Add batches to settle payments

Currently all individual PaymentObjects need to be settled on chain individually. We have plans for moving to batching for V1. We should implement this.

Make enum for 'success' and 'failure' in response messages

Right now we treat these as strings. They should really be an Enum type.

Persist command sequences & shared objects

Persist and recover command sequences in VASPPairChannel

Define the full OffChainVASP to provide channels bettween VASPs

Test for coverage

Test for coverage and add additional tests to reach 100%

Integrate into the logging framework

We need to log event, error and warnings into a unified logging framework once one is defined.

[Feature Request] Adding customized locking support for distributed env

There are two critical sections in Off Chain API:

1. Sequencing commands

Problem:

Off Chain API resolves synchronization issue by using a reentrant RLock. However, as discussed before, when we run it in multiple hosts (or any forms of non-shared memory), we still have the synchronization issue.

Proposed Solution:

To solve this problem, we can allow users to implement their own locking mechanism by taking users' own locks (in the form of context manager, for example). We can also use RLock as the default if the user does not provide any.

2. Processing commands

Problem:

process_command_success_async is the main entry point to process a command, invoking user-defined business logics, which may be not idempotent. If a command is handed to multiple threads/instances to process, result may be undesired (for example, in a distributed VASP, which instance should do retry_process_commands?)

Proposed Solution:

Add and expose a customized lock to users in the beginning of process_command_success_async to avoid that. (and I guess process_command_failure_async should be fine?)

Check that we end up in a valid status upon processing a payment

Test that the new status is valid upon completing a payment processing in payment_logic.py (ie. in the finally block of payment_process).

[PaymentLogic] Only return signature if own VASP asks for it

In the payment_logic process_payment the condition:

if role == 'receiver' and other_status == Status.needs_recipient_signature:

is not quite right. We should instead only return the signature if our own VASP states it should be returned.

Serialization and parsing

At some point we will want to talk to a network, and we will need to serlialise and parse requests, replies, commands, objects etc. The baseline is to use JSON with escaping byte strings as base64/58. We also need validation to ensure what we parse is the right type.

[Core] New command - sync / async: return a future with final result

Right now the new command of Vasp (in core.py) waits until the payment is created, but not until the payment is resolved (to settle or to abort). The proposal is to return a future of the very final result once the payment is either settled or aborted. This will allow someone to create a new command with a payment, and await the very final decision!

Handle call backs after triggering `BusinessAsyncInterupt`

In payment_logic.py, register call-back for when an async operation is done; ie. after a BusinessAsyncInterupt is triggered.

Enhance concurrency and synchronization handling

Today we use the global DB lock (underlying is python rlock) when handling requests/responses, with two purposes:

1. as a mutex to only allow one runtime in the critical section, in order to avoid data races
2. as a pre-DB cache to save DB connection buildings etc.

This works well for running the code in a single host, but has the following limitations:

1. not work for multiple hosts deployment (lock can't be shared across hosts)
2. limits concurrency (lock has coarse granularity)

Given that we greatly simplified the request/response processing logic (by removing global ordering), we now can explore some systematic methods to resolve the issues above:

1. define a lock interface and allow users to plug in their lock implementations (we can provide rlock as the basic/default)
2. use finer granularity lock, based on payment or object

We discussed different concurrency level and briefly summarized them into two types:

1. single process (roughly equivalent to single host)
2. multi processes (roughly equivalent to multiple hosts, aka distributed system)
   We believe the above proposal works for both scenarios.

Refactor code to export correct abstractions based on sample service

Now that we have implemented the sample service we realize:

• We need cleaner interfaces to handle commands
• Cleaner interfaces to handle re-tries of interrupted commands.
• Better unit testing to ensure faults to not exhibit themselves on integration tests.

Include warnings in low-level protocol sequencer and execution to support conformance testing.

In many places in the lower level execution framework and sequencer we quietly ignore deviations from the protocol. However, this is the reference implementation of these protocols, and others may use our implementation to do conformance testing for their implementations.

We should add soft warnings everywhere where we detect the other side is deviating from the protocol, and have a way to turn on the warnings to inform others who are trying to build a conformant implementation that there might be a mistake in their code.

Will we allow partial updates to command?

I have this question because PaymentCommand.get_object() takes dependencies (as base versions). As of today there does not seem to be actual partial updates to objects (as we avoid to do so). If we do not have plans in the future, maybe we don't need to load the dependencies at all ?

https://github.com/libra/off-chain-reference/blob/master/src/offchainapi/payment_command.py#L46

Obligation release should happen in case of exception

                        # Crash-recovery: Once a request is ordered to
                        # be sent out we can consider this command
                        # done.
                        if self.obligation_exists(other_address_str, seq):
                            self.release_command_obligation(
                                other_address_str, seq)

should also happen in exception handling l273-281 of https://github.com/novifinancial/off-chain-api/blob/master/src/offchainapi/payment_logic.py

Remove `validate_recipient_signature` from BusinessContext?

Should we remove def validate_recipient_signature(self, payment): from the business context? It made sense when we didn't want to handle signature verification in the OffChainAPI, but now we do it to authenticate VASPs (and we even use the compliance key for that).