Improving DataLayerService read performance about knix

[ruichuan]

created a new branch "feature/datalayer_lease" for this feature.

[manuelstein]

It's not a lease, because that would imply the local data layer [cache] has a temporal ownership of the global data, which it doesn't - any other local data layer can write the same identifier to the global backend, which renders the local cache content invalid. It's an expiration timer typically labelled TTL.

[iakkus]

As an alternative to the above solution, one could utilize Redis as a caching layer in front of Riak. See https://riak.com/products/riak-kv/redis-integration/index.html.

In this alternative, the DataLayerService would become a translation layer for storage operations:

1. If the operation is a KV operation, then the DataLayerService will redirect it to Redis (which has its own synchronization with Riak).
2. If the operation is a CRDT operation, then the DataLayerService will redirect it to Riak.
3. Other operations related to other features (e.g., triggerable tables) need to be figured out, but will most likely use a combination of Redis and Riak.

As a result of these modifications, the DataLayerService will not cache anything; thus, offloading cache-related problems like invalidation to Redis. Furthermore, its logic will be simplified, so that it can become part of each sandbox. As an implementation optimization regarding JVMs, it can be merged with the queue service.

@ruichuan, @paarijaat: Did I miss anything?

[iakkus]

As far as I can tell, the data layer service is only used by sandboxes (including management) and the storage frontend, which is going to be removed (see #28). After that, the data layer service will be the only remaining component in the bare metal setup that requires the fluentbit server to send the logs to elasticsearch (in fact, the fluentbit server is not utilized in the knative setup anyway).

If the data layer service moves inside the sandbox, we can also get rid of the extra fluentbit server in bare metal, because the sandbox logs are already sent to elasticsearch. That move would affect the way the management service is bootstrapped, though.

[ruichuan]

@iakkus thanks, it captures our discussion well. The data layer service (DLS) is a bit more than a simple translation layer of redirecting requests to riak/redis for processing. DLS needs to use respective client libraries to organize operations in certain ways for correct processing. I agree that the new `riak_redis' approach can simplify DLS.

Regarding where to put DLS (either standalone or inside a sandbox), we can decide later. It anyways is a simple move.

[ruichuan]

switched this work to "feature/redis_caching" branch