Setup for throughput optimization ? about hsds

[jreadey]

I've been working directly with @PjEdwards on this issue, so I will just summarize the results of our investigation here.

The server returns 503 ("server busy") when there are more than 100 inflight requests per node. The idea is that the client will throttle back when it sees this so that the workload on the clients and server can be roughly balanced.

In the particular tests that were run, each client request to the SN node (for a hyperslab selection) generated 100's of SN requests to DN nodes (each touched chunk resulted in a additional SN to DN request) so it was likely to hit this limit with a modest number of client requests.

Regarding CPU usage, in my testing I saw high cpu utilization per node which indicates that the limits on inflight requests look about right. You can run: $ docker stats $(docker ps --format={{.Names}}) to see the container CPU utilization.

To improve throughput there are a few options. Best approach will depend on the particulars of the client and hardware set:

1. Reduce request rate when the 503 errors reach a certain threshold. Going past a small percentage of 503 errors will result in a large amount of wasted effort on the server
2. Move to a larger instance and run HSDS with more nodes
3. Run multiple HSDS servers behind a load balancer
4. Run multiple servers in a Kubernetes cluster

Also, for AWS deployments I'll be adding support for using AWS Lambda rather than using the DN nodes (for read requests). Lambda allows higher amount of parallelism (up to a 1000 requests) than would be practical with just SN and DN nodes. Stay tuned!