asonnino / key-transparency Goto Github PK

Type	Sub-type
End-to-end Evaluation	Common case
End-to-end Evaluation	Crash-faults
Witnesses only evaluation	Common case
Witnesses only evaluation	Crash-faults
Micro benchmarks (re-run)

End-to-end Evaluation

Setup: One benchmark client submitting requests at a fixed rate to the IdP. The IdP interacts with a geo-distributed (5 AWS regions) committee of witnesses.
Baseline: Batch-size of 1024, 10 witnesses, updates size of 64 bytes, every run is the average of 3 independent runs.

Common Case

Graph 1: X-axis=throughput and Y-axis=latency; setup a benchmark client and vary the input load until the latency spikes. Repeat with a committee size of 10, 20, 50 witnesses.
Graph 2: Using the same experimental data of Graph 1 (no extra experiments), set a maximum latency (depending on the result of Graph 1); plot X-axis=committee and Y-axis=throughput.

Crash-faults

Graph 3: X-axis=throughput and Y-axis=latency; setup a benchmark client and vary the input load until the latency spikes. Repeat with a committee size of 10 containing 0, 1, 3 dead witnesses.

Witnesses only evaluation (Optional)

It aims to benchmark the witnesses (without the IdP) to identify the bottleneck (witnesses or IdP). If the witness are not the bottleneck, they can run on cheaper hardware and futures efforts should be devoted to parallelize/distribute the IdP.

Setup: No IdP, one benchmark client per witness (collocated with a witness). Witnesses are geo-distributed like in the previous experiments.

Graph 4: X-axis=throughput and Y-axis=latency; setup all benchmark clients and vary their input load until the latency spikes. The total load is shared among all benchmark clients. Repeat with a committee size of 10, 20, 50 witnesses.

Graph 5: Using the same experimental data of Graph 4, set a maximum latency (depending on the result of Graph 4); plot X-axis=committee and Y-axis=throughput.