natolichris / diablo-benchmark Goto Github PK

Diablo primary panics because of division by zero in extreme cases.
It would be better to detect these edge cases and give a meaningful value instead of panic.

The workload configuration file has two mandatory fields:

• secondaries
• threads
  The "secondaries" field is related to setup and not to workload. It should be specified as an operand of the diablo primary.
  The "threads" field is related to the implementation of clientinterface. It should be up to the blockchain integrator to decide how to send the workload to nodes.

Describe the bug
Run the etherum example at Master branch

To Reproduce
Steps to reproduce the behavior:

1. ./scripts/sample/start_testnet.sh
2. ./diablo primary -c scripts/sample/workloads/sample_contract_store.yaml -cc scripts/sample/blockchain-configs/ganache-using-keyfile.yaml -a "0.0.0.0:8323" --level DEBUG
3. ./diablo secondary -m ":8323" -c scripts/sample/workloads/sample_contract_store.yaml -cc scripts/sample/blockchain-configs/ganache-using-keyfile.yaml --level DEBUG --timeout 2

Expected behavior
Run succssfully.

Screenshots
Primary:
2022-09-19T15:02:05.395+0800 INFO Secondary 1 / 1 connected {"Addr:": "127.0.0.1:37920"}
2022-09-19T15:02:05.395+0800 DEBUG SendSync {"len": 9}
2022-09-19T15:02:05.398+0800 DEBUG GOT REPLY FROM 127.0.0.1:37920

2022-09-19T15:02:05.398+0800 INFO Benchmark secondaries all connected. {"secondaries": 1}
2022-09-19T15:02:05.399+0800 DEBUG tps interval {"total": 5, "thread": 5}
2022-09-19T15:02:05.399+0800 DEBUG tps interval {"total": 5, "thread": 5}
2022-09-19T15:02:05.399+0800 DEBUG tps interval {"total": 5, "thread": 5}
2022-09-19T15:02:05.399+0800 DEBUG tps interval {"total": 5, "thread": 5}
2022-09-19T15:02:05.399+0800 DEBUG tps interval {"total": 5, "thread": 5}
2022-09-19T15:02:05.399+0800 DEBUG tps interval {"total": 5, "thread": 5}
2022-09-19T15:02:05.399+0800 INFO Generating workload {"workloadType": "contract", "threadsTotal": 1, "totalTransactions per worker": 25}
2022-09-19T15:02:05.422+0800 ERROR failed to generate workload {"error": "json: cannot unmarshal array into Go struct field .Contracts.Abi of type string"}
2022-09-19T15:02:05.422+0800 DEBUG Closing Secondary 0 @ 127.0.0.1:37920

Secondary
2022-09-19T15:02:05.394+0800 DEBUG Loaded accounts {"accounts": 2000}
2022-09-19T15:02:05.395+0800 WARN Overwriting config timeout (0) with flag 2
2022-09-19T15:02:05.395+0800 DEBUG Connection OK {"ADDR": ":8323"}
2022-09-19T15:02:05.395+0800 INFO Secondary init
2022-09-19T15:02:05.395+0800 DEBUG Performing Initial Read
2022-09-19T15:02:05.395+0800 DEBUG Received Command Message {"CMD": "AQAAAAAAAAAB"}
2022-09-19T15:02:05.395+0800 INFO Got command from primary {"CMD": "PREPARE"}
2022-09-19T15:02:05.398+0800 DEBUG Connect and Init of workload handler and client interface OK {"ID": 0}
2022-09-19T15:02:05.398+0800 DEBUG OK sent to master
2022-09-19T15:02:05.398+0800 DEBUG Performing Initial Read
2022-09-19T15:02:05.422+0800 WARN failed to read {"err": "EOF"}
2022-09-19T15:02:05.422+0800 DEBUG Closing Connection to primary

Desktop (please complete the following information):

• OS:Ubuntu 21
• • go version: go1.17 linux/amd64
    -- npm: npm: '8.5.5',
    node: '17.9.0',
    -- Ganache CLI v6.12.2 (ganache-core: 2.13.2)

Additional context
Add any other context about the problem here.

We need to integrate Libra/Diem

IT could be interesting to integrate Avalanche

Ethereum clients send transactions only to their "primaryID" blockchain node.
This creates a workload imbalance. We should instead spread the workload evenly among the blockchain nodes (or add an option somewhere for the user to indicate what he wants).

Describe the bug
I'm trying to use the instructions in the https://diablobench.github.io/fresh-install at the Install yourself sections and use the
final command ./bin/eurosys workload-native-10.yaml setup.txt

I’m facing several issues with go during the deploy, because minion (v2) set the go version to 1.17.8 (while in diablo-v1 use 1.14 or greater). The step is when v2 take from repo v1 the Makefile and make these step:

go mod tidy
make reqs
make diablo

what I noticed are problems with:
golang.org/x/lint
golang.org/x/tools/go/ast/astutil
golang.org/x/tools/internal/typeparams
golang.org/x/tools/go/gcexportdata

Screenshots

Docker Container Environment

• OS: Both Ubuntu 20 and 22.04

Others
Question:
I'm trying to understand how you use minion (v2) in the paper at https://arxiv.org/abs/2311.09440, because i notice that from the "minion paper" you switch from AWS to a simple testbed that i think use just SSH.
Since i'm going to do the same, i need to understand the steps to not use AWS. In particular, are the guidelines given in the FreshInstallation/InstallYourself section useful? With setup.txt file to be used in order to execute the experiment i need on more than one machine/container simply with ssh
@NatoliChris @gauthier-voron @lebdron

Also, I'm using the Dockerfile below to solve other dependencies issues that happen during the deploy, but then i stop on the bug above.

FROM ubuntu:20.04

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update -y && \
    apt-get install -y \
        openssh-client openssh-server \
        net-tools netcat \
        cloud-utils \
        curl wget \
        git python3-pip \
        bash \
        perl-base libconfig-yaml-perl libcapture-tiny-perl libbz2-dev libreadline-dev libssl-dev libffi-dev libsqlite3-dev libncurses5-dev libncursesw5-dev liblzma-dev \
        sudo

SHELL ["/bin/bash", "-c"]

RUN echo "ALL ALL=(ALL:ALL) NOPASSWD: ALL" > /etc/sudoers.d/nopasswd

COPY id_ed25519 /root/.ssh/id_ed25519
COPY id_ed25519.pub /root/.ssh/id_ed25519.pub
COPY id_ed25519.pub /root/.ssh/authorized_keys

RUN echo "StrictHostKeyChecking no" >> /root/.ssh/config

RUN chmod 700 /root/.ssh && \
    chmod 600 /root/.ssh/id_ed25519.pub && \
    chmod 600 /root/.ssh/authorized_keys && \ 
    chmod 600 /root/.ssh/config

RUN sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config && \
    sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config 

EXPOSE 22

RUN curl -L https://install.perlbrew.pl | bash
RUN source /root/perl5/perlbrew/etc/bashrc
RUN exec bash; \
    perlbrew install --notest --force perl-5.34.0; \
    perlbrew switch perl-5.34.0

# Install other Perl dependencies
RUN wget -O YAML-1.24.tar.gz http://search.cpan.org/CPAN/authors/id/T/TI/TINITA/YAML-1.24.tar.gz; \
    tar -xzvf YAML-1.24.tar.gz; \
    cd YAML-1.24; \
    perl Makefile.PL; \
    make; \
    make install; \
    cd /; \
    wget -O IO-Tee-0.65.tar.gz http://search.cpan.org/CPAN/authors/id/N/NE/NEILB/IO-Tee-0.65.tar.gz; \
    tar -xzvf IO-Tee-0.65.tar.gz; \
    cd IO-Tee-0.65; \
    perl Makefile.PL; \
    make; \
    make install; \
    cd /; \
    cpan -i Capture::Tiny; \
    cpan -i XML::Simple; \
    cpan YAML;  
    cpan JSON

RUN curl https://pyenv.run | bash; \
    echo 'export PATH="$HOME/.pyenv/bin:$PATH"' >> /root/.bashrc; \
    echo 'eval "$(pyenv init --path)"' >> /root/.bashrc; \
    echo 'eval "$(pyenv virtualenv-init -)"' >> /root/.bashrc; \
    source /root/.bashrc
RUN exec bash; \
    pyenv install -s 3.10.6; \
    pyenv global 3.10.6; \
    pip install PyYAML

RUN apt-get clean; rm -rf /var/lib/apt/lists/

The specification of smart contract functions is not generic enough.
The notion of "ftype" is not present on every blockchain. The types of argument can change from a smart contract implementation to another.

I do not think we can go much further than a pair "contract name/path" and "interaction name". It would be up to the blockchain integrator to decide how to interpret the interaction name to make something sensitive (at worst, encoding the parameter in the interaction name e.g. "IncrementCounterBy:1")

Describe the bug
A benchmark of Solana on 4 nodes + 1 diablo node for 1000 tx/s for 2 minutes does not work (i.e. takes 12 minutes to produce a 0 tx/s throughput result)

To Reproduce
Clone minion script (commit: 95cfd11982b5480d3fb743b5d6a57c6f4318c5bc)
Checkout develop branch
Spawn 5 machines: ./bin/minion boot --verbose --key=your-key --size=5 --region=eu-central-1
Change the 'install-diablo' script so it fetches develop branch (and remove old install before git clone)
Run solana: ./bin/minion run --verbose --install=install-solana --install=install-solang solana --install=install-diablo workload-1000.yaml the-fleet-id
with 'workload-1000.yaml' being:

name: "Workload 1000"
description: "Constant 1000 tps"
secondaries: 0
threads: 0
bench:
  type: "simple"
  txs:
    0:   1000
    119: 1000

Expected behavior
Produces a result directory with non zero throughput in less than 20 minutes

Hey, I saw currently readme is for the old version. I want to try the newest version of diablo. Can public a new version of readme file or run example doc on Ethereum.
Regards.

When a secondary has more than one thread, its throughputs are averaged to give the secondary whole throughput.
It would make more sense to add them.

We need to integrate the Solana blockchain.

Some of the metrics average / median values are computed in secondary nodes. This is useless and adds work for blockchain integrators.
Let's put all of this in the primary as a blockchain agnostic code.

I made an attempt in bac3917 at core/results/event.go

We should put heavy premade workload and accounts file in a dedicated repository.
Not everyone is interested in these workloads and it only complexifies the repository structure (aside of making git clone slower on crappy networks)

It would be nice to have CI and tests.

The workload configuration file does not make possible to express complex workload types without using "premade" type.
Examples of such workloads could be:

• using more than one contract
• changing the proportion of contract interaction over time
• changing how well the workload is spread among the nodes over time
• adding the notion of conflicting transactions (with changing proportion over time)
• setting optional behaviors (e.g. poll blocks / subscribe to tx commit)

When the benchmark is done, it sequentially gets the results from all the secondaries, and (!) it synchronously waits for the results from each secondary in the same loop. It means that, the primary does not stop the secondary until it gets the result from the current secondary in the loop.

Here are the lengths of arrays with throughput windows from all the secondaries in one of the experiments with the duration of 120 seconds.

[123, 126, 129, 131, 133, 134, 137, 141, 147, 147]

For the first secondary, we get pretty much the expected length — 123 windows of 1 second duration. But for all the subsequent secondaries, the number of windows increases. Therefore, the timeout value for the first secondary was 0 seconds, but for the last one, we see additional 27 second timeout.

Current API to integrate new blockchain is very Ethereum specific (see unused functions in Hyperledger Fabric / API mismatch in Algorand).
We should put effort in a new API for a more abstract object (and add a set of "toolbox" functions if necessary).

I created an attempt for this in bac3917 at blockchains/interface.go with bridges in blockchain/clientinterfaces/bridge.go and blockchain/workloadgenerators/bridge.go