tribler / bami

BAMI 🍜 protocol prototype: Ultimate Toolbox for Collaboration

Tribler

Towards making Bittorrent anonymous and impossible to shut down.

👉 Download the Latest Release 👈

Tribler aims to give anonymous access to content. We are trying to make privacy, strong cryptography, and authentication the Internet norm.

We implemented our own enhanced Tor-like network for anonymous torrent downloading, with hidden services based seeding and end-to-end encryption.

Since 2004, we have been working to ensure that "the only way to take Tribler down is to take The Internet down" (but a single software bug could end everything).

Socials, Support and Contribution

Want to hang out with other Tribler users?

• Join our !

Have you found a bug or do you have a feature request?

• If nobody else has already, open an issue!

Lost in development?

• Check out our for development and building instructions!

---

---

License

This file is part of Tribler, Copyright 2004-2024. Tribler is licensed under the GNU General Public License, version 3 (GPL-3.0), as published by the Free Software Foundation on 29 June 2007. Tribler is free software: you can redistribute it and/or modify it under the terms of this license. Tribler is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. For more details, see the full terms and conditions in the LICENSE.txt file at the root of this repository, or visit https://www.gnu.org/licenses/gpl-3.0.html.

bami's Issues

Move next peer selection strategies to a separate file

The strategies to select a peer during the frontier gossip are currently located in the gossip.py. Even though the are an essential part of the gossip strategies, they can be moved to a separate file for a cleaner code organisation.

Should the click dependency be a dev dependency?

Currently, the click library seems to be required in console.py. Not only is this console implementation incomplete, I'm also not sure if click should be considered a project dependency. We might consider making click a dev dependency instead, or remove it at all if the console will not be implemented further.

The "smart" peer selection strategy requires explanation/comments

BAMI currently has the SmartPeerSelectionStrategy implemented. However, there is no explanation whatever how this strategy is considered smart.

Revise the need for a personal ledger with hash pointers

I'm not entirely sure if the current design requires users to explicitly store a personal ledger, i.e., maintaining a chain where each block links to the previous one with a hash pointer. Even if we would drop the hash pointer to the previous block in ones chain, double-spends can still be detected.

The future of BAMI

I wrote this issue to organize a few thoughts on the future direction of the BAMI repository. The repository did come up a few times in our discussions and meetings, yet there are no concrete plans on how we can proceed with this codebase. I believe with sufficient effort and focus, this repository can become instrumental in our labs' experiments.

BAMI essentially started as an implementation of Plexus and shifted towards being an “accountability toolbox”. The idea of Plexus, however, has changed significantly (both code-wise and conceptually) whereas this repository has remained untouched since last year. At the same time, I see an increasing need for a Python framework with distributed system data structures so we can quickly prototype any ideas that we would have.

Whereas IPv8 provides basic, yet very robust networking services, I would envision that BAMI provides developers with primitives to build decentralized applications. This would, for example, include commonly used data structures like:

• Bloomfilters and its variants
• Merkle trees/dags
• Skiplists (?)

All implementations should be accompanied by representative unit tests. Unlike the policy with the Tribler repository, this repository can contain dead code or unused data structures.

Since BAMI builds upon IPv8, developers should be able to quickly combine these data structures with their customized overlays. BAMI would then also provide the following services:

• Sophisticated methods for peer discovery and organization
• Overlay peer services such as witnessing
• Routing services

Compared to libp2p, we would avoid a monolithic library while at the same time provide an academic toolbox with primitives for P2P research. This approach would solve two problems:

• Repository bloat: right now, we have many different repositories under the Tribler organization. TrustChain is integrated into the AnyDex market, making it hard to find. We should, for example, integrate TrustChain in BAMI.
• Redundant engineering efforts: it is a given that future students will re-implement (parts of) our existing implementations, maybe because they are underdocumented or hard to find. A unified framework would hopefully address this issue.

I would also like BAMI to support simulation, e.g., by integrating my discrete event asyncio loop.

Thoughts/comments? :D

Error when running CI tests

ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    coverage[toml] from https://files.pythonhosted.org/packages/e4/6a/fa167da3ac886b4777bfe37a86a2d664637dff12837dd6187ce9cdd5eb70/coverage-5.3-cp38-cp38-manylinux1_x86_64.whl#sha256=0f313707cdecd5cd3e217fc68c78a960b616604b559e9ea60cc16795c4304259
Session tests failed.
Ran multiple sessions:
* black: failed
* tests: failed
Error: Process completed with exit code 1.

The arguments to create_subcom are not clear

The create_subcom method currently has the following signature:

def create_subcom(self, *args, **kwargs) -> BaseSubCommunity:

From this signature alone, is not clear what arguments are expected here. The call to this method gave me a bit more information, but I think the expected arguments should be added to the signature:

subcom = self.subcom_factory.create_subcom(
                subcom_id=subcom_id, max_peers=self.settings.subcom_max_peers
            )

Add contributing guidelines

We should add a CONTRIBUTING.md document that describes the intended development process.

Suggestion to move start_discovery outside BamiCommunity

The start_discovery method contains logic to add a particular walking strategy to IPv8. This method, however, is currently not used anywhere in the BAMI module. It is used by the experiments, however, with the default arguments.

I wonder if it would be cleaner to move this method outside the start_discovery, e.g., to Gumby?

The StateVote type is currently unused

Can we use Python 3.8 for nox?

Currently, Python 3.7 is specified as interpreter version in noxfile.py. However, I don't have Python 3.7 installed on my Mac, nor do I want to install an older Python version (I have 3.8, installed with Brew). Can we either use Python 3.8 with Nox, or specify that Nox always uses the latest Python version installed on the system?

BAMI should address uncontrolled transaction creation (DoS attack)

A particular weakness of TrustChain is that there is no defence against users that aim to deplete computer resources of others through a DoS attack, e.g., through the unsolicited dissemination of messages to others. While this particular issue is currently not high on the priority list, BAMI should contain a basic mechanism that blocks/ignores users that are suspected to perform a DoS attack.

Warning of coroutine never awaited on tests

When running the tests on the latest repo this is the warning I got:

=============================== warnings summary ===============================
tests/payment/test_community.py::TestMint::test_invalid_mint_tx_value_out_of_range[20]
  /home/user/.cache/pypoetry/virtualenvs/bami-O6MDcsrF-py3.8/lib64/python3.8/site-packages/ipv8/messaging/serialization.py:166: RuntimeWarning: coroutine 'interval_runner' was never awaited
    return super(DefaultStruct, self).pack(*data), self.size

-- Docs: https://docs.pytest.org/en/latest/warnings.html

So far I have not changed any files in the local repository. This is after installing the project's dependencies.

Edit: This was done after being recommended in #45 to set up the dev environment in order to deal with the linked issue

Tests kick in twice when creating a PR from a branch on the main repo

For example, see #42

The SecurityMode and SyncMode enums are unused

As the title says. Will these enums eventually be used?

The audit (witness) logic should be moved to a lower level

Currently, the audit logic is located in the application-specific payment community. I think, however, that this is a lower-level functionality and should either move to the backbone community, or be enabled in sub-communities as a mix-in class.

Enabling the frontier gossip could be a setting

We could consider making the frontier gossip a setting. Currently to enable frontier gossip, I have to explicitly call the start_frontier_gossip_sync in the join_subcommunity_gossip method. Moving this to a setting, e.g., named enable_frontier_gossip, is more convenient for developers and requires less coding when creating custom communities.

Create setup and unload methods in mixins

Sometimes, mixins require more state management, e.g., to start a queue. It would be helpful if a setup and unload method can be implemented in these mixins to achieve this. This can be done similarly to how setup_messages works.

The initial_mint_value does not exist

In PaymentCommunity (line 234), the initial_mint_value attribute is used but nowhere defined:

value = self.settings.initial_mint_value

Separate the settings for the backbone community and the payment community

Currently, the BamiSettings contains settings for both the backbone community (e.g., gossip parameters) and the payment community (e.g., max_wait_time and max_wait_block). To avoid mixing applications-specific logic with low-level primitives, we should distinguish between these settings, e.g., by putting them in separate files.

Revise the community prefix in Plexus

In Plexus, there is the community prefix (which is currently used to distinguish between the community with auditors and regular peers. However, I'm not sure if we want this distinction since it adds yet another layer of complexity. Instead, we might want to use a different community ID to distinguish between different (but related) communities.

The set_vals method is implemented five times in the tests

The set_vals method that is responsible for initialising an overlay in the tests is implemented five times. All these implementations are very similar. We can save a lot of code if we can just implement this method twice.

Update the TrustChain implementation to match the algorithm described in the publication

The TrustChain algorithm described in the latest publication has a few key differences compared to the original implementation:

• Each record has multiple back-pointers which decreases the fraud detection time (see the experiment in Figure 10).
• Support for sharing inconsistencies/fraud (pairs of records) with other users.

The code used in the paper can be found here but is tightly integrated with the simulation code. We should extract the core parts and include them in BAMI.

Clean and simple community: Reciprocity Community

We need to create a very simple community for recording reciprocity: given and taken.
This community will be used for deployment in Tribler as Proof of Concept.

The deadline is 20 of October 2020

Document or remove the need for getcontext in the Payment community

In the Payment community, we are currently using a call to getcontext to store the asset precision. However, it is not clear why this is required. Therefore, we should either document it if it is required (e.g., during experiments), or remove it when it is not required.

StateSyncMixin is currently unused

It seems that the StateSyncMixin class is currently unused. Is there any plan to use this in the main code base?

The PaymentState is not persistent

The PaymentState class is not persistent across different BAMI runs. I see this as an absolute requirement when we integrate BAMI in Tribler. Maybe we can initialize the PaymentState from the other database where we store the raw states?

We should revise the requirements to create a new community

Right now, if I want to create a new community, I have to implement the following methods:

    def apply_confirm_tx(self, block: BamiBlock, confirm_tx: Dict) -> None:
        pass

    def apply_reject_tx(self, block: BamiBlock, reject_tx: Dict) -> None:
        pass

    def apply_witness_tx(self, block: BamiBlock, witness_tx: Any) -> None:
        pass

    def block_response(self, block: BamiBlock, wait_time: float = None, wait_blocks: int = None) -> BlockResponse:
        return BlockResponse.DELAY

    def build_witness_blob(self, chain_id: bytes, seq_num: int) -> Optional[bytes]:
        pass

    def create_subcom(self, *args, **kwargs) -> BaseSubCommunity:
        return None

    def discover(self, subcom: BaseSubCommunity, target_peers: int = -1, **kwargs) -> None:
        pass

    def join_subcommunity_gossip(self, sub_com_id: bytes) -> None:
        pass

    def witness_tx_well_formatted(self, witness_tx: Any) -> bool:
        pass

That's quite some work, even for a simple community. Furthermore, what is expected from each method is not clearly described and I often have to look at how they are called to get an idea of what they are doing. This makes the design of new communities quite complicated.

Suggestions

• Many of the methods could return a default value by default, while dropping the abstractmethod decorator of these methods. For example, block_response could by default return IGNORE (see #9). create_subcom could by default return a new LightSubCommunity instance.

I will update this issue later when I have some concrete suggestions on how to improve this 👍

Fail the PR pipeline when black has not been applied

Sometimes, I forget to run the black formatter, resulting in inconsistent code formatting. In this situation, it would be helpful if the PR tool errors and advises me to run the formatter.

Add hot start for BAMI

Read the blocks from a database and reconstruct indexes.

Implementation of block_response is abstract but has to be called by the developer itself

I found that the implementation of the block_response is required. Therefore, I would expect that this implementation is critical when running a simple community derived from the backbone. However, the PaymentCommunity reveals that the developer has to call the block_response manually when processing incoming blocks. I think it would be helpful to move the primitives for confirming/responding to blocks to the backbone. For example, we can provide a method in the backbone, should_counter_sign, that is called on every incoming block, and return True when we respond to that block. Then, the block_response method is called.

The different types in utils.py requires explanation

I noticed there are several types (e.g., Dot, Links, Ranges) that would greatly benefit from some documentation. Since I see that these types are occasionally used in the source code, I think that a few lines of comments for each type is a worthwhile time investment. 👍

Add an IGNORE option to BlockResponse

Currently, there are three options on how to handle an incoming block: confirm, reject, and delay. The confirm and reject responses each create a new block, and the delay option adds the incoming block to the counter_signing_block_queue again. When creating a very basic community, I'm looking for an option to simply ignore incoming blocks. Therefore, I'm suggesting an ignore option, where nothing is done with the incoming block.

Community discovery

We need a simple crawler to discover which communities there are in BAMI.

BAMI is not compatible with the latest version of IPv8

Egbert recently replaced the master peer in a community with a 20-bytes community ID. This breaks compatibility with BAMI. When running the BAMI tests with the latest version of IPv8, I get the following error:

self = <ipv8.test.mocking.endpoint.AutoMockEndpoint object at 0x109c05610>
listener = <tests.payment.test_community.FakePaymentCommunity object at 0x109ae36d0>
prefix = b'\x00\x02'

    def add_prefix_listener(self, listener, prefix):
        """
        Add an EndpointListener to our listeners, only triggers on packets with a specific prefix.
    
        :raises: IllegalEndpointListenerError if the provided listener is not an EndpointListener
        """
        if not isinstance(listener, EndpointListener):
            raise IllegalEndpointListenerError(listener)
        if not len(prefix) == self.prefixlen:
>           raise RuntimeError("Tried to register a prefix of length %d, required to be of length %d!"
                               % (len(prefix), self.prefixlen))
E           RuntimeError: Tried to register a prefix of length 2, required to be of length 22!