GithubHelp home page GithubHelp logo

scout's Introduction

Overview

Scout is a library enabling two users to contact each other over the Internet using only their respective public keys. The Bittorrent distributed hash table (DHT) is used to store and retrieve contact information. Scout can also be used to store and retrieve short messages from the DHT so that peers can communicate even if they are never online at the same time.

Prerequisites

Scout requires Boost 1.58 or newer and Libsodium 1.0 or newer

Building

Scout uses Boost.Build version 2 to build. For information on installing BBv2 see the Boost.Build manual. By default scout uses the versions of boost and libsodium which are installed on the system. If you are building on Windows you will need to specify the paths to the boost and libsodium source trees using the BOOST_ROOT and SODIUM_ROOT environment variables. Example commands to build scout on Windows:

C:\scout> set BOOST_ROOT=C:\boost_1_60_0
C:\scout> set SODIUM_ROOT=C:\libsodium-1.0.8
C:\scout> bjam toolset=msvc-14

Setting up a DHT session

Most users will want to use the dht_session class to easily set up a DHT node which can be used with the rest of scout's functions. To start a node create an instance of dht_session and call the start function.

scout::dht_session ses;
ses.start();

The start function will start a DHT node in a separate thread and return immediately. To stop the DHT node call the stop function.

ses.stop();

This function will block until the dht node thread has exited. If stop is not called explicitly it will be called from the dht_session destructor.

Generating a key pair

Scout provides the generate_keypair function to generate a new ed25519 key pair.

std::pair<scout::secret_key, scout::public_key> keypair = scout::generate_keypair();

Callbacks

The scout API involves many callback functions. When using the dht_session class it is important to keep in mind that callbacks will be invoked in the DHT node's thread rather than the main thread of your application. This means you need to be careful when accessing your application's data structures from a callback. Ideally callbacks will carry a copy of any data they might need to store in the DHT and post notifications to the main application event loop for new data retrieved from the DHT.

Storage lifetime

Data stored in the DHT can only be expected to remain there for up to two hours. It is recommended that data be stored/synchronized roughly once an hour.

Synchronizing contact information

Scout stores contact information as a vector of entries. Each entry must be assigned an id which is unique within that vector. The contents of the entries are left up to the application. Scout encrypts the entry vector before storing it in the DHT so applications do not need to encrypt each entry's contents.

To communicate entries between peers, scout uses a synchronize operation which retrieves the existing vector of entries from the DHT then writes a new vector with whatever updates the application specifies. To synchronize with a peer you need to have a shared secret to use as a key. Scout provides a key exchange function with uses Diffie-Hellman to generate a shared secret from the user's private key and a remote peer's public key.

scout::secret_key shared_secret = scout::key_exchange(my_secret_key, remote_public_key);
ses.synchronize(shared_secret, entries, entry_updated, finalize_entries, sync_finished);

The entries vector should contain the entries which the application is currently aware of. The entry_updated callback will be invoked when a new or updated entry is retrieved from the DHT. The finalize_entries callback will be invoked after all updates have been retrieved and before the updated entry vector is stored in the DHT, it provides the application a final opportunity to update the entries. The sync_finished callback is invoked once all store requests have completed, any resources associated with the operation may be freed by this function.

Storing offline messages

Scout supports storing messages in the DHT so that a peer can retrieve them later even if the originator has gone offline. Messages are limited to 1000 bytes each. Scout does not encrypt message contents, the application is expected to have it's own message encryption scheme. Messages are stored in the DHT using the hash of their content as the key, thus the content of a message cannot be changed. A series of messages are stored as a linked list which can be retrieved using just the hash of the most recently stored message. Message lists are always retrieved in last-in-first-out order.

The first step is to create a message list.

scout::list_head message_list;

You can then add one-or-more messages to the list.

scout::list_token msg_token = message_list.push_front(message_contents);

The list_token contains the hash of the next message in the list, which corresponds to the previously added message. It should be stored alongside the message contents so that the message can be periodically stored in the DHT.

ses.put(msg_token, message_contents, put_finished);

The msg_token must match the one returned from push_front for the given message. The storage backing the message contents must remain valid until the put_finished callback is invoked.

Retrieving offline messages

To retrieve a list of offline messages you first must obtain the hash of the first message. The sender can get this hash from the list_head.

hash head_hash = message_list.head();

Typically this hash will be included in the contents of an entry. Once the receiving peer has the hash it can retrieve the first message.

ses.get(head_hash, message_received);

The message_received callback is passed the message contents along with the hash of the next message in the list.

scout's People

Contributors

bommuraj2012 avatar ssiloti avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

scout's Issues

Synchronization has unstable behavior

Sync command in examples/client_test prints out only some but not all of the entries added, and it fails to update the content of an entry with an existing id. Below is a sequence of commands I used, and the corresponding output.

Generate 2 keys

PS > .\client_test.exe gen-key 1.key
PS > .\client_test.exe dump-key 1.key
Public key: 142cc76d0bd5f5434ccbcb45590913e33f9f9922d2e84b90d4ae7cc4f7f3b405
PS > .\client_test.exe gen-key 2.key
PS > .\client_test.exe dump-key 2.key
Public key: 612379ef06ecf63c4689d075870769e4e1e2e7f778ffb4a5aa11e0b7d508166f

Sync 4 items using 1.key. Note that when syncing 3 sit, 1 ipsum and 2 dolor were left out.

PS > .\client_test.exe sync 1.key 612379ef06ecf63c4689d075870769e4e1e2e7f778ffb4a5aa11e0b7d508166f 0 Lorem 2> 0.log
0 Lorem
PS > .\client_test.exe sync 1.key 612379ef06ecf63c4689d075870769e4e1e2e7f778ffb4a5aa11e0b7d508166f 1 ipsum 2> 1.log
0 Lorem
1 ipsum
PS > .\client_test.exe sync 1.key 612379ef06ecf63c4689d075870769e4e1e2e7f778ffb4a5aa11e0b7d508166f 2 dolor 2> 2.log
0 Lorem
1 ipsum
2 dolor
PS > .\client_test.exe sync 1.key 612379ef06ecf63c4689d075870769e4e1e2e7f778ffb4a5aa11e0b7d508166f 3 sit 2> 3.log
0 Lorem
3 sit

Sync 4 items using 2.key. Note that many items were left out.

PS > .\client_test.exe sync 2.key 142cc76d0bd5f5434ccbcb45590913e33f9f9922d2e84b90d4ae7cc4f7f3b405 4 amet 2> 4.log
0 Lorem
1 ipsum
3 sit
4 amet
PS > .\client_test.exe sync 2.key 142cc76d0bd5f5434ccbcb45590913e33f9f9922d2e84b90d4ae7cc4f7f3b405 5 consectetur 2> 5.log
0 Lorem
1 ipsum
3 sit
4 amet
5 consectetur
PS > .\client_test.exe sync 2.key 142cc76d0bd5f5434ccbcb45590913e33f9f9922d2e84b90d4ae7cc4f7f3b405 6 adipiscing 2> 6.log
0 Lorem
1 ipsum
3 sit
4 amet
6 adipiscing
PS > .\client_test.exe sync 2.key 142cc76d0bd5f5434ccbcb45590913e33f9f9922d2e84b90d4ae7cc4f7f3b405 7 elit 2> 7.log
0 Lorem
1 ipsum
3 sit
4 amet
7 elit

Update existing entries. Note that 0th entry's value was not successfully set to Scout.

PS > .\client_test.exe sync 1.key 612379ef06ecf63c4689d075870769e4e1e2e7f778ffb4a5aa11e0b7d508166f 0 Scout 2> 0u.log
0 Scout
1 ipsum
3 sit
4 amet
PS > .\client_test.exe sync 2.key 142cc76d0bd5f5434ccbcb45590913e33f9f9922d2e84b90d4ae7cc4f7f3b405 7 felis 2> 7u.log
0 Lorem
1 ipsum
3 sit
4 amet
7 felis

It appears that in this particular run, only 0 Lorem, 1 ipsum, 3 sit, and 4 amet were successfully added. Also, syncing an entry with an existing ID does not seem to override/update it's value. Here are all the log files from commands above: log.zip

Build instructions

I'm not familiar with the boost build system. Can some basic build instructions get added to the README?

Compile Errors

I'm getting the following errors when I try to build. Any idea what I might be doing wrong?

building boost from source directory:  C:\boost_1_62_0
building libsodium from source directory:  C:\libsodium-1.0.11
OS = NT
Performing configuration checks

    - 32-bit                   : yes (cached)
    - arm                      : no  (cached)
    - mips1                    : no  (cached)
    - power                    : no  (cached)
    - sparc                    : no  (cached)
    - x86                      : yes (cached)
    - symlinks supported       : yes (cached)
...patience...
...found 1552 targets...
...updating 8 targets...
compile-c-c++ bin\msvc-14\debug\threading-multi\src\dht_session.obj
dht_session.cpp
Please define _WIN32_WINNT or _WIN32_WINDOWS appropriately. For example:
- add -D_WIN32_WINNT=0x0501 to the compiler command line; or
- add _WIN32_WINNT=0x0501 to your project's Preprocessor Definitions.
Assuming _WIN32_WINNT=0x0501 (i.e. Windows XP target).
c:\users\amd\documents\github\scout\include\utils.hpp(23): fatal error C1083: Cannot open include file: 'sodium/crypto_box.h': No such file or directory

    call "C:\Users\AMD\AppData\Local\Temp\b2_msvc_14.0_vcvarsall_x86.cmd" >nul
cl /Zm800 -nologo @"bin\msvc-14\debug\threading-multi\src\dht_session.obj.rsp" 

...failed compile-c-c++ bin\msvc-14\debug\threading-multi\src\dht_session.obj...
compile-c-c++ bin\msvc-14\debug\threading-multi\src\file.obj
file.cpp
C:\Users\AMD\Documents\GitHub\scout\include\utils.hpp(23): fatal error C1083: Cannot open include file: 'sodium/crypto_box.h': No such file or directory

    call "C:\Users\AMD\AppData\Local\Temp\b2_msvc_14.0_vcvarsall_x86.cmd" >nul
cl /Zm800 -nologo @"bin\msvc-14\debug\threading-multi\src\file.obj.rsp" 

...failed compile-c-c++ bin\msvc-14\debug\threading-multi\src\file.obj...
compile-c-c++ bin\msvc-14\debug\threading-multi\src\scout.obj
scout.cpp
C:\Users\AMD\Documents\GitHub\scout\include\utils.hpp(23): fatal error C1083: Cannot open include file: 'sodium/crypto_box.h': No such file or directory

    call "C:\Users\AMD\AppData\Local\Temp\b2_msvc_14.0_vcvarsall_x86.cmd" >nul
cl /Zm800 -nologo @"bin\msvc-14\debug\threading-multi\src\scout.obj.rsp" 

...failed compile-c-c++ bin\msvc-14\debug\threading-multi\src\scout.obj...
compile-c-c++ bin\msvc-14\debug\threading-multi\src\upnp-portmap.obj
upnp-portmap.cpp
Please define _WIN32_WINNT or _WIN32_WINDOWS appropriately. For example:
- add -D_WIN32_WINNT=0x0501 to the compiler command line; or
- add _WIN32_WINNT=0x0501 to your project's Preprocessor Definitions.
Assuming _WIN32_WINNT=0x0501 (i.e. Windows XP target).
C:\Users\AMD\Documents\GitHub\scout\include\utils.hpp(23): fatal error C1083: Cannot open include file: 'sodium/crypto_box.h': No such file or directory

    call "C:\Users\AMD\AppData\Local\Temp\b2_msvc_14.0_vcvarsall_x86.cmd" >nul
cl /Zm800 -nologo @"bin\msvc-14\debug\threading-multi\src\upnp-portmap.obj.rsp" 

...failed compile-c-c++ bin\msvc-14\debug\threading-multi\src\upnp-portmap.obj...
compile-c-c++ bin\msvc-14\debug\threading-multi\src\utils.obj
utils.cpp
C:\Users\AMD\Documents\GitHub\scout\include\utils.hpp(23): fatal error C1083: Cannot open include file: 'sodium/crypto_box.h': No such file or directory

    call "C:\Users\AMD\AppData\Local\Temp\b2_msvc_14.0_vcvarsall_x86.cmd" >nul
cl /Zm800 -nologo @"bin\msvc-14\debug\threading-multi\src\utils.obj.rsp" 

...failed compile-c-c++ bin\msvc-14\debug\threading-multi\src\utils.obj...
...skipped <pbin\msvc-14\debug\threading-multi>scout.dll for lack of <pbin\msvc-14\debug\threading-multi>src\dht_session.obj...
...skipped <pbin\msvc-14\debug\threading-multi>scout.lib for lack of <pbin\msvc-14\debug\threading-multi>src\dht_session.obj...
...skipped <pbin\msvc-14\debug\threading-multi>scout.pdb for lack of <pbin\msvc-14\debug\threading-multi>src\dht_session.obj...
...failed updating 5 targets...
...skipped 3 targets...

Sync command in examples/client_test.cpp not working as expected

In the ses.synchronize() call, the finalize callback has a for loop that looks like it should print each entry to cout after updates to the vector have been received from the DHT. However, when I use the sync command, no entries (not even the one I just created) get printed to cout. I have added print lines to verify that the finalize callback is actually getting called, and it appears as though there are no entries being stored in the vector anymore by that point.

failed to load DHT state: The system cannot find the file specified

I'm getting an odd error when I try to use the "put" command in client_test.

PS C:\Users\AMD\Documents\GitHub\scout\examples\bin\msvc-14.0\debug\link-static\threading-multi> .\client_test.exe put "testing testing"
DHT: Generating a random node ID: "2900000023480000BE18000084670000E14A0000"
DHT: Enable(enabled=0, rate=1024) [bootstrap=1]
failed to load DHT state: The system cannot find the file specified
DHT: Loaded 0 nodes and ID "2900000023480000BE18000084670000E14A0000" from disk
DHT: DhtImpl() [bootstrap=1]
dht router is at "router.utorrent.com"
dht router is at "router.bittorrent.com"
DHT: Enable(enabled=1, rate=8000) [bootstrap=1]
adding UPnP port mapping
Usage: client_test <command> [args]
Commands:
gen-key <file name>            - generate a key pair and store it in the specified file
dump-key <file name>           - print the public key from the specified file
sync <file name> <public key> <entry #> <string> - Synchronize entries using the secret key from the specified file and the speficied
   public key of a remote peer. Add or update the entry with the specified id with
   the specified value.
put <string>                   - Store the given string in the DHT as an offline message
get <hash>                     - get the message with the given hash
local ip: 10.10.10.32

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.