GithubHelp home page GithubHelp logo

daily-co / synthetic-network Goto Github PK

View Code? Open in Web Editor NEW
78.0 6.0 9.0 5.11 MB

Docker containers on a synthetic network. Run applications in a context that lets you manipulate their network conditions.

License: Apache License 2.0

Dockerfile 0.96% Makefile 1.38% Nix 0.76% JavaScript 20.52% HTML 1.19% CSS 0.52% Shell 4.41% Rust 70.26%

synthetic-network's Introduction

Synthetic Network

Docker containers on a synthetic network. Run applications in a context that lets you manipulate their network conditions.

Dependencies

Overview

$ make
SYNTHETIC_NETWORK ?= 10.77.0.0/16
CONTAINER_NAME_INTERACTIVE ?= syntheticnet-interactive
CONTAINER_NAME_CHROME ?= syntheticnet-chrome
TESTHOST ?= <hostname>:<address> (add /etc/hosts entry to container)
help: # Print this help message
image: # Build Docker image: syntheticnet
image-vnc: # Build Docker image: syntheticnet:vnc
image-chrome: image-vnc # Build Docker image: syntheticnet:chrome
rush: # Build rush
minimal: rush # Build Docker image: syntheticnet:minimal
run-interactive: image synthetic-network # Debug syntheticnet container. Prereq: create-synthetic-network
run-chrome: image-chrome synthetic-network # Run syntheticnet:chrome. Prereq: create-synthetic-network
synthetic-network: # Specify SYNTHETIC_NETWORK (this rule is documentation)
create-synthetic-network: synthetic-network # Create Docker network: synthetic-network

Run Chrome using Synthetic Network in VNC

  1. ensure Docker for Mac/Windows/Linux is running

$ make create-synthetic-network # You only need to do this once
$ make run-chrome
...
πŸŽ› Synthetic network GUI will listen on http://localhost:3000

πŸ“Ί Point your VNC client at localhost:5901
...
  1. open TigerVNC and navigate to 127.0.0.1::5901

Resolving test domains within the container

$ TESTHOST=my-test-domain.dev:192.168.0.1 make run-chrome

Build Container Image

Build syntheticnet image

$ make image

with VNC:

$ make image-vnc

It is also possible to build a minimal image with:

$ make minimal

The minimal image doesn't have a frontend to update the user network proxy (rush) settings (see example below). Instead, one can create a configuration file and place it at /opt/etc/synthetic_network.json which can be done by creating a new container image based on the minimal image and copying a file to that particular location. If no configuration file is provided the example settings will be used (see rush -h below).

Rush settings example

rush/ is the packet processing framework we use to handle bandwidth, packet loss, jitter, etc. This is an example configuration:

{
  "default_link": {
    "ingress": {
      "rate": 10000000,
      "loss": 0,
      "latency": 0,
      "jitter": 0,
      "jitter_strength": 0,
      "reorder_packets": false
    },
    "egress": {
      "rate": 10000000,
      "loss": 0.02,
      "latency": 0,
      "jitter": 0,
      "jitter_strength": 0,
      "reorder_packets": false
    }
  },
  "flows": []
}

Where we define a 10Mbps ingress and egress network and a 2% packet loss on egress. We can also specify settings per each protocol, an example can be obtained running rush help:

make rush
cd rush
cargo build --release
./target/release/rush -h

Scripting the Synthetic Network

const SyntheticNetwork = require('synthetic-network/frontend')

const synthnet = new SyntheticNetwork({hostname: "localhost", port: 3000})

await synthnet.get() // Get current configuration

// Double ingress rate
var current_ingress_rate = synthnet.default_link.ingress.rate()
synthnet.default_link.ingress.rate(current_ingress_rate*2)

await synthnet.commit() // Apply new configuration

// Add a flow
synthnet.addFlow('udp', {protocol: 'udp'})
synthnet.flows.udp.link.ingress.rate(500000)
synthnet.flows.udp.link.egress.rate(500000)
synthnet.flows.udp.link.egress.loss(0.01)
await synthnet.commit()

// Print ingress traffic statistics
const ingress_profile = await synthnet.profiles.ingress.get()
for (var flow in ingress_profile.flows)
  console.log(flow, ingress_profile.flows[flow].packets)

// ...

See also: frontend/udp_rate_sine_demo.js

Further reading

Check out the reports under doc/ for details.

The packet processing framework we use to do network conditioning can be found under rush/. Its README points to a screen cast series covering its design and implementation.

synthetic-network's People

Contributors

aconchillo avatar cbrianhill avatar dependabot[bot] avatar eugeneia avatar jasonm avatar kwindla avatar markbackman avatar vipyne avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

synthetic-network's Issues

Modeling real-world networks

@steely-glint was wondering on Twitter how well network testing tools such as Synthetic Network simulate real-world network characteristics such as buffering that real congested networks do.

Seems like a great opportunity to compare notes: what do we do, and what do networks do? What could we do to be more like networks?

Right now we have these building blocks:

  • RateLimiter: forwards packets up to a bps limit, and drops all packets exceeding the limit (no buffering!). It uses a token bucket algorithm and replenishes tokes every 100us.
  • Loss: drops a fraction of all incoming packets at random and forwards the rest (no buffering!)
  • Latency: buffers packets for X ms before forwarding. (Never drops or reorders packets!)
  • Jitter: like Latency but buffers packets for a random duration up to X ms. Can be configured to reorder packets. (Never drops packets!)

At the moment they are combined as our QoS path for each direction of each flow like so:

image

@steely-glint had some specific use-cases in mind that I think would be interesting to think about how to model in SyntheticNetwork:

I am trying to mimic 2 common domestic situations.

  1. where a user moves from room to room forcing WiFi to change modulation modes.
  2. where a second family member starts streaming Netflix and halves the available bandwidth.

I worry about the delay. I suspect that changing the modulation also changes the time to transmit a packet, so the delay would change too.
Buffer preserved in both cases ?

We might be able to model 1) already by scripting RateLimiter and Latency in concert. However we don’t have an explicit way to talk about buffers, It might make sense to decouple simulation of buffering and delay, and make the buffer sizes scriptable as well. Imagine this:

image

We could differentiate between wifi latency and geographic network latency, and we could change RateLimiter and Latency to read from Buffers as permitted/needed. If the Buffer size would be scriptable we could possibly simulate arbitrary network configurations?

As for 2) we might want to add contenders to a flow. For example:

image

The Join building block we already have. LoadGen could be interesting to write. It should probably simulate real traffic with pauses/intervals etc, and probably we should make sure that we don’t actually send the generated packets out in the end! ;-)

Anyways that’s the braindump / food for thought. Looking forward to hear what others think! :-)

ARM support

Feature request

Perhaps more a question or discussion than feature request, but: do you have any experience running this on ARM platforms e.g. Apple M1 or AWS Graviton?

Looks like MediaSoup may support ARM ref1 ref2.

The asm code in rush referenced in #2 / #3 indeed comes with both x86_64 and aarch64 targets.

Why you need this

I primarily develop on an M1 Mac. This is very much in the "nice-to-have" territory, but I am curious if others are successfully using this on ARM.

Alternatives you've considered

The main issues I encountered with ARM, and workarounds I tried, in case it's useful:

The chrome image installs google-chrome-stable on Debian, but Google only provides Chrome for Linux amd64. Docker for Mac provides emulation via qemu (support for which should be regarded as "best effort" only). For me, amd64 (x86) Chrome in Docker (via make run-chrome) failed to launch with this error.

I was not successful in finding an older version of Chrome that successfully launched via apt or deb for Debian 9 (per this repo's Dockerfile FROM node:12.20.2) or Debian 11 (the newest stable version; easy to update by changing this repo's Dockerfile to use FROM node:12-bullseye).

I also tried various versions of Chromium under arm64, but was not successful.

I ended up running it successfully on an x86 machine, but thought I'd share notes in case this is useful πŸ˜„

Unable to run Google Chrome on synthetic-net on amd64

Description:

When attempting to run Google Chrome on the synthetic-net on amd64 Docker image using the make run-chrome command, an error message is displayed saying that Google Chrome cannot be found. This issue appears to be related to missing dependencies.

Steps to reproduce:

  1. Clone the daily-co/synthetic-network repository
  2. Run make run-chrome on an amd64 system
  3. Observe the error message saying that Google Chrome cannot be found

Expected result:

Google Chrome should launch successfully on the synthetic-net amd64 Docker image.

Actual result:

Google Chrome cannot be found on the synthetic-net amd64 Docker image.

Suggested solution:

Add a step to the Dockerfile to install the missing dependencies required for Google Chrome to run properly. This will ensure that the make run-chrome command works as expected on all architectures.

Thank you for your attention to this matter.

cargo build --release: cannot find macro `asm` in this scope

Expected behavior

Hello! When I clone this repo and run make create-synthetic-network and then make run-chrome, I would expect the latter command to build the Docker image and exit successfully per the README.

Describe the bug (unexpected behavior)

Instead, when I clone this repo and run make run-chrome the docker build (...) errors while running cargo build --release:

Click to expand a shell session of the build and error log
~/dev $ git clone https://github.com/daily-co/synthetic-network.git
Cloning into 'synthetic-network'...
remote: Enumerating objects: 1446, done.
remote: Counting objects: 100% (1446/1446), done.
remote: Compressing objects: 100% (1187/1187), done.
remote: Total 1446 (delta 215), reused 1445 (delta 215), pack-reused 0
Receiving objects: 100% (1446/1446), 4.77 MiB | 6.21 MiB/s, done.
Resolving deltas: 100% (215/215), done.

1s ~/dev $ cd synthetic-network/

0s (main) ~/dev/synthetic-network $ ls
CODE_OF_CONDUCT.md Dockerfile         Makefile           aarch64.nix        doc                run-in-vnc.sh      setup.sh
CONTRIBUTING.md    LICENSE            README.md          default.nix        frontend           rush               synth-chrome

0s (main) ~/dev/synthetic-network $ make create-synthetic-network
export SYNTHETIC_NETWORK=10.77.0.0/16
docker network create synthetic-network --subnet=10.77.0.0/16
a349f6a728c19feddf76a4af9f021d4b967d30aa5045d0ffcd5df6ebffe52cc3

7s (main) ~/dev/synthetic-network $ make run-chrome
docker build -t syntheticnet:vnc --build-arg VNC=true .
[+] Building 100.4s (14/17)
 => [internal] load build definition from Dockerfile                                                                                                                                                 0.0s
 => => transferring dockerfile: 1.08kB                                                                                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/node:12.20.2                                                                                                                                      1.9s
 => [auth] library/node:pull token for registry-1.docker.io                                                                                                                                          0.0s
 => [internal] load build context                                                                                                                                                                    1.0s
 => => transferring context: 12.72MB                                                                                                                                                                 1.0s
 => [ 1/12] FROM docker.io/library/node:12.20.2@sha256:0140c4e17aa1f3fd78b939e3c7ac45accdd5274d0194046b484029a5e46b9db6                                                                             38.6s
 => => resolve docker.io/library/node:12.20.2@sha256:0140c4e17aa1f3fd78b939e3c7ac45accdd5274d0194046b484029a5e46b9db6                                                                                0.0s
 => => sha256:0140c4e17aa1f3fd78b939e3c7ac45accdd5274d0194046b484029a5e46b9db6 776B / 776B                                                                                                           0.0s
 => => sha256:2571706a0e9a90a55ea51746239ff8acd489e5c2ce2f119273245ccbc0a94392 2.21kB / 2.21kB                                                                                                       0.0s
 => => sha256:3aab5c2e02a68fcfa5e8c766e1689c6790580f0d51ee647ab67ea57b21fa6dc2 7.83kB / 7.83kB                                                                                                       0.0s
 => => sha256:296c9f0bf5f2cf24e87bc5abd674fc486e8df419d4aa2d362453f64d38900504 43.18MB / 43.18MB                                                                                                    10.1s
 => => sha256:84d546321c8e11789c75187b5b592633207519d75f96e8f03e8378d2da4a72a9 10.18MB / 10.18MB                                                                                                     3.9s
 => => sha256:f91c8ffe219846d18819503633eb252121c21cd20dbff74278b85c4029370992 4.10MB / 4.10MB                                                                                                       1.8s
 => => sha256:df8d698654f0ee6c5be4a19593a2d7a793c4bed9cb10de0c87d1d1c2ce6ba368 47.73MB / 47.73MB                                                                                                    11.2s
 => => sha256:eb8f7931e7aa1d31b83a8db251eb9107d989c9244f8bd7a542788aef3b6d1975 201.71MB / 201.71MB                                                                                                  32.0s
 => => sha256:6516b36f67a70cc6094778e92c2002789a89825b2769c9d5ce9f6427c5c813c2 4.20kB / 4.20kB                                                                                                      10.4s
 => => extracting sha256:296c9f0bf5f2cf24e87bc5abd674fc486e8df419d4aa2d362453f64d38900504                                                                                                            2.3s
 => => sha256:8592970063e59260afa874fb5f28a75d3e4742e7edaaafc1ba690c492e2039d4 23.70MB / 23.70MB                                                                                                    15.0s
 => => sha256:934b7a74b39bb36b47f580e3c33386d8edf20b9f9bc885442b1e78875d253ddf 2.38MB / 2.38MB                                                                                                      11.8s
 => => sha256:68a1166726ff6c90bb9e39a88d2245a71adb142bc16a8731e84ac31b4762ccbc 292B / 292B                                                                                                          12.1s
 => => extracting sha256:84d546321c8e11789c75187b5b592633207519d75f96e8f03e8378d2da4a72a9                                                                                                            0.7s
 => => extracting sha256:f91c8ffe219846d18819503633eb252121c21cd20dbff74278b85c4029370992                                                                                                            0.2s
 => => extracting sha256:df8d698654f0ee6c5be4a19593a2d7a793c4bed9cb10de0c87d1d1c2ce6ba368                                                                                                            3.1s
 => => extracting sha256:eb8f7931e7aa1d31b83a8db251eb9107d989c9244f8bd7a542788aef3b6d1975                                                                                                            5.2s
 => => extracting sha256:6516b36f67a70cc6094778e92c2002789a89825b2769c9d5ce9f6427c5c813c2                                                                                                            0.0s
 => => extracting sha256:8592970063e59260afa874fb5f28a75d3e4742e7edaaafc1ba690c492e2039d4                                                                                                            0.8s
 => => extracting sha256:934b7a74b39bb36b47f580e3c33386d8edf20b9f9bc885442b1e78875d253ddf                                                                                                            0.1s
 => => extracting sha256:68a1166726ff6c90bb9e39a88d2245a71adb142bc16a8731e84ac31b4762ccbc                                                                                                            0.0s
 => [ 2/12] RUN apt-get update && apt-get install -y     iproute2 ethtool iputils-ping iperf3 python3 lsof tcpdump net-tools                                                                         5.7s
 => [ 3/12] ADD run-in-vnc.sh /opt/lib/                                                                                                                                                              0.0s
 => [ 4/12] RUN if [ -n "true" ] ; then apt-get install -y tigervnc-standalone-server ratpoison ; else echo "No VNC for you" ; fi                                                                   13.1s
 => [ 5/12] RUN apt-get install -y curl build-essential                                                                                                                                              1.9s
 => [ 6/12] RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs     | sh -s -- --default-toolchain nightly -y                                                                             26.2s
 => [ 7/12] ADD rush /opt/lib/rush                                                                                                                                                                   0.1s
 => [ 8/12] WORKDIR /opt/lib/rush                                                                                                                                                                    0.0s
 => ERROR [ 9/12] RUN cargo build --release                                                                                                                                                         12.7s
------
 > [ 9/12] RUN cargo build --release:
#14 0.695    Compiling libc v0.2.68
#14 0.695    Compiling proc-macro2 v1.0.24
#14 0.695    Compiling unicode-xid v0.2.1
#14 0.695    Compiling syn v1.0.60
#14 0.695    Compiling getrandom v0.2.2
#14 0.697    Compiling cfg-if v1.0.0
#14 0.700    Compiling memchr v2.3.3
#14 0.845    Compiling serde_derive v1.0.123
#14 0.873    Compiling ryu v1.0.5
#14 1.092    Compiling serde v1.0.123
#14 1.125    Compiling signal-hook v0.3.6
#14 1.198    Compiling byteorder v1.3.4
#14 1.206    Compiling ppv-lite86 v0.2.10
#14 1.207    Compiling serde_json v1.0.62
#14 1.312    Compiling lazy_static v1.4.0
#14 1.337    Compiling regex-syntax v0.6.17
#14 1.403    Compiling itoa v0.4.7
#14 1.435    Compiling once_cell v1.3.1
#14 1.687    Compiling thread_local v1.0.1
#14 1.854    Compiling aho-corasick v0.7.10
#14 2.162    Compiling quote v1.0.9
#14 2.342    Compiling signal-hook-registry v1.3.0
#14 2.470    Compiling rand_core v0.6.2
#14 2.700    Compiling rand_chacha v0.3.0
#14 3.014    Compiling rand v0.8.3
#14 3.564    Compiling regex v1.3.6
#14 11.58    Compiling rush v0.1.0 (/opt/lib/rush)
#14 12.45 error: cannot find macro `asm` in this scope
#14 12.45    --> src/checksum.rs:151:5
#14 12.45     |
#14 12.45 151 |     asm!("
#14 12.45     |     ^^^
#14 12.45     |
#14 12.45     = note: consider importing one of these items:
#14 12.45             std::arch::asm
#14 12.45             core::arch::asm
#14 12.45
#14 12.61 error: could not compile `rush` due to previous error
------
executor failed running [/bin/sh -c cargo build --release]: exit code: 101
make: *** [image-vnc] Error 1

Steps to reproduce

$ git clone https://github.com/daily-co/synthetic-network.git
$ cd synthetic-network/
$ make run-chrome

Screenshots

System information

I am not familiar with Rust programming. I found this thread that seems surface-similar to the error I received, and so I thought perhaps it was an architecture-specific problem; I am running on arm64 on an M1 Mac. However I see the same error on an x86 Mac.

OS and arch for both systems:

# m1 mac, macOS Monterey 12.1
$ uname -a
Darwin MacBook-Pro.local 21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:41 PST 2021; root:xnu-8019.61.5~1/RELEASE_ARM64_T6000 arm64
$ docker --version
Docker version 20.10.11, build dea9396

# x86 mac, macOS Monterey 12.0.1
$ uname -a
Darwin Jasons-MacBook-Pro-2.local 21.1.0 Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:23 PDT 2021; root:xnu-8019.41.5~1/RELEASE_X86_64_x86_64
$ docker --version
Docker version 20.10.11, build dea9396

BUG: executor failed running [/bin/sh -c cargo build --release]: exit code: 101

Expected behavior

As shown in the README.md, the following should be printed when `make run-chrome' command is run.

πŸŽ› Synthetic network GUI will listen on http://localhost:3000

πŸ“Ί Point your VNC client at localhost:5901

Describe the bug (unexpected behavior)

However, the following checksum mismatch error is printed.

 => ERROR [ 9/12] RUN cargo build --release                                                                     1.1s
------
 > [ 9/12] RUN cargo build --release:
#14 1.030 error: the listed checksum of `/opt/lib/rush/vendor/byteorder/CHANGELOG.md` has changed:
#14 1.030 expected: 51f0eb3b6139fc1a908d41a7b3cba7d58d684700986b3518f82e5af254c39e8e
#14 1.030 actual:   3ff0b140f3f49df66dc452b6df345710d2db900d9477d18c045bd48eab5a45c4
#14 1.030
#14 1.030 directory sources are not intended to be edited, if modifications are required then it is recommended that `[patch]` is used with a forked copy of the source
------
executor failed running [/bin/sh -c cargo build --release]: exit code: 101
Makefile:18: recipe for target 'image-vnc' failed
make: *** [image-vnc] Error 1

Steps to reproduce

  1. Clone the git repo
  2. Run make
  3. Then, make create-synthetic-network
  4. And, make run-chrome

System information

  • Device: Intel(R) Core(TM) i5-1035G1 CPU @ 1.00GHz 1.19 GHz, 16GB RAM
  • OS, version: Windows 10 Home Single Language, Version: 21H1, OS build: 19043.2006
  • Browser version: Not used yet.
  • Docker version: Docker Desktop 4.12.0, command line: Docker version 20.10.17, build 100c701
  • VNC (if applicable): Not used yet.

Looking forward to getting started with using synthetic-network.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.