atifs / vmxdocker

Run Juniper Networks vMX in a Docker Container

Run Juniper Networks vMX in a Docker Container

WARNING: I no longer maintain this project. It served me well in the early days of vMX 14.x and 15.x and ultimately led to the creation of the vMX based lightweight 4over6 solution: https://github.com/Juniper/vmx-docker-lwaftr

Overview

Juniper Networks vMX virtual router can be installed on bare metal servers by running an installation script. A different solution is used here by creating a vmx Docker image that can be instantiated one or more times via Docker.

Each vMX container is given a list of interface, memory and vCPU count as well as an initial "zero touch" config plus an actual vMX distribution tar file (not provided here). The list of interfaces can contain virtual bridges and physical interfaces. The container will create required virtual bridges and attach physical interfaces as needed. Access to Intel 82599 based 10G Ethernet ports is handled via Snabb Switch by providing their PCI addresses instead of their interface names. Based on the CPU's capability, the container runs either the standard or lite vPFE version (though its possible to force the use of the lite version).

This is currently in prototype stage and not recommended for production use.

Requirements

• Juniper Networks vMX distribution tar file. Download the latest vMX package from http://www.juniper.net/support/downloads/?p=vmx#sw. A valid user account is required to access and download the file. This file must be made available to the container at startup via a mounted volume.
• Bare metal linux server with Docker installed. Currently tested with Ubuntu 14.04 and 15.04. The kernel must have HugePages reserved by setting the following options in /etc/default/grub:

# cat /etc/default/grub
...
GRUB_CMDLINE_LINUX_DEFAULT="hugepages=12000"
...
# update-grub
# reboot

• Optional one or more Intel 82599 based 10G Ethernet ports. If used, the kernel must also have intel_iommu disabled:

# cat /etc/default/grub
...
GRUB_CMDLINE_LINUX_DEFAULT="hugepages=12000 intel_iommu=off"
...
# update-grub
# reboot

Qemu and snabb will get downloaded and compiled during the creation of the vmx docker image, hence there are no requirements on the server itself to have qemu or even developer tools installed.

Download the vMX Image

$ docker pull marcelwiget/vmx
$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
marcelwiget/vmx     latest              a9c492840cd6        2 hours ago         432.3 MB

Running the vMX Container

Starting with vMX 15.1F3, its possible to specify a full Junos configuration file that is used during startup via CONFIG. The former CFG variable is still used to pass commands to the CLI after boot, e.g. to enter the license key. See the example vmx1.cfg (though it contains an invalid license key)

$ docker run --name vmx1 --rm --privileged --net=host \
  -v $PWD:/u:ro \
  --env TAR="vmx-15.1F3.11.tgz" \
  --env CFG="vmx1.cfg" \
  --env CONFIG="vmx1-conf.txt" \
  --env DEV="br0 br0" \
  --env VCPMEM="2000" \
  -i -t marcelwiget/vmx:latest

The second example vmx2 launches the first vMX FRS release attached to two Intel 82599 based 10G interfaces. The Docker option '--net=host' isn't required here, because the 10G ports are directly addressed via their PCI addresses and the management interfaces of vRE and vPFE are bridged to the eth0 interfaces that Docker connects to the docker0 bridge in the host.

$ docker run --name vmx2 --rm --privileged \
  -v $PWD:/u:ro \
  --env TAR="vmx-14.1R5.4-1.tgz" \
  --env CFG="vmx2.cfg" \
  --env DEV="0000:05:00.0 0000:05:00.1" \
  --env PFE="lite" \
  --env MEM="5000" --env VCPU="3" \
  -i -t marcelwiget/vmx:latest

--name The name must be unique across containers on the same server (e.g. vmx1)

--rm
Destroy the container after termination (use -d to run as daemon instead)

-d
Optional instead of --rm: Launch the Container in detached mode, making it possible to launch vMX fully unattended, while allowing the user to re-attach to the console via 'docker attach '.

--privileged
Required to allow creation of virtual bridges and tap interfaces and mounting of hugetables as a filesystem

--net=host
Required to allow interface and virtual bridge access across containers and between the host and containers. It also allows the binding of fxp0 to docker0 (--net is optional when using 10GE ports only and fxp0 isn't required)

--v $PWD:/u
Provides access to vmx tar and config file in the current directory from within the container. The destination directory must always be /u and the source directory can be adjusted as needed.

--env TAR=""
Specify the filename of the vMX distribution tar file provided in /u to the container (see --v option)

--env CONFIG=""
Optional. Specify a config file to be loaded at startup. Requires Junos Release 15.1F3 or higher. A metadata drive is built, containing the specified configuration file and added to the vRE at boot time.

--env CFG=""
Optional. Specify a config file that allows zero-touch provisioning of the vMX. See an example further down. It is possible to set a license key as well, but large configs should be transferred via netconf/ssh, because the content of the file is sent to the virtual serial based console with a 1 sec delay after each line. For Junos releases 15.1F3 and higher it is recommended to just use this CFG option to set a license key and use the CONFIG option to pass a full Junos configuration file.

--env DEV=" ... "
Space separated ordered list interface list given to the vMX. Possible interfaces are physical network interfae names (e.g. eth0, p2p1, etc), virtual bridges (which will be automatically created) and PCI addresses of Intel 82599 based 10 Gigabit Ethernet ports. All interface types can be mixed.

--env VCP="<vcp/jinstall*img>"
Optional. Specify a virtual disk image for the VCP/vRE instead of taking it from the TAR file. Can be used to run just the VCP image without any vPFE.

--env PFE="lite"
Optional. If set to "lite", the lite version of the vPFE is used, even if the CPU would allow the use of the high performance vPFE image from the provided vMX distribution tar file.

--env VCPMEM=""
Optional. set the amount of memeory in MB given to the vRE image. default is 2000 (16.1F requires only 1G).

--env MEM=""
Optional. Set the amount of memory in MB given to the vPFE image. default is 8000 (15.1F requires at least 8GB).

--env VCPU=""
Optional. Set the number of vCPU to be used by the vPFE. Default is 5. The vRE image is hard set in launch.sh to 1 vCPU.

-i
Keep STDIN open even if not attached. Required to keep tmux happy, even when not attached.

-t
Allocate a pseudo-TTY. Required for proper operation.

Example: 2 vMX connected via virtual bridge

Launch 2 vMX containers named vmx1 and vmx2 with configs vmx1.cfg and vmx2.cfg and connect them via a virtual bridge br0 (which will be automatically created and destroyed as needed):

Router config files:

  $ cat vmx1.cfg
  root
  cli
  conf
  set interface fxp0.0 family inet address 172.17.42.5/24
  set system root-authentication plain-text-password
  juniper1
  juniper1
  set system host-name vmx1
  set system service ssh
  set system service netconf ssh
  set interface ge-0/0/0.0 family inet address 10.10.10.1/24
  commit and-quit
  mwiget-mba13:vmxdocker mwiget$ cat vmx2.cfg
  root
  cli
  conf
  set interface fxp0.0 family inet address 172.17.42.6/24
  set system root-authentication plain-text-password
  juniper1
  juniper1
  set system host-name vmx2
  set system service ssh
  set system service netconf ssh
  set interface ge-0/0/0.0 family inet address 10.10.10.2/24
  commit and-quit

Launch both vMX's:

  #!/bin/bash
  docker run --name vmx1 -d --privileged --net=host \
    -v $PWD:/u:ro \
    --env TAR="vmx-14.1R5.4-1.tgz" \
    --env CFG="vmx1.cfg" \
    --env DEV="br0" \
    --env PFE="lite" \
    --env MEM="8000" --env VCPU="5" \
    -i -t marcelwiget/vmx:latest

  docker run --name vmx2 -d --privileged --net=host \
    -v $PWD:/u:ro \
    --env TAR="vmx-14.1R5.4-1.tgz" \
    --env CFG="vmx2.cfg" \
    --env DEV="br0" \
    --env PFE="lite" \
    --env MEM="5000" --env VCPU="5" \
    -i -t marcelwiget/vmx:latest

Attach to either vmx via

  $ docker attach vmx1

It is also possible to launch the container directly in interactive mode, so progress can be monitored and the router console is accessible:

  docker run --name vmx1 --rm --privileged --net=host \
    -v $PWD:/u:ro \
    --env TAR="vmx-14.1R5.4-1.tgz" \
    --env CFG="vmx1.cfg" \
    --env DEV="br0" \
    --env PFE="lite" \
    --env MEM="5000" --env VCPU="5" \
    -i -t marcelwiget/vmx:latest

Example vMX with two 82599 based 10GE ports

This example assumes a back-to-back cable to be connected between both ports.

  $ cat vmx3.cfg
  root
  cli
  conf
  set interface fxp0.0 family inet address 172.17.42.7/24
  set system root-authentication plain-text-password
  juniper1
  juniper1
  set system host-name vmx3
  set system service ssh
  set system service netconf ssh
  set interface ge-0/0/0.0 family inet address 10.10.10.1/24
  set routing-instance R1 instance-type virtual-router
  set routing-instance R1 interface ge-0/0/0.0
  set interface ge-0/0/1.0 family inet address 10.10.10.2/24
  set routing-instance R2 instance-type virtual-router
  set routing-instance R2 interface ge-0/0/1.0
  commit and-quit

Launch the vMX as follows:

  docker run --name vmx3 --rm --privileged \
    -v $PWD:/u:ro \
    --env TAR="vmx-14.1R5.4-1.tgz" \
    --env CFG="vmx3.cfg" \
    --env DEV="0000:04:00.0 0000:04:00.1" \
    --env PFE="lite" \
    --env MEM="5000" --env VCPU="5" \
    -i -t marcelwiget/vmx:latest

IMPORTANT: Instead of detaching from tmux via ^BD, use docker's method of detaching from an interactive docker session via ^P^Q. Failing to do so will kill the vMX and the virtual interfaces and bridges will be cleaned up. See http://docs.docker.com/articles/basics/ for details.

Building the vMX Docker Image

• Clone this repository to the linux server

   git clone https://github.com/mwiget/vmxdocker.git
   cd vmxdocker
  

• Building the image

   docker build -t marcelwiget/vmx:latest .
  

  The vmx image is now available for launch:

mwiget@va:~$ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE marcelwiget/vmx latest 14bcc6fdcb4f 35 minutes ago 432.3 MB