Aim of the FPGA Integration for OpenStack [*] Cloud project is to bring FPGA accelerators as a resources available through Docker containers for the OpenStack users.
There are lots of shell session snippets, which have follows certain convention. Typical Unix/Linux environment is multiuser. To distinguish between ordinary user and privileged ones, shell indicates that by using different shell prompt. That's why in shell snippets, the $ in the beginning of command line indicates non-privileged user, for example:
1 $ whoami # this is a comment!
2 ubuntu
So the prompt in this case is $
and the command was whoami
. Line 2 have
no sign at the beginning means output of the command - user name. Note, that
# sign is used for comments, which means that such thing will be ignored by
shell.
Besides $ prompt can be also #, which means, that commands following it
have to be executed as root
(privileged) user:
1 $ sudo su -
2 # date # this will print current date
3 Wed Aug 31 13:23:12 CEST 2016
4 # whoami
5 root
We have changed user to privileged one using su
command (hence the #
prompt sign), and executed date
and whoami
commands as a root user.
Lines 3 and 4 represents output of the commands. Note, that shell sessions
in this document have no numbered lines, so the above example would look like
this:
$ sudo su -
# date # this will print current date
Wed Aug 31 13:23:12 CEST 2016
# whoami
root
For this very demo of the FPGA integration in OpenStack, there would be 3 machines needed. The minimum requirements for each of them are:
- Processor with 2+ cores
- 4GB Ram
- 10GB drive
- 2 NICs (optionally - management is all we need for the demo purposes)
One of the node should have FPGA installed, however it is not necessary for demonstrating OpenStack part, there could be mock command used.
If needed, virtual machines might be used as well.
Software assumption:
- Command line application for burning/erasing/getting status is required (fpga-cli.py is provided for getting the idea about expected interface)
- Liberty: Ubuntu 14.04
- Mitaka: Ubuntu 14.04
- Newton: Ubuntu 16.04, RedHat 7.2
Note that the following guide is an example based on Ubuntu operating system.
The recommended installation source is the server version of Ubuntu. This demo was prepared using either 14.04 LTS or 16.04 LTS version. The installation is straightforward, although it might require providing some information (like proxy servers) depending on environment. The best way of installing the system is to keep it minimal. For what it's worth, it might be useful to install OpenSSH server on each node.
Several configuration options should be verified:
/etc/hostname
- for each node provide unique host name (for example "controller", "compute1", "compute2")/etc/network/interfaces
- provide the management and public network configuration./etc/hosts
- Idea is, that nodes should be pingable on the management network using their hostnames - for example:··· 192.168.0.10 controller 192.168.0.11 compute1 192.168.0.12 compute2 ···
Following command executed on
compute1
:# ping controller PING controller (192.168.192.10) 56(84) bytes of data. 64 bytes from controller (192.168.192.10): icmp_seq=1 ttl=64 time=0.160 ms 64 bytes from controller (192.168.192.10): icmp_seq=2 ttl=64 time=0.221 ms 64 bytes from controller (192.168.192.10): icmp_seq=3 ttl=64 time=0.157 ms
The main installation process is described in the OpenStack documentation, with the following assumptions:
- Services which are installed are narrowed down to:
- Keystone
- Nova (on controller and on computes)
- Glance
- nova-docker [1] should be installed
Additional package nova-docker and docker itself is required on compute nodes, along with the and following changes:
# # Depending on OpenStack version, use appropriate branch: # # stable/liberty - for Liberty release # # stable/mitaka - for Mitaka release # # master - for Newton release # git clone https://github.com/openstack/nova-docker -b stable/liberty # cd nova-docker # patch -Np1 -i "[/path/to/this/repository]/patches/nova_docker_[OpenStack version].patch" # pip install . # # this one is optional; useful if you want to perform simple test # docker pull busybox # docker save -o busyimg busybox
Alter the /etc/nova/nova.conf
on compute nodes:
[DEFAULT] ... compute_driver=novadocker.virt.docker.DockerDriver
And the /etc/glance/glance-api.conf
on controller node:
[DEFAULT] ... container_formats=ami,ari,aki,bare,ovf,ova,docker
Follow docker installation guide, which basically are the following steps:
# apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 \ --recv-keys 58118E89F3A912897C070ADBF76221572C52609D # # Use either trusty or xenial repository # echo 'deb https://apt.dockerproject.org/repo ubuntu-trusty main' >> \ /etc/apt/sources.list # apt-get update # apt-get purge lxc-docker # apt-get install docker-engine
Change the /etc/nova/nova-compute.conf
to look like following on the compute
nodes:
[DEFAULT] compute_driver=novadocker.virt.docker.DockerDriver
Add users nova
and ubuntu
to group docker:
# usermod -a -G docker nova
Since networking is not relevant at the moment for this moment (but that's the subject to change, obviously), installing nova-network is enough (on compute nodes):
# apt-get install nova-network
For confidence, there are scripts for automate the process of installation under
build_scripts
directory.
After having up and running OpenStack, it is time to install the modifications and configure stack to be FPGA aware.
On controller alter line containing
container_formats
in file/etc/glance/glance-api.conf
to looks like that:container_formats = ami,ari,aki,bare,ovf,ova,docker,fpga
Clone this repository
Now, patch installed nova files with provided patches:
# cd /usr/lib/python2.7/dist-packages/nova # nova_ver=$(dpkg -l |grep -w python-nova | sed -e "s/ii\s\+python-nova\s\+2:\([0-9.]\+\).*/\1/g") # echo $nova_ver 12.0.5 # patch -Np1 -i "[/path/to/this/repository]/patches/ubuntu_[14.04 or 16.04]-nova-${nova_ver}.patch" patching file compute/resource_tracker.py patching file db/sqlalchemy/migrate_repo/versions/303_add_fpga_field.py patching file db/sqlalchemy/migrate_repo/versions/304_add_fpga_instance_field.py patching file db/sqlalchemy/models.py patching file objects/block_device.py patching file objects/compute_node.py ...
Append following lines on
/etc/nova.conf
on[DEFAULT]
section on FPGA node:[DEFAULT] ... fpga_access = True fpga_simulation_mode = False
Alter
/etc/nova/nova.conf
to have the options changed or included in section[DEFAULT]
on controller node:[DEFAULT] ... scheduler_available_filters = nova.scheduler.filters.all_filters scheduler_available_filters = nova.scheduler.filters.fpga_filter.FpgaFilter scheduler_default_filters = RamFilter,ComputeFilter,AvailabilityZoneFilter,ImagePropertiesFilter,FpgaFilter
Issue necessary migration (only on controller):
# nova-manage db sync
Optional. You can install
fpga-cli.py
command frombin
directory to/usr/bin
, if you are installing without real FPGA hardware, or for some reason you don't have the real command available or you just want to wrap the real commands into script or executable with compatible interface. For the last case it will need modification of the code.There is a need for modifying rootwrap configuration, for enabling stub command to be used by compute node. Append following line for
/etc/nova/rootwrap.d/compute.filters
and/etc/nova/rootwrap.d/network.filters
:fpga-cli.py: CommandFilter, fpga-cli.py, root
and provide configuration for it in
/etc/nova/nova-compute.conf
in section[DEFAULT]
:[DEFAULT] ... fpga_exec = fpga-cli.py
If you are using program which implements same interface as
fpga-cli.py
defines (see below for description), replacefpga-cli.py
with proper executable file name. It is expected, that such executable will be placed somewhere in$PATH
variable, so it will be easily reachable.Create images and new flavor. First image have artificial format of fpga, and should contain zip archive (bitfiles with certain accelerator, additional files, and manifest file), which should be propagated to image metadata (this process is not done here). Second image is the system image (here: simple busybox image, we created earlier), which should contain all the tools required for accelerator use, and, what is important, it should have
docker_devices
key, which contain list of devices from/dev
filesystem, which should be passed to the container. Flavour metadata should point to right accelerator binaries. Below are example how match those three entities together for LZO compression accelerator.FPGA IP-Core files as zip archive:
# glance image-create --id dd834aa4-f950-40e6-8c23-9dab7f3f0138 \ --name lzo_compression --disk-format raw --container-format fpga \ --file lzo_compression.zip # glance image-update \ --property manifest='$(cat manifest.json)' \ dd834aa4-f950-40e6-8c23-9dab7f3f0138
where
manifest.json
file is the manifest file, which packagelzo_compression.zip
contains.Provided
id
is not necessary, but must be identical to the one with the one on flavor metadata.Docker image with system and appropriate software to use accelerator:
# docker save ubuntu_lzo | glance image-create \ --id 064704cb-b416-4acf-b149-b7272e1a9a20 --name ubuntu_lzo \ --disk-format raw --container-format docker # glance image-update \ --property docker_devices='/dev/fpga1,/dev/fpga0,/dev/fpga2' \ 064704cb-b416-4acf-b149-b7272e1a9a20
New flavor. Note, that passed to
hw:fpga_ip_id
key value is the same as FPGA package image created above:# nova flavor-create fpga-lzo 6 512 1 1 # nova flavor-key fpga-lzo set \ "hw:fpga_ip_id"= "dd834aa4-f950-40e6-8c23-9dab7f3f0138"
Important
Instead of flavor, information passed with the
hw:fpga_ip_id
key might be passed to the proper docker image in Glance, so that it can be only two entities, not the three. This however might be dangerous, because images can be created by users, while flavors not. Such user created image might contain malicious IP, wich may even phisically destroy FPGA device.
Now restart all nova services on all nodes and you are done. To boot newly created flavor with "fpga" image, just issue the commands:
# nova boot --flavor 6 --image ubuntu_lzo dcr1
Integration with the OpenStack code base have, as described in the above instructions, assumptions:
- Support for Docker containers only, thus nova-docker has to be used
- Some kind of command line tool for programming, erasing and getting the status of FPGA with interface described below
- On system level accelerator programmed on FPGA should expose any kind of interface which might be passed to container (i.e. device from /dev filesystem, socket, pipe etc)
As for the OpenStack code base, nova components was changed as follows:
compute/resource_tracker.py
- added new config option for accessing FPGA by compute host
- added new method for updating fpga resources
- added new
scheduler/filters/fpga_filter.py
- call for
_update_fpga_resource
from_update_available_resource`
db/sqlalchemy/migrate_repo
- added two migrations for new fields in tablescompute_nodes
andinstances
db/sqlalchemy/models.py
- added
fpga_regions
andfpga_regions_used
fields forComputeNode
model - added
fpga_device
field forInstance
model
- added
- several objects have bumped their versions due to change of
Instance
andComputeNode
classes:BlockDeviceMapping
BlockDeviceMappingList
ComputeNodeList
FixedIP
FixedIPList
FloatingIP
FloatingIPList
InstanceListv1
SchedulerRetries
Service
ServiceList
scheduler/host_manager
-HostState
class was updated to make use offpga_regions
andfpga_regions_used
fields- unit tests where adapted to above changes
fpga
- new module that contains FPGA programming/erasing logiccompute/manager
- triggers methods fromfpga
module to program/erase FPGA
Nova-docker driver, was adapted to accept list of devices, file: novadocker/virt/docker/driver.py
.
All actions regarding interaction with the FPGA going through the command line utility, which interface is described in next section.
There should be command line utility available, let's call it fpga-cli.py
,
which will be used for programming, erasing and getting status of the FPGA.
Such utility should provide following interface:
burn
. This argument require another one which is identifier of an IP-Core image stored in glance service. Underneath logic should be able to fetch such image and as a result of programming there should be returned an unique identifier, which will help to find and identify the right region for erase procedure. This could be an uuid or any other string, which will not exceed 256 characters. For example:# fpga-cli.py burn image-id a0399bc1-cb67-4548-b0b8-aa95a91402d3
In case of error, it will return non-zero value, for example:
# fpga-cli.py burn bad-image-id; echo $? Error: cannot program `bad-image-id' - no matching hardware found 64
erase
. Another argument is required, and it should be identifier returned by successfulburn
command. No output is returned, besides exit code, which in case of success is 0. For example:# fpga-cli.py erase a0399bc1-cb67-4548-b0b8-aa95a91402d3; echo $?
In case of error, it will return non-zero value, for example:
# fpga-cli.py erase bad_id; echo $? Error: cannot erase FPGA device with id `bad_id' - unknown region 33
status
. Command for providing information about FPGA:# fpga-cli.py status Used regions: 1/2
Which means, that we have an FPGA have two regions, while one of it is occupied. Error situation will return non-zero exit code:
# fpga-cli.py status; echo $? Error: FPGA device is not accesible 127
This work is on Apache 2.0 license. See LICENSE for details.
Current version of this work is 0.1, and is treated as alpha/PoC stage.
[*] | Other names and brands may be claimed as the property of others |
[1] | Until virtualization is not completed, there will be docker driver used as a workaround for utilizing acceleration provided by FPGA under guest. |