GithubHelp home page GithubHelp logo

canonical / colcon-in-container Goto Github PK

View Code? Open in Web Editor NEW
26.0 5.0 1.0 167 KB

Colcon extension to build a colcon workspace in a container

License: GNU General Public License v3.0

Python 100.00%

colcon-in-container's Introduction

colcon-in-container

colcon verb extension to build and test inside a fresh and isolated ROS environment and transfer the results back to the host.

With this extension, developers can build ROS packages for any ROS 2 distributions directly from colcon independently of the host. With it, one can validate 'builds' and 'tests' making sure all the dependencies are properly listed in their package.xml on any ROS 2 distribution. Validating packages and workspace in an isolated and ephemeral environment is key for distributing and packaging software.

Quickstart

  • Install the tool:
pip3 install -U git+https://github.com/canonical/colcon-in-container
  • Install and initialize LXD:
sudo snap install lxd
lxd init --auto

โš ๏ธ If you have Docker installed, mind that it causes connectivity issues

  • Then call colcon with the build-in-container verb:
colcon --log-level=info build-in-container --ros-distro jazzy

See the usage section for advanced information on installation and tool usage.

How it works

colcon build-in-container

colcon build-in-container performs the following steps to build your workspace:

  • Start an ephemeral LXD container
  • Install a fresh ROS 2
  • Upload your workspace inside the container
  • Install your build time rosdep dependencies (make sure to keep your package.xml updated!)
  • Build you workspace inside the container
  • Download the build/ and install/ directories of the built workspace on your host under: build_in_container and install_in_container/

colcon test-in-container

colcon test-in-container performs the following steps to build your workspace:

  • Start an ephemeral LXD container
  • Install a fresh ROS 2
  • Upload your workspace as well as your build_in_container and install_in_container/ directories inside the container
  • Install your test time rosdep dependencies (make sure to keep your package.xml updated!)
  • Test you workspace inside the container
  • Download the test results directory of the built workspace on your host under: test_results_in_container/

colcon release-in-container

colcon release-in-container performs the following steps to release your workspace's packages:

  • Start an ephemeral LXD container
  • Install a fresh ROS 2
  • Install the necessary software to generate a Debian file
  • Upload your workspace inside the container
  • Install your build time rosdep dependencies (make sure to keep your package.xml updated!)
  • Generate the Debian for each of the selected packages
  • Download the results of the releases on your host under: release_in_container/

Usage

Installation

colcon in-container

To use the extension you will need to install it and also install and initialize LXD. Both steps are described below.

This being a colcon extension, make sure to have colcon installed.

The extension can be installed via pip using the URL with the following command:

pip3 install -U git+https://github.com/canonical/colcon-in-container

Due to potential conflict with the python dependencies cryptography and pyopenssl between a ROS install and the tool, a python virtual environment is recommended.

LXD

LXD can be installed with snap:

sudo snap install lxd

Alternative installation methods are available in the LXD documentation.

Initialize LXD with:

lxd init --auto

Multipass

As an alternative to LXD, one can use multipass as an environment provider.

multipass is support on Linux, Windows and MacOS.

multipass can be installed following the documentation.

Multipass environment variable

multipass VMs attributes can be specified over environment variables:

  • COLCON_IN_CONTAINER_MULTIPASS_CPUS, default="2"
  • COLCON_IN_CONTAINER_MULTIPASS_MEMORY, default="2G"
  • COLCON_IN_CONTAINER_MULTIPASS_DISK, default="256G"

colcon build-in-container

Basic usage:

colcon build-in-container

Advanced usage:

colcon --log-level=info build-in-container --ros-distro jazzy --colcon-build-args "--cmake-args -DCMAKE_BUILD_TYPE=Release" --debug

Usage help:

$ colcon build-in-container --help

usage: colcon build-in-container [-h] [--ros-distro ROS_DISTRO] [--colcon-build-args *] [--debug] [--shell-after] [--paths [PATH [PATH ...]]]

Call a colcon build command inside a fresh container.

options:
  -h, --help            show this help message and exit
  --ros-distro ROS_DISTRO
                        ROS version, can also be set by the environment variable ROS_DISTRO.
  --colcon-build-args *
                        Pass arguments to the colcon build command
  --debug               Shell into the environment in case the build fails.
  --shell-after         Shell into the environment at the end of the build or if there is an error. This flag includes "--debug".
  --provider {lxd, multipass}      Environment provider.

By default, build-in-container uses the ROS version from the ROS_DISTRO environment variable. This can be overwritten with the option --ros-distro allowing one to compile for a different ROS distribution than the one associated with the host OS.

colcon test-in-container

Basic usage:

colcon test-in-container

Advanced usage:

colcon --log-level=info test-in-container --ros-distro jazzy --colcon-test-args "--cmake-args -DCMAKE_BUILD_TYPE=Release" --debug

Usage help:

$ colcon test-in-container --help

usage: colcon test-in-container [-h] [--ros-distro ROS_DISTRO] [--colcon-test-args *] [--debug] [--shell-after] [--paths [PATH [PATH ...]]]

Call a colcon test command inside a fresh container.

options:
  -h, --help            show this help message and exit
  --ros-distro ROS_DISTRO
                        ROS version, can also be set by the environment variable ROS_DISTRO.
  --colcon-test-args *  Pass arguments to the colcon test command. Arguments matching other
                        options must be prefixed by a space.
  --debug               Shell into the environment in case the build fails.
  --shell-after         Shell into the environment at the end of the build or if there is an
                        error. This flag includes "--debug".
  --provider {lxd, multipass}      Environment provider.

By default, buil and test in-container use the ROS version from the ROS_DISTRO environment variable. This can be overwritten with the option --ros-distro allowing one to test for a different ROS distribution than the one associated with the host OS.

colcon release-in-container

Basic usage:

colcon release-in-container

Advanced usage:

colcon --log-level=info release-in-container --ros-distro jazzy --bloom-generator rosdebian --debug

Usage help:

$ colcon release-in-container --help

usage: colcon release-in-container [-h] [--ros-distro ROS_DISTRO] [--bloom-generator {debian,rosdebian}] [--debug] [--shell-after] [--paths [PATH [PATH ...]]]

Generate Debian package inside a fresh container using bloom and fakeroot.

options:
  -h, --help            show this help message and exit
  --ros-distro ROS_DISTRO
                        ROS version, can also be set by the environment variable ROS_DISTRO.
  --bloom-generator {debian,rosdebian}
                        Pass arguments to the bloom-generate command
                        Pass arguments to the colcon build command
  --debug               Shell into the environment in case the build fails.
  --shell-after         Shell into the environment at the end of the build or if there is an error. This flag includes "--debug".
  --provider {lxd, multipass}      Environment provider.

By default, release-in-container uses the ROS version from the ROS_DISTRO environment variable. This can be overwritten with the option --ros-distro allowing one to release for a different ROS distribution than the one associated with the host OS.

Use cases

The colcon in-container extension can be used to:

  • Build, test and release ROS 2 package in a clean environment
  • Build, test and release a ROS 2 package for a different ROS distribution
  • Make sure that your package.xml are up to date
  • Build, test and release a ROS 2 workspace with a ROS 2 version you haven't installed
  • And more!

Troubleshooting

If you have issues with pylxd and openssl:

Pylxd doesn't work from the apt debian or the pip package. You must be installed from source

Please open a GitHub issue if you face any issue with the tool.

colcon-in-container's People

Contributors

guillaumebeuzeboc avatar giusebar avatar artivis avatar

Stargazers

Lee Trout avatar Daisuke Nishimatsu avatar Patrick Roncagliolo avatar Adam Sasine avatar Yang Gao avatar  avatar Iftach Naftaly avatar Alexander Entinger avatar BrnBlrg avatar Russ avatar  avatar  avatar Kalle avatar  avatar Ng Chun Lin avatar Ashutosh Ramola avatar Rajendra Singh avatar Yong avatar Miguel Xochicale, PhD avatar Alexis Paques avatar  avatar KasperCJ avatar Gabriele Sisinna avatar  avatar Andre Nguyen avatar Silvio Traversaro avatar

Watchers

Mirko Ferrati avatar Silvio Traversaro avatar  avatar  avatar  avatar

Forkers

gavanderhoorn

colcon-in-container's Issues

Interest in Docker support?

First: thank you for making this package available.

As I had a need to build packages in Docker containers I spent some time adding support for that. I've pushed those changes to my fork: main...gavanderhoorn:colcon-in-container:docker_provider.

At this point I believe it has feature parity with the LXD and Multipass providers, in addition to a Docker-specific approach for #23 (allow builds to be based on a user-provided Docker image, which could contain more than just the base development/build environment).

It has worked well enough for me, so I thought I'd let you know about it.

Is pre-downloading images required?

Hi, I'd like to try this out, it looks useful! But on my first attempt I get:

colcon --log-level=debug build-in-container --ros-distro rolling
DEBUG:colcon:Parsed command line arguments: Namespace(log_base=None, log_level=10, verb_name='build-in-container', ros_distro='rolling', colcon_build_args='', debug=False, shell_after=False, provider='lxd', ignore_user_meta=False, metas=['./colcon.meta'], base_paths=['.'], packages_ignore=None, packages_ignore_regex=None, paths=None, packages_up_to=None, packages_up_to_regex=None, packages_above=None, packages_above_and_dependencies=None, packages_above_depth=None, packages_select_by_dep=None, packages_skip_by_dep=None, packages_skip_up_to=None, packages_select_build_failed=False, packages_skip_build_finished=False, packages_select_test_failures=False, packages_skip_test_passed=False, packages_select=None, packages_skip=None, packages_select_regex=None, packages_skip_regex=None, packages_start=None, packages_end=None, allow_overriding=[], mixin_files=None, mixin=None, verb_parser=<colcon_mixin.mixin.mixin_argument.MixinArgumentDecorator object at 0x7eb0f9e81270>, verb_extension=<colcon_in_container.verb.build_in_container.BuildInContainerVerb object at 0x7eb0fa691a20>, main=<bound method BuildInContainerVerb.main of <colcon_in_container.verb.build_in_container.BuildInContainerVerb object at 0x7eb0fa691a20>>, mixin_verb=('build-in-container',))
INFO:colcon.colcon_core.location:Using log path 'log/build-in-container_2024-05-07_19-54-23'
[0.118s] INFO:colcon.colcon-in-container:Downloading the image then creating the LXD instance
[0.130s] ERROR:colcon:colcon build-in-container: Failed getting remote image info: Failed getting image: The requested image couldn't be found for fingerprint "ubuntu/jammy/cloud/amd64"
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/colcon_core/command.py", line 533, in verb_main
    rc = context.args.main(context=context)
  File "/home/dhood/.local/lib/python3.10/site-packages/colcon_in_container/verb/build_in_container.py", line 99, in main
    self.provider = ProviderFactory.create(context.args.provider,
  File "/home/dhood/.local/lib/python3.10/site-packages/colcon_in_container/providers/provider_factory.py", line 47, in create
    return provider(ros_distro)  # type: ignore
  File "/home/dhood/.local/lib/python3.10/site-packages/colcon_in_container/providers/lxd.py", line 89, in __init__
    self.instance = self.lxd_client.instances.create(config, wait=True)
  File "/home/dhood/.local/lib/python3.10/site-packages/pylxd/models/instance.py", line 343, in create
    client.operations.wait_for_operation(response.json()["operation"])
  File "/home/dhood/.local/lib/python3.10/site-packages/pylxd/models/operation.py", line 57, in wait_for_operation
    operation.wait()
  File "/home/dhood/.local/lib/python3.10/site-packages/pylxd/models/operation.py", line 94, in wait
    response = self._client.api.operations[self.id].wait.get()
  File "/home/dhood/.local/lib/python3.10/site-packages/pylxd/client.py", line 207, in get
    self._assert_response(
  File "/home/dhood/.local/lib/python3.10/site-packages/pylxd/client.py", line 178, in _assert_response
    raise exceptions.LXDAPIException(response)
pylxd.exceptions.LXDAPIException: Failed getting remote image info: Failed getting image: The requested image couldn't be found for fingerprint "ubuntu/jammy/cloud/amd64"

is there some lxd image I should download ahead of time, or something?

Tests should be built and ran independently

Tests shouldn't be built during the build phase. This could be done with --catkin-skip-building-tests.
Additionally, during the test-in-container, we should seperate the build of the tests from the run.
Ideally with the following workflow:

  • create a ros lxc and rosdep install only the test-deps
  • take a lxc snapshot
  • rosdep install the build deps
  • build the tests
  • copy the tests artefacts to a temporary folder on the host
  • restore the lxc snapshot
  • copy again the tests to the lxc
  • run the tests

Every command executed in multipass is executed with sudo

The difference between multipass and lxd is that lxd only has a root user, while multipass has a root and an ubuntu one.
By default, in multipass every command is executed as ubuntu user.

So far we have force sudo on every command to have the same process as LXD.
We might want to change that.

Originally posted by @artivis in #16 (comment)

Install packages produced by colcon release_in_container to satisfy dependencies of subsequently discovered packages.

mkdir -p tryit/src
cd tryit/src
git clone [email protected]:Fields2cover/ortools_vendor
git clone [email protected]:Fields2cover/Fields2cover
colcon --log-level=info release-in-container --ros-distro humble --bloom-generator rosdebian --debug 
INFO:colcon.colcon_core.location:Using log path 'log/release-in-container_2024-06-10_16-40-00'
[1.752s] INFO:colcon.colcon-in-container:Downloading the image then creating the LXD instance
[7.522s] INFO:colcon.colcon-in-container:Waiting for ROS 2 to be installed
[191.008s] INFO:colcon.colcon-in-container:Initialising rosdep
[192.088s] INFO:colcon.colcon-in-container:Updating rosdep
[214.000s] INFO:colcon.colcon_core.package_discovery:Crawling recursively for packages in '/home/aosmw/lxd'
[214.031s] INFO:colcon.colcon-in-container:Discovered 2 packages, uploading the selected ones in the instance
[214.031s] INFO:colcon.colcon-in-container:uploading src/ortools_vendor into the instance /ws/src/ortools_vendor
[214.725s] INFO:colcon.colcon-in-container:uploading src/Fields2cover into the instance /ws/src/Fields2cover
[216.111s] INFO:colcon.colcon-in-container:Installing dependencies with rosdep
[336.569s] INFO:colcon.colcon-in-container:Bloom generating ortools_vendor
[338.204s] INFO:colcon.colcon-in-container:Generating binary for ortools_vendor

[2935.279s] INFO:colcon.colcon-in-container:Saving results for ortools_vendor
[2935.894s] INFO:colcon.colcon-in-container:Package ortools_vendor released!
[2935.894s] INFO:colcon.colcon-in-container:Installing dependencies with rosdep
[2936.507s] INFO:colcon.colcon-in-container:Bloom generating fields2cover
[2937.115s] ERROR:colcon.colcon-in-container:Release package fields2cover failed: Failed to run command: functools.partial(<bound method ReleaseInContainerVerb._bloom_generate of <colcon_in_container.verb.release_in_container.ReleaseInContainerVerb object at 0x7462e6a254b0>>, 'fields2cover', 'rosdebian') in container.Error code is 1
[2937.116s] WARNING:colcon.colcon-in-container:Debug was selected, entering the instance

The actual error was due to a cmake check where Fields2cover tests for a function provided by a newly packaged ortools_vendor.

Manually installing the ortools_vendor package with dpkg -i /ws/release/ortools_vendor/ros-humble-ortools-vendor_9.9.0-0jammy_amd64.deb

and then manually restarting the Fields2cover build showed that it satisfied the dependency.

Maybe there is a way to inject the /ws/release directory into the places searched by rosdep.

`release-in-container` appears to assume `package_name` == pkg directory

As per title.

Looking at _bloom_generate(..) fi:

def _bloom_generate(self, package_name, generator):
logger.info(f'Bloom generating {package_name}')
return self.provider.execute_commands([
f'cd /ws/src/{package_name}',
f'bloom-generate {generator}'])

in case package a is actually in dir b (on disk), this fails for me (as cd /ws/src/a will not work).

Technically there is no requirement for a ROS package to be hosted in a directory with the same name as the one listed in the manifest. It's often the case though, but would be nice to not assume it.

Failed to run cloud-init with error: 1.

mkdir tryit
cd tryit
python3 -mvenv venv
source venv/bin/activate
pip3 install --upgrade pip
pip3 install git+https://github.com/canonical/colcon-in-container
mkdir src
cd src
git clone myrosrepo
cd ..
colcon --log-level=info release-in-container --ros-distro humble --bloom-generator rosdebian --debug 
INFO:colcon.colcon_core.location:Using log path 'log/release-in-container_2024-06-10_14-41-44'
[0.123s] INFO:colcon.colcon-in-container:Downloading the image then creating the LXD instance
[1.571s] INFO:colcon.colcon-in-container:Waiting for ROS 2 to be installed
[470.152s] ERROR:colcon.colcon-in-container:Failed to run cloud-init with error: 1.
[470.152s] WARNING:colcon.colcon-in-container:Debug was selected, entering the instance
root@colcon-in-container:/ws#
root@colcon-in-container:/ws# cloud-init status
status: error
root@colcon-in-container:/ws# cloud-init collect-logs
version:  /usr/bin/cloud-init 24.1.3-0ubuntu1~22.04.4

Wrote /ws/cloud-init.tar.gz
root@colcon-in-container:/ws# tar -zxvf cloud-init.tar.gz 
tar -zxvf cloud-init.tar.gz
cd cloud-init
root@colcon-in-container:/ws/cloud-init-logs-2024-06-10# cat journal.txt  | grep -C3 ERROR
Jun 10 04:42:28.906350 colcon-in-container snapd[175]: overlord.go:515: Released state lock file
Jun 10 04:42:28.906350 colcon-in-container snapd[175]: daemon stop requested to wait for socket activation
Jun 10 04:42:28.907474 colcon-in-container systemd[1]: snapd.service: Deactivated successfully.
Jun 10 04:44:52.956665 colcon-in-container cloud-init[276]: 2024-06-10 04:44:52,955 - gpg.py[ERROR]: Failed to obtain gpg key C1CF 6E31 E6BA DE88 68B1 72B4 F42E D6FB AB17 C654
Jun 10 04:44:52.956665 colcon-in-container cloud-init[276]: Traceback (most recent call last):
Jun 10 04:44:52.956665 colcon-in-container cloud-init[276]:   File "/usr/lib/python3/dist-packages/cloudinit/gpg.py", line 101, in recv_key
Jun 10 04:44:52.956665 colcon-in-container cloud-init[276]:     naplen = next(sleeps)
--
Jun 10 04:46:46.483889 colcon-in-container systemd[1]: Finished Download data for packages that failed at package install time.
Jun 10 04:46:53.886977 colcon-in-container snapd[2046]: api_snaps.go:427: Installing snap "git" revision unset
Jun 10 04:47:33.947019 colcon-in-container snapd[2046]: api_snaps.go:427: Installing snap "python3-pip" revision unset
Jun 10 04:47:54.005176 colcon-in-container python3[1716]: ["2024-06-10T04:47:54.001", "ERROR", "ubuntupro.http", "_readurl_urllib", 171, "timed out", {"exc_info": "Traceback (most recent call last):\n  File \"/usr/lib/python3.10/urllib/request.py\", line 1348, in do_open\n    h.request(req.get_method(), req.selector, req.data, headers,\n  File \"/usr/lib/python3.10/http/client.py\", line 1283, in request\n    self._send_request(method, url, body, headers, encode_chunked)\n  File \"/usr/lib/python3.10/http/client.py\", line 1329, in _send_request\n    self.endheaders(body, encode_chunked=encode_chunked)\n  File \"/usr/lib/python3.10/http/client.py\", line 1278, in endheaders\n    self._send_output(message_body, encode_chunked=encode_chunked)\n  File \"/usr/lib/python3.10/http/client.py\", line 1038, in _send_output\n    self.send(msg)\n  File \"/usr/lib/python3.10/http/client.py\", line 976, in send\n    self.connect()\n  File \"/usr/lib/python3.10/http/client.py\", line 1448, in connect\n    super().connect()\n  File \"/usr/lib/python3.10/http/client.py\", line 942, in connect\n    self.sock = self._create_connection(\n  File \"/usr/lib/python3.10/socket.py\", line 845, in create_connection\n    raise err\n  File \"/usr/lib/python3.10/socket.py\", line 833, in create_connection\n    sock.connect(sa)\nTimeoutError: timed out\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/lib/python3/dist-packages/uaclient/http/__init__.py\", line 167, in _readurl_urllib\n    resp = request.urlopen(req, timeout=timeout)\n  File \"/usr/lib/python3.10/urllib/request.py\", line 216, in urlopen\n    return opener.open(url, data, timeout)\n  File \"/usr/lib/python3.10/urllib/request.py\", line 519, in open\n    response = self._open(req, data)\n  File \"/usr/lib/python3.10/urllib/request.py\", line 536, in _open\n    result = self._call_chain(self.handle_open, protocol, protocol +\n  File \"/usr/lib/python3.10/urllib/request.py\", line 496, in _call_chain\n    result = func(*args)\n  File \"/usr/lib/python3.10/urllib/request.py\", line 1391, in https_open\n    return self.do_open(http.client.HTTPSConnection, req,\n  File \"/usr/lib/python3.10/urllib/request.py\", line 1351, in do_open\n    raise URLError(err)\nurllib.error.URLError: <urlopen error timed out>"}]
Jun 10 04:47:54.006043 colcon-in-container python3[1716]: ["2024-06-10T04:47:54.005", "ERROR", "ubuntupro.lib.esm_cache", "main", 17, "Error updating the cache: Failed to connect to https://contracts.canonical.com/v1/resources?architecture=amd64&kernel=6.5.0-1023-oem&series=jammy&virt=lxc\ntimed out\n", {}]
Jun 10 04:47:54.032409 colcon-in-container systemd[1]: esm-cache.service: Deactivated successfully.
Jun 10 04:47:54.032713 colcon-in-container systemd[1]: Finished Update the local ESM caches.
Jun 10 04:48:13.990177 colcon-in-container snapd[2046]: api_snaps.go:427: Installing snap "build-essential" revision unset

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.