realorangeone / zoloto

A fiducial marker system powered by OpenCV - Supports ArUco and April

Zoloto

A fiducial marker system powered by OpenCV - Supports ArUco and April

Documentation

Installation

pip install zoloto

OpenCV

OpenCV should be installed manually, ideally through your system package manager. This makes it easier to customize your OpenCV installation for your system, or use the optimal settings for your OS / hardware. Note that you may need to install opencv-contrib as well as opencv.

If you'd rather have one installed automatically, install the extra opencv:

pip install zoloto[opencv]

Note that this version lacks hardware acceleration. See the README for more details.

For storage-constrained environments, there's also opencv-contrib-python-headless, which should be installed manually.

Examples

from pathlib import Path

from zoloto import MarkerType
from zoloto.cameras import ImageFileCamera


with ImageFileCamera(Path("my-image.png"), marker_type=MarkerType.ARUCO_6X6) as camera:
    camera.save_frame("my-annotated-image.png", annotate=True)
    print("I saved an image with {} markers in.".format(len(camera.get_visible_markers())))

More examples

Zoloto ships with a CLI (aptly named zoloto), which contains some helpful utils for working with Zoloto and fiducial markers.

Development setup

./scripts/setup.sh will create a virtual environment, and install all the required development dependencies into it.

Note that this will not install a version of OpenCV for you. For that, run ./scripts/setup.sh opencv.

There are some additional useful scripts to assist:

• ./scripts/test.sh: Run the unit tests and linters
• ./scripts/fix.sh: Automatically fix issues from black and isort
• ./scripts/benchmark.sh: Run benchmarks (these can take a couple minutes depending on your hardware)

zoloto's Issues

Should BaseCamera be abstract?

I think that it definitely could be, and we wouldn't lose 3.5 support by implementing that, but will make it harder to misuse the library.

Use OpenCV 4

We want to use OpenCV 4, so we're using a more modern version, which contains a fair number of improvements: https://opencv.org/opencv-4-0/

Unfortunately, there are currently no wheels available for the Raspberry Pi (something we need). #54 reverted back to OpenCV 3, with minimal changes. It's hoped that once OpenCV 4 wheels are available, they'll be a lot more performant, which is great!

Unfortunately, this is blocked on piwheels/packages#28 and piwheels/piwheels#189.

Drop support for Python 3.5

Python 3.5 will be EoL in September (src), so there's no point keeping it around.

Removing support for this will also make typing a bit easier, as 3.6 massively improved Python's typing story.

Camera parameter: `marker_size`

This would be really useful so that you no longer need to subclass Camera in order to use it.

Python 3.10

I am unable to setup Zoloto on Python 3.10 as numpy doesn't support it until 1.22

Numpy release notes: https://numpy.org/doc/stable/release/1.22.0-notes.html

Additional Flake8 plugins

I'd like to propose adding some extra flake8 plugins:

• flake8-debugger - Checks for pdb / ipdb statements
• flake8-docstrings - Especially important for #129
• flake8-commas - Enforces commas, helps reduce merge conflicts in lists.
• flake8-todo - Checks for TODO statements. (I use these a lot when developing a PR)

Use and re-raise OpenCV errors

On almost all the OpenCV calls, there's an additional argument, which is a boolean stating whether the statement executed successfully.

Currently, these values are simply ignored, usually unpacked with _. We should be using these to raise our own exceptions.

This would make errors much easier to catch, and likely allows further tests to be written to account for these edge cases.

Fastcache is not recommended for Python 3.5+

See here: https://github.com/pbrady/fastcache#performance-warning

Specify options as attributes

Many of the arguments passed to the constructor are unlikely to change with each camera instance, so should probably be class attributes, that the constructor then validates as necessary.

For example:

class MyCamera(Camera):
    marker_dict = MarkerDict.DICT_6X6_250
    calibration_file = Path("calibrations.xml")

If a user requires these arguments to be passed into the constructor, they can override the constructor and assign the attributes manually:

class MyCamera(Camera):
    marker_dict = MarkerDict.DICT_6X6_250
    
    def __init__(self, calibration_file: Path):
        self.calibration_file = calibration_file
        super().__init__()

This yields a much simpler implementation for the end user, and means there's no need to pass unnecessary arguments to the constructor.

It's unfortunate something like https://github.com/python-attrs/attrs doesn't exist to achieve this, but the code behind it will be very minimal.

This should aim to be done alongside #26, as this change enforces that the provided camera classes not be used directly.

ujson should be a dev dependency

There are no usages of ujson in the zoloto module, so it should probably be a dev / test dependency only.

API Documentation

There should be some auto-generated API docs, likely based off Sphinx.

I've already registered https://zoloto.readthedocs.io/en/latest/, and linked it to the repository.

Stop requiring subclassing for camera construction

Having to define subclasses for cameras is a strange API.

For quick things, there's really no need, unless you're actually trying to override functionality.

Correctly typed constructor arguments and regular instantiation would work perfectly fine.

zoloto.cameras is not public accessible

This renders the library unusable unfortunately

Investigate Cython / PyPy

The majority of the performance-critical code is already performed in C(++), either by Numpy or OpenCV, however there are still parts of the Python side of things which might benefit from being generated as Cython.

It's also possible that this ends up being a large time sync into adding, for little to no gain.

Adding support for PyPy would be pretty nice too.

May have implications on #203

Difference between `opencv-contrib-python` and `opencv-contrib-python-headless`

As I understand it from the opencv-python project, the -headless variation of a package simply has less features. Why then does Zoloto explicitly fail tests should the non -headless variant of the package be used (the test simply verifies that opencv-contrib-python is not present, and does not check any actual module behaviour)?

What impact / bugs would arise in the library should the non -headless variant be used that warrants explicitly failing tests should it be present?

Image Buffering causes incorrect data

Zoloto seems to cache images for about 4 calls of process_frame. This results in the data persisting after the environment has changed.

It is not clear if the results of the calculation, or the actual image is being cached.

This can be replicated using the j5 implementation of Zoloto:

from time import sleep

from sbot import Robot

r = Robot(wait_start=False)

while True:
	print(len(r.camera.see()))
	sleep(1)

Token is removed after 2 seconds.

> 1
> 1
> 1
> 1
> 1
> 1
> 0
> 0
> 0
> 0
> 0

Enumeration of available cameras.

It would be really useful to support enumeration of available cameras.

Annotate Axes on images

https://docs.opencv.org/4.1.0/d9/d6a/group__aruco.html#ga16fda651a4e6a8f5747a85cbb6b400a2

Determine if we are using the jpeg or YUYV (raw) feed from webcams

Raw has less noisy, jpeg possibly faster.

Maybe add an option?

Command which validates calibrations

We should have a command which validates that a calibration file exists, is valid, and ideally produces sane-looking results (likely by passing them through OpenCV and confirming nothing fails)

Deliver examples in package

The examples/ package contains a couple generally useful scripts, which I find handy during development, and for demos.

It might be an idea to vendor these in to the package (as part of the CLI) so they can be used downstream.

Source bundle on PyPi does not include tests

The Zoloto source bundle on PyPi does not include tests. This means that build-from-source distributions which use PyPi as the python package source of truth cannot easily build and test the package.

This could be resolved by including the tests in the source bundle, but not in the wheels or other prebuilt packages.

Camera discovery prints OpenCV errors

VIDEOIO ERROR: V4L2: Could not obtain specifics of capture window.
VIDEOIO ERROR: V4L: can't open camera by index 1
/dev/video1 does not support memory mapping
VIDEOIO ERROR: V4L: can't open camera by index 2
VIDEOIO ERROR: V4L: can't open camera by index 3
VIDEOIO ERROR: V4L: can't open camera by index 4
VIDEOIO ERROR: V4L: can't open camera by index 5
VIDEOIO ERROR: V4L: can't open camera by index 6
VIDEOIO ERROR: V4L: can't open camera by index 7

I'm aware this issue is known, but I would like to track it in an issue so we can keep an eye on it.

This currently forcefully outputs text for each camera, which is annoying. This will be fixed once opencv/opencv#15111 is released into OpenCV wheels.

RFC: API refactor away from primarily camera classes

As with all good things, I've been tempted to rewrite the API and implementation and change things massively.

The primary changes being:

• Extract the image processing from the BaseCamera into primary functions
• Remove the ImageFileCamera and MarkerCamera. Replace with a process_image function, a more generic process_frame function, and some form of get_marker_image. VideoFileCamera will probably stay as-is, as that still makes a bit of sense as a "camera",
• Ensure these new methods are part of the public API
• Remove get_visible_markers. It's an additional API to maintain (and variants of the below) with little to no benefit ([marker.id for marker in camera.process_frame()] works absolutely fine).

API examples

def process_image_eager(file: Path, marker_dict, detector_params: CalibrationParameters, marker_size: int) -> Generator[EagerMarker, None, None]: ...
def process_image(file: Path, marker_dict, detector_params: CalibrationParameters, marker_size: int) -> Generator[Marker, None, None]: ...

def process_frame(data: ndarray, marker_dict, detector_params: CalibrationParameters, marker_size: int) -> Generator[Marker, None, None]: ...
def process_frame_eager(data: ndarray, marker_dict, detector_params: CalibrationParameters, marker_size: int) -> Generator[EagerMarker, None, None]: ...

def get_marker_image(border_size: int, marker_id: int, marker_size: int, border_colour: Tuple[int, int, int]) -> ndarray: ...

• Lots of the options to the above are shared, I wonder if it's worth having a struct to collect them together, similar to CalibrationParameters.?

Implementation probably also requires #202 (and thus #220).

PyPI package is broken

[EnvCommandError]
Command ['/home/dan/.cache/pypoetry/virtualenvs/j5-viTGjnwY-py3.6/bin/pip', 'install', '--no-deps', 'zoloto==0.4.0'] errored with the following return code 1, and output: 
Collecting zoloto==0.4.0
  Downloading https://files.pythonhosted.org/packages/0f/29/f6cbeb0dc53300024ac8d6f55a4abadba09828c0ae4d07a3e00621c58db4/zoloto-0.4.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-ovbvx2hy/zoloto/setup.py", line 6, in <module>
        with open("requirements.txt") as f:
    FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-ovbvx2hy/zoloto/
You are using pip version 18.1, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Please consider using standard syntax in your setup.py, or switch back to poetry where everything worked just fine.

Calibration Script

We need to re-add this.

Auto release to PyPI and Ship wheels

We should ship wheels for the various OSs rather than just the source distribution. We can use GitHub actions to build on release to automatically generate these easily and deploy automatically.

Deploying automatically would reduce deployment overhead.

Run CI on macOS and Windows

The intention behind Zoloto is for it to be cross-platform, so we should run tests on those platforms too!

GitHub actions makes this incredibly simple, and it's already somewhat setup for it. Probably wants some local testing too.

Module Not Found: cv2

When installing Zoloto from pip on Mac, opencv-python isn't installed, leading to a ModuleNotFoundError.

Support for variable capture resolution

Either as an Enum of resolutions, or a tuple.

Probably should be an argument to process_frame

Remove internal uses of `ndarray`

ndarray is annoying, and not very type safe, especially in its uses in OpenCV.

I think the best option here is to just remove as much ndarray as possible.

This has the benefit of reducing this issue as much as possible, but also enables deeper typing.

I'm expecting to have to create a Frame and Pixel type, and reuse some of the existing ones in coords.py.

Also, array is a method, not a type.

See also #282

Unable to setup on Apple Silicon

Zoloto does not work on Apple Silicon as numpy is restricted to < 1.21

Numpy release notes: https://numpy.org/doc/stable/release/1.21.0-notes.html

Fail when constructing a `Camera` with an invalid source

When constructing any cameras, if the argument is known to be invalid, we should fail immediately, rather than waiting until the user tries to process the image.

Calibration script is broken

Traceback (most recent call last):
  File "/home/dan/.cache/pypoetry/virtualenvs/zoloto-py3.6/bin/zoloto-calibrate", line 11, in <module>
    load_entry_point('zoloto', 'console_scripts', 'zoloto-calibrate')()
  File "/home/dan/Code/zoloto/zoloto/cli/calibrate.py", line 92, in main
    wait_for_markers(camera)
  File "/home/dan/Code/zoloto/zoloto/cli/calibrate.py", line 29, in wait_for_markers
    visible_markers = camera.get_visible_markers(frame)
TypeError: get_visible_markers() takes 1 positional argument but 2 were given

Remove explicit OpenCV dependency

Obviously, this still uses OpenCV, and that's not going anywhere.

However, removing the python dependency on opencv-contrib-python-headless allows the package to be used with a system version of OpenCV, which likely has better compiler optimisations for the hardware.

It would also mean viewer.py could be removed, and replaced with the native OpenCV implementation, which would be nice.

I think we should move the python OpenCV bindings to extras, such that zoloto[opencv] and zoloto[opencv-headless]` work.

For ease of use, we'd likely need to revert (and modify the implementation of) 940d26a to defend certain implementations.

Rename `marker_dict` to `marker_type`

To keep a distinction, the marker_dict is really just a key to be used ton construct what becomes the actual marker dictionary, so we should rename it. This should make the code far less confusing!

Should be done after #118 merges.

Support Marker Dictionaries with FEC

With regards to Zoloto replacing libkoki, during a discussion with @rgilton:

Just read the relevant parts of the Aruco paper. It sounds like it's pretty much equivalent to libkoki, with one exception, which would need careful testing. In libkoki we employ a FEC and a CRC - the FEC allows for some bit errors, while the CRC protects against false positives (every 4-cornered shape in the environment needs filtering out - e.g. Windows, doors, and also just shapes that appear in noisy backgrounds - this is basically the main problem in this space).

What Aruco does is to look for the marker that appears most similar to the marker it's seen. Aruco does limit the similarity using a maximum hamming distance, which is good, but it should be thoroughly tested to ensure that false positives are not reported.

If making Aruco detect libkoki markers turns out to be a useful, it'll be slightly more involved than just adding a new dictionary to it, as there would need to be processing of the FEC. Probably easiest to do this by getting Aruco to do the preliminary marker detection in the image, without filtering for codes, and then doing the decoding/filtering in python.

To Do

• - Ensure that Original Aruco, New aruco and AprilTags are not suitable
• - Consider the scope of implementing koki marker decoding / filtering within this project

Use __all__ in __init__.py

This is nice in a couple of ways, but most importantly, you no longer need the # noqa

from zoloto.coords import Coordinates, Orientation, Spherical, ThreeDCoordinates  # noqa
from zoloto.exceptions import MissingGUIComponents
from zoloto.marker import Marker  # noqa
from zoloto.marker_dict import MarkerDict  # noqa

License

We could really do with a licence on here.

Missing cv2 dependency

Installing zoloto into a fresh virtualenv doesn't install any open cv packages, yet they're needed for it to be used at all.

Steps:

$ python3.9 -m venv venv 
$ . venv/bin/activate

(venv) $ pip install -U pip setuptools wheel

Requirement already satisfied: pip in ./venv/lib/python3.9/site-packages (21.2.4)
Collecting pip
  Using cached pip-22.0.3-py3-none-any.whl (2.1 MB)
Requirement already satisfied: setuptools in ./venv/lib/python3.9/site-packages (58.1.0)
Collecting setuptools
  Using cached setuptools-60.9.3-py3-none-any.whl (1.1 MB)
Collecting wheel
  Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel, setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 58.1.0
    Uninstalling setuptools-58.1.0:
      Successfully uninstalled setuptools-58.1.0
  Attempting uninstall: pip
    Found existing installation: pip 21.2.4
    Uninstalling pip-21.2.4:
      Successfully uninstalled pip-21.2.4
Successfully installed pip-22.0.3 setuptools-60.9.3 wheel-0.37.1

(venv) $ pip install zoloto

Collecting zoloto
  Using cached zoloto-0.8.0-py3-none-any.whl (23 kB)
Collecting numpy<1.21
  Downloading numpy-1.20.3-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.4/15.4 MB 9.1 MB/s eta 0:00:00
Collecting pyquaternion>=0.9.2
  Downloading pyquaternion-0.9.9-py3-none-any.whl (14 kB)
Collecting cached-property>=1.5
  Downloading cached_property-1.5.2-py2.py3-none-any.whl (7.6 kB)
Installing collected packages: cached-property, numpy, pyquaternion, zoloto
Successfully installed cached-property-1.5.2 numpy-1.20.3 pyquaternion-0.9.9 zoloto-0.8.0

(venv) $ python -c 'import zoloto'

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/venv/lib/python3.9/site-packages/zoloto/__init__.py", line 1, in <module>
    from zoloto.coords import Coordinates, Orientation, Spherical, ThreeDCoordinates
  File "/tmp/venv/lib/python3.9/site-packages/zoloto/coords.py", line 4, in <module>
    from cv2 import Rodrigues
ModuleNotFoundError: No module named 'cv2'

Possible bad line in pyproject.toml

zoloto/pyproject.toml

Line 55 in e298adb

black = {version = "^19.3b0", python = "^3.6", allows-prereleases = true}

See srobo/j5#396 for more details

PyPI

This should be on PyPI, it's somewhat annoying to add as a dependency to j5 currently. Especially as there are no release versions.

Expose camera metadata

e.g Model, Serial Number, USB ID?

Can be done on linux using Udev

Fallback to old method on Windows / Mac?

Cryptic Exception when marker dict not supplied

I'd suggest making this a required kwarg, e.g def __init__(arg1, arg2, *, markerdict=bees)

  File "/home/dan/.cache/pypoetry/virtualenvs/j5-py3.6/src/zoloto/zoloto/cameras/base.py", line 13, in __init__
    kwargs["marker_dict"]
KeyError: 'marker_dict'
Exception ignored in: <bound method BaseCamera.__del__ of <__main__.ZolotoCamera object at 0x7f90619e6748>>
Traceback (most recent call last):
  File "/home/dan/.cache/pypoetry/virtualenvs/j5-py3.6/src/zoloto/zoloto/cameras/base.py", line 107, in __del__
    self.close()
  File "/home/dan/.cache/pypoetry/virtualenvs/j5-py3.6/src/zoloto/zoloto/cameras/camera.py", line 20, in close
    self.video_capture.release()
AttributeError: 'ZolotoCamera' object has no attribute 'video_capture'

Support OpenCV >=4.1

It would be great to support 3.8 for this package.

I've had to install Python 3.6 from the AUR to use it :(

Add check to see if calibrations file exists.

We should throw a FileNotFound or similar if a calibrations file is given and doesn't exist.

For reference, the current error is:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dan/Code/j5/j5/components/marker_camera.py", line 54, in see
    return self._backend.get_visible_markers(self._identifier)
  File "/home/dan/Code/j5/j5/backends/hardware/zoloto/camera_board.py", line 76, in get_visible_markers
    zmarker.cartesian.x,
  File "/home/dan/.cache/pypoetry/virtualenvs/j5-py3.6/src/zoloto/zoloto/marker.py", line 63, in cartesian
    return ThreeDCoordinates(*self._tvec)
  File "/home/dan/.cache/pypoetry/virtualenvs/j5-py3.6/src/zoloto/zoloto/marker.py", line 85, in _tvec
    _, tvec = self._get_pose_vectors()
  File "/home/dan/.cache/pypoetry/virtualenvs/j5-py3.6/src/zoloto/zoloto/marker.py", line 74, in _get_pose_vectors
    [self.__pixel_corners], self.__size, *self.__camera_calibration_params
cv2.error: OpenCV(4.1.0) /io/opencv/modules/calib3d/src/calibration.cpp:1070: error: (-215:Assertion failed) CV_IS_MAT(objectPoints) && CV_IS_MAT(imagePoints) && CV_IS_MAT(A) && CV_IS_MAT(rvec) && CV_IS_MAT(tvec) in function 'cvFindExtrinsicCameraParams2'

PEP 396 -- Module Version Numbers

Whilst this PEP is optional, I feel like it is best practise to expose an __version__

Org for long term location

In order to enable future open source contributions, it would be better to have the canonical source of this repo in a location where there is more than one maintainer.

Improve Documentation

The current documentation is a bit crap.

• The API documentation is incomplete
• There is no "tutorial-style" content
• It requires some hacks to even get it to build, because autodoc doesn't like the docstrings of OpenCV

This needs some time looking into. I'm also not married to Sphinx if something else gives the features we need.

Migrate CI to GitHub actions

There was a time when CI was running on GitHub actions (#128), but due to tkinter issues, it was reverted (#130).

By moving over, we can automatically run CI for macOS and windows (currently untested), and speed up our build times due to not having concurrency limits.

Required by #203