arenaxr / apriltag-js-standalone

Apriltag detector using the apriltag C library and compiled to WASM using emscripten.

Apriltag WASM Detector Standalone Repo

Apriltag detector using the apriltag C library at https://github.com/AprilRobotics/apriltag, and compiled to WASM using emscripten.

This is the main WASM apriltag detector source, with additional tests and a standalone javascript application that displays the detector output. This allows to develop and test the detector, and then transfer the source to the main ARENA-core source.

Apriltags in the browser

Contents

• apriltag: submodule of the apriltag library source repository (https://github.com/AprilRobotics/apriltag)
• bin: where the resulting binaries are placed
• docs: doxygen documentation of the detector C source; see the docs.
• html: standalone javascript application that displays the detector output; live here.
• log: where valgrind logs are placed
• src: the detector source
• test: cmocka tests
• test/tag-imgs: test input images

Quick Start

Install make, gcc, emscripten, cmocka, valgrind, and doxygen. Cmocka and valgrind are only necessary to run the tests and memory checks. Doxygen is needed if you want to build the documentation.

To compile and run tests, use make:

make <target>

The Makefile has the following targets:

• all: Builds the example binary (atagjs_example) and the WASM files (apriltag_wasm.js).
• atagjs_example (default): Creates a binary (at bin/atagjs_example) of an example program that get the detector output by giving it image files. The image files are indicated as arguments to the program (requires gcc).
• apriltag_wasm.js: Builds the WASM detector (requires emscripten). The resulting files (apriltag_wasm.js and apriltag_wasm.wasm) are placed under the [html(html) folder so they are run with the javascript example there.
• tests: Builds the cmocka test runner as executes it (requires cmocka).
• valgrind: Runs the test program under valgrind for several input images in test/tag-imgs (requires valgrind).
• clean: Cleans non-source files.
• help: outputs description of targets.

Detector Details

The apriltag detector uses the tag36h11 family (pre-generated tags). For tag pose estimation, tag sizes must be known. Use set_tag_size(tagid, size) to tell the detector about the size of a known tag. If the size of a tag is not provided (by calling set_tag_size(tagid, size)), tags are assumed to be of 150 mm.

See pre-generated tags here: https://github.com/arenaxr/apriltag-gen

Detector API

The C detector documentation is here. The detector calls are documented in apriltag_js.c. A usage example can be found at atagjs_example.

When running in a browser, the C code is compiled to WASM and wrapped by the javascript class Apriltag using emscripten's cwrap(). The detector C calls are private to the Apriltag class, which exposes the following calls:

• Apriltag() constructor. Accepts a callback that will be called when the detector code is fully loaded:

let apriltag = Apriltag(() => {
  console.log("Apriltag detector ready.");
});

• The detect() call receives a grayscale image (grayscaleImg) with dimensions given by the arguments imgWidth and imgHeight in pixels:

apriltag.detect(grayscaleImg, imgWidth, imgHeight)

detect() will return an array of JSON objects with information about the tags detected.

Example detection:

[
{
 "id": 151,
 "size": 0.1,
 "corners": [
   { "x": 777.52, "y": 735.39},
   { "x": 766.05, "y": 546.94},
   { "x": 578.36, "y": 587.88},
   { "x": 598, "y": 793.42}
 ],
 "center": { "x": 684.52, "y": 666.51 },
 "pose": {
   "R": [
     [ 0.91576, -0.385813, 0.111941 ],
     [ -0.335306, -0.887549, -0.315954 ],
     [ -0.221252, -0.251803, 0.942148 ] ],
   "t": [ 0.873393, 0.188183, 0.080928 ],
   "e": 0.000058,
   "asol":{
      "R":[
         [ 0.892863, -0.092986, -0.440623 ],
         [ 0.077304, 0.995574, -0.053454 ],
         [ 0.443644, 0.013666, 0.896099 ] ],
      "t":[ 0.040853, -0.032423, 1.790318 ],
      "e":0.000078
   }
 }
}
]

Where:

• id is the tag id,
• size is the tag size in meters (based on the tag id)
• corners are x and y corners of the tag (in fractional pixel coordinates)
• center is the center of the tag (in fractional pixel coordinates)
• pose: is the pose estimation, only returned if return_pose = 1
  • R is the rotation matrix (column major)
  • t is the translation
  • e is the object-space error of the pose estimation
  • asol is the alternative solution candidate, only returned if return_solutions = 1 (see: apriltag_pose.h)

• Use set_tag_size(tagid, size) to tell the detector about the size of a known tag. This size is used when computing the tag's pose and should be set before calling detect(), where
  • tagid is the id of the apriltag
  • size is the size of the tag in meters

apriltag.set_tag_size(5, 0.1); // set the size of tag with id 5 to 0.1 meters

• Use set_camera_info(fx, fy, cx, cy) to tell the detector the camera parameters used when computing the tag's pose. The camera parameters should be set before calling detect(), where
  • fx, fy is the focal length, in pixels
  • cx, cy is the principal point offset, in pixels

apriltag.set_camera_info(997.28, 997.28, 636.91, 360.51);

• Set the detector maximum number of detections, if it should return pose estimates and details about alternative solutions with set_max_detections(maxDetections), set_return_pose(returnPose) and set_return_solutions(returnSolutions), where
  • maxDetections is the maximum number of detections (0=return all)
  • returnPose indicates if pose estimates are returned, (0=do not return; 1=return)
  • returnSolutions indicates if the alternative pose estimates solution is returned, (0=do not return; 1=return)

// return all detections
apriltag.set_max_detections(0);
// return pose estimate
apriltag.set_return_pose(1);
// return pose estimate alternative solution details
apriltag.set_return_solutions(1);

Javascript example

This is an example javascript code snippet that shows how to call detect(), using a video frame already in an html canvas. Before this code, we also need to assign an instance of the Apriltag class to the apriltag variable used in the code and, if we are getting the pose from the detector, we would also need to call apriltag.set_camera_info(fx, fy, cx, cy) to set the correct camera parameters.

// get the video frame
let ctx = canvas.getContext("2d"); // canvas is an html canvas with the video frame
let imageData = ctx.getImageData(0, 0, ctx.canvas.width, ctx.canvas.height);
let imageDataPixels = imageData.data;

// this is the grayscale image we will pass to the detector
let grayscalePixels = new Uint8Array(ctx.canvas.width * ctx.canvas.height);

// convert to grayscale
for (var i = 0, j = 0; i < imageDataPixels.length; i += 4, j++) {
  let grayscale = Math.round((imageDataPixels[i] + imageDataPixels[i + 1] + imageDataPixels[i + 2]) / 3);
  grayscalePixels[j] = grayscale; // single grayscale value
}

// call detect() passing the grayscale image in grayscalePixels. NOTE: **apriltag** is a previously created instance of ```Apriltag```
detections = await apriltag.detect(grayscalePixels, ctx.canvas.width, ctx.canvas.height); // Important: pass a width and height matching the grayscalePixels array size

// do something with the detections returned by detect() ...

See the full example in the html folder, live at https://arenaxr.github.io/apriltag-js-standalone/.

Detector Options

• Change detector options with set_max_detections(maxDetections), set_return_pose(returnPose) and set_return_solutions(returnSolutions). See Detector API for details.

Defaults

The detector is initialized with the following options defined in the Apriltag constructor:

this._opt = {
  // Decimate input image by this factor
  quad_decimate: 2.0,
  // What Gaussian blur should be applied to the segmented image; standard deviation in pixels
  quad_sigma: 0.0,
   // Use this many CPU threads (no effect)
  nthreads: 1,
  // Spend more time trying to align edges of tags
  refine_edges: 1,
  // Maximum detections to return (0=return all)
  max_detections: 0,
  // Return pose (requires camera parameters)
  return_pose: 1,
  // Return pose solutions details
  return_solutions: 1
}

You can edit the source file to change these defaults too.

apriltag-js-standalone's Issues

Arbitrary tag size

Support tags with arbitrary sizes (instead of current ID-based fixed size).

Are tag families 41h12 or 49h12 supported

Are tag families 41h12 or 49h12 supported or are there any plans to support them?

demo can't access camera in current browsers

Methods of accessing the camera have changed. e.g.

const constraints = { video: {width: {exact: 320}, height: {exact: 240} } }
navigator.mediaDevices.getUserMedia(constraints)
  .then((stream) => { 
  	out("Ready.") 
  	player.srcObject = stream;
    })
  .catch((err) => {
  	out(err+". For live results, allow webcam access (click left of URL) then rerun")
    })