firepick1 / firesight

Pick and Place Vision Library

FireSight

FireSight is a factory that builds an OpenCV image processing pipeline from a declarative pipeline specfication. When processed, the pipeline transforms an optional input image and returns a recognized data model with information recognized during each pipeline stage. Both pipeline specification and data model are represented using the JSON data format, which is easy to generate and parse.

Intended for pick-and-place vision recognition, FireSight provides the power of OpenCV in an easy to use JSON format. FireSight lets users customize their vision recognition solutions with minimal effort. FireSight also provides a scriptable command line utility that permit quick exploration of alternate pipeline implementations.

FireSight is available as a C++ library as well as a standalone runtime under the MIT license. It has been tested on:

• Raspbian (Raspberry Pi Debian)
• ChromeBook Pixel (Crouton+Ubuntu)
• Windows 7 Professional ThinkPad T410

Installation

• Build FireSight (Linux)
• Build FireSight (Windows) | Depreciated*

Examples

Recognize holes in cam.jpg:

firesight -p json/pipeline0.json -i img/cam.jpg

The -p pipeline0.json argument identifies the JSON pipeline configuration. In this case, the pipeline consists of two stages (named "s1" and "s2"). The first stage recognizes holes within the given range of diameters. The second writes the working image to "pipeline0.jpg":

[
	{"name":"s1", "op":"HoleRecognizer", "diamMin":22.6, "diamMax":29.9, "show":1},
	{"name":"s2", "op":"imwrite", "path":"target/pipeline0.jpg"}
]

The -i cam.jpg argument specifies the image to process. This parameter is optional but may prove useful. Alternatively, the pipeline itself can use imread to load an image.

The pipeline returns a JSON model representing the information recognized at each pipeline stage:

{
  "s1":{
    "holes":[
      {
        "xmin":608,
        "xmax":632,
        "xavg":619.8985595703125,
        "ymin":45,
        "ymax":69,
        "yavg":57.265720367431641,
        "pts":493,
        "ellipse":490.87387084960938,
        "covar":0.671875
      },
      {
        "xmin":180,
        "xmax":205,
        "xavg":192.58755493164062,
        "ymin":30,
        "ymax":55,
        "yavg":42.214008331298828,
        "pts":514,
        "ellipse":530.92919921875,
        "covar":1.3974609375
      },
      {
        "xmin":287,
        "xmax":311,
        "xavg":298.81341552734375,
        "ymin":34,
        "ymax":58,
        "yavg":45.660377502441406,
        "pts":477,
        "ellipse":490.87387084960938,
        "covar":0.8798828125
      },
      {
        "xmin":500,
        "xmax":525,
        "xavg":512.39361572265625,
        "ymin":41,
        "ymax":64,
        "yavg":52.455375671386719,
        "pts":437,
        "ellipse":490.08847045898438,
        "covar":0.82421875
      },
      {
        "xmin":393,
        "xmax":417,
        "xavg":404.7484130859375,
        "ymin":38,
        "ymax":62,
        "yavg":49.585624694824219,
        "pts":473,
        "ellipse":490.87387084960938,
        "covar":0.89453125
      },
      {
        "xmin":73,
        "xmax":97,
        "xavg":84.7353515625,
        "ymin":26,
        "ymax":51,
        "yavg":38.303031921386719,
        "pts":495,
        "ellipse":510.50881958007812,
        "covar":0.045166015625
      }
    ]
  },
  "s2":{
    "result":true
  }
}

firesight's Issues

Build fails on Rpi2

I'm trying to build on an Rpi2 that's running the standard Debian Wheezy. Here's the error I'm getting. I can't figure it out. Any thoughts? I'd like to try to make this part of ChiliPeppr.

[ 3%] Building CXX object CMakeFiles/firesight.dir/bgSub.cpp.o
In file included from /home/pi/FireSight/bgSub.cpp:7:0:
/home/pi/FireSight/FireSight.hpp:61:12: error: cannot declare field ‘firesight::HoleRecognizer::mser’ to be of abstract type ‘cv::MSER’
/usr/local/include/opencv2/features2d.hpp:208:20: note: because the following virtual functions are pure within ‘cv::MSER’:
/usr/local/include/opencv2/features2d.hpp:217:26: note: virtual void cv::MSER::detectRegions(cv::InputArray, std::vector<std::vector<cv::Point > >&, std::vectorcv::Rect_&)
/usr/local/include/opencv2/features2d.hpp:221:26: note: virtual void cv::MSER::setDelta(int)
/usr/local/include/opencv2/features2d.hpp:222:25: note: virtual int cv::MSER::getDelta() const
/usr/local/include/opencv2/features2d.hpp:224:26: note: virtual void cv::MSER::setMinArea(int)
/usr/local/include/opencv2/features2d.hpp:225:25: note: virtual int cv::MSER::getMinArea() const
/usr/local/include/opencv2/features2d.hpp:227:26: note: virtual void cv::MSER::setMaxArea(int)
/usr/local/include/opencv2/features2d.hpp:228:25: note: virtual int cv::MSER::getMaxArea() const
/usr/local/include/opencv2/features2d.hpp:230:26: note: virtual void cv::MSER::setPass2Only(bool)
/usr/local/include/opencv2/features2d.hpp:231:26: note: virtual bool cv::MSER::getPass2Only() const
/home/pi/FireSight/bgSub.cpp: In member function ‘bool firesight::Pipeline::apply_backgroundSubtractor(json_t_, json_t_, firesight::Model&)’:
/home/pi/FireSight/bgSub.cpp:59:84: error: cannot allocate an object of abstract type ‘cv::BackgroundSubtractorMOG2’
/usr/local/include/opencv2/video/background_segm.hpp:76:20: note: because the following virtual functions are pure within ‘cv::BackgroundSubtractorMOG2’:
/usr/local/include/opencv2/video/background_segm.hpp:62:26: note: virtual void cv::BackgroundSubtractor::apply(cv::InputArray, cv::OutputArray, double)
/usr/local/include/opencv2/video/background_segm.hpp:65:26: note: virtual void cv::BackgroundSubtractor::getBackgroundImage(cv::OutputArray) const
/usr/local/include/opencv2/video/background_segm.hpp:79:25: note: virtual int cv::BackgroundSubtractorMOG2::getHistory() const
/usr/local/include/opencv2/video/background_segm.hpp:80:26: note: virtual void cv::BackgroundSubtractorMOG2::setHistory(int)
/usr/local/include/opencv2/video/background_segm.hpp:82:25: note: virtual int cv::BackgroundSubtractorMOG2::getNMixtures() const
/usr/local/include/opencv2/video/background_segm.hpp:83:26: note: virtual void cv::BackgroundSubtractorMOG2::setNMixtures(int)
/usr/local/include/opencv2/video/background_segm.hpp:85:28: note: virtual double cv::BackgroundSubtractorMOG2::getBackgroundRatio() const
/usr/local/include/opencv2/video/background_segm.hpp:86:26: note: virtual void cv::BackgroundSubtractorMOG2::setBackgroundRatio(double)
/usr/local/include/opencv2/video/background_segm.hpp:88:28: note: virtual double cv::BackgroundSubtractorMOG2::getVarThreshold() const
/usr/local/include/opencv2/video/background_segm.hpp:89:26: note: virtual void cv::BackgroundSubtractorMOG2::setVarThreshold(double)
/usr/local/include/opencv2/video/background_segm.hpp:91:28: note: virtual double cv::BackgroundSubtractorMOG2::getVarThresholdGen() const
/usr/local/include/opencv2/video/background_segm.hpp:92:26: note: virtual void cv::BackgroundSubtractorMOG2::setVarThresholdGen(double)
/usr/local/include/opencv2/video/background_segm.hpp:94:28: note: virtual double cv::BackgroundSubtractorMOG2::getVarInit() const
/usr/local/include/opencv2/video/background_segm.hpp:95:26: note: virtual void cv::BackgroundSubtractorMOG2::setVarInit(double)
/usr/local/include/opencv2/video/background_segm.hpp:97:28: note: virtual double cv::BackgroundSubtractorMOG2::getVarMin() const
/usr/local/include/opencv2/video/background_segm.hpp:98:26: note: virtual void cv::BackgroundSubtractorMOG2::setVarMin(double)
/usr/local/include/opencv2/video/background_segm.hpp:100:28: note: virtual double cv::BackgroundSubtractorMOG2::getVarMax() const
/usr/local/include/opencv2/video/background_segm.hpp:101:26: note: virtual void cv::BackgroundSubtractorMOG2::setVarMax(double)
/usr/local/include/opencv2/video/background_segm.hpp:103:28: note: virtual double cv::BackgroundSubtractorMOG2::getComplexityReductionThreshold() const
/usr/local/include/opencv2/video/background_segm.hpp:104:26: note: virtual void cv::BackgroundSubtractorMOG2::setComplexityReductionThreshold(double)
/usr/local/include/opencv2/video/background_segm.hpp:106:26: note: virtual bool cv::BackgroundSubtractorMOG2::getDetectShadows() const
/usr/local/include/opencv2/video/background_segm.hpp:107:26: note: virtual void cv::BackgroundSubtractorMOG2::setDetectShadows(bool)
/usr/local/include/opencv2/video/background_segm.hpp:109:25: note: virtual int cv::BackgroundSubtractorMOG2::getShadowValue() const
/usr/local/include/opencv2/video/background_segm.hpp:110:26: note: virtual void cv::BackgroundSubtractorMOG2::setShadowValue(int)
/usr/local/include/opencv2/video/background_segm.hpp:112:28: note: virtual double cv::BackgroundSubtractorMOG2::getShadowThreshold() const
/usr/local/include/opencv2/video/background_segm.hpp:113:26: note: virtual void cv::BackgroundSubtractorMOG2::setShadowThreshold(double)
/home/pi/FireSight/bgSub.cpp:75:46: error: ‘CV_LOAD_IMAGE_GRAYSCALE’ was not declared in this scope
/home/pi/FireSight/bgSub.cpp:77:46: error: ‘CV_LOAD_IMAGE_COLOR’ was not declared in this scope
/home/pi/FireSight/bgSub.cpp:100:34: error: ‘CV_BGR2GRAY’ was not declared in this scope
/home/pi/FireSight/bgSub.cpp:105:24: error: ‘class cv::BackgroundSubtractor’ has no member named ‘operator()’
/home/pi/FireSight/bgSub.cpp:107:29: error: ‘class cv::BackgroundSubtractor’ has no member named ‘operator()’
CMakeFiles/_firesight.dir/build.make:54: recipe for target 'CMakeFiles/_firesight.dir/bgSub.cpp.o' failed
make[2]: *** [CMakeFiles/_firesight.dir/bgSub.cpp.o] Error 1
CMakeFiles/Makefile2:60: recipe for target 'CMakeFiles/_firesight.dir/all' failed
make[1]: *** [CMakeFiles/_firesight.dir/all] Error 2
Makefile:133: recipe for target 'all' failed
make: *** [all] Error 2
ERROR : make failed (RC=2)
pi@raspi2 ~/FireSight $

build script changes

Hey, Karl

So as it turns out not all systems contain lsb_release and have Debian like tags. So I propose a new universal way of checking Raspberry Pi versions in the build script, we should add g++ checking as well. Also do we need jansson build docs? if not we should add -DJANSSON_BUILD_DOCS=OFF to build as well. There is currently zero support for recognizing armv7l when installing opencv binaries from the repo. I have tested the below code on all raspberry pi variants but the pi 3 and it works flawlessly,

elif [ "$(cat /proc/cpuinfo | grep -ci -m1 'BCM2709')" == "1" ]; then
echo "SYSTEM : Raspberry Pi 2"
echo "COMMAND : sudo -E unzip -d /usr/local rpi-opencv.2.4.8.zip"
sudo -E unzip -d /usr/local rpi-opencv.2.4.8.zip

elif [ "$(free | awk '/^Mem:/{print $2}')" > "246000" ]; then
echo "SYSTEM : Raspberry Pi 1 (512MB)"
echo "COMMAND : sudo -E unzip -d /usr/local rpi-opencv.2.4.8.zip"
sudo -E unzip -d /usr/local rpi-opencv.2.4.8.zip

elif [ "$(free | awk '/^Mem:/{print $2}')" < "246000" ]; then
echo "SYSTEM : Raspberry Pi 1 (256MB)"
echo "COMMAND : sudo -E unzip -d /usr/local rpi-opencv.2.4.8.zip"
sudo -E unzip -d /usr/local rpi-opencv.2.4.8.zip

EDIT: Sorry for re opening and closing, was a mistake on this damn android app

MatchGrid assertion with small grid

Using the following pipeline:

[
    {"op":"matchTemplate",
        "name":"match",
        "template":"template.png",
        "corr":"{{corr||0.8}}",
        "threshold":"{{threshold||0.6}}",
        "method":"{{method}}",
        "angle":"{{angle}}",
        "output":"{{output}}" },
    {"op":"matchGrid", 
        "name":"grid1", 
        "calibrate":"{{calibrate}}", 
        "scale":"{{scale}}",
        "sep":"[1.5,1.5]",
        "model":"match" },
    {"op":"drawRects", "name":"draw1", "radius":5, "model":"grid1", "color":"{{colorUndistort||[32,255,32]}}"},
    {"op":"undistort", "name":"undistort", "model":"grid1" },
    {"op":"drawRects", "name":"draw2", "radius":2, "model":"match", "color":"{{colorMatch||[32,32,255]}}"}
]

And the following grid / template:

I get the following assertion:

flyinglotus1983@lunix /media/sf_Computer_Vision/2015-08-22 $ firesight -p matchGrid.json -i endo-uncorrected.png -o output.png
firesight: /home/flyinglotus1983/git/FireSight/stages/calibrate.cpp:446: void GridMatcher::calcGridIndexes(): Assertion `0 <= c && c < gridIndexes.cols' failed.
Aborted

I've tried to look through the code, but I can't seem to find out what's causing it. We don't get this assertion with a larger grid. But I'm not sure what's causing it to fail with a smaller grid. The documentation and lack of comments in the code make this a bit confusing to follow. Any help would be appreciated.

Errors when compiling

Hello, I'm trying to compiling the sources and I had the folling error:

FireSight/FireSight.hpp:61:12: error: cannot declare field ‘firesight::HoleRecognizer::mser’ to be of abstract type ‘cv::MSER’
MSER mser;
^

FireSight/bgSub.cpp: In member function ‘bool firesight::Pipeline::apply_backgroundSubtractor(json_t_, json_t_, firesight::Model&)’:
/opt/programacion/framework/FireSight/bgSub.cpp:59:84: error: invalid new-expression of abstract class type ‘cv::BackgroundSubtractorMOG2’
ractor = new BackgroundSubtractorMOG2(history, varThreshold, bShadowDetection);
^
FireSight/bgSub.cpp:105:24: error: ‘class cv::BackgroundSubtractor’ has no member named ‘operator()’
pSubtractor->operator()(bgImage, fgMask, learningRate);
^
FireSight/bgSub.cpp:105:24: error: ‘class cv::BackgroundSubtractor’ has no member named ‘operator()’
pSubtractor->operator()(bgImage, fgMask, learningRate);
^

CMakeFiles/_firesight.dir/build.make:62: fallo en las instrucciones para el objetivo 'CMakeFiles/_firesight.dir/bgSub.cpp.o'
make[2]: *** [CMakeFiles/_firesight.dir/bgSub.cpp.o] Error 1
CMakeFiles/Makefile2:178: fallo en las instrucciones para el objetivo 'CMakeFiles/_firesight.dir/all'
make[1]: *** [CMakeFiles/_firesight.dir/all] Error 2
Makefile:149: fallo en las instrucciones para el objetivo 'all'
make: *** [all] Error 2
ERROR : make failed (RC=2)

I attach the entire output here: http://pastebin.com/TyzrxG6g
Thank you people, and sorry for my bad english!

On RPi estimate-px-per-mm takes about 6s for 400x400 image

The current FirePick Raspberry Pi FireSight takes about 6 seconds to execute estimate-px-per-mm. This is a lot of time and one solution would be to support the specification of a region-of-interest to reduce computational load and speed up processing. To specify the ROI, we could add to following parameter to the stage:

"roi":[150,0,100,400]

Such an ROI would be about 4x faster.

PRIORITY: Medium (it takes about 10-20 tries to get a good image that shows correct resolution)