petercorke / machinevision-toolbox-matlab

Machine Vision Toolbox for MATLAB

Synopsis

Machine Vision Toolbox for MATLAB® release 4.

The Machine Vision Toolbox (MVTB) provides many functions that are useful in machine vision and vision-based control. It is a somewhat eclectic collection reflecting my personal interest in areas of photometry, photogrammetry, colorimetry. It includes over 100 functions spanning operations such as image file reading and writing, acquisition, display, filtering, blob, point and line feature extraction, mathematical morphology, homographies, visual Jacobians, camera calibration and color space conversion. With input from a web camera and output to a robot (not provided) it would be possible to implement a visual servo system entirely in MATLAB.

An image is usually treated as a rectangular array of scalar values representing intensity or perhaps range. The matrix is the natural datatype for MATLAB and thus makes the manipulation of images easily expressible in terms of arithmetic statements in MATLAB language. Many image operations such as thresholding, filtering and statistics can be achieved with existing MATLAB functions.

Advantages of the Toolbox are that:

• the code is mature and provides a point of comparison for other implementations of the same algorithms;
• the routines are generally written in a straightforward manner which allows for easy understanding, perhaps at the expense of computational efficiency. If you feel strongly about computational efficiency then you can always rewrite the function to be more efficient, compile the M-file using the MATLAB compiler, or create a MEX version;
• since source code is available there is a benefit for understanding and teaching.

Code Examples

Binary blobs

>> im = iread('shark2.png');   % read a binary image of two sharks
>> idisp(im);   % display it with interactive viewing tool
>> f = iblobs(im, 'class', 1)  % find all the white blobs
f =
(1) area=7827, cent=(172.3,156.1), theta=-0.21, b/a=0.585, color=1, label=2, touch=0, parent=1
(2) area=7827, cent=(372.3,356.1), theta=-0.21, b/a=0.585, color=1, label=3, touch=0, parent=1
>> f.plot_box('g')  % put a green bounding box on each blob
>> f.plot_centroid('o');  % put a circle+cross on the centroid of each blob
>> f.plot_centroid('x');

Binary blob hierarchy

We can load a binary image with nested objects

>> im = iread('multiblobs.png');
>> idisp(im)

and request the blob label image which we then display

>> [label, m] = ilabel(im);
>> idisp(label, 'colormap', jet, 'bar')

Camera modelling

>> cam = CentralCamera('focal', 0.015, 'pixel', 10e-6, ...
    'resolution', [1280 1024], 'centre', [640 512], 'name', 'mycamera')
cam = 
name: mycamera [central-perspective]                    
  focal length:   0.015                                 
  pixel size:     (1e-05, 1e-05)                        
  principal pt:   (640, 512)                            
  number pixels:  1280 x 1024                           
  pose:           t = (0, 0, 0), RPY/yxz = (0, 0, 0) deg

and its intrinsic parameters are

>> cam.K
ans =
   1.0e+03 *

    1.5000         0    0.6400
         0    1.5000    0.5120
         0         0    0.0010

We can define an arbitrary point in the world

>> P = [0.3, 0.4, 3.0]';

and then project it into the camera

>> cam.project(P)
ans =
   790
   712

which is the corresponding coordinate in pixels. If we shift the camera slightly the image plane coordiante will also change

>> cam.project(P, 'pose', SE3(0.1, 0, 0) )
ans =
   740
   712

We can define an edge-based cube model and project it into the camera's image plane

>> [X,Y,Z] = mkcube(0.2, 'pose', SE3(0, 0, 1), 'edge');
>> cam.mesh(X, Y, Z);

or with a fisheye camera

>> cam = FishEyeCamera('name', 'fisheye', ...
'projection', 'equiangular', ...
'pixel', 10e-6, ...
'resolution', [1280 1024]);
>> [X,Y,Z] = mkcube(0.2, 'centre', [0.2, 0, 0.3], 'edge');
>> cam.mesh(X, Y, Z);

Bundle adjustment

Color space

Plot the CIE chromaticity space

showcolorspace('xy')
lambda = [460:10:540 560:20:600];
[x,y]=lambda2xy(lambda*1e-9);
hold on
plot_point([x y]', 'printf', {' %d', lambda}, 'ko', 'MarkerFaceColor', 'k', 'MarkerSize', 6)

Load the spectrum of sunlight at the Earth's surface and compute the CIE xy chromaticity coordinates

lambda = [400:5:700] * 1e-9; % visible light
sun_at_ground = loadspectrum(lambda, 'solar');
>> lambda2xy(lambda, sun_at_ground)
ans =
    0.3327    0.3454
>> colorname(ans, 'xy')
loading rgb.txt
ans =
    'antiquewhite4'

Hough transform

im = iread('church.png', 'grey', 'double');
edges = icanny(im);
h = Hough(edges, 'suppress', 10);
lines = h.lines();

idisp(im, 'dark');
lines(1:10).plot('g');

lines = lines.seglength(edges);

lines(1)

k = find( lines.length > 80);

lines(k).plot('b--')

SURF features

We load two images and compute a set of SURF features for each

>> im1 = iread('eiffel2-1.jpg', 'mono', 'double');
>> im2 = iread('eiffel2-2.jpg', 'mono', 'double');
>> sf1 = isurf(im1);
>> sf2 = isurf(im2);

We can match features between images based purely on the similarity of the features, and display the correspondences found

>> m = sf1.match(sf2)
m = 
644 corresponding points (listing suppressed)
>> m(1:5)
ans = 
 
(819.56, 358.557) <-> (708.008, 563.342), dist=0.002137
(1028.3, 231.748) <-> (880.14, 461.094), dist=0.004057 
(1027.6, 571.118) <-> (885.147, 742.088), dist=0.004297
(927.724, 509.93) <-> (800.833, 692.564), dist=0.004371
(854.35, 401.633) <-> (737.504, 602.187), dist=0.004417
>> idisp({im1, im2})
>> m.subset(100).plot('w')

Clearly there are some bad matches here, but we we can use RANSAC and the epipolar constraint implied by the fundamental matrix to estimate the fundamental matrix and classify correspondences as inliers or outliers

>> F = m.ransac(@fmatrix, 1e-4, 'verbose')
617 trials
295 outliers
0.000145171 final residual
F =
    0.0000   -0.0000    0.0087
    0.0000    0.0000   -0.0135
   -0.0106    0.0116    3.3601
>> m.inlier.subset(100).plot('g')
>> hold on
>> m.outlier.subset(100).plot('r')
>> hold off

where green lines show correct correspondences (inliers) and red lines show bad correspondences (outliers)

Fundamental matrix

What's new

• Travis CI is now running on the code base
• All code related to pose representation has been split out into the Spatial Math Toolbox. This repo is now a dependency.

Installation

Install from shared MATLAB Drive folder

This will work for MATLAB Online or MATLAB Desktop provided you have MATLAB drive setup.

1. Click on the appropriate link below and an invitation to share will be emailed to the address associated with your MATLAB account:

• RVC 2nd edition RTB10+MVTB4 (2017)
• RVC 1st edition: RTB9+MVTB3 (2011)

2. Accept the invitation.
3. A folder named RVC1 or RVC2 will appear in your MATLAB drive folder.
4. Use the MATLAB file browser and navigate to the folder RVCx/rvctools and double-click the script named startup_rvc.m

Note that this is a combo-installation that includes the Robotics Toolbox (RTB) as well.

Install from github

You need to have a recent version of MATLAB, R2016b or later.

The Machine Vision Toolbox for MATLAB has dependency on two other GitHub repositories: spatial-math and toolbox-common-matlab.

To install the Toolbox on your computer from github follow these simple instructions.

From the shell:

mkdir rvctools
cd rvctools
git clone https://github.com/petercorke/machinevision-toolbox-matlab.git vision
git clone https://github.com/petercorke/spatial-math.git smtb
git clone https://github.com/petercorke/toolbox-common-matlab.git common
make -C vision

The last command builds the MEX files. Then, from within MATLAB

>> addpath rvctools/common  %  rvctools is the same folder as above
>> startup_rvc

The second line sets up the MATLAB path appropriately but it's only for the current session. You can either:

1. Repeat this everytime you start MATLAB
2. Add the MATLAB commands above to your startup.m file
3. Once you have run startup_rvc, run pathtool and push the Save button, this will save the path settings for subsequent sessions.

Downloading the example images

The Robotics, Vision & Control book (2nd edition) uses a number of example images and image sequences. These are bulky and not really appropriate to keep on Github but you can download them. There are two zip archives:

Archive	Size	Contents
images-RVC2a.zip	74M	All images, seq/*, mosaic/*, campus/*
images-RVC2a.zip	255M	Chapter 14: bridge-l/*, bridge-r/*

Each will expand into the ./images folder which is the default location that MVTB searches for images and sequences.

To download the main (and smaller) archive

cd rvctools/vision
wget petercorke.com/files/MVTB/images-RVC2a.zip
unzip images-RVC2a

To download the second (and larger) archive

cd rvctools/vision
wget petercorke.com/files/MVTB/images-RVC2b.zip
unzip images-RVC2b

Download the contributed code

Some MVTB functions are wrappers of third-party open-source software. Working versions, some patched, can be downloaded below. If you are not using MacOS you will need to rebuild the code.

Archive	Size	Contents
contrib.zip	16M	vl_feat, graphseg, EPnP, camera calibration
contrib2.zip	5M	SIFT, SURF

The packages, and their home pages are

Tool	Author	MVTB function
OpenSurf		isurf
SIFT		isift
vl_feat	A. Vedaldi and B. Fulkerson	imser, isift, BagOfWords
graphseg	P. Felzenszwalb, D. Huttenlocher	igraphseg
Efficient perspective-n-point camera pose estimation (EPnP)	V. Lepetit, F. Moreno-Noguer, P. Fua	CentralCamera.estpose
Camera Calibration Toolbox for MATLAB	Jean-Yves Bouget	calib_gui

Online resources:

• Home page
• Discussion group

Please email bug reports, comments or code contribtions to me at [email protected]

Contributors

Contributions welcome. There's a user forum at http://tiny.cc/rvcforum

License

This toolbox is released under GNU LGPL.

machinevision-toolbox-matlab's Issues

exemplars

Need to have the 'label' flag otherwise it won't display

icorner.m - Issue on the implementation of 'suppress' option

eported by [email protected], Feb 2, 2014
What steps will reproduce the problem?

1. b1 = iread('building2-1.png','grey','double');
2. C1 = icorner(b1, 'nfeat', 200);
3. C2 = icorner(b1, 'nfeat', 200, 'suppress', 20);

What is the expected output? What do you see instead?
For the first call of "icorner" it should return the first 200 strongest corners.
For the second call it should return the first 200 strongest corners that do not overlap regarding a circle with the radius of 20 pixels. (the corner should be more dispersed in the output image).
But, both calls return exactly the same first 200 strongest corners.

What version of the product are you using? On what operating system?
Vision-3.3, Windows 8.1 x64, and Matlab R2012a

Please provide any additional information below.
Reviewing the "icorner" source code I found the problem:
line 280: d = sqrt( sum((features.v'-y).^2 + (features.u'-x).^2) );
This line compute the sum of the distances between current corner and the corners already saved on the solution. In this case "d" is a scalar.
But, we need to compute the individual distances between the current corner and each corner part of the solution. In this case "d" it will be an array. Regarding to the minimum value from "d", we can take a decision if we keep the corner or not.
Fix: d = sqrt((features.v'-y).^2 + (features.u'-x).^2);
Because of this modification, we need to do one more thing: add a new index to the loop. We already have one for parsing the list of corners (which in the current implementation is "i"), and we need one more for the output array.
We need to keep increasing "i" on each iteration, but "j" (the second index) only if we keep the current corner.(is part of the solution)
Attached to this report, is also the icorner.m modified file.

Best Regards,
Andrei

Shadowing of function col2im

The function col2im is shadowed by the col2im function of image Toolbox from matlab.

Error occured if current directory is not the toolbox directory and official image toolbox is installed.

SE3 Error

When I try to run the live script from chapter 15, this part from the second section

P = mkgrid( 2, 0.5, 'pose', SE3(0,0,3) );

gives me the following error

`The class SE3 has no Constant property or Static method named 'check'. Error in mkgrid (line 72)

p=SE3.check(opt.pose)*p;`

Why is this happening and what is the solution.

Thanks

Julia package port

Is there any plan for porting this toolbox to Julia language?

More corner detectors

Add more corner detectors, leverage OpenCV?

Alternatively, wrap CVST

colorspace

merge rg_ticks functionality into colourspace, also do ticks for xy space
deprecate rg_ticks

incomplete file fo census.m

Reported by nibesh.mastran, Apr 19, 2013
What steps will reproduce the problem?

1. opening the census.m file.IT is not complete.

What is the expected output? What do you see instead?

What version of the product are you using? On what operating system?

Please provide any additional information below.

output of igraphseg

The labels should ideally be sequential starting at 1. We have the same problem in ilabel(), perhaps factor out that code and apply it to this problem. Maybe the new MATLAB Map container is the answer.

The function or variable 'numrows' is not defined.

An error occurred when I called the function "homography" Can help me solve some of this problem, thank you!

No startup file in common

It seems there are no start up files in common.

Invariant command

Hello, I was following along to the Robotics, Vision and Control Fundamental Algorithms in MATLAB second edition book, chapter 10.4.2 in shadow removal.

Using the command invariant(im, 0.7, 'noexp'); in MATLAB I get unrecognized function.
After looking into both MVTB and RTB I cannot find this function anywhere. I searched online as well but had no luck. I was hoping by posting here someone might be able to help me recover this function.

esttheta also does not show up in either toolbox

Thanks

mvtbdemo

fairly broken

SE3 Error

When I try to run the live script from chapter 25, this part from the second section

pbvs.run(5);

gives me the following error

The class SE3 has no Constant property or Static method named 'check'.
Error in PBVS/init (line 110)
vs.camera.T = SE3.check(vs.T0); % set camera back to its initial pose
Error in VisualServo/run (line 122)
vs.init();

Why is this happening and what is the solution.

Thanks

can't use SE3

thanks for your work,I am reading your book,and practice ,and I fellow your code in book,but I meet a error.When I use the function --CentralCamera in chap15,it told me error to use SE3.>> cam = CentralCamera('default');
Wrong use of SE3
The specified 'SO3' superclass contains a parse error, can not be found in MATLAB's search path, or is hidden by another file with the same name.
error Camera/set.T (line 717)
c.T = SE3(Tc);
error Camera (line 138)
c.T = eye(4,4);
error CentralCamera (line 169)
c = c@Camera(varargin{:});

could you help me solve this problem?thanks very much!

Bug in istereo with vshift

Two bugfixes (I think) in istereo:

1. 108: variable was given as vshift instead of opt.vshift in 2 locations
2. 109 and 110: zero-indexing-related error produces inconsistent image sizes --> stereo_match doesn't work. My solution:

    if opt.vshift ~= 0
        if opt.vshift > 0
            L = L(1:end-opt.vshift+1,:);
            R = R(opt.vshift:end,:);
        else
            opt.vshift = -opt.vshift;
            L = L(opt.vshift:end,:);
            R = R(1:end-opt.vshift+1,:);
        end
    end

ICP calls for "closest" mex function, in conflict with empty "closest.m" file

Hi,

When using the ICP function, a call is made to the closest function (which is a .mex file). However, I suppose for printing help purposes, there's the closest.m file with only comments. In Matlab R2018b, however, this file makes Matlab throw an error as it can be seen:

T = icp(SAT, sat_hat)
Attempt to execute SCRIPT closest as a function:
/Users/Maltergate/Library/Application Support/MathWorks/MATLAB Add-Ons/Toolboxes/Machine Vision
Toolbox for MATLAB/vision/closest.m

Error in icp (line 107)
        [corresp,distance] = closest(D, Mk);

I am using Matlab R2018b, on Mac (Catalina) and I use the toolbox version 4.2.1.

Error occured when inversing the colorname toolbox

Hi Mr Peter,

I'm having a problem when I'm trying to inverse the "colorname([0.2 0.3 0.4])".

I got an error such as the image below.

Can you please guide me on how to solve this problem?

Thank you.

icorner

icorner with patch returns unit vector descriptors

PointFeature.match subtracts these, is this right.
Modify closest to do dot product, not just subtraction.

Do the Zisserman trick on descriptors.