elliotwoods / vvvv.nodes.emgucv

DEPRECIATED!!! please instead use https://github.com/elliotwoods/VVVV.Packs.Image

Introduction

https://github.com/elliotwoods/VVVV.Nodes.EmguCV

A new link type for VVVV 'CVImageLink' which lets you work with images on the CPU (previously achieved with DirectShow filters). CVImageLink includes lots of lovely magic, like threading, double buffering, automatic conversion, yada yada

Perhaps all you want to know is 'is it f*&cking fast'? the answer is yes :)

This is designed for general use of OpenCV functions + also video playback / capture.

Elliot

Credits

As always, big thanks for vux for his undeniable contribution to the VVVV plugin ecosystem, both in terms of generating plugins for users and making new things possible for plugin devs (like me!) Thanks to the VVVV devs for making such a lovely plugin architecture and a great piece of software (www.vvvv.org)

Otherwise: Me - Elliot Woods alg - Vadim Smakhtin

License

This plugin is distributed under the LGPL license (as much as it can be)

Since we wrap EmguCV, currently we inherit the GPL license from EmguCV.

EmguCV is dual license (GPL/commercial). So to the extent to which that affects this code, this also implies that your code must be GPL if you utilise this plugin base Suggest move to opencvsharp http://code.google.com/p/opencvsharp/ or discussion with EmguCV about a licensing model for the VVVV community.

Isntallation

TODO: make notes for quick installation :)

Operating notes

Known issues

• Changing slice counts in image inputs/outputs will sometimes cause crashes. Fixing this is somewhat of a priority, and relates to things being properly disposed (which is also of general importance).
• Changing slice counts on processor nodes (generators, filters, destinations) doesn't result in input properties being updated for the instances.
• We don't handle stride (useful for non '4 byte' image types at non power of 2 resolutions)

Video playback

This isn't necessarilly the best route for video playback, but can also work quite nicely.

Notes for video playback:

• This doesn't employ any fancy hardware optimisation of video codecs. If you need to chunk big video files, suggest sticking with FileStream + clever codecs.
• No audio, probably never will have (although other nodes which output CVImage type could give audio).
• Works with pretty much all the AVI's i threw at it (which isn't that many on this PC). Feedback welcome!

Types

We are now using CVImageLink which is a polymorphic (accepts different formats of image), double threaded (handles threading, locking) and auto converting (e.g. for RGBA8 for GPU) image type.

To use this type yourself, check out examples of where we use CVImageInputSpread and CVImageInputSpreadWith

It can be used for lots of things. We can make nodes in minutes to supply video from lots of sources (e.g. CLEye, Point Grey, BlackMagic) and then use them with the full chain of CV / texture utils.

Threading

Current design is 'Highly Threaded' or 'Background' as described in the threading options list at http://vvvv.org/forum/replacing-directshow-with-managed-opencv.-video-playback-capture-cv

Each node has it's own thread, every exchange of image data between threads results in a double buffer This obviously leads to many threads + much memory usage (perhaps in the future users will be able to select the global threading model at runtime)

The node thread has an output buffer (single). When the thread is ready to send (i.e. all of its inputs are fresh) then it calls FOutputBuffer.SetImage(...) which pushes the image downstream. FOutput should not have any internal buffers, instead passing through its input directly

Links are double buffers

Memory usage

Examples of memory usage:

• 640*480 ~= 300KB (VGA mono)
• 6404803 ~= 1MB (VGA colour)
• 64048016 ~= 5MB (VGA colour + alpha, 32bit float)
• 192010803 ~= 6MB (HD colour video frame)

Generally each slice at each node = 2 * the above (double buffered)

Todo

Interfaces

• ICaptureNode
• ICaptureInstance
• IFilterNode
• IFilterInstance

Node suggestions

General

• Add
• Subtract
• Queue
• Cons

Files

• ImageLoad Loads a set of images into RAM as ImageRGB's. Either use OpenCV's image loader or .NET's Bitmap class (probably quicker)
• ImageSave
• VideoSave Built into CV so should be decent performance

Tracking

• Contour

CameraCalibration

• StereoCalibrate

Future

• AsImage Convert texture to image (AsVideo in existing DirectShow). - requires texture input on plugins (i.e. not possible yet)

vvvv.nodes.emgucv's Issues

Structured Light

Basic structured light implementation in VVVV

Polymorphism on processing classes

e.g.

CVImageOutputSpread and CVImageInputSpread

ProcessInput and ProcessInputOutput

VideoIn is not activating and not reporting status

At "Camera ID 0" VideoIn node is not activating and not reporting any status. That issue was spotted only on last revision, previously all was ok. On "Camera ID -1" VideoIn is activating, but performance if really slow, about 2 fps at MainLoop.

Threading issue?

Get an issue allocating images in OpenNI's Image node
Is likely to be a common issue

Happens on first initialise (but only under certain conditions)

ImageRGB throws:

System.TypeInitializationException was unhandled by user code
Message=The type initializer for 'Emgu.CV.CvInvoke' threw an exception.
Source=Emgu.CV
TypeName=Emgu.CV.CvInvoke
StackTrace:
at Emgu.CV.CvInvoke.cvCreateImageHeader(Size size, IPL_DEPTH depth, Int32 channels)
at Emgu.CV.Image2.AllocateData(Int32 rows, Int32 cols, Int32 numberOfChannels) in C:\Emgu\emgucv-windows-x86 2.3.0.1416\Emgu.CV\Image.cs:line 259 at Emgu.CV.Image2..ctor(Int32 width, Int32 height) in C:\Emgu\emgucv-windows-x86 2.3.0.1416\Emgu.CV\Image.cs:line 209
at VVVV.Nodes.EmguCV.ImageUtils.CreateImage(Int32 width, Int32 height, TColourFormat format) in C:\kimchiandchips\VVVV.Research\VVVV.Nodes.EmguCV\src\Utilities\ImageUtils.cs:line 70
at VVVV.Nodes.EmguCV.CVImage.Allocate() in C:\kimchiandchips\VVVV.Research\VVVV.Nodes.EmguCV\src\Image\CVImage.cs:line 76
at VVVV.Nodes.EmguCV.CVImage.Initialise(Size size, TColourFormat format) in C:\kimchiandchips\VVVV.Research\VVVV.Nodes.EmguCV\src\Image\CVImage.cs:line 34
at VVVV.Nodes.OpenNI.OpenNIImageNode.Evaluate(Int32 SpreadMax) in C:\kimchiandchips\VVVV.Research\VVVV.Nodes.OpenNI\OpenNI\OpenNIImageNode.cs:line 104
InnerException: System.DllNotFoundException
Message=Unable to load DLL 'opencv_core231': The specified module could not be found. (Exception from HRESULT: 0x8007007E)
Source=Emgu.CV
TypeName=""
StackTrace:
at Emgu.CV.CvInvoke.cvRedirectError(CvErrorCallback errorHandler, IntPtr userdata, IntPtr prevUserdata)
at Emgu.CV.CvInvoke..cctor() in C:\Emgu\emgucv-windows-x86 2.3.0.1416\Emgu.CV\PInvoke\CvInvoke.cs:line 59
InnerException:

various bugs

objectTracking test patches:doesn't work
positionXY and scaleXY pin output nil values...
if i recreate it from scratch it works!
vvvv26 still crash on double clic (opening new node)
or sometime when deleting nodes

ImageLoad + AsTexture

Looks like loaded image pushed to GPU only when AsTexture is reconnected to the rendering object. If you reload image and not reconnect AsTexture- all will be black if you loading first time and texture will appeared distored, if you loaded it second time. I think during refactoring you just discarded my small fix to it.

Create interfaces

Test out an interface model for some of the nodes

e.g. FindBoards, ObjectDetect

IInstanceInput needs CVImageLink not CVImageInput?

Since it should perhaps have its own internal CVImageInput
CVImageInput deals with things like is the image fresh, etc.

CV's camera capture functions use VFW

Consider implementing a DirectShow based capture for better performance (so far it isn't an issue)

Initialise default values on a IFilterInstance

default values are currently taken internally, but should be taken from the node
this is only an issue if more image slices are added on the input

VideoIn node should use IGeneratorNode / IGeneratorInstance

Readme is incorrect

1. The is no OpenGL dll's in Prerequisites.zip, only OpenCV dll's.
2. EmguCV and OpenCV dll's must be placed to the same folder, where VVVV.Nodes.EmguCV.dll is located.
3. The is no need to place anything to the plugins folder. We can place folder with VVVV.Nodes.EmguCV.dll and support dlls anywhere.

Enabled 'thread per spread'

currently default is thread per node
it shouldn't be too difficult to have an ISpread in ProcessInputOutputThreaded to allow thread per spread

Merge Image type

to polymorphic type which accepts

Grayscale, RGB, RGBA
bool, byte, ushort, float

Also create associated nodes.

VideoIn thread not closing properly

Causes VVVV to hold on quit

imageload

TTY renderer :
ERR imageLoad can't load image file
black renderer

Texture Loading Node and Alpha channel

I'm i'mplementing texture loading via EmguCV classes, currently all is working, but no alpha channel. If i understanding right, our ImageRBG class can't provide ARGB texture? If this is right, how alpha channel problem will be solved? Or our framework will not support alpha?

P.S. I will commit ImageLoadNode this evening by Moscow time. Also i will add changes to FaceTracking Node.

Tag things that need Reinitialising in AsTexture

Currently Reinitialise reinitialises all textures for that node
we can actually check within the reinitialise function whether each texture needs refreshing or not
let's do that! :)

Line #40 in ProcessInputOutputThreaded

This shouldn't need to be here
track it down :)

Add file save/load to CalibrateCamera

This save/load will also be in use elsewhere. suggest "ISaveable" interface
Or go for a more OpenCV filesaving approach?

AsTexture bugs

1. Huge slowdown on big Images.
2. If one of the input slices is changed, all textures is recreating. Tested with ImageLoad node.

Auto thread locks

Currently the developer must lock the thread themselves within the process
this is performed automatically in the case of GetImage(CVImage out), i.e. the lock is performed automatically during the copy/conversion
but when access is performed via CvMat then the user must lock manually

the locking could be performed by IProcessor (doesn't exist yet, i mean ProcessFilter, ProcessDestination`, etc)
but then we'd miss out on some opportunities to leave the image unlocked (e.g. if it only needs to be locked for a conversion)

so a suggestion could be:
Have IInstanceAutoLock
For these instances, the lock is kept throughout the whole of Process()
for ordinary instances, the locks are performed manually

also for a sanity check, i'm going to check the read lock of a CVImageInput when .CvMat is called