knime-ip / knip

KNIME Image Processing Extension

KNIME Image Processing Extension

The KNIME Image Processing Extension adds new nodes to KNIME Analytics Platform to, e.g. read more than 100 different kinds of images (thanks to the Bio-Formats library), apply well-known methods for preprocessing, image segmentation, and classification. Most of the included nodes operate on multi-dimensional image data (e.g. videos, 3D images, multi-channel images or even a combination of them), which is made possible by ImgLib2. In addition, several nodes to calculate image features (e.g. zernike-, texture- or histogram features) for segmented images (e.g. a single cell) are included. These feature vectors can be used to apply machine learning methods in order to train and apply a classifier.

The KNIME Image Processing Extension currently provides about 90 nodes for (pre)processing, filtering, segmentation, feature extraction, various views, ....

For more information how to install/use KNIME Image Processing see: https://www.knime.com/community/image-processing

Example Workflows can be found at: https://hub.knime.com/knime/spaces/Examples/latest/99_Community/01_Image_Processing/

Package Organization

• org.knime.knip.core: Logic/Algorithms/DataStructures/Views independent of KNIME
• org.knime.knip.base: KNIME Image Processing Nodes wrapping core and providing dedicated KNIME Image Processing functionality (NodeModels etc).
• org.knime.knip.feature: Eclipse feature for org.knime.knip.core and org.knime.knip.base
• org.knime.knip.testing: KNIME Image Processing Testing nodes, e.g. for regressions tests.
• org.knime.knip.testing.feature: Eclipse feature for KNIME Image Processing Testings nodes.
• org.knime.knip.io: Image Reader/Image Writer for KNIME Image Processing.
• org.knime.knip.update: Eclipse update site for KNIME Image Processing and depedencies.
• org.knime.knip.tracking: TrackMate Tracking integration.

For Developers

See this repository for instructions.

knip's Issues

Trackmate Tracker: Missing values in feature rows resultieren in ClassCastExceptions

Slice Loop Start: Problem with labelings

Node fails when only labelings are processed.

Img Manipulations: Make sure ColorTables are also adapted

Cropping, Converting, Changing Dimensions etc also affects the ColorTables of an ImgPlus, e.g. converting an ImgPlus from 16bit to 8bit results in another ColorTable. We have to test and fix the corresponding nodes.

Trackmate Node fixes

• Improve description of features
• Enable support for calibration

Slice Loop Start: Fails on invalid input

SliceLoopStart fails on invalid input (just connect any table without any image) with a NullPointerException. There should certainly be a better error handling:

java.lang.NullPointerException
at org.knime.knip.base.nodes.util.slicelooper.SliceIteratorLoopStartNodeModel.execute(SliceIteratorLoopStartNodeModel.java:315)
at org.knime.core.node.NodeModel.execute(NodeModel.java:700)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:552)
at org.knime.core.node.Node.invokeNodeModelExecute(Node.java:1097)
at org.knime.core.node.Node.execute(Node.java:948)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:518)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:120)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:240)

Refactoring KNIME Image Processing

Hi @AndreasBurger @angrauma @Squareys @DanielSeebacher @gab1one,

all further changes / PR will go into https://github.com/knime-ip/knip/tree/refactoring until its merged into master. Reasons why I can't merge, yet:

ImgLib2 related:

[ ] ImgLib2 Bug with Unsynchronized Unsigned12BitType
[ ] ImgLib2 Bug with Extend/Convert imglib/imglib2#82
[ ] ImgLib2-ROI Snapshot and ImgLib2-ROI will heavily change at the hackathon
[ ] No implementation for NTreeImg BitType anymore.

KNIME Image Processing related:

[ ] Adapt Testflows to use mean of images (#87)
[ ] Create Testflows only used for serialization, deserialization
[ ] Create Testflows only used to test convertes (e.g. Inverter/Converter Node + Gauss)

Branch-Cleanup:

[ ] https://github.com/knime-ip/knip/tree/descriptors @DanielSeebacher: Is there any code which is not in imagej-ops besides the UI creation? do we need to transfer anything to the new knip-core?
[ ] https://github.com/knime-ip/knip/tree/imgsaver @gab1one: We need to wait until we have everything in place at knip-core. Can you please merge the current master into this branch?
[ ] https://github.com/knime-ip/knip/tree/scifio-imgsaver @gab1one whats this?
[ ] https://github.com/knime-ip/knip/tree/readpdfs @DanielSeebacher do you still need this?
[ ] https://github.com/knime-ip/knip/tree/knipimg can be deleted
[ ] https://github.com/knime-ip/knip/tree/InteractiveLabelingEditorBugfix @AndreasBurger
[ ] https://github.com/knime-ip/knip/tree/interactiveAnnotatorBezierTool @angrauma
[ ] https://github.com/knime-ip/knip/tree/sls-adjustdim @angrauma
[ ] https://github.com/knime-ip/knip/tree/KnimeService @Squareys I moved the org.knime.knip.scijava project to https://github.com/knime-ip/knip-scijava. Can you double check that this makes sense?

Force ImageType in ImgReader?

Should it be possible to force the ImageReader to use a specific ImageType? To clarify, the following dataset (patentbase) consists of 2000 binary images and has a size of 80,1 MB.

If this dataset is loaded using the Image Reader and saved, the workflow needs over 9GB of space on the hard disk, since every image is now uncompressed and of type UnsignedByteType. Since we know beforehand that we will only read binary images, we could tell the ImageReader to load the images as BitType, which should theoretically reduce the hard drive usage by one-eighth.

Image Reader: Metainfo only

The image reader, or a custom new node, should be able to output only the meta information of an image without reading all of it.

An alternative solution and / or side-project could be removing the need for an input image at the ImageJ Macro Node.

Maximum Finder: Improve Documentation

The current documentation is not pretty expressive. Please work with examples here.

Move to ImageJ-Ops: Boundary/Contour Tracing Algorithms

see text

Table Cell Viewer: Better presentation of table and view

It would be great if the Table Cell Viewer remembered the width of the table vs cell contents.

Moreover, there seems to be a bug when trying to change the size of the table: if I first click on the image and then try to increase the table size, the ruler does not move at all.

Image Viewer: Merged view

It would be great if there was a merged + color table based view.
In Zeiss Zen blue, I can select for each of N channels which of them should be shown in an overlay.

Additionally, the black and white value (= min and max shown) and the gamma per channel can be set by adjusting the line on the histograms within the screenshots. The first and last box shaped handle of the line modifies the black and white value, respectively. The middle handle modifies them both at once like sliding a window. The second and fourth handle do not move left and right as the others, but up and down and they modify the gamma value. Moreover, the black/white value can also be changed with the vertical bar at the end of the lines for convenience.

The "best fit" works as a contrast stretcher, excluding a default of 2% black pixels and 0.01% white pixels and can be applied to the current x,y(,z)?(,t)? subvolumes.

The gamma and best fit would be an optional extra, but the merged overlay and being able to set the black/white value would help a lot!

Update: another optional, but handy feature is storing the image display settings (in Zen blue, they can even be copy+pasted with or without assigning colors, but I don't expect this to be necessary in Knime)

Remove loci dependencies from scifio imagewriter

see #57 (comment)

ImgPlus- / LabelingCell

Unify class hierarchy and introduce new cell types:

LabelingCell and ImgPlusCell share tons of code and therefore we could unify them. Additionally, several nodes (e.g. Cropper, Splitter, etc) can be used by both LabelingCell and ImgPlusCell (as these nodes are independend from the content type and work on RandomAccessibleInterval<?>).

One problem might be, that we have to be backwards compatible.

Splitter: Allow cropping of images with varying dimensions

Select: X,Y,Channel to be splitted

Input: X,Y,Channel and X,Y,Channel,Time

Currently: Error as differing dimensions

Actually: Possible, as the number of resulting objects is the same, just the dimensionality of the objects differs.

Open First Image in TableCellViewer / Interactive Segmentation View per default

Once you opened the TableCellViewer / Interactive SegmentationView, the first image should be displayed directly (we don't want the user to make an additional click here).

Slice Loop Start: Column Selection doesn't work

Column Selection is not working if the input structure of the table changes. (e.g. have img+labeling and then only labeling). Please test it!

Image Parameter Adjuster: issues with channel, Z and zoom

The Image Parameter Adjuster Node does not update the values for Z and channel. The channel variable was even beyond the image dimensions.
Zoom level is initialized with 0, but should be 1.

Thanks for your help!

InteractiveLabelingEditor highlight bug

After switching the displayed image while highlighting is on, returning to a previously highlit image with disabled highliting shows an image without labels.

Create Externalizers for LabelingCell and ImgPlusCell

Currently LabelingCell and ImgPlusCell don't have their own externalizers. Therefore their externalization mechanism can't be changed.

New Node: Segment Counter

Node which allows you to simply calculate the number of labels per labeling. Additionally, the user should be able to set a filter (Label/Group).

Outputs of the nodes may be the following:

(a) Labeling -> Total Segment Count (1:1)
(b) Labeling -> Segment Count per Group (1:n)

WählbyCellClumpSplitter: Border Pixels are removed even if valid

After splitting, all segments which previously touched the border, do not touch the border anymore.

New Node: Image Segment Features

New node using imagej-ops feature implementations. Should be based on knip-scijava

Interactive Segmentation View: Right-Click Context-Menu should be available everywhere

In hilite mode the "context-menu" which appears on "right-click" should appear anywhere, no matter where you perform the right-click in the viewer.

Unify Caching System

Currently we have serveral caching implementations. Actually we could unify them using only one (e.g. apache JCS). We could use a SciJava-Service here.

Image Reader: Allow copying of path

if you copy a path into the text field, the file chooser should "jump" to the location and not set a filter.

Add Butterworth High/Low Pass Filters

http://tech.knime.org/forum/knime-image-processing/highlow-pass-filters

We could simply add a kernel to the convolution kernel creator.

Reorganization: knip-tracking

Move org.knime.knip.tracking into its own repository, s.t. it can be versioned and installed independently.

InteractiveLabelingEditor Nullpointer Exception

Clicking outside the displayed image of the Interactive Labeling Editor results in a NullPointerEx.

Would it be possible make net.imglib2.img.ImagePlusAdapter available via knip plugin

It seems the class net.imglib2.img.ImagePlusAdapter is not made available in knip plugin. Or am I doing something wrong with the plugin? I think the imglib2-ij.jar is not included in the current build.

LabelingArithmetic: Make outtype configurable

If you e.g. merge to labelings of BitType the outtype must be > bittype as you may have three labels (labels from img A, labels from img B and empty labeling) in the resulting labeling.

Rendered image shifts for no apparent reason

When changing transparency or interacting with the displayed image in the Interactive Labeling Editor while zoomed in, the image shifts one pixel.

New Node: Label Mapper Node

will follow.

New Node: Track Features

Integrate Track features / measurements from TrackMate

Image Reader: Fail if selected column does not exist

Recommendation from Thorsten:

Zum anderen verhält er sich nicht korrekt, wenn ich eine Spalte vom
optionalen Eingang konfiguriert habe und die Spalte gibt es später nicht
mehr oder der Eingang ist gar nicht mehr verbunden. Solchen Fälle muss
der Knoten im configure überprüfen und dann fehlschlagen. Im Moment
ignoriert er das aber einfach und gibt eine leere Tabelle aus.

InteractiveLabelingEditor BoundingBoxRenderer bug

When adding labels to a labeling, the corresponding bounding boxes are not updated immediately.

Nullpointer in TrackMateTracker

Under certain conditions the Trackmate Tracker throws a nullpointer exception.

Add a prominent link to http://tech.knime.org/community/image-processing#exampleworkflows

Whether it is in the README or in the description of the repository or even that of the GitHub organization, I think there is value in having a prominent link to the tutorials, as well as a prominent link to an idiot-proof installation walk-through.

NullPointerException when calling .randomAccess()

I have created an ArrayImg using the ArrayImgFactory. The ArrayImg was created, it isn't null but when I call .randomAccess() a NullPointerException is thrown. Below I have added a simple class with a main method which reproduces this error.

import net.imglib2.img.array.ArrayImg;
import net.imglib2.img.array.ArrayImgFactory;
import net.imglib2.img.array.ArrayRandomAccess;
import net.imglib2.img.basictypeaccess.array.ByteArray;
import net.imglib2.type.NativeType;
import net.imglib2.type.numeric.RealType;

public class Error {

    public static <T extends RealType<T> & NativeType<T>> void main(
            String[] args) {

        // create an ArrayImg using the ArrayImgFactory
        ArrayImg<T, ByteArray> arrayImg = new ArrayImgFactory<T>()
                .createByteInstance(new long[] { 50, 50 }, 1);

        // arrayImg isn't null
        System.out.println((arrayImg != null));

        // NullPointerException is thrown here
        ArrayRandomAccess<T> randomAccess = arrayImg.randomAccess();
    }

}

The expected output is

true
Exception in thread "main" java.lang.NullPointerException
    at net.imglib2.img.array.ArrayRandomAccess.<init>(ArrayRandomAccess.java:84)
    at net.imglib2.img.array.ArrayImg.randomAccess(ArrayImg.java:109)
    at Error.main(Error.java:21)

Image Reader: Allow to read in subsets of series

see: http://tech.knime.org/forum/knime-image-processing/feature-request-image-reader-read-multiple-series

Image Converter takes too long

Maybe instead of creating a new image we could just use some kind of Converter for the randomaccess instead of creating a new image.

New Node: Interactive Labeling Editor

Allows you to edit the labels of a labeling.

Image Writer: merged channels

The image writer should support "merging" the channels the same way as I proposed for the image viewer (#56). My goal is to export them in a presentation-ready format, e.g. as PNG with a scale bar included.
Contrast adjustment would be nice, but I assume that should be done with a prefixed normalizer node or image parameter adjuster.

SetImageMetadata: Copies Image Information (which is not required at all).

see above. We should think about the the possibility, that an ImgPlusCell just wraps another ImgPlusCell (to avoid duplication of data serialization).

InteractiveLabelingEditor rename error

InteractiveLabelingEditor throws NullPointerException when trying to rename a label

Slice Loop Start: Support Images / Labeling with different sizes, but same iteration intervals

Given: Img1 (X,Y,Z,Time,Channel) another Img2 (X,Y,Z,Channel).

All dimensions are the same, but Time (Missing in Img2). User wants to perform loop over Channel: Not possible since now, but should be possible, as the number of intervals is the same.

New Node: Thinning with Pruning

Thinning Nodes need a pruning step.

Update Testflows

Update Testflows:

• Store Mean of image instead of images
• Testflow for Don't Save Start / End Nodes
• Testflow for SliceLoopStart / End
• Explicit Testflows for Serialization of all kind of labelings / image factories and types

InteractiveLabelingEditor performs unnecessary calculations

When executing, the Interactive Labeling Editor performs unnecessary calculations on images that weren't touched by the user.

New Node: Track Hilite Propagator

When a point is hilited the complete track should be hilited or vice-versa.