jni / affinder Goto Github PK

A complement to #9

Just pulled the latest master branches of napari, magicgui and affinder and getting this Error when clicking the start button:

(napari) hilsenst@itservices-XPS-15-9500:~/Github/napari$ affinder 
ERROR:root:Unhandled exception:
Traceback (most recent call last):
  File "/home/hilsenst/Github/magicgui/magicgui/widgets/_bases/value_widget.py", line 44, in <lambda>
    lambda *x: self.changed(value=x[0] if x else None)
  File "/home/hilsenst/Github/magicgui/magicgui/events.py", line 603, in __call__
    self._invoke_callback(cb, event)
  File "/home/hilsenst/Github/magicgui/magicgui/events.py", line 625, in _invoke_callback
    cb_event=(cb, event),
  File "/home/hilsenst/Github/magicgui/magicgui/events.py", line 619, in _invoke_callback
    cb(event)
  File "/home/hilsenst/Github/magicgui/magicgui/widgets/_function_gui.py", line 175, in _disable_button_and_call
    self.__call__()
  File "/home/hilsenst/Github/magicgui/magicgui/widgets/_function_gui.py", line 249, in __call__
    value = self._function(*bound.args, **bound.kwargs)
  File "/home/hilsenst/Github/affinder/affinder/affinder.py", line 147, in start_affinder
    close_affinder()
  File "/home/hilsenst/Github/magicgui/magicgui/widgets/_function_gui.py", line 241, in __call__
    raise TypeError(msg) from None
TypeError: missing a required argument: 'layers' in call to 'close_affinder(layers, callback)'.
To avoid this error, you can bind a value or callback to the parameter:

    close_affinder.layers.bind(value)

Or use the 'bind' option in the magicgui decorator:

    @magicgui(layers={'bind': value})
    def close_affinder(layers, callback): ...

ERROR:root:Unhandled exception:

Many image alignment methods fail because they rely on global optimisation of a non-convex metric. With affinder, we can let the user quickly provide a close-enough estimate of initial alignment, after which automatic alignment methods have a better chance of being accurate.

Therefore, it would be nice to add automatic registration methods to affinder that users can activate after an initial alignment.

Hello @jni
I recently installed Napari in the context of the NEUBIAS Pasteur course (30/05-02/06/2023).
We got a very nice introduction and tutorial on creating our own Napari plugins by Robert Haase and Kevin Yamauchi.
In my case, I am really interested in using Napari for a project about 3D microscopy image registration.
I discovered your plugin "affinder" and like it a lot. However, as far as I understand, so far it is only available for 2D, while I would definitely be interested in a 3D version.
Thus, I have the following questions:

• Do you know if there is another equivalent plugin that works in 3D?
• Is there any plan to extend affinder to the 3D? By you or other contributors?
• Do you know if someone is by chance already working on it?
  In case of NO to all three questions above, I might consider to work on it with another colleague of mine.
  Do you have any guidelines and recommandations ?

Thank you very much by advance for your reply.

Best regards,

If a user accidentally closes the affinder widget, the callbacks on the layers will stick around and it will be hard to remove them. It would be good if closing the widget was equivalent to stopping affinder. I don't know yet whether there is an event one can catch when the widget is closed, but there were some discussions about this either in the napari repo or in the Zulip.

According to NEP-29, we no longer need to support Py3.7. And Py3.10 has been out for a few months so we should definitely support it!

It turns out that (unexperienced) users tend to mistake the "Select File" button for a way to load images. This easily leads to inadvertently overwriting image files with the affine matrix.

Could solve or at least mitigate this issue by renaming the button, limiting the selection to .txt files or both.

The description page @DragaDoncila made here is lovely. why not just make it the readme? There's nothing in there that isn't generally applicable to the plugin, and it would make for a much nicer presentation on github, pypi, etc...

Either

• by showing the two layers (layer groups) next to each other, with control points connected by lines (as suggested by Volker), similar to the elephant example in https://ch.mathworks.com/discovery/image-registration.html, or
• by showing the difference between two images diff_img = np.abs(img1 - img2) (this only works for images from the same modality/contrast range, though)

This imagesc thread asks some very fundamental questions about affinder that should be answered in the docs:

https://forum.image.sc/t/affinder-support-request/79861

specifically:

• where is the affinder data stored "in flight"
• how can affinder be integrated into a larger workflow
• how can we monitor residuals and edit point pairs without messing up affinder's callbacks too much

Hi @jni,

just a brief placeholder for something that I noticed today.

When using a dataset with channel order (c,y,x) and 3 channels, napari loads this as if it was a time-series or Z-stack.
After clicking 3 points with affinder I noticed that it didn't switch to the other layer. After clicking a 4th point it crashed.
Looking at the error message I interpret the behaviour as affinder thinking it needs to align volumes or some other dimension misinterpretation.

As some users may want to register a single frame to a time series, or several multi-channel time-series on top of each other
the interpretation of the dimension becomes important. I heard there is a guardian of dimensions keeping an eye on such matters.

Can try and create a more detailed report in the coming days, but I guess as affinder is still WIP maybe this is already on your radar.

If not done already:

Not only save the transform file, but also the control points specified by the user, for later (re-)processing (e.g., using a different transformation model).

During a call with @andreasmarnold, we discussed what else we needed in affinder before publishing. The big issue he identified is that it is very hard to recover from misclicks: if you inadvertently add an incorrect point (or for example, put down the first three points in the wrong order in the first layer switch), the transform is very distorted and it becomes very hard to recover, even if deleting the points.

This could be fixed by an affinder-level history, where each point added and resulting transform is recorded. When hitting undo, we would switch back to the previous state. Even for long click sequences this would be a small amount of data so not too onerous to keep track of.

Offer the user to optionally specify coordinates (e.g. as .csv or .npy) that will be transformed "live" using the computed transform

If we add a layer with even small-ish scale values, adding points results in massive circles on the screen (at least in some platforms). We should use canvas minimum and maximum point sizes to prevent this.

Thank you for this great plugin!

I would like to reuse the saved transformation matrix from Affinder but after using it with skimage.transform.warp I get slightly different results. Could you help me figure out what I'm missing?

Napari 0.4.15
Affinder 0.2.2
skimage 0.19.2

When switching layers before the affine transform has been estimated, the zoom/position shift can be quite jarring, even making finding the layer difficult. The behaviour should instead be as follows:

• upon starting: set the zoom to fit the reference layer
• when enough points (ndim+1) have been selected in the reference: set the zoom to fit the moving layer
• when switching back to the reference layer, alternate between:
  • set the zoom to fit the reference layer
  • don't change the zoom: the next point should be in approximately the same position in the moving layer

Currently affinder is a standalone function. It should also provide a napari dock widget hook implementation so that napari users can install it.

The PyData Sphinx Theme has received a lot of updates recently and it's looking awesome, with lots of momentum. Let's use it!

https://pydata-sphinx-theme.readthedocs.io/en/stable/index.html

#47 added alignment of non-image layers to affinder. We should add tests to make sure we don't break that feature going forward. This should be done after #46 merges.

See napari/napari#3769 for instructions.

... although I'm not sure if there is a functional settings framework in place for napari already?

It's useful to have a sample dataset to demonstrate affinder. We should provide image pairs for each of the transforms supported by affinder (affine, euclidean/rigid, similarity). See the sample_data contribution reference here:

https://napari.org/dev/plugins/contributions.html#contributions-sample-data

When trying to transform images that have a scale <1, the resulting transformed image is wrong.
Is it possible that this is a result of defining the affine in relation to the reference image in apply_tf (line 66)?

Here's an example to quickly reproduce the problem. However, it will be necessary to use the plugin.

import napari
from skimage import transform
from skimage.data import astronaut

# Create reference image
reference_image = astronaut()
# Create moving image by resizing reference image and rotating it by 90 degrees
moving_image = transform.rotate(reference_image, 90, resize=True)
moving_image = transform.resize(moving_image, (1024, 1024), order=2)

# open viewer and add images with arbitrary "nonsense" scale < 1
viewer = napari.Viewer()
viewer.add_image(reference_image, name='ref', scale=(0.16, 0.16))
viewer.add_image(moving_image, name='mov', scale=(0.02, 0.02))

# Use affinder to find affine transformation and then apply it to the moving image

# The transformed image contains nothing of the original image but is the right shape
print(f"shape of transformed image: {viewer.layers[-1].data.shape}")
print(f"scale of transformed image: {viewer.layers[-1].scale}")

Add a status label, such as "Select control points in layer X"

scikit-image/scikit-image#5188 allows estimation of n-dimensional affine transforms. We can either vendor that or wait for the next skimage release, but either way, the affinder code should be n-dimensional. This means we need ndim+1 points, rather than the currently-hardcoded "3", to estimate the initial transform.

Sometimes we want to align a projection to its 3D volume, or a 2D shape with measurements that change over time, e.g. the segmentation of a neuron aligned with a calcium imaging time series. What to do is different in each case (treat the 2D image as a very thin 3D slice, or treat it as "limited" to the two right-aligned dimensions), and this could be controlled by an option switch.

This is different to #67 (but it is related).

@jni you can add a URL to the right hand sidebar description on your repo page. Can you add the documentation url there? https://jni.github.io/affinder/

You have to look for the little gear icon in the right hand sidebar, next to the "About" repo info. You click it to edit the repo description and url.

Then there's a spot you can add the URL

Currently, affinder uses least squares to estimate the transformation between all user selected keypoints. However, it might not always be the best choice, as some user selected points may be more accurately placed than others (due e.g. to the sharpness of image features in different parts of the image). Additionally, users may want to use pre-computed points (see #11, for example), which could come from automated methods). Using RANSAC to match points will increase the reliability of found alignments.

Shapes layers have feelings affines too!

Specifically, I worked with someone who had a 2D+t image layer and a 2D shapes layer (see also #41), and it was not possible to use affinder to align them.

Since all napari layers natively support affine transforms, we should be able to align any layer with any other using affinder.

We missed the moving points layer transformations in #58. This is fixed in #94, but it would be good to add a test to make sure we don't miss it again.

Currently there's no built-in way to save the found affine transforms. We should find a way to set saving options within the plugin.

in testing out npe2 convert, I realized that python setup.py bdist_wheel doesn't seem to be working here. It's not including the affinder module (indeed, the wheel on pypi lacks the actual module) https://pypi.org/project/affinder/#files

I think you need to add packages = find: to your [options] (see for example)

[options]
packages = find:
...

fortunately, this is in the cookiecutter! 😅

Now that we can save transforms, it might be useful to use them as a new starting point, or simply to use affinder to view the transformed data after a reboot.

If a user starts affinder and then adds a layer, that layer is not updated in the dropdown list of available layers. It works when the affinder window is restarted but ideally the layers would refresh automatically.

Instead, proposed by @sofroniewn, leave space for a sub-widget in the existing widget and show/hide this sub-widget.

How does this interact with magicgui though?

this line creates an instance of a widget and therefore a Qt application, which has downstream consequences for napari, among other things. It should be refactored to use magic_factory.

CC @tlambert03

This is really a napari issue, but I find it really hard to place the markers pixel-perfectly accurate due to the points layer cursor.
I could zoom in a lot more, but that makes the overall workflow more cumbersome as I need to constantly zoom in and zoom out again as to quickly navigate without losing the overall overview.

I think with a one line-thine translucent cross hair that is fully transparent at the centre pixel one can place points more accurately.
The same is actually true for the points themselves, they are rather large.

Again, I realize these are all napari issues but none of these have bothered me other than in this particular context of affinder.

When opening the "Start affinder" widgets without any active layers in the viewer:

import napari


viewer = napari.Viewer()

qtwidget, widget = viewer.window.add_plugin_dock_widget(
        'affinder', 'Start affinder'
        )

napari.run()

I get this error:

Traceback (most recent call last):
  File "C:\Users\arnoldam\custom_code\affinder\examples\error-example.py", line 8, in <module>
    qtwidget, widget = viewer.window.add_plugin_dock_widget(
  File "C:\Users\arnoldam\miniconda\envs\napari-main\lib\site-packages\napari\_qt\qt_main_window.py", line 811, in add_plugin_dock_widget
    wdg = _instantiate_dock_widget(
  File "C:\Users\arnoldam\miniconda\envs\napari-main\lib\site-packages\napari\_qt\qt_main_window.py", line 1465, in _instantiate_dock_widget
    return wdg_cls(**kwargs)
  File "C:\Users\arnoldam\miniconda\envs\napari-main\lib\site-packages\magicgui\type_map\_magicgui.py", line 487, in __call__
    self._widget_init(widget)
  File "C:\Users\arnoldam\custom_code\affinder\src\affinder\affinder.py", line 120, in _on_affinder_main_init
    _update_unique_choices(widget.moving, widget.reference.current_choice)
  File "C:\Users\arnoldam\custom_code\affinder\src\affinder\affinder.py", line 104, in _update_unique_choices
    index = choice_names.index(choice_name)
ValueError: '' is not in list

Process finished with exit code 1

Somehow the code evolved to use a magicgui function not for a gui but for its ability to have bound variables, here:

and here:

This is generally a gross abuse of magicgui and the Python programming language. 😂 Since the main widget is stateful, the correct thing to do is to make a class subclassing FunctionGui.

e.g. the mean of the matched point's residuals under the current transformation

Currently, affinder only works if both layers start out with the identity matrix as their .affine. We should instead make sure affine transforms can be composed ad-infinitum.

(Related: #21)

During a pair session today @andreasmarnold came up with what I think is a really good idea: during alignment, have a table widget showing all the points added so far in each layer, together with the associated residuals. Then, one can go back and remove high-residual points from the table, which may have been added less precisely than others. This is easier than manually going through points, especially because the conditions on the "move layers" callback (which alternately bring the reference and moving images and points to the front) are not very sophisticated, and can get weirdly out of sync if you start removing points in the "wrong" order. By removing points in corresponding pairs through the table, we circumvent this issue altogether.

The magicgui table widget might be usable here. It might be a matter of:

• Adding a(n initially empty) table as a return value from the start_affinder function
• Add a callback to add rows to the table as points get added
• Add a callback for when row values get deleted to actually delete the whole row and the corresponding points in the points layers

The last bit is a bit tricky because by default, selecting a row in a magicgui table and pressing backspace clears the row values, rather than deleting the row. We'll have to look at the internal representation of the table to see how easy it is to delete a row "in place". I also expect that people will expect that editing the values in the table actually edits the point coordinates, but let's call that a stretch goal for now. 😂

CC @tlambert03 in case you have some comments about that proposed implementation / the Table widget API. 🙏