Keypoints not properly transformed through Augmentations about kornia

[johnnv1]

Not all augmentations have support for keypoints yet (see #941), contributions are welcomed. @shijianjian do we have somewhere a list with what augmentations support each case?

[shijianjian]

In theory, all the augmentations inherits from the GeometricAugmentation class should be properly supported. It should work for the demo code you provided. It worth checking where went wrong if you got some time. @alexanderswerdlow

[alexanderswerdlow]

Upon digging a little deeper, I think I figured out the issue which is pretty simple albeit unintuitive for an end-user. Keypoints can be transformed to be moved out of the image but then moved back into the image by subsequent transformations [e.g., Translate] with zeros for padding. The end result of this is that keypoints become detached from their original content as seen below.

If you have a series of augmentations as is common, it doesn't seem like there's a simple way to determine if a keypoint is no longer a valid one; perhaps returning a mask to denote which ones are valid or setting invalid keypoints to a special value [-1, NaN] would be a solution.

The example below is with RandomResizedCrop -> RandomTranslate.

[shijianjian]

Ah. Yes, I remember the design now. Since we use the same point transformation for bounding boxes, we do not remove those invalid keypoints. Those need to be kept otherwise the boxes will not have four corners.

I would vote for having a visibility field in the keypoint data structure. probably 0 for invisible and 1 for visible. Do you think it is easy to add @alexanderswerdlow ?

[edgarriba]

geometrically speaking the keyword visibility can be a bit too image specific. To follow the recent generic geometry data structures I would considervalid/is_valid that will become also more idiomatic so that we can have something like.

pts = [....]
pts_filtered = [p for pts if p.valid]

besidesi, i'd also consider expanding the Keypoint data structure possibly based or inspired by Vector2

kornia/kornia/geometry/vector.py

Line 95 in 40d07ec

class Vector2(TensorWrapper):

[alexanderswerdlow]

I don't think it should be very difficult but I'm not very familiar with Kornia (this is my first time using it actually).

I added a basic implementation but I'm not sure how robust it is and whether it works for all geometric augmentations. The few I tested (Shear, Translate, Crop, Rotate, Flip) seem to work though. It's a little hacky as it checks the output_size field, specifically for cropping, so a more general implementation (maybe that can apply to 3D) would be better, but unfortunately I don't have the capacity for that atm.

I think a mask is preferable to making each point into an object with a field, at least for my use-case with a very dense grid of key points.

It also might make sense to set the masked out key points to some special value so it's clear to the user that they need to ignore these values, unless there's a use case for them. On the surface, key points are ostensibly used for correspondence so it's hard to see why someone would want this broken.

[shijianjian]

link to #2689