kornia / kornia

Geometric Computer Vision Library for Spatial AI

---

English | 简体中文

Website • Docs • Try it Now • Tutorials • Examples • Blog • Community

Kornia is a differentiable computer vision library for PyTorch.

It consists of a set of routines and differentiable modules to solve generic computer vision problems. At its core, the package uses PyTorch as its main backend both for efficiency and to take advantage of the reverse-mode auto-differentiation to define and compute the gradient of complex functions.

Overview

Inspired by existing packages, this library is composed by a subset of packages containing operators that can be inserted within neural networks to train models to perform image transformations, epipolar geometry, depth estimation, and low-level image processing such as filtering and edge detection that operate directly on tensors.

At a granular level, Kornia is a library that consists of the following components:

Component	Description
kornia	a Differentiable Computer Vision library, with strong GPU support
kornia.augmentation	a module to perform data augmentation in the GPU
kornia.color	a set of routines to perform color space conversions
kornia.contrib	a compilation of user contrib and experimental operators
kornia.enhance	a module to perform normalization and intensity transformation
kornia.feature	a module to perform feature detection
kornia.filters	a module to perform image filtering and edge detection
kornia.geometry	a geometric computer vision library to perform image transformations, 3D linear algebra and conversions using different camera models
kornia.losses	a stack of loss functions to solve different vision tasks
kornia.morphology	a module to perform morphological operations
kornia.utils	image to tensor utilities and metrics for vision problems

Installation

From pip:

pip install kornia
pip install kornia[x]  # to get the training API !

Other installation options

From source:

python setup.py install

From source with symbolic links:

pip install -e .

From source using pip:

pip install git+https://github.com/kornia/kornia

Examples

Run our Jupyter notebooks tutorials to learn to use the library.

🚩 Updates

• ✅ Image Matching Integrated to Huggingface Spaces. See Gradio Web Demo.
• ✅ Face Detection Integrated to Huggingface Spaces. See Gradio Web Demo.

Cite

If you are using kornia in your research-related documents, it is recommended that you cite the paper. See more in CITATION.

@inproceedings{eriba2019kornia,
  author    = {E. Riba, D. Mishkin, D. Ponsa, E. Rublee and G. Bradski},
  title     = {Kornia: an Open Source Differentiable Computer Vision Library for PyTorch},
  booktitle = {Winter Conference on Applications of Computer Vision},
  year      = {2020},
  url       = {https://arxiv.org/pdf/1910.02190.pdf}
}

Contributing

We appreciate all contributions. If you are planning to contribute back bug-fixes, please do so without any further discussion. If you plan to contribute new features, utility functions or extensions, please first open an issue and discuss the feature with us. Please, consider reading the CONTRIBUTING notes. The participation in this open source project is subject to Code of Conduct.

Community

• Forums: discuss implementations, research, etc. GitHub Forums
• GitHub Issues: bug reports, feature requests, install issues, RFCs, thoughts, etc. OPEN
• Slack: Join our workspace to keep in touch with our core contributors and be part of our community. JOIN HERE
• For general information, please visit our website at www.kornia.org

Made with contrib.rocks.

kornia's Issues

[JIT] Image warp and conversion warnings

JIT tests throws the following warnings that should be fixed: https://pastebin.com/XQPURJFy

TGM v0.1.0 Roadmap

This is the roadmap for torchgeometry v0.1.0 focuses on multi-view reprojection primitives that work on images and feature maps.

• Feel free to volunteer yourself if you are interested in trying out some items(they do not have to be on the list).

Features

• API
  • Pinhole
    • inverse_pose 001097c
    • pinhole_matrix 063b0c6
    • inverse_pinhole_matrix 063b0c6
    • scale_pinhole 063b0c6
    • homography_i_H_ref 063b0c6
  • Conversions
    • pi fb56c9c
    • rad2deg fb56c9c
    • deg2rad fb56c9c
    • convert_points_from_homogeneous f13c462
    • convert_points_to_homogeneous f13c462
    • transform_points 1a1511c
    • angle_axis_to_rotation_matrix 4c929d5
    • rtvec_to_pose 4c929d5
    • rotation_matrix_to_angle_axis f999165
    • rotation_matrix_to_quaternion f999165
    • quaternion_to_angle_axis f999165
  • Warping
    • HomographyWarper 2e54229
    • DepthWarper cd03569
  • Utilities
    • tensor_to_image 3ad7d05
    • image_to_tensor 3ad7d05
    • inverse 1a1511c
• Testing
  • docker kit ae8d9e1
  • automated test with TravisCI c213e3e
  • lint tests c213e3e
  • cuda tests (v0.1.1)
• Documentation
  • sphinx documentation b9251ab
  • development e800c36
• Examples:
  • homography regression ce1524d
  • image warped by depth f999165
  • jupiter notebooks 3c96835

SO3 & SE3

@edgarriba, Have you seen the GVNN library. It is vision geometry Pytorch library that is written in Lua. @chaneyw was suggesting to add SO3, SE3 transformations as well. Pl let me know what do you think.

[FEAT] Implement Conv SAME

Import Conv SAME from: https://github.com/rwightman/pytorch-image-models/blob/ema-cleanup/models/conv2d_same.py

This will be useful for filtering to avoid black borders

[Feature] Transformation

Hi, I would like to contribute a few functions related to transformation into this repo.
I think just calling things pose is very vague, pose is an abstract term that can have many representations. The current pose is represented using a homogeneous transformation, and it should be named that way (instead of just calling it pose, which can be represented by xyz and quaternion or xyz and euler angles).
Also whenever one mentions such a transformation, it is always better to also mention the two frames (a passive transformation).

The other thing I don't understand is what is eps in all these transformation functions?

[FEAT] rename compose_transformation and relative_transformation signatures

As we started to discuss in #79 ,
@versatran01 proposes to rename the following function signatures:

compose_transformations(...) -> boxplus(...)
relative_transformation(...) -> boxminus(...)

Links to this change reference:

• https://arxiv.org/abs/1107.1119
• https://frc.ri.cmu.edu/~hpm/project.archive/reference.file/Smith,Self&Cheeseman.pdf

function 'convert_points_from_homogeneous' may not work

This package is a great job.

However, in the function convert_points_from_homogeneous, I found:

# we check for points at infinity
    z_vec: torch.Tensor = points[..., -1:]
    scale: torch.Tensor = torch.tensor(1.) / torch.clamp(z_vec, EPS)

    return scale * points[..., :-1]

This code does not work if the z_vec is negative, because its value is restricted by EPS. However, the last element of the homogeneous coordinate vector can be negative.

So, is that a bug?

[FEAT] Verify conversion to homogeneous

We should double check the following behavior: opencv/opencv#14411

[CONTRIB] LAF-related stuff

Hi,

Would you be interested in incorporating things for local affine frames ?
E.g. https://github.com/ducha-aiki/affnet/blob/master/LAF.py

If yes, we can discuss and I can prepare PR after the CVPR deadline

`normalize_transform_to_pix` strange behaviour

found that normalize_transform_to_pix when the grid in the HomographyWarper is in normalized coordinates doesn't produce the correct result.

However, when we disable normalize_transform_to_pix and we force the grid to be in pixel coordinates and we convert the transformed grid to normalized coordinates we get the correct results.

TGM v0.2.0 Roadmap

This is the roadmap for torchgeometry v0.2.0 focuses mainly importing GVNN, documentation improvement and website. Some basic functionalities for image processing will also be part of this release.

Feel free to volunteer yourself if you are interested in trying out some items(they do not have to be on the list). Please, take a read on the CONTRIBUTING notes.

• GVNN
  • Linear Transformations
    • relative_pose 6130892
  • SO3 Layer [link]
    • TransformationRotationSO3
    • Transform3DPoints_R
  • SE3 Layer [link]
    • TransformationMatrix3x4SO3
    • Transform3DPoints_Rt
  • Common
    • PinHoleCameraProjectionBHWD
    • ReverseXYOrder
  • Optical Flow [link]
  • Disparity [link]
  • Projection Layer [link]
  • Lens Distorsion [link]
  • Nonrigid SO3 [link]
  • Nonrigid SE3 [link]
  • M-estimators [link]
• LAF: Local Affine Features
  • CornersHarris @edgarriba
• Image Filtering
  • GaussianBlur 0ff6f5e
  • Blur
  • Sobel @edgarriba
  • PyrDown @edgarriba
• Image Color Space: borrowed from tf.image
  • rgb_to_grayscale
  • hsv_to_rgb
  • rgb_to_hsv
  • rgb_to_yiq
  • rgb_to_yuv
  • yiq_to_rgb
  • yuv_to_rgb
• Contrib
  • SpatialSoftArgmax2d cf7bb29
  • ROIAlign @IssamLaradji
  • SoftArgmax
• Losses
  • SSIM 7f890f4
  • DepthSmoothnessLoss 42a1d22
  • DiceLoss 9b0fddf
  • TverskyLoss 89246d2
  • FocalLoss ffe4cb1
  • MultiScaleSSIM
  • PSNR
• Metrics
  • ConfusionMatrix 3678dcf
  • MeanIoU 3678dcf
• Installation
  • conda packages
• Documentation
  • create CONTRIBUTING file 0ab5c8a
  • add tutorials to sphinx docs 06bdd03
  • add examples to sphinx docs 39eb046
  • create front-end website
  • add installation and extended project background to website
  • fix jupyter notebooks tracebacks
• Test
  • add pytest-cov and Codecov check to CI 909b529
  • add mypy check to CI 6cbaef3
  • add docs to readthedocs dbfaa1a
  • refactor test framework. Goal: designing proper classes @carlosb1 #38
  • CUDA tests (looking for alternatives for GPU servers with Jenkins)
• Tutorials
  • add code to CI
  • Beginner
    • improve rotation tutorial with extra explanation, add batched version.
    • improve perspective tutorial with extra explanation, add batched version.
    • improve gaussian blur tutorial with extra explanation, add batched version.
    • add sobel tutorial
    • add box blur, pyrdown tutorial
  • Intermediate
    • add pipeline example: use sequential to implement vision pipelines.
    • add JIT tutorial
  • Advanced
    • move existing homography tutorial to docs
    • move existing depth tutorial to docs
    • add SSIM example to learn input image

[FEAT] Switch torch asserts

We currently have custom assertion to check equal torch tensors

import utils # test utilities
assert utils.check_equal_torch(trans_02_hat, trans_02)

this can be replaced by:

from torch.testing import  assert_allclose
assert_allclose(rans_02_hat, trans_02)

[ENHANCEMENT] Create conda packages

• We should provide conda packages for each release.
• Update installation docs showing conda installation procedure.

[BUG] Does perspective transform support source points outside the image dimensions

Hi,

I was experimenting with homography transform and one issue, I was consistently getting different results as compared to opencv. in warp perspective where the calculation of 'M' was correct.

Before I check my code or discuss more about this issue, could you please answer if the source points outside image dimensions are supported ? Because I have such "source" points which are supported in OpenCV but not in TGM ?

[BUG] Regarding relative_pose

Either this math is wrong or I misunderstood some of the notation here.

P_1^{2} = \begin{bmatrix} R_2 R_1^T & R_1^T (t_2 - t_1) \ \mathbf{0} &
1\end{bmatrix}

I believe the correct forms should be (assuming when you say pose, it is wrt to some common frame, thus T_1 is actually T_1^W, T_2 is actually T_2^W. Then the relative transformation from 1 to 2 is T_2^1 which is

T_2^1 = T_1 ^ -1 * T_2 = R_1^T * R_2 | R_1^T * (t_2 - t_1)

So the rotation is flipped.

or

T_1^2 = T_2 ^ -1 * T_1 = R_2^T * R_1 | R_2^T * (t_1 - t_2)

[DOCS] Update data types links

Some data types are yet not well linked. Need to check the source code and update types according to mypy if possible.

Example link

  Args:
    M (Tensor): transformation matrix. # WRONG
    M (torch.Tensor): transformation matrix. # CORRECT

[Festure Request] Suggested enhancement for SpatialSoftArgmax2d

@edgarriba : First of all, very nice repository!

I have a suggested enhancement that would help users implement recently published papers. I was inspecting the class that impelments softargmax2d

class SpatialSoftArgmax2d

and thought that you might want to add a temperature parameter for your class that a client can then use to specify the peakiness of their desired distribution.

Install issue keeps happening no matter how I install

import torchgeometry
Traceback (most recent call last):
File "", line 1, in
File "torchgeometry/init.py", line 3, in
from torchgeometry import core
File "torchgeometry/core/init.py", line 1, in
from .homography_warper import HomographyWarper, homography_warp
File "torchgeometry/core/homography_warper.py", line 39
height: int,
^

[COMMUNITY] Call for contributions / tutorials / examples

Hello, community!

We’re putting out a call for contributions in an effort to make the docs even more useful, with interactive Gradio demos hosted on Hugging Face 🤗

As Kornia is a Computer Vision library, we want users to be able to see the functionalities of our modules as easily as possible. We’re tackling this by making demos to showcase particular features, and then embedding them directly into the docs where possible. For example, see the demo included in the kornia.enhance documentation: https://kornia.readthedocs.io/en/latest/enhance.html#interactive-demo

With our Docs ‘n Demos Sprint, we’re opening this initiative up to the community! The process is simple:

1. Choose from one of the features and examples listed below, or suggest one yourself! Let us know in the comments on this issue.
2. Go to the Kornia organization on Hugging Face, and click “Request to join this org”.
3. Open an issue for your feature, and mention this issue in the description.
4. Build a simple Gradio app to showcase the feature, like this one here: Demo + Code
5. Deploy your Gradio app to Hugging Face Spaces. Make sure to choose "kornia" as the owner.
6. Open a PR with a link to your Gradio app embedded in the docs

.. raw:: html

    <gradio-app space="kornia/edge_detector"></gradio-app>

Visit the demo on `Hugging Face Spaces <https://huggingface.co/spaces/kornia/edge_detector>`_.

Visit the Gradio docs to learn about embedding your app.

Below are some options that we think lend themselves well to creating demos:

• Filtering Operators tutorial
• Image Stitching
• Edge Detection tutorial
• Tracking
• 2D Line fitting
• Image denoising with Total Variation
• Extracting and Combining Tensor Patches
• ZCA whitening
• Geometry image transforms in batch
• Line segment matching
• Image equalization
• Image resize/antialias exploration
• Homography warping
• Morphological operators
• Image registration
• Semantic Segmentation
• Random Mosaic
• Face Detection

Feel free to ask any questions here, or on the Github Discussions.

For further contributions, we also recommend looking at our development notes in CONTRIBUTING.

Happy coding !
The Kornia team

TGM v0.1.1 Roadmap (short-term)

This is the roadmap for short-term torchgeometry v0.1.1 focuses PyTorch 1.0.0 support, bug fixes and code clean up.

Feel free to volunteer yourself if you are interested in trying out some items(they do not have to be on the list).

Features

• Support PyTorch 1.0.0 3ee14c8

• Bug fixes

  • test_homography_i_H_ref crashes 0aba15d

• Test

  • Support Codecov
  • Add type checking to functions
  • Enable batched version to functions

• Installation

  • pip packages

[BUG] Bug in quaternion_to_axis_angle

There is a bug in quaternion_to_axis_angle when the input quaternion is the quaternion corresponding to a 0 rotation ([1,0,0,0]). Specifically the problem is introduced by the division at line 408 in conversions.py

Minimal example to reproduce it

import torch
import torchgeometry as tgm
x = torch.tensor([1,0,0,0]).float()
aa = tgm.quaternion_to_angle_axis(x)

[JIT] Unit test for torch script

If we decouple the pytest functions into two parts, one for functionality to test and the other one for testing as in the following example, we can carry out tests for both pytorch and torch script in a single test function.

If it is OK, I am going to decouple them prior to writing tests for torch script.

Do you have any suggestions?

def rad2deg(x_rad):
    # convert radians/degrees
    x_deg = tgm.rad2deg(x_rad)
    x_deg_to_rad = tgm.deg2rad(x_deg)

    return x_deg, x_deg_to_rad


@pytest.mark.parametrize("device_type", TEST_DEVICES)
@pytest.mark.parametrize("batch_shape", [
    (2, 3), (1, 2, 3), (2, 3, 3), (5, 5, 3), ])
def test_rad2deg(batch_shape, device_type):
    # trace
    dummy_input = torch.rand(batch_shape)
    rad2deg_traced = torch.jit.trace(rad2deg, dummy_input)

    # generate input data
    x_rad = tgm.pi * torch.rand(batch_shape)
    x_rad = x_rad.to(torch.device(device_type))

    # convert radians/degrees
    x_deg, x_deg_to_rad = rad2deg(x_rad)
    x_deg_traced, x_deg_to_rad_traced = rad2deg_traced(x_rad)

    # compute error
    error = utils.compute_mse(x_rad, x_deg_to_rad)
    assert pytest.approx(error.item(), 0.0)

    error_traced = utils.compute_mse(x_rad, x_deg_to_rad_traced)
    assert pytest.approx(error_traced.item(), 0.0)

    # functional
    assert torch.allclose(x_deg, tgm.RadToDeg()(x_rad))
    assert torch.allclose(x_deg_traced, tgm.RadToDeg()(x_rad))

    # evaluate function gradient
    assert gradcheck(tgm.rad2deg, (utils.tensor_to_gradcheck_var(x_rad),),
                     raise_exception=True)

[BUG] Using Multi GPU support

While using tgm, I have come across one or two places in the code where the creation of new tensor has to be done on the device in the context otherwise the multi gpu training gives an error.

Should I give a pull request or perhaps you could think of better ways to solve this.

for example in torchgeometry/core/conversions.py in line 97-100, I changed the code as follows

device_points = points.device
scale: torch.Tensor = torch.tensor(1.) / torch.clamp(z_vec, eps)
scale = scale.to(device_points)

HomographyWarper gradcheck test fails

Apparently we might have an issue when we compute the gradient using the HomographyWarper module since the Pytorch gradcheck routine doesn't succeed as expected.

link to the unittest:

https://github.com/arraiy/torchgeometry/blob/master/test/test_homography_warper.py#L49-L70

I've asked** to the Pytorch slack channel and they pointed my the following link in order to solve the issue:

https://discuss.pytorch.org/t/custom-top-eigenvector-function/16510/6

** slack question

[FEAT] Add tests coverage for warp_affine

the function warp_affine doesn't have a proper unittest.

Coverage report: https://codecov.io/gh/arraiyopensource/torchgeometry/src/25cc06ec56334055dcf3a1a38eee619fa40f644a/torchgeometry/core/imgwarp.py#L105...149

Benchmark

It could be interesting to evaluate benchmark against optimized impl. like https://docs.nvidia.com/cuda/npp/group__nppi.html

[FEATURE] Rendering/rastering stuff

Hi guys! Thank you for great library
Is there any plans to implement 3d model rendering/rastering? Probably like https://github.com/tensorflow/graphics

[BUG] test_image crashes with PyTorch 1.0.1

Noticed that there's either a bug somewhere in test_image.py or in last PyTorch release 1.0.1.
The crash happens when updating to pytorch-cpu==1.0.1 in TravisCI.

UPDATE (11/02/2019):

Confirmed that after installing pytorch-cpu==1.0.1 through the script setup_travis_env.sh there is a crash only in TravisCI. Tried to reproduce in local and seems to work as expected. Poltergeist in new stable release v1.0.1?

Link to the PR (#72) and build:
https://travis-ci.com/arraiyopensource/torchgeometry/jobs/176774886)

How to reproduce the issue:

git clone https://github.com/arraiyopensource/torchgeometry
cd torchgeometry
./setup_travis_env.sh
source path.bash.inc
pytest test

You should get green lights, TravisCI don't.

Screenshot of the log:

possible bug in Homography

@edgarriba To calculate homography matrix you have solved for Ah=x0 where h is [8x1] vector and then you append 1 to this vector.
I would try solving for Ah=0 where h is [9x1] then divide each element of h by the last one. Something similar to opencv

Please let me know if it works.

[FEATURE] Add borderType option to kornia.filters like OpenCV

kornia.filters operations have no option like borderType on OpenCV.
In my experience, borderType is the most critical argument in the arguments that is not in kornia.filters.
To reproduce it, I suggest the changes like below:

class BoxBlur(nn.Module):
    def __init__(self, kernel_size: Tuple[int, int], border_type='zero': str) -> None:
        super(BoxBlur, self).__init__()
        self.kernel: torch.Tensor = _get_box_filter(kernel_size)
        self.padding: Tuple[int, int] = _compute_zero_padding(kernel_size)
        self.border_type = border_type

    def forward(self, input: torch.Tensor):  # type: ignore
        if not torch.is_tensor(input):
            raise TypeError("Input type is not a torch.Tensor. Got {}"
                            .format(type(input)))
        if not len(input.shape) == 4:
            raise ValueError("Invalid input shape, we expect BxCxHxW. Got: {}"
                             .format(input.shape))
        # prepare kernel
        b, c, h, w = input.shape
        tmp_kernel: torch.Tensor = self.kernel.to(input.device).to(input.dtype)
        kernel: torch.Tensor = tmp_kernel.repeat(c, 1, 1, 1)

        return conv2d(input, kernel, padding=self.padding, padding_mode=self.border_type, stride=1, groups=c)

[PIP] Update pytorch required version

We should update setup.py to accept PyTorch >= v1.0.0
https://github.com/arraiyopensource/torchgeometry/blob/master/setup.py#L32

[FEAT] DepthWarper using inverse depth

We started the discussion in #79 whether the DepthWarper should warp by depth or inverse depth.

@ClementPinard @versatran01 Propose to use inverse depth for numerical stability issues.
Screenshot of one the arguments for that:

Currently the warper works as follows:

# module
pinhole_dst = PinholeCamera(...)
pinhole_src = PinholeCamera(...)
warper = DepthWarper([pinhole_dst, ], height, width)
warper.compute_projections(pinhole_src)
img_src = warper(depth_src, img_dst)
# functional
img_src = depth_warp([pinhole_dst, ], pinhole_ref, depth_src, (height, width))

I was testing the API, and might be much simple to receive a single pinhole instance as destination. Notice that the internal members extrinsics and intrinsics are already batched.

Let's continue the discussion here.

[FEAT] Batch support for image_to_tensor/tensor_to_image

Current implemetation only works for (C, H, W) ->(H, W, C), maybe is good to have (B, C, H, W) ->(B, H, W, C).
Also, if we define this function works on image, should we limit the number of channels to only 1 and 3? seem other number of channels do not make sense, at least cannot be plot via matplotlib...

[DOCS] normalize_pixel_coordinates input shape is wrong

docs state that input shape is BxHxWx2, however it handles tensors with shape (*, 2)
link

[DOCS] Doc link is broken.

[FEATURE] Add Yapf as pre-commit hook

We should have a script to pre-format the code before sending a pull request:
https://github.com/google/yapf/tree/master/plugins

[DOCS] Points clipping

When we use use Transform points i.e. from 3d points to the camera plane it is not possible to do visibility clipping after https://github.com/arraiyopensource/kornia/blob/master/kornia/geometry/linalg.py#L212 caused by clipping ambiguities in the return values after the internal embedded normalisation at that line.

Also w could assume 0 values without clipping.

See http://medialab.di.unipi.it/web/IUM/Waterloo/node51.html.

See also https://computergraphics.stackexchange.com/questions/5922/why-does-opengl-perform-clipping-in-clip-coordinates

https://gamedev.stackexchange.com/questions/65789/why-is-clip-space-always-referred-to-as-homogeneous-clip-space

Clipping in
http://www.unchainedgeometry.com/jbloom/pdf/homog-coords.pdf

UPDATE

add to docs a reference note saying that some points might not be project onto the image plane. The user must be responsible for filtering out those points. Verify nans/infs.

[FEATURE] Dstnn implementation

We might want to introduce dstnn under contrib as discussed in #138 (comment)

test_transform

I'm getting error when I run test_transforms. I'm missing any steps?

root@deepdev06:/code/torchgeometry# python -m torchgeometry.test.test_transforms
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/opt/conda/lib/python3.7/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/code/torchgeometry/torchgeometry/__init__.py", line 3, in <module>
    from .depth_warper import DepthWarper
  File "/code/torchgeometry/torchgeometry/depth_warper.py", line 29
    point = torch.FloatTensor([[x], [y], [1.0], [invd]])).to(x.device)
                                                        ^
SyntaxError: invalid syntax

[TEST] Speed up CI builds and GPU support

Each time a pull request is sent, the current Travis script builds the environment for each of the stages.

We should be able to cache the conda packages and reuse it for all the stages.

[PACKAGE] move version number to a file

Isolate the package version a single file. Similar to: https://github.com/allenai/allennlp/blob/master/allennlp/version.py

Contributor covenant

Can we adopt contributor covenant?

[examples] MyHomography simplification

suggestion of @szagoruyko

code from: https://github.com/arraiy/torchgeometry/blob/master/examples/homography_regression/main.py#L35-L46

class MyHomography(nn.Module):
    def __init__(self):
        super(MyHomography, self).__init__()
        self.homo = nn.Parameter(torch.Tensor(3, 3))

        self.reset_parameters()

    def reset_parameters(self):
        torch.nn.init.eye_(self.homo)

    def forward(self):
        return torch.unsqueeze(self.homo, dim=0)  # 1x3x3

the whole class is equivalent to

homo = torch.eye(3).requires_grad_(True)

[FEAT] Improve kornia.color

I open this since we might want to implement the following operators in kornia.color module. The motivation is that torchvision only supports this for PIL images, we want to give support for torch.Tensor:

Assigned -> @priba

• kornia.color.adjust_brightness #195
• kornia.color.adjust_contrast torchvision ref
• kornia.color.adjust_hue torchvision ref
• kornia.color.adjust_saturation torchvision ref
• kornia.color.adjust_gamma torchvision ref

Extra:

• kornia.color.zca_whitening numpy ref

[DOCS] Fix homography warper docs

As discussed and proved in #157 we need to:

1. Replace in documentation dst_homo_src -> src_homo_dst.
   1.1 Add integration test for 1) - probably the code I shared in #157
2. Call kornia.normalize_pixel_coordinates in here when normalized_coordinates is set to true.
   2.1 Add a unit test show normalized vs pix coords.

[FEAT]Quaternion to Rotation matrix

quaternion_to_rotation_matrix

https://github.com/KieranWynn/pyquaternion/blob/master/pyquaternion/quaternion.py#L965

[FEAT] Add more conversions functions

As discussed, we should add the following conversion operators:

• denormalize_coord #153
• quaternion_to_rotation_matrix #155
• quaternion_log_to_exp #155
• quaternion_exp_to_log #155
• quaternion_relative_angle
• composeRT

perspective transform not giving same result

Hi,

I checked perspective transform using same source and destination points and the perspective transform should be giving the same array back but for some reason, it does not. I wanted to calculate a metric using perspective transform and the metric gave different results when compared to opencv's perspective transform.

Here is my code to reproduce.
`
import torch
import torchgeometry as tgm
import numpy as np
import cv2

read the image with OpenCV

image = cv2.imread('./data/bruce.png')[..., (2,1,0)]
print(image.shape)
image = np.array(image, dtype=np.uint8)
img = torch.from_numpy(image.transpose(2, 0, 1))
img = torch.unsqueeze(img.float(), dim=0) # BxCxHxW
print('img size ', img.size())

the source points are the region to crop corners

points_src = torch.FloatTensor([[
[125, 150], [562, 40], [562, 282], [54, 328],
]])

the destination points are the image vertexes

h, w = 372, 600 # destination size
points_dst = torch.FloatTensor([[
[125, 150], [562, 40], [562, 282], [54, 328],
]])

compute perspective transform

M = tgm.get_perspective_transform(points_src, points_dst)

warp the original image by the found transform

img_warp = tgm.warp_perspective(img, M, dsize=(h, w))
##################THIS SHOULD GIVE EQUAL AS TRUE #############
print('images are equal or not ', torch.equal(img, img_warp))

convert back to numpy

image_warp = img_warp.byte().cpu().numpy()[0].transpose(1, 2, 0)
print('image warp size ', image_warp.shape)

draw points into original image

for i in range(4):
center = tuple(points_src[0, i].long().numpy())
image = cv2.circle(image.copy(), center, 5, (0, 255, 0), -1)
`

Meaning of the Homography in HomographyWarper

Hello,
I'm doing some tests with the HomographyWarper function to find out how it uses the homography to warp the tensor.
In the following code I'm trying to track a point in a image. Both the image and the point are warped using the same homography. The warped point must be mapped at the same location of the warped image:

#Load a sample image
img = io.imread("sintel.jpg")
img = img / 255.0

h = img.shape[0]
w = img.shape[1]

H = torch.tensor([[1.0, 0.0, 0.5],
                  [0.0, 1.0, 0.0],
                  [0.0, 0.0, 1.0]])

#Test point chosen in the sample image
point = torch.ones(3, 1)
point[0] = 66
point[1] = 372

#Normalize between [-1, 1] the point coordinates 
point[0] = ((point[0] * 2.0 / (h - 1)) - 1)
point[1] = ((point[1] * 2.0 / (w - 1)) - 1)

#Warp the point
point_ = torch.matmul(H, point)
point_ = point_ / point_[2]

#Normalize back the point coordinates
point_[0] = ((point_[0] + 1) * (h - 1)) / 2
point_[1] = ((point_[1] + 1) * (w - 1)) / 2

H = H.unsqueeze(0)
H = H.type(torch.float32)

img = img_np2tensor(img).type(torch.float32)

#Warp the image
warper = tgm.HomographyWarper(260, 615, normalized_coordinates=True)
warped = warper(img, H)

warped = img_tensor2np(warped)

In this case the image is translated to the left while the point is moved down, so the point moved in a different way with respect the image.
If I use this homography to move the point I match the image movement:

H_point = torch.tensor([[1.0, 0.0, 0.0],
                  [0.0, 1.0, -0.5],
                  [0.0, 0.0, 1.0]])

Why does this happen?
The normalized coordinates means that the top left corner of the image has indices [-1,-1], and the bottom right corner has indices [1,1], right?
It seems like in a point [v1, v2, 1] the first value (v1) describes the column index and the second (v2) describes the row index.
There's also something wrong with the sign of the homography values...

[Feature request] Depth Smooth Loss

Hello, I happened to have worked on depth smoothness,
and even if when they are used they are called depth smooth, the value that is smoothed is actually the inverse depth. So maybe we should add it to the right terminology, otherwise it might get confusing and people will try to lower spatial gradient of depth instead of inverse depth.

The loss is taken from https://github.com/tensorflow/models/blob/master/research/struct2depth/model.py#L625-L641

But you can see here, that they apply it on "disparity" (calling it inverse depth is better since disparity is only valid for stereo vision)
https://github.com/tensorflow/models/blob/master/research/struct2depth/model.py#L464-L465

A quick example on why it is important not to confuse the two is the scene with inifinite horizontal plane, where depth goes to infinity (along with its gradient).

http://perso.ensta-paristech.fr/~pinard/images/plane.png
http://perso.ensta-paristech.fr/~pinard/images/depth_vs_disp.png

Another point to discuss is the fact that depth smoothness can also be applied in a 2nd order fashion, eg as here : https://github.com/tinghuiz/SfMLearner/blob/master/SfMLearner.py#L156-L167

These loss are very close the Total Variation loss, so maybe we can call them TVloss and TVVloss.

More generally, I'm currently writing a report on depth smoothness constraint for autosupervised depth training, I'll keep you updated on it, but the general idea is that it's probably better to try to minimize the square of gradients to have a something similar to diffusion operation, the same L2 weight decay is better than L1 weight decay.