ziatdinovmax / pyroved

Invariant representation learning from imaging and spectral data

pyroVED

---

pyroVED is an open-source package built on top of the Pyro probabilistic programming framework for applications of variational encoder-decoder models in spectral and image analyses. The currently available models include variational autoencoders with translational, rotational, and scale invariances for unsupervised, class-conditioned, and semi-supervised learning, as well as im2spec-type models for predicting spectra from images and vice versa. More models to come!

Documentation and Examples

The documentation of the package content can be found here.

The easiest way to start using pyroVED is via Google Colab, which is a free research tool from Google for machine learning education and research built on top of Jupyter Notebook. The following notebooks can be executed in Google Colab by simply clicking on the "Open in Colab" icon:

• Mastering the 1D shifts in spectral data

• Disentangling image content from rotations

• Learning (jointly) discrete and continuous representations of data

• Semi-supervised learning from data with orientational disorder

• im2spec: Predicting 1D spectra from 2D images

Installation

Requirements

• python >= 3.6
• pytorch >= 1.8
• pyro-ppl >= 1.6

Install pyroVED using pip:

pip install pyroved

Latest (unstable) version

To upgrade to the latest (unstable) version, run

pip install --upgrade git+https://github.com/ziatdinovmax/pyroved.git

Reporting bugs

If you found a bug in the code or would like a specific feature to be added, please create a report/request here.

Development

To run the unit tests, you'll need to have a pytest framework installed:

python3 -m pip install pytest

Then run tests as:

pytest tests

If this is your first time contributing to an open-source project, we highly recommend starting by familiarizing yourself with these very nice and detailed contribution guidelines.

pyroved's Issues

Refactor the models classes due to repetition

There appears to be quite a bit of repetition in the classes in models.py, especially in e.g. the __init__. I'd suggest refactoring, and moving a lot of the shared code into the base class (and then adding documentation to the base class, of course).

Add self.encode_and_predict method to iVAE

It will run Gaussian Process regression to interpolate between existing encodings in order to predict 'labels' on new inputs X.

@utkarshp1161

Originally posted by @ziatdinovmax in #53 (reply in thread)

Suggestion: implement mypy type checking

While it is a huge pain to implement when using custom libraries, e.g. torch, since everything is typed in the code anyway, it might be worthwhile to add a proper mypy type check into the CI pipeline. That said, I really am no expert on using stubs so it could be a challenge/time consuming.

Edit: actually it doesn't look too bad: https://realpython.com/python-type-checking/#adding-stubs

Question: import style modification

I am not sure which is best practice, since the way that you're handling imports does make sense. However, I think for readability, it might make more sense to precisely specify imports.

For example, consider the __init__.py in pyroved.utils,

from .coord import (generate_grid, generate_latent_grid,
                    generate_latent_grid_traversal, transform_coordinates)
from .data import init_dataloader, init_ssvae_dataloaders
from .nn import (get_activation, get_bnorm, get_conv, get_maxpool,
                 set_deterministic_mode, to_onehot)
from .prob import get_sampler
from .viz import plot_grid_traversal, plot_img_grid, plot_spect_grid

__all__ = ['generate_grid', 'transform_coordinates', 'generate_latent_grid',
           'get_sampler', 'init_dataloader', 'init_ssvae_dataloaders',
           'get_activation', 'get_bnorm', 'get_conv', 'get_maxpool',
           'to_onehot', 'set_deterministic_mode', 'get_sampler',
           'plot_img_grid', 'plot_spect_grid', 'plot_grid_traversal',
           'generate_latent_grid_traversal']

It too me a minute to figure out where set_deterministic_mode was when reading trvae.py,

from pyroved.utils import (
    generate_grid, generate_latent_grid, get_sampler,
    plot_img_grid, plot_spect_grid, set_deterministic_mode,
    to_onehot, transform_coordinates
)

In other words, I would modify the imports here to say

from pyroved.utils.nn import set_deterministic_mode

and remove the code in the __init__.py, for example. However, I'm happy to be convinced that your current way is better. What do you think?

Extend to multi-channel data

Currently, all the VED/VAE models work only with a single channel data (e.g. grayscale images but not color images). The extension to multiple channels should be pretty straightforward if there is a use case.

VAE with invariance to 3D rotations

This should be easy to implement by using von Mises Fisher distribution (or the qualitatively similar Projected Normal distribution) instead of the Normal distribution for sampling a latent angle. One will also need to modify the outputs of the encoder NN such that it outputs a "concentration" parameter in addition to z_loc and z_scale.

Improve documentation

The following modules may benefit from more extensive documentation: conv.py, fc.py, and pretty much all modules in the utils.

Manifold traversals for 1D data in joint iVAE

Currently, the manifold_traversal() in joint iVAE supports only plotting the 2D images. It needs to be expanded to 1D spectral data since there's an increasing interest in applying joint iVAE to this type of data.

Suggestion: use e.g. `with warnings.catch_warnings()` in favor of module-level warnings suppression

pyroVED/pyroved/nets/conv.py

Line 20 in 1854047

filterwarnings("ignore", module="torch.nn.functional")

I'd recommend against module-level warnings suppression in favor of

with warnings.catch_warnings():
    warnings.simplefilter('ignore')

at the precise spot where the warnings are thrown.

Thoughts?

Suggestion: consistent numpy documentation style

See here. I prefer this style personally and it definitely is just a personal choice, but if there are no objections this is how I'll start documenting!

Regarding post analysis of trained vae.

Hi @ziatdinovmax ,

Say I defined a vae = pv.models.iVAE(in_dim, latent_dim=2, invariances=['r'], seed=0)

1. vae.manifold2d(d=8,cmap="jet") ---> plots the decoded latent space on grid, how to save it to a png file directly on disc. I know
   we can pass savefig = True in atomai but couldnt find here.

2. Is there a function or modular way to create latent images ?

dx_pri and dy_pri unclear documentation

pyroVED/pyroved/models/trvae.py

Line 123 in 1854047

dx_pri = tt(kwargs.get("dx_prior", 0.1))

What are these objects?

User should be able to override default device even if a GPU is accessible

pyroVED/pyroved/models/base.py

Line 26 in 1854047

self.device = 'cuda' if torch.cuda.is_available() else 'cpu'