diffeqml / torchdyn Goto Github PK

If i follow the tutorial, using torchdyn 0.2.2.1:

from torchdyn.models import *

I get this error:

ModuleNotFoundError: No module named 'torchdyn.models'

I am sure I must be overlooking something obvious, but I am trying to work through the torchdyn tutorials and I am not able to find plot_2D_depth_trajectory( ) .

The model that I am training is very similar to ANODE as defined in torchdyn/tutorials/04_augmentation_strategies.ipynb

Given a time step in [0,1], I would like to get the "flow" of the model at that time step - more specifically the flow for a batch of data (I hope my language is correct!). Chen (2018) writes that "Most ODE solvers have the option to output the state z(t) at multiple times." Is there a way to do this torchdyn when using neuralDE and Augmenter? I was hoping to pass these flows into another function -- thanks

Describe the solution you'd like

If my vector field function is in the form of f(z(t), x(t)), where z the solution of ODE where x(t) is a known control input, can we have a model in TorchDyn for this type ODE. Or this has been considered? Any hints or suggestions?

J.

Tutorial for this paper. odeint_hybrid is already in.

@frankschae On our end, it would be good to extend the NHA tutorial with adjoint sensitivity through the events. ping since you're working on sensitivity + NHDE and you might be able to provide guidance on implementation tricks later on.

This page doesn't show the plots themselves (see bottom of the page):
https://torchdyn.readthedocs.io/en/latest/tutorials/05_generalized_adjoint.html
Would be nice if they were included like in other tutorials.

Some NeuralSDE tests break with the new torchsde version (test_strato_sde)

I am working on a project where I sample a set of n-dimensional points from a Gaussian distribution (of learnt parameters) as follows and then evaluate those points based on a loss function to update model parameters with gradient descent.

mu, std = self.lin_1(z), self.lin_2(z)
eps = torch.Tensor(*img_shape).normal_()
return self.act((eps.cuda() * std) + mu)

I would like to transform the Gaussian distribution for being able to sample those points from a more complex learnt distribution. In other words, the model needs to learn how to best transform points obtained from the Gaussian distribution.

I would be glad if you can suggest the best normalizing flows method (transform) to employ considering the following scalability requirements (whether or not it is available in this repo). Thank you very much in advance for your suggestion.

• I am sampling 100K-dimensional points with a batch-size of 5K; hence, the scalability is crucial.
• The method should be memory efficient and fast to train on a RTX series desktop Nvidia GPU.
• There should not ideally be an additional regularization parameter to my current loss function.
• Expressiveness of the method is not as important as scalability and robustness in the training.

Not sure if this is a bug or misuse on my side. If I run the quick start tutorial adding gpus to the trainer, the model is switched back to cpu after training:

# quickstart tutorial setup

learn = Learner(model)
trainer = pl.Trainer(max_epochs=3, gpus=1,
                     progress_bar_refresh_rate=1
                    )
print("before train:", model.device)
trainer.fit(learn)
print("after train:", model.device)

model

GPU available: True, used: True
TPU available: False, using: 0 TPU cores
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

  | Name  | Type     | Params
-----------------------------------
0 | model | NeuralDE | 82    

before train: cuda:0

HBox(children=(HTML(value='Training'), FloatProgress(value=1.0, bar_style='info', layout=Layout(flex='2'), max…


after train: cpu

Just as I upgrade to pytorch 1.6, original torchdyn code cannot run under multiple GPUs. The error is as following:

Traceback (most recent call last):
  File "manipulation/ffjord.py", line 116, in <module>
    trainer.fit(learn, train_dl)
  File "/home/b146466/anaconda3/envs/xjj/lib/python3.7/site-packages/pytorch_lightning-0.9.1rc1-py3.7.egg/pytorch_lightning/trainer/states.py", line 48, in wrapped_fn
    result = fn(self, *args, **kwargs)
  File "/home/b146466/anaconda3/envs/xjj/lib/python3.7/site-packages/pytorch_lightning-0.9.1rc1-py3.7.egg/pytorch_lightning/trainer/trainer.py", line 1016, in fit
    results = self.accelerator_backend.train()
  File "/home/b146466/anaconda3/envs/xjj/lib/python3.7/site-packages/pytorch_lightning-0.9.1rc1-py3.7.egg/pytorch_lightning/accelerators/dp_backend.py", line 99, in train
    results = self.trainer.run_pretrain_routine(model)
  File "/home/b146466/anaconda3/envs/xjj/lib/python3.7/site-packages/pytorch_lightning-0.9.1rc1-py3.7.egg/pytorch_lightning/trainer/trainer.py", line 1237, in run_pretrain_routine
    self.train()
  File "/home/b146466/anaconda3/envs/xjj/lib/python3.7/site-packages/pytorch_lightning-0.9.1rc1-py3.7.egg/pytorch_lightning/trainer/training_loop.py", line 396, in train
    self.run_training_epoch()
  File "/home/b146466/anaconda3/envs/xjj/lib/python3.7/site-packages/pytorch_lightning-0.9.1rc1-py3.7.egg/pytorch_lightning/trainer/training_loop.py", line 484, in run_training_epoch
    batch_output = self.run_training_batch(batch, batch_idx)
  File "/home/b146466/anaconda3/envs/xjj/lib/python3.7/site-packages/pytorch_lightning-0.9.1rc1-py3.7.egg/pytorch_lightning/trainer/training_loop.py", line 838, in run_training_batch
    self.hiddens
  File "/home/b146466/anaconda3/envs/xjj/lib/python3.7/site-packages/pytorch_lightning-0.9.1rc1-py3.7.egg/pytorch_lightning/trainer/training_loop.py", line 1005, in optimizer_closure
    training_step_output = self.accelerator_backend.training_step(args)
  File "/home/b146466/anaconda3/envs/xjj/lib/python3.7/site-packages/pytorch_lightning-0.9.1rc1-py3.7.egg/pytorch_lightning/accelerators/dp_backend.py", line 111, in training_step
    output = self.trainer.model(*args)
  File "/home/b146466/anaconda3/envs/xjj/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/b146466/anaconda3/envs/xjj/lib/python3.7/site-packages/pytorch_lightning-0.9.1rc1-py3.7.egg/pytorch_lightning/overrides/data_parallel.py", line 83, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/b146466/anaconda3/envs/xjj/lib/python3.7/site-packages/pytorch_lightning-0.9.1rc1-py3.7.egg/pytorch_lightning/overrides/data_parallel.py", line 147, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/b146466/anaconda3/envs/xjj/lib/python3.7/site-packages/pytorch_lightning-0.9.1rc1-py3.7.egg/pytorch_lightning/overrides/data_parallel.py", line 288, in parallel_apply
    raise output
  File "/home/b146466/anaconda3/envs/xjj/lib/python3.7/site-packages/pytorch_lightning-0.9.1rc1-py3.7.egg/pytorch_lightning/overrides/data_parallel.py", line 245, in _worker
    output = module.training_step(*input, **kwargs)
  File "manipulation/ffjord.py", line 74, in training_step
    trJ_z = self.forward(x)
  File "manipulation/ffjord.py", line 61, in forward
    return self.model(self.augmenter(x.view(x.shape[0], -1)))
  File "/home/b146466/anaconda3/envs/xjj/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "./torchdyn/models/neuralde.py", line 81, in forward
    out = odeint(x)
  File "./torchdyn/models/neuralde.py", line 115, in _adjoint
    return self.adjoint(self.defunc, x, self.s_span, rtol=self.rtol, atol=self.atol, method=self.solver)
  File "/home/b146466/anaconda3/envs/xjj/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "./torchdyn/sensitivity/adjoint.py", line 93, in forward
    self.flat_params = flatten(func.parameters()) ; self.s_span = s_span
  File "./torchdyn/sensitivity/adjoint.py", line 23, in flatten
    return torch.cat([el.contiguous().flatten() for el in iterable])
RuntimeError: There were no tensor arguments to this function (e.g., you passed an empty list of Tensors), but no fallback function is registered for schema aten::_cat.  This usually means that this function requires a non-empty list of Tensors.  Available functions are [CPU, CUDA, QuantizedCPU, Autograd, Profiler, Tracer, Autocast]

The compatibility problem is also present in torchdiffeq and just got fixed (rtqichen/torchdiffeq#119). I think you probably have the same issue as torchdiffeq. Hope it helps.

Describe the bug
It's not really a bug but it can kind of easily cause a bug down the line. ;)

In cell 9 and 10 of your CNF tutorials (07a and 07b) you reverse the flow of the CNF so that you can sample from it and plot it by changing s_span of the model:

model[1].s_span = torch.linspace(1, 0, 2)

After that you never revert the flow back to its original state, which could be achieved by running:

model[1].s_span = torch.linspace(0, 1, 2)

I think it would be great to have some additional clarification about that step.

I based some on my own code on these tutorials, where after plotting the network I continued training. Obviously, the resulting network was garbage. :(

Step to Reproduce

Steps to reproduce the behavior:

1. Run either of the CNF tutorials.
2. After visualizing the result, repeat the training procedure
3. Visualize it again and you will see odd results.

Expected behavior

Have some kind of clarification about ensuring that the flow points in the right direction. Maybe there is even a way to abstract away some of the specifics from the user by providing an API to change between both "modes".

Planned features:

• Redesign of the plotting API. More modular and less convoluted.
• Add more basis functions for Galerkin Neural ODEs (e.g. RBFs)
• Base plug-and-play regularization nn.Module (RNODE and beyond)
• .backward_trajectory method for NeuralDE

Minor changes:

• Set up benchmark folder for more involved applications of torchdyn

Miscellaneous:

• Docs: improve examples
• More extensive test coverage (and coverage badges) ~80%
• More extensive CI testing (Linux python 3.7/3.8/3.9, Windows python 3.8)

Hi, I am relatively new to using machine learning to solve ODE.

Can you kindly help me understand how to implement second-order ODE with torchdyn?

This issue it to keep a track of the conversation that started on the slack channel

Describe the bug

I attempted to deepcopy a DEFunc but deepcopy failed because only leaf tensors can be deepcopied (see screenshot below).

I tracked it back to the DEFunc.dxds attribute which seems to be just a slice of the input x (or the entire input) to DEFunc.forward, see this line.

Step to Reproduce

Steps to reproduce the behavior:

import torchdyn
import torchdyn.models
import copy
import torch
import torch.nn as nn

# create Vanilla NeuralDE
f_forward = nn.Sequential(nn.Linear(2, 16), nn.Tanh(), nn.Linear(16, 2))
model = torchdyn.models.NeuralDE(
    f_forward,
    sensitivity="adjoint",
    solver="dopri5",
)

# create some fake data of batch size 1
x = torch.rand(1, 2)

# do a deepcopy, nothing fails because we never tracked dxds (no forward pass)
model_copy1 = copy.deepcopy(model)

# do a forward+backward pass
y = model(x)
loss = y.sum()
loss.backward()

# do another deepcopy, fails because model.defunc tracks dxds
model_copy2 = copy.deepcopy(model)

Expected behavior

Keeping track of pytorch tensors as plain attributes of classes can induce all kinds of unexpected behaviors. copy.deepcopy is just one example of that but another example would for example be if we moved the model from one device to another. While all properly registered tensors would also be moved to a different device model.defunc.dxds would be left at the original device.

I would suggest one of the following:

• I would properly register dxds as buffer using torch.nn.Module.register_buffer and then use a detach().clone() operation to track dxds during the forward pass without creating any memory leaks or potentially breaking any kind of upstream tasks like copy.deepcopy.

• I could not find any place in the codebase where dxds was used. So maybe just removing it all together might be easier?

Screenshots

We run an automated script in test/validate_tutorials.py to ensure all notebooks are always runnable. Tutorials in modules not included in the script might not run due to API changes. Older tutorials have not all been updated yet.

Good first issue to get familiar with the codebase or a specific concept / model class of interest.

In the example notebooks (at least, the ones I've been through; I haven't looked at all of them fully) it seems like the loss during training only uses the final point of each trajectory. For things like classification problems, this makes sense. However, if I'm training a neural ODE to approximate some other ODE that I have sampled trajectories from, using only the final point of the trajectory doesn't make sense. In particular, suppose my data is like x_{t0}, x_{t1}, ..., x_{tn} for a single trajectory from time t0 to tn. I see a couple of options here:

1. use each point in the sampled trajectory as an input. The labels are the next points. I.e. my input points are x_{t0}, x_{t1}, ..., x_{t(n-1)} and my labels are x_{t1}, x_{t2}, ..., x_{tn}. If the ODE is time-invariant and the time difference between each point is the same, this works just using y_hat = self.model(x) (where self is the Learner) like in the examples. If that's not the case, it seems like self.model.trajectory is required; more on this in 2.
2. To predict a whole trajectory, we can use self.model.trajectory and pass in a tensor of all time points that we need. This seems to work for me with a basic model, but fails when I try, e.g., to do input-layer augmentation like in the demo notebook (because then self.model is an nn.Sequential which doesn't have .trajectory). We could work around this by having Learner.forward call the augmentation layer first, then get a trajectory with the NeuralDE, then run the layer that converts each point back into the space of the original variables.

There isn't anything that I can't get to work here, but since I didn't see any examples using model.trajectory for training, I just wanted to check if this seems right or if there's a better way to do this.

Hi, nice work. I wonder if it's possible to add adaptive checkpoint adjoint (ACA) solver as an option in this repo?
We recently published a paper in ICML2020 and enable NODE to achieve (even outperform) ResNet on Cifar10, the code is published at https://github.com/juntang-zhuang/torch_ACA, and with paper https://arxiv.org/abs/2006.02493. I'm very glad if we can contribute to this nice project together and boost DiffEq in machine learning. Please contact me if you are interested, please also tell me if you have any ideas such as requirements on API.

I am going over 2nd Torchdyn Tutorial and I encountered this error:

This error happens when I run this command:

I think that this error is caused by PytorchLighting. However, I am a newbie to PytorchLighting. Thus, I don't know how to debug it

Extended tutorial on using MSLs to speed up Neural CDEs via root tracking.

@Juju-botu @shuheng-liu @cbhua

Issue to track progress on the planned addition to neurodiffeq.

Introduce torchdyn as a backend for torchcde. This would make it easier to further experiment with Neural CDEs + multiple shooting methods, and would streamline our prototypes so that the tutorials can be easier. Pretrained Neural CDEs will then be added to our model zoo.

@patrick-kidger pinging so that you can keep track of progress.

Standardize the two classes. NeuralSDEs should have an option to use our internals or torchsde as a backend. Related to this for implementation of our internal solver SDE suite.

Major:

• HybridNeuralDE API: modular ODE-RNN and friends
• VariationalNeuralDE API
• NeuralSDE compatibility (torchsde)
• Cheap trace eval.: https://papers.nips.cc/paper/9187-neural-networks-with-cheap-differential-operators.pdf
• Adaptive checkpointing: https://arxiv.org/abs/2006.02493

Minor:

• A variety of TimeSeriesDataset
• Optimal control example
• Improved image classification architecture and tutorial
• DC and Stable normalizing flow tutorials

Miscellaneous:

• Performance tests

Describe the bug

Y = torch.zeros(x.shape[0], *h.shape)
if self.reverse: x_t = x_t.flip(0)
for t, x_t in enumerate(x):
    h, c = self.jump_func(x_t, h, c)
    h = self.flow(h)
    Y[t] = h
Y = self.out(Y)

In the line 23~29 of hybrid.py, the Y remains as CPU tensor after it is initialized even if HybridNeuralDE parameters and input tensor are placed on GPU.
This causes an error in the line 29 when a model is placed on GPU.
I am currently using torch==1.7.1 and torchdyn==0.2.2.1

I tend to work with relatively small datasets, so I have found one nice advantage of working with Jax is that I can train an ensemble of networks on a single GPU via Will Whitney's approach.

I see that functorch has made it possible to now do this in pytorch.

Question: Would it be feasible to make use of functorch's implementation to train an ensemble of neuralodes on a single GPU?

Clone parts of the odeint and ODE solver implementations to get an internal, simplified sdeint API. Complete the EulerMaruyama placeholder in solvers.py.

Is there a way to define a time dependent function as a NeuralDE? The s_span would be used as time variable in the function evaluation. This is supported in torchdiffeq but I don't see how to do it using your library.

I'm looking to taking in a sequence of values and output a sequence as well.
So instead of doing flatten and computing the final output from neural ode, I would like to make use of each intermediate value that neural ode outputs and produce a loss with a target label.

Is there an example of this somewhere using TorchDyn?

Thanks

@Zymrael I'm finally done with the training infrastructure I told you about. I see you're making progress on this stuff. I sent an email to kaist. I sent some details of the architecture inside of that email.

I need your help with the various networks now so I can do proper trail runs.

Let me know your thoughts.

Good first issue for someone interested in getting familiar with implicit models and DEQs. Those interested are encouraged to get in touch directly as we have internal private prototypes that can be used as a starting point.

This might be a bad question --

Is there a way to initialize the model/vector field so that if no training was performed, you would simply get the input as the output?

Good first issue for someone interested in getting familiar with diffusion models and score-matching SDEs. Those interested are encouraged to get in touch directly as we have internal private prototypes that can be used as a starting point.

I'm having trouble changing the number of Gaussians above 2 in code I'm slightly modifying from the tutorials.

d = ToyDataset()
n_samples = 1 << 14
n_gaussians = 4
X, yn = d.generate(n_samples // n_gaussians, 'gaussians', n_gaussians=n_gaussians, std_gaussians=0.5, radius=4, dim=2)
X = (X - X.mean())/X.std()

When I try to compute loss loss = nn.CrossEntropyLoss()(y_hat, y), for any value of n_samples larger than 2, I get a error saying IndexError: Target 3 is out of bounds.

Any suggestions as to what I am missing?

Thank you so much!

• DataControl with CNF requires testing. Compatible but unreliable

• Galerkin layers with CNF requires testing. Compatible but unreliable

• Loading NeuralDE via torch.load or .load_state_dict sometimes causes issue due to Adjoint class. Can be fixed by dropping excess state_dict keys.

• Training issue in notebook 05

• Issue with 02_classification on Windows due to pytorch-lightning issues. See #2358. EDIT: Should be fixed.

We plan to kickstart our pretrained model zoo with weights of a SOTA CNF trained on CIFAR10 and ImageNet.

Hi guys!
I want to contribute to this project, I tried joining the slack channel, but was not able to.
Could you please have a look at it? It says the invite has expired.

I'm working on sequential MNIST with an RNN and using DepthCat to have access to time of the ODE. I set s_span = torch.linspace(0, 27, 28) for [0, 27] time interval. However, when extracting t from the state in forward(), I get t values greater than 27. This does not happen with the rk4 solver so is the time interval enforced properly? Here's logs:

Printing modules
[Neural DE:
	- order: 1        
	- solver: dopri5
	- integration interval: 0.0 to 27.0        
	- num_checkpoints: 28        
	- tolerances: relative 0.0001 absolute 0.0001        
	- num_parameters: 128        
	- NFE: 0.0
        
Traceback (most recent call last):
  File "pixel_mnist_rnn.py", line 181, in main
    rnn, lossfun, test_loader, device=DEVICE, repeats=num_repeats)
  File "/mypath/data/workdir/data_utils.py", line 137, in test
    out = model(x)
  File "/mypath/envs/neuralodes/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _cal
l_impl
    result = self.forward(*input, **kwargs)
  File "pixel_mnist_rnn.py", line 116, in forward
    x = self.ode(init)
  File "/mypath/envs/neuralodes/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _cal
l_impl
    result = self.forward(*input, **kwargs)
  File "/mypath/envs/neuralodes/lib/python3.6/site-packages/torchdyn/models/neuralde.py", line 81, in forwar
d
    out = odeint(x)
  File "/mypath/envs/neuralodes/lib/python3.6/site-packages/torchdyn/models/neuralde.py", line 109, in _auto
grad
    atol=self.atol, method=self.solver)[-1]
  File "/mypath/envs/neuralodes/lib/python3.6/site-packages/torchdiffeq/_impl/odeint.py", line 75, in odeint
    solution = solver.integrate(t)
  File "/mypath/envs/neuralodes/lib/python3.6/site-packages/torchdiffeq/_impl/solvers.py", line 30, in integ
rate
    solution[i] = self._advance(t[i])
  File "/mypath/envs/neuralodes/lib/python3.6/site-packages/torchdiffeq/_impl/rk_common.py", line 194, in _a
dvance
    self.rk_state = self._adaptive_step(self.rk_state)
  File "/mypath/envs/neuralodes/lib/python3.6/site-packages/torchdiffeq/_impl/rk_common.py", line 255, in _a
daptive_step
    y1, f1, y1_error, k = _runge_kutta_step(self.func, y0, f0, t0, dt, t1, tableau=self.tableau)
  File "/mypath/envs/neuralodes/lib/python3.6/site-packages/torchdiffeq/_impl/rk_common.py", line 76, in _ru
nge_kutta_step
    f = func(ti, yi, perturb=perturb)
  File "/mypath/envs/neuralodes/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _cal
l_impl
    result = self.forward(*input, **kwargs)
  File "/mypath/envs/neuralodes/lib/python3.6/site-packages/torchdiffeq/_impl/misc.py", line 191, in forward
    return self.base_func(t, y)
  File "/mypath/envs/neuralodes/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _cal
l_impl
    result = self.forward(*input, **kwargs)
  File "/mypath/envs/neuralodes/lib/python3.6/site-packages/torchdyn/models/defunc.py", line 51, in forward
    else: x = self.m(x)
  File "/mypath/envs/neuralodes/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _cal
l_impl
    result = self.forward(*input, **kwargs)
  File "/mypath/envs/neuralodes/lib64/python3.6/site-packages/torch/nn/modules/container.py", line 117, in f
orward
    input = module(input)
  File "/mypath/envs/neuralodes/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _cal
l_impl
    result = self.forward(*input, **kwargs)
  File "pixel_mnist_rnn.py", line 71, in forward
    pixels = self.data[torch.arange(x.shape[0]), t.long().squeeze(), :]
IndexError: index 28 is out of bounds for dimension 1 with size 28

In torchdiffeq, the network forward falls into forward(self, t, x) form. However in this software, network is only allowed to take in a single input and the t variable seems to be only available for the network with DepthCat module, as I understand.

However, DepthCat is only one way of using t, others like ConcatSquash in FFJORD paper need the t parameter as free input rather than concatenated to original input.

I wonder how I should implement ConcatSquash with your software?

Hi, I was playing with your FFJORD tutorial (really nice tutorial btw). And I'm curious about if there is a way to print the number of integration steps taken during the training or sampling process.

The number of function evaluations (nfe) can not be of the float type. Thus, could you change the initialization of the nfe from <0.> to <0>?

In the code snippet below you can see that self.nfe is initialized to <0.>

class DEFunc(nn.Module):
    """Differential Equation Function Wrapper. Handles auxiliary tasks for NeuralDEs: depth concatenation,
    higher order dynamics and forward propagated integral losses.

    :param model: neural network parametrizing the vector field
    :type model: nn.Module
    :param order: order of the differential equation
    :type order: int
   """
    def __init__(self, model, order=1):
        super().__init__()
        self.m, self.nfe,  = model, 0.
        self.order, self.intloss, self.sensitivity = order, None, None

Just curious if this works with padded sequences?
https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.pad_sequence.html

I have sequences of variable length and I would like to use torchdyn for classification.
Currently I am doing windowing, but I would rather use padding.

Additional Description

I have been using Neural ODEs recently for one particular deep learning task. In this task, I have to do 17 function evaluations, i.e. s_span = torch.linspace(0, x, 17). However, I found that setting x in s_span affects the training. When x is large it learns faster. Thus, could you explain the intuition behind setting x in the s_span.

Example:

In the plot below blue is when x = 1 and brown is when x = 5

Thanks!

I don't understand the theoretical part of the paper stable neural flows and I think there might be something wrong with it.

Can someone explain the proof of Proposition 1 in detail?

If there won't be any answers, I suggest to precautionary remove the stable model.

I am seeing in the docs (classification or Lagrangian neural nets tutorials) that the ODE is instantiated as NeuralDE. When I try it in my environment, with torchdyn 0.2.0, I get NameError: name 'NeuralDE' is not defined. However, the tutorials in github say it's NeuralODE, and that works for me.

Are the docs wrong or NeuralDE and NeuralODE are two different things and I'm doing something wrong?

Thanks!

Seems new requirements make the install break.... not sure what's going on but this is what I get:

Installing pip dependencies: | Ran pip subprocess with arguments:
['/home/andriy/miniconda3/envs/myenv/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/andriy/Projects/TunNickel/condaenv.yuq4gswn.requirements.txt']
Pip subprocess output:
Obtaining torchdyn from git+https://github.com/DiffEqML/torchdyn.git@43d22ce5e12dc1a298ef8c11e64f015b05a60765#egg=torchdyn (from -r /home/andriy/Projects/TunNickel/condaenv.yuq4gswn.requirements.txt (line 4))
  Skipping because already up-to-date.
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Requirement already satisfied: torch>=1.6.0 in /home/andriy/miniconda3/envs/myenv/lib/python3.9/site-packages (from torchdyn->-r /home/andriy/Projects/TunNickel/condaenv.yuq4gswn.requirements.txt (line 4)) (1.7.1)
Requirement already satisfied: scikit-learn in /home/andriy/miniconda3/envs/myenv/lib/python3.9/site-packages (from torchdyn->-r /home/andriy/Projects/TunNickel/condaenv.yuq4gswn.requirements.txt (line 4)) (0.24.0)
Requirement already satisfied: matplotlib in /home/andriy/miniconda3/envs/myenv/lib/python3.9/site-packages (from torchdyn->-r /home/andriy/Projects/TunNickel/condaenv.yuq4gswn.requirements.txt (line 4)) (3.3.3)

Pip subprocess error:
ERROR: Could not find a version that satisfies the requirement dgl>=0.4.1 (from torchdyn)
ERROR: No matching distribution found for dgl>=0.4.1

failed

CondaEnvException: Pip failed

Here ie my environment.yml file. To reproduce the error simply do conda env create -f environment.yml:

name: myenv
channels:
  - conda-forge
  - pytorch
  - defaults
dependencies:
  - ipywidgets
  - altair 
  - vega_datasets
  - python=3.8
  - numpy
  - jupyter
  - jupyterlab
  - matplotlib
  - pydantic
  - pip
  - pytest
  - scikit-learn
  - pytorch
  - torchvision 
  - torchaudio
  - cudatoolkit=11.1
  - pip:    
    - wandb
    - pytorch-lightning
    - tqdm
    - "--editable=git+https://github.com/DiffEqML/torchdyn.git@43d22ce5e12dc1a298ef8c11e64f015b05a60765#egg=torchdyn"

    

Really enjoying the library so far --

I was wondering if there was any thoughts about implementing the regularization method introduced in Kelly 2020. I would be more than happy to assist in the implementation.

Thanks

A planned feature for the next version of torchdyn is a model zoo containing models such as Stable Neural ODEs, Hamiltonian Neural ODEs, Lagrangian Neural ODEs, as well as CNFs and their many variants.

These will be wrappers to nn.Module interfacing the generic neural network defining the dynamics with the DEFunc class, which handles augmentation, depth-variance, data-control and other auxiliary features that allow for mixing and matching of Neural ODE variants.

The design choice follows from the observation that the models above all require additional steps at each function evaluation of the solver e.g gradient computations for Hamiltonian Neural ODEs, Jacobian / hessian calculations for Lagrangian Neural ODEs, handling log-probability propagation for CNFs. It seems thus natural to include these as wrappers, which can then even be stacked and passed to a DEFunc and then the NeuralDE itself.

We are looking for feedback on the above, as we expect CNFs and the other models to be a popular feature in torchdyn. Check new_release_dev for more information on the WIP version.

Also accepting PRs with proof of concept implementations or tutorials on the topic.

Because pytorch currently gets downgraded to 1.7.1 when we install torchdyn, this creates an issue with torchtext

For those who might face a similar problem, the solution is to add the following line

!pip install torchtext==0.8.1

It seems that there are package dependencies which are not installed using setup.py. In particular it looks like matplotlib and scikit-learn are required.

Create a fresh environment and then, following the instructions in the README, install torchdyn with:

git clone https://github.com/DiffEqML/torchdyn.git

cd torchdyn

python setup.py install

However if you then try: from torchdyn.models import * it raises an error as matplotlib is not installed.

Similarly if you install matplotlib into the environment and then run from torchdyn.datasets import * you get an error as sklearn is not installed.

After installing both you these you can from torchdyn import * without error, so perhaps they are the only missing dependencies.

It would be great if these could be installed with setup.py install.