havakv / torchtuples

Training neural networks in PyTorch

torchtuples

torchtuples is a small python package for training PyTorch models. It works equally well for numpy arrays and torch tensors. One of the main benefits of torchtuples is that it handles data in the form of nested tuples (see example below).

Installation

torchtuples depends on PyTorch which should be installed from HERE.

Next, torchtuples can be installed with pip:

pip install torchtuples

Or, via conda:

conda install -c conda-forge torchtuples

For the bleeding edge version, install directly from github (consider adding --force-reinstall):

pip install git+git://github.com/havakv/torchtuples.git

or by cloning the repo:

git clone https://github.com/havakv/torchtuples.git
cd torchtuples
python setup.py install

Example

import torch
from torch import nn
from torchtuples import Model, optim

Make a data set with three sets of covariates x0, x1 and x2, and a target y. The covariates are structured in a nested tuple x.

n = 500
x0, x1, x2 = [torch.randn(n, 3) for _ in range(3)]
y = torch.randn(n, 1)
x = (x0, (x0, x1, x2))

Create a simple ReLU net that takes as input the tensor x_tensor and the tuple x_tuple. Note that x_tuple can be of arbitrary length. The tensors in x_tuple are passed through a layer lin_tuple, averaged, and concatenated with x_tensor. We then pass our new tensor through the layer lin_cat.

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.lin_tuple = nn.Linear(3, 2)
        self.lin_cat = nn.Linear(5, 1)
        self.relu = nn.ReLU()

    def forward(self, x_tensor, x_tuple):
        x = [self.relu(self.lin_tuple(xi)) for xi in x_tuple]
        x = torch.stack(x).mean(0)
        x = torch.cat([x, x_tensor], dim=1)
        return self.lin_cat(x)

    def predict(self, x_tensor, x_tuple):
        x = self.forward(x_tensor, x_tuple)
        return torch.sigmoid(x)

We can now fit the model with

model = Model(Net(), nn.MSELoss(), optim.SGD(0.01))
log = model.fit(x, y, batch_size=64, epochs=5)

and make predictions with either the Net.predict method

preds = model.predict(x)

or with the Net.forward method

preds = model.predict_net(x)

For more examples, see the examples folder.

torchtuples's Issues

Can I use lr_scheduler of Pytorch in model.fit ?

First of all, thank you for your great works !!

Can I use lr_scheduler of Pytorch in model.fit ?
I've noticed that there is lr_scheduler.py in this repo, however, I couldn't find how to use it in examples.
If it's possible, then how can I modify the following code to use the scheduler for learning rate?

log = model.fit(x_train, y_train, batch_size, epochs, callbacks, val_data=val)

Many thanks

Use AdamW from pytorch

Should change torchtuples.optim.AdamW to instead use the version in pytorch

File path in Windows

The generated file paths include multiple colons ":" which doesn't work on windows.

Remove callbacks that are no longer correct

Remove old callbacks that can no longer be run.

Using LSTM instead of MLPVanilla

In order to use LSTM instead of MLPVanilla with the CoxTime and CoxPH models, I have the following model class. It works mechanically, but I want to make sure that the implementation is theoretically correct. I'm trying to make each patient the input sequence for the LSTM model and the hidden and cell states can be transferred within that sequence, not on the whole batch of patients as a sequence. Would you be able to share some insights?

from torch import nn

class LSTMCox(nn.Module):
    def __init__(self, embedding_dim, hidden_dim, n_layers, output_size):
      super(LSTMCox, self).__init__()
      self.n_layers = n_layers
      self.hidden_dim = hidden_dim
      self.embedding_dim = embedding_dim
      
      self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers)
      self.fc = nn.Linear(hidden_dim, output_size)
      self.activation = nn.ReLU()

    def forward(self, input):
      input = input.view(len(input), 1, self.embedding_dim)

      lstm_out, _ = self.lstm(input)
      lstm_out = lstm_out.contiguous().view(len(input), -1)

      out = self.fc(lstm_out)
      out = self.activation(out)

      return out

net = LSTMCox(in_features, 512, 1, 1)
model = CoxPH(net, tt.optim.Adam)
model.optimizer.set_lr(0.01)
log = model.fit(x_train, y_train, batch_size, epochs, callbacks, val_data=val, val_batch_size=batch_size)

Stop wrapping optimizers to simplify use fo torch optimizers

Right now all optimizers are wrapped, so to access a torch optimizer object we need to call model.optimizer.optimizer. It might makes sense to be able to get the torch optimizer with model.optimizer.

If we continue to wrap torch optimizers, maybe make it a wrapper objects model.optimizer_wrapper.

Add conda recipe

Adding a conda recipe would make distribution simpler, especially in combination with packages that require C/C++ libraries.

Cheers,
Sarthak

Use torch DataLoader if torchtupeles DataLoaderSlice is not supported

For versions of torch that cannot take advantage of torchtuples.data.DataLoaderSlice we should default to torch.utils.data.DataLoader with a warning instead of raising an error.

Make it simpler to access train/val metrics

Now one needs to call model.val_metrics.scores['loss']['score'][-1] and model.train_metrics.scores['loss']['score'][-1] to get the last training/validation scores. This should be easier to access.

Also, should probably make the same change to the MonitorMetrics callback.

Update to pytorch 1.2

Update to use pytorch 1.2 instead of 1.1.