viorik / convlstm Goto Github PK

Hello,

Thanks for sharing your implementation, I have one question. Is it possible to use a single tensor as input (instead of a table of tensors) ? If it is, then SeqLen x BatchSize x Featsize ( Featsize = NumFeatMaps x Width x Height) is the correct dimension order?

Thanks!

@viorik, I am getting the above mentioned error whenever I am running the file model-demo-ConvLSTM.lua

Here is the complete error :

/home/ubuntu/torch/install/bin/luajit: model-demo-ConvLSTM.lua:44: attempt to index global 'opt' (a nil value)
stack traceback:
model-demo-ConvLSTM.lua:44: in main chunk
[C]: in function 'dofile'
...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:131: in main chunk
[C]: at 0x00406670

Actually I am getting this error whenever I am running any lua file of ConvLSTM that uses opt.
The same error was coming for decoder.lua

After some googling, I found similar question here : http://stackoverflow.com/questions/12161266/why-does-my-lua-class-report-an-attempt-to-index-global-object-error

So, after adding "require opts-mnist" in decoder.lua, the error is gone. But new error comes up on doing so in model-demo-ConvLSTM.lua

/home/ubuntu/torch/install/bin/luajit: model-demo-ConvLSTM.lua:48: attempt to call field 'UntiedConvLSTM' (a nil value)
stack traceback:
model-demo-ConvLSTM.lua:48: in main chunk
[C]: in function 'dofile'
...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:131: in main chunk
[C]: at 0x00406670

Kindly look into this!

Hi @viorik I'm trying to apply trained model, and got folowing errors.

user@lin01:~/LSTM/ConvLSTM$ qlua main-demo-ConvLSTM.lua
Loading model from fileoutputs_mnist_line/model_3000.bin
Training done
Start quantitative evaluation
main: Loaded 2000 test sequences
qlua: bad argument #2 to '?' (out of range)
stack traceback:
    [C]: at 0x7f53878e79c0
    [C]: at 0x7f534116a2d0
    [C]: in function '__index'
    ...r/torch/install/share/lua/5.1/rnn/SequencerCriterion.lua:42: in function 'updateOutput'
    main-demo-ConvLSTM.lua:161: in function 'main'
    main-demo-ConvLSTM.lua:177: in main chunk

I narrowed down, that in evaluation phase you are loading 19 images from database to inputTable

for i = 1,data:size(1)-1 do
      table.insert(inputTable, data[i]:cuda())
    end

And one remained image to targetTensor
target:resizeAs(data[1]):copy(data[data:size(1)])

That is different from what is done in learning phase, and thus criterion:update output, that would like to iterate overtables, noticed that second arg is not a vector it report off-by-one error.
I dont have enough expertise currently to suggest a fix, thus i would like to hear from you. I think idea was to get an error score by loading 19 pictures to a model , and then do a prediction, and compare to 20th one, however cannot fix it myself.

I have modified sources, so line numbers are not correct, the problem is here
torch/install/share/lua/5.1/rnn/SequencerCriterion.lua: SequencerCriterion:updateOutput
self.output = self.output + criterion:forward(input, targetTable[i])
Where target table has size=1

Please help.

Hi there

I installed stn(https://github.com/qassemoquab/stnbhwd.git) with replacement BilinearSamplerBHWD.lua with ConvLSTM/BilinearSamplerBHWD.lua
and i also installe extracunn(https://github.com/viorik/extracunn.git)

When i run the main-mnist.lua i got the error like below.
I think nn.BilinearSamplerBHWD is missing or having problem.

Can you tell me how can i solve this problem?

Best Regards,
DH

.../torch/install/share/lua/5.1/stn/BilinearSamplerBHWD.lua:1: stnbhwd package not preloaded
stack traceback:
[C]: in function 'error'
...ricon/Project/torch/install/share/lua/5.1/trepl/init.lua:378: in function 'require'
main-mnist.lua:9: in main chunk
[C]: in function 'dofile'
...ject/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00406670

I have a question about the cell state (or memory activation), i.e. variable c in Equation (1) of the main paper. When t=1, c_1 depends on c_0. I would like to manually assign the value of c_0 during both training and evaluation stage. Is there a way to do this?

i think this code thought is like Generative Adversarial Nets,but using in video. its very good!

function ConvLSTM:__init(inputSize, outputSize, rho, kc, km, stride, batchSize)
Description for above constructor says

inputSize - number of input feature planes
outputSize - number of output feature planes
rho - recurrent sequence length
kc - convolutional filter size to convolve input
km - convolutional filter size to convolve cell; usually km > kc

I am confused between kc and km, How convolve cell filter and convolve input filter is different?
As I can understand from "model-demo-ConvLSTM.lua", Here UntiedConvLSTM is initialized as

if opt.untied then
net:add(nn.UntiedConvLSTM(opt.nFiltersMemory[1],opt.nFiltersMemory[2], opt.nSeq, opt.kernelSize, opt.kernelSizeMemory, opt.stride))

with
opt.kernelSize = 7 opt.kernelSizeMemory = 7

Is it simply referring to 7x7 (readas: kcXkm) convolution kernel? Am I missing something?
section NETWORK PARAMETERS from "patio-temporal video autoencoder with differentiable memory" says

The memory module (LSTM ) has 64 filters, size 7 × 7, and the optical flow
regressor Θ has 2 convolutional layers,

which is also pointing to the kc x km convolution kernel.

My question is, do kc and km serve the same purpose as KW and KH in below code snippet?
module = nn.SpatialConvolution(nInputPlane, nOutputPlane, kW, kH, [dW], [dH], [padW], [padH])

I have another newbe doubt
If I use convLSTM with Sequencer, will 'rho' parameter still be valid?
As the Sequencer will additionally call forget() before each call to forward and if we use seqlen < rho than, it will forget previous step after every seqlen no matter what "rho" is.

Thank You

Hi. You use the convolution with NO bias in ConvLSTM.lua, and the normal convolution with bias in UntiedConvLSTM.lua. Could you please explain about what's the difference with or without bias? Thank you!

Is the convLSTM block compatible with BiSequencer of the rnn package? To my understanding, there seems to be no problem. But, my test encountered the following error:

all/share/lua/5.1/extracunn/SpatialConvolutionNoBias.lua:94: attempt to call field 'SpatialConvolutionMMNoBias_updateOutput' (a nil value)

(extracunn has been installed.)

Thanks!

Hi,

I successfully ran ConvLSTM with mini-batches on single GPU, but it failed when I tried to run it on multiple GPUs. The error is as follows,

/lua/5.1/nn/CAddTable.lua:16: bad argument #2 to 'add' (sizes do not match at /tmp/luarocks_cutorch-scm-1-7971/cutorch/lib/THC/THCTensorMathPointwise.cu:121)

I carefully checked the size of tensors before using CAddTable, and found it matched. So I am confused what happened?

Any one has an idea? Thanks a lot.

Hi,
When I try to run main-mnist.lua, following errors happened:

/home/sodabeta/torch/install/bin/luajit: /home/sodabeta/torch/install/share/lua/5.1/torch/File.lua:366: unknown object
stack traceback:
        [C]: in function 'error'
        /home/sodabeta/torch/install/share/lua/5.1/torch/File.lua:366: in function 'readObject'
        /home/sodabeta/torch/install/share/lua/5.1/torch/File.lua:391: in function 'load'
        /home/sodabeta/convlstm/ConvLSTM/data-mnist.lua:4: in function 'getdataSeq_mnist'
        main-mnist.lua:23: in function 'main'
        main-mnist.lua:162: in main chunk
        [C]: in function 'dofile'
        ...beta/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
        [C]: at 0x00406670

I downloaded the dataset file from the link provided, and I'm not sure if the downloaded dataset file is broken, and it should be helpful if you could provide the md5sum.
Please let me know if you have any ideas about the problem. Thanks!

Hi,
Would you happen to have the dataset_fly_64x64_lines_test and train datasets available in h5 or a non-Torch format? I'm getting unknown object errors trying to load these files.

Many thanks!

Hi Viorik,

Your code is not compatible with new rnn update. Nicholas has tried update it but there is some issue with the extracunn's SpatialConvolutionMMNoBias. If you could help in figuring out why that error is coming then it will be great!

Thanks
Mansi

Hi Viorik,

Thanks for sharing the code. In order to run the code, could you please also share the data used?
Thanks!!!

Hello @viorik ,
As I understand, in function buildGate(), input2gate and output2gate only have same size if you setting the stride = 1 (then the width and height of input will be preserved). But in case the stride is not 1, then the input2gate and output2gate don't have same size. In your paper, you said the input is dxhxw and output is dmxhxw, this's only true if stride=1. Am I right?

Hello,
I am attempting to use batches with element research's rnn lib in conjunction with yours. The rnn lib requires that when the sequencer is being used tables of tensors are to be passed to the sequencer's forward() method. When wrapping a ConvLSTM with a sequencer the following error occurs:

CAddTable.lua:16: bad argument #2 to 'add' (sizes do not match ...)

The input to the rnn sequencer is a lua table:
{ tensor(batch, color, height, width), tensor(batch, color, height, width), tensor(batch, color, height, width) ... }

Which is the standard format for the rnn lib.

I ensured that I specified a batchSize when creating the ConvLSTM.

An interesting note is that the ConvLSTM runs correctly in batch mode when the sequence length is 1! (only 1 entry in the input lua table, instead of many)

Please let me know if you have any ideas!
Best wishes,
Michael

I am getting following error while training Network containing UntiedConvLSTM()

~/torch/install/share/lua/5.1/nn/ConcatTable.lua:55: bad argument #2 to 'add' (sizes do not match at /home/rmalav/torch/extra/cutorch/lib/THC/generic/THCTensorMathPointwise.cu:10)

After some testing, I found that network is unable to backpropogate (model:forward() works fine, But model:backward() throws above error). I have created a simple Jupyter Notebook File just to show the problem.

I think I am missing something obvious. Can you help me with it?
Thank You.

Hello viorik @viorik
Recently I have been trying to implement conv-lstm for my video processing task. I have read your "spatio-temporal video autoencoder" paper and It is really a good one. After reading your source code, I had several problems about the data in main.lua.
The program "getdataSeq_mnist.lua" is mainly for generation of samples, right? But I'm not certain about the data format: here 'data' is a 5-D dim? 1 for samples, and 2 for sequence length? 3 for channels? 4/5 for width/height. If so, I cannot understand the meanings of
sample = dataset[t] data = sample[1] for i = 1,data:size(1)-1 do table.insert(inputTable, data[i]:cuda()) end
Here sample should be a sample, and data should be a sequence?? right? If so, data:size(1) should equal to 1 ? I'm sorry but I'm confused about the data format here.
Could you please explain it a little? Thanks.

thank you for your implement, if there is a version of tensorflow?