hunkim / pytorchzerotoall

Simple PyTorch Tutorials Zero to ALL!

PyTorchZeroToAll

Quick 3~4 day lecture materials for HKUST students.

Video Lectures: (RNN TBA)

• Youtube
• Bilibili

Slides

• Lecture Slides @GoogleDrive

If you cannot access the GoogleDoc for somehow, please check out pdf files in slides. However, slides in GoogleDrive are always latest. We really appreciate your comments.

Previous Lectures

• cf., http://bit.ly/TF_HKUST (3 day crash course using TensorFlow)

pytorchzerotoall's Issues

PyTorchDataParallel tutorial?

@chsasank, are you potentially interested in this kind of tutorial: https://github.com/hunkim/PyTorchZeroToAll/blob/master/tutorials/PyTorchDataParallel.ipynb

It seems current tutorial (http://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html) does not have enough information/examples.

Cheers!

Question about slide 09 Softmax Classifier

As you mentioned in 28th line "09_01_softmax_loss.py"

27 # Input is class, not one-hot
28 Y = Variable(torch.LongTensor([0]), requires_grad=False)

however, whichever category I choose not just class 0 (the first dimension) , Loss1 and Loss2 are always stay the same.

for example:
I choose class 1 (the second dimension), the losses are still 0.41 and 1.84

Y=Variable(torch.LongTensor([1]),requires_grad=False)

PyTorch Loss1 = 0.41703000664711
PyTorch Loss2 = 1.840616226196289

how can i pay you a coffee?

seriously, your videos helped me tremendously. do you have PayPal or something?

Error in 13_3_char_rnn.py

Training for 100 epochs...
Traceback (most recent call last):
File "F:/pytorch/PyTorchZeroToAll-master/13_3_char_rnn.py", line 142, in
loss = train(lines[0]) # Batch size is 1
File "F:/pytorch/PyTorchZeroToAll-master/13_3_char_rnn.py", line 117, in train
loss += criterion(output, target[c])
File "C:\Users\hh\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "C:\Users\hh\Anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 862, in forward
ignore_index=self.ignore_index, reduction=self.reduction)
File "C:\Users\hh\Anaconda3\lib\site-packages\torch\nn\functional.py", line 1550, in cross_entropy
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
File "C:\Users\hh\Anaconda3\lib\site-packages\torch\nn\functional.py", line 1403, in nll_loss
if input.size(0) != target.size(0):
RuntimeError: dimension specified as 0 but tensor has no dimensions

Process finished with exit code 1

12_4_hello_rnn_emb: AttributeError: 'Model' object has no attribute 'num_layers'

How plot the accuracy and loss history for 13_2_rnn_classification.py

How plot the accuracy and loss history for 13_2_rnn_classification.py thanks

License of this repository?

Thank you so much for making it easy to try out the code examples. As title says, what is the license of this repository?

Expected object of scalar type Long but got scalar type Float for argument #2 'target'

I followed the concept given above ,
took the data,normalized ,split into train and test data and when training ,it says that it want type long but got float32

Any help would be appreciated

small bug

line 53 of 12_1_rnn_basics.py should be
inputs = inputs.transpose(dim1=0, dim2=1)

How to introduce mini batch training in the seq2seq example?

As you stated in your comments you don't handle batches in your seq2seq example which makes the training very slow. Can you please give us an indication of how to introduce it?

Error on 10_1_cnn_mnist.py

Hi Prof. Kim

When I ran 10_1_cnn_mnist.py on Python 3.5 (Ubuntu 16.04.5 LTS), I met the following error:

10_1_cnn_mnist.py:49: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
return F.log_softmax(x)
Traceback (most recent call last):
File "10_1_cnn_mnist.py", line 92, in
train(epoch)
File "10_1_cnn_mnist.py", line 69, in train
100. * batch_idx / len(train_loader), loss.data[0]))
IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number

Look forward to hearing from soon. Many thanks,
Jianliang

gzip file cannot open

Having trouble opening the gzip files in the data folder.
Got this error message.
OSError: Not a gzipped file (b'\n\n')

error when running 09_2_softmax_mnist.py

Traceback (most recent call last):
File "/home/cxy/chenxiuyi/PyTorchZeroToAll/09_2_softmax_mnist.py", line 94, in
test()
File "/home/cxy/chenxiuyi/PyTorchZeroToAll/09_2_softmax_mnist.py", line 81, in test
test_loss += criterion(output, target, size_average=False).data[0]
File "/home/cxy/anaconda2/envs/conda_conda_torch/lib/python2.7/site-packages/torch/nn/modules/module.py", line 224, in call
result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'size_average'

So I use "test_loss += criterion(output,target).data[0]". It's ok! I want to know why you use "size_average=False".

Best regards

Why can't I open the link of slides?

I can open the google.com,but i can't open the link of slides ，
Is there anything wrong with it?

Exercices answers

Hello I really appreciate the slides, the content is awesome and the way of explaining things is sooooo clear. Could you please share with us exercises solutions to compare our answers?

With 2 epochs, the data are loaded twice, why?

Hi Dr. Kim,

Why do we need to load data twice into torch tensors with 2 epochs? What the difference of the epoch for dataloader and the epoch for neural network? Sorry, maybe my questions are silly, as I only started learning neural network and PyTorch 2 days ago.

Many thanks,
Best wishes,
Jianliang

TypeError: 11_3_hello_rnn_gpu.py on GPU

Anyone can fix this?

Traceback (most recent call last):
  File "11_3_hello_rnn_gpu.py", line 83, in <module>
    outputs = rnn(inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "11_3_hello_rnn_gpu.py", line 65, in forward
    out, _ = self.rnn(x, h_0)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py", line 162, in forward
    output, hidden = func(input, self.all_weights, hx)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/_functions/rnn.py", line 351, in forward
    return func(input, *fargs, **fkwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/_functions/rnn.py", line 244, in forward
    nexth, output = func(input, hidden, weight)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/_functions/rnn.py", line 84, in forward
    hy, output = inner(input, hidden[l], weight[l])
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/_functions/rnn.py", line 113, in forward
    hidden = inner(input[i], hidden, *weight)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/_functions/rnn.py", line 19, in RNNTanhCell
    hy = F.tanh(F.linear(input, w_ih, b_ih) + F.linear(hidden, w_hh, b_hh))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 553, in linear
    return torch.addmm(bias, input, weight.t())
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/variable.py", line 924, in addmm
    return cls._blas(Addmm, args, False)
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/variable.py", line 920, in _blas
    return cls.apply(*(tensors + (alpha, beta, inplace)))
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/_functions/blas.py", line 26, in forward
    matrix1, matrix2, out=output)
TypeError: torch.addmm received an invalid combination of arguments - got (int, torch.cuda.FloatTensor, int, torch.FloatTensor, torch.cuda.FloatTensor, out=torch.cuda.FloatTensor), but expected one of:
 * (torch.cuda.FloatTensor source, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
 * (torch.cuda.FloatTensor source, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
 * (float beta, torch.cuda.FloatTensor source, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
 * (torch.cuda.FloatTensor source, float alpha, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
 * (float beta, torch.cuda.FloatTensor source, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
 * (torch.cuda.FloatTensor source, float alpha, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
 * (float beta, torch.cuda.FloatTensor source, float alpha, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
      didn't match because some of the arguments have invalid types: (int, torch.cuda.FloatTensor, int, !torch.FloatTensor!, torch.cuda.FloatTensor, out=torch.cuda.FloatTensor)
 * (float beta, torch.cuda.FloatTensor source, float alpha, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
      didn't match because some of the arguments have invalid types: (int, torch.cuda.FloatTensor, int, !torch.FloatTensor!, torch.cuda.FloatTensor, out=torch.cuda.FloatTensor)

L1 basic file

In the basic file: https://github.com/hunkim/PyTorchZeroToAll/blob/master/01_basics.py
@49cab2b

Isn't line 27 should be loss(y_pred_val, y_val)

poor performance of diabetes prediction

Why is the performance of using 07_diabetes_logistic.py to predict diabetes pretty poor?

lec 14 video

hi, i watched your video recently and found they are soooooo helpful for me!

will you upload lec 14 video to youtube channel? i really want to learn seq2seq model!

The videos of these lectures are incomplete?

Thank you for sharing the videos ,but why i cannot see after 13 lectures?
Expecially the language model for the pytorch.

Request: Generator Fitting

Fantastic videos!!

One request if that is ok - I would love to see the following topics:

1. How to use data loader and fit a model where the data can not be held in memory - ala fit_generator in Keras.

That is the most important, I can not find much to demonstrate this online.
Also,
2) Multivariate time series modeling with RNN

Hello, I have a few questions that I would need to understand better

First, let me thank you for your deep-tutorials about ML, I really think it helped a lot of people to get a better understanding.
Secondly, if we take a look at this: https://github.com/hunkim/PyTorchZeroToAll/blob/master/01_basics.py
is the purpose of that code, just to setup everything as you show in youtube video? If I understand correctly, w is just to help us "walk-over" some field of different numbers right?
Well my confusion raises from not being capable of understanding why we get such graph and why w at 2 should have 0 loss :/
Hopefully you can understand where my issue is :)

Best of regards,
TK

Replace deprecated `Variable` class

PyTorchZeroToAll/03_auto_gradient.py

Line 7 in edd7ba1

w = Variable(torch.Tensor([1.0]), requires_grad=True) # Any random value

The class torch.autograd.Variable is deprecated.

This line must be updated to

w = torch.tensor([1.0], requires_grad=True) # Any random value

How to save Trained Model???

Thanks for this project.
I following by this example. but I have a question.
How to save Training model and load model?
For example, 13-2 RNN Classification,
When I start file, Always wait for Training Times.
I want to start training and saving model. So, Whenever I load trained model.
( Just I want to save my times..)
Thanks for Guide.

where can I reference exercise code?

Call for comments for seq2seq and charrnn

Could you please check if this code is easy/correct enough to show the basic concepts?

https://github.com/hunkim/PyTorchZeroToAll/blob/master/13_1_seq2seq.py
https://github.com/hunkim/PyTorchZeroToAll/blob/master/12_5_char_rnn.py

@yunjey @kkweon, thanks in advance!

Error message on dim

RuntimeError Traceback (most recent call last)
in ()
25
26 # The given dimensions dim0 and dim1 are swapped.
---> 27 inputs = inputs.transpose(3, dim1=1, dim2=2)
28 # Propagate input through RNN
29 # Input: (seq_len, batch_size, input_size) when batch_first=False (default)

RuntimeError: transpose() got an unexpected keyword argument 'dim2'

RuntimeError: dimension specified as 0 but tensor has no dimensions

line 78 of 12_2_hello_rnn.py
What's wrong with it? loss += criterion(output, label)

$ python 12_2_hello_rnn.py
Model(
(rnn): RNN(5, 5, batch_first=True)
)
predicted string: lTraceback (most recent call last):
File "12_2_hello_rnn.py", line 78, in
loss += criterion(output, label)
File "/home/jiapei/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/jiapei/.local/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 759, in forward
self.ignore_index, self.reduce)
File "/home/jiapei/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 1442, in cross_entropy
return nll_loss(log_softmax(input, 1), target, weight, size_average, ignore_index, reduce)
File "/home/jiapei/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 1328, in nll_loss
if input.size(0) != target.size(0):
RuntimeError: dimension specified as 0 but tensor has no dimensions

Does `batch_first` really also work on the shape of RNN's hidden vector?

Hi, in your code for 12_1_rnn_basics, you mentioned that

#(batch, num_layers * num_directions, hidden_size) for batch_first=True

However, in pytorch's docs, it just said:

batch_first – If True, then the input and output tensors are provided as (batch, seq, feature)

So I got confused which one was right? Then I printed the shape of RNN's hidden vector after executing this line, which turned out to be [1, 3, 2].

So could you please tell me which one is right, and explain why the output is [1, 3, 2]?

Thank you very much.