ne7ermore / torch-light

Deep-learning by using Pytorch. Basic nns like Logistic, CNN, RNN, LSTM and some examples are implemented by complex model.

---

This repository includes basics and advanced examples for deep learning by using Pytorch.
Basics which are basic nns like Logistic, CNN, RNN, LSTM are implemented with few lines of code, advanced examples are implemented by complex model.
It is better finish Official Pytorch Tutorial before this.

Continue updating...

Tutorial

Get tutorial series in Blog if know Chinese

Tabel of Pytorch Examples

1. Basics

• Cbow
• N-Gram
• CNN Text classfication
• LSTM Text classfication

2. Reinforcement Training

• AlphaGo-Zero
• Image-Cap
• Reinforced Translate
• Toy

3. NLP

• Poetry VAE-NLG
• Seq2seq
• BiLSTM CRF NER
• LSTM CNNs CRF
• Chinese Poetry NLG
• BiMPM
• Pair Ranking Cnn
• BiLSTM CRF
• Capsule Text classfication
• Retrieval Based Chatbots
• Hierarchical for Summarization and Classification
• Deep SRL
• BERT
• Relation Network
• Information Extraction
• Pointer Network
• coreference

4. Vision

• yolo-v3
• DenseNet
• Neural Style
• DC Gan
• Facial Beauty Prediction

5. Special Things

• Customize

6. Speech

• Voice Conversion

Getting Started

clone code

$ git clone [email protected]:ne7ermore/torch-light.git

train

$ cd torch-light/project
$ python3 main.py

or

$ cd torch-light/project
$ python3 corpus.py
$ python3 main.py

or

$ cd torch-light/project
$ python3 corpus.py
$ python3 train.py

Citation

If you find this code useful for your research, please cite:

@misc{TaoTorchLight,
  author = {Ne7ermore Tao},
  title = {torch-light},
  publisher = {GitHub},
  year = {2020},
  howpublished = {\url{https://github.com/ne7ermore/torch-light}}
}

Contact

Feel free to contact me if there is any question (Tao [email protected]). Tao Ne7ermore/ @ne7ermore

Dependencies

• Python 3.5
• PyTorch 0.2.0
• Numpy 1.13.1

torch-light's Issues

请问 alpha zero训练时，出现如下错误 是什么原因啊

Game - 104 | data length - 21
kl - nan | lr_multiplier - 2.25 | loss - nan

Traceback (most recent call last):
File "train.py", line 125, in
t.run()
File "train.py", line 62, in run
print(f"Game - {step} | data length - {self.sample(game.play())}")
File "D:\theory\courses\Machine_Learning\torch-light\alpha-zero\game.py", line 152, in play
pi, next_node = mc.search(self.board, node, temperature=1)
TypeError: 'NoneType' object is not iterable

AttributeError: 'NoneType' object has no attribute 'data'

Trying to train the retrieval-based-chatbots.
There has an error.

Expect for your reply.

▶ python3 train.py
==============================arguments==============================
logdir: logdir
batch_size: 64
lr: 0.001
dropout: 0.5
emb_dim: 200
first_rnn_hsz: 200
fillters: 8
kernel_size: (3, 3)
match_vec_dim: 50
second_rnn_hsz: 50
use_cuda: False
max_cont_len: 10
max_utte_len: 50
dict_size: 156260
============================================================
Traceback (most recent call last):
  File "train.py", line 88, in <module>
    model = Model(args)
  File "/Users/viosey/Workspace/Dev/ML/PyTorch/torch_light/retrieval-based-chatbots/model.py", line 32, in __init__
    self._reset_parameters()
  File "/Users/viosey/Workspace/Dev/ML/PyTorch/torch_light/retrieval-based-chatbots/model.py", line 40, in _reset_parameters
    self.transform_A.bias.data.fill_(0)
AttributeError: 'NoneType' object has no attribute 'data'

your deep-srl is NER?

wait- your deep-srl model is for NER task? and I didnt see the viterbi hard constraint part?

Model not found

https://github.com/ne7ermore/torch-light/blob/master/vae-nlg/train.py#L20

the data parameter seems is the directory of a model, but there is only two corpus and a w2v

Thx

run train.py error

When I run train.py, show type error:

Traceback (most recent call last):
File "D:/github/torch_light/Image-Cap/train.py", line 235, in
loss = pre_train_actor()
File "D:/github/torch_light/Image-Cap/train.py", line 120, in pre_train_actor
target, _ = actor(hidden, labels)
File "D:\Anaconda3\envs\python36\lib\site-packages\torch\nn\modules\module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "D:\github\torch_light\Image-Cap\model.py", line 91, in forward
emb_enc = self.lookup_table(labels[:, i])
File "D:\Anaconda3\envs\python36\lib\site-packages\torch\nn\modules\module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "D:\Anaconda3\envs\python36\lib\site-packages\torch\nn\modules\sparse.py", line 103, in forward
self.scale_grad_by_freq, self.sparse
File "D:\Anaconda3\envs\python36\lib\site-packages\torch\nn_functions\thnn\sparse.py", line 57, in forward
output = torch.index_select(weight, 0, indices)
TypeError: torch.index_select received an invalid combination of arguments - got (torch.cuda.FloatTensor, int, !torch.cuda.IntTensor!), but expected (torch.cuda.FloatTensor source, int dim, torch.cuda.LongTensor index)

An error when I run the train.py of LSTM-CNNs-CRF

Hi,I'm new to pytorch,when I run the train.py, I got error below, I try such ways to slove, bur all not work. How this error means,?Thanks.
`D:\workspace\Python\NLP\torch_light-master\LSTM-CNNs-CRF\model.py:158: UserWarning: nn.init.xavier_uniform is now deprecated in favor of nn.init.xavier_uniform_.
init.xavier_uniform(self.char_ebd.weight)
D:\workspace\Python\NLP\torch_light-master\LSTM-CNNs-CRF\model.py:31: UserWarning: nn.init.xavier_uniform is now deprecated in favor of nn.init.xavier_uniform_.
init.xavier_uniform(self.transitions)
D:\Program\Anaconda3\lib\site-packages\torch\cuda_init_.py:116: UserWarning:
Found GPU0 Quadro K600 which is of cuda capability 3.0.
PyTorch no longer supports this GPU because it is too old.

warnings.warn(old_gpu_warn % (d, name, major, capability[1]))
Train Processing: 0it [00:00, ?it/s]------------------------------------------------------------------------------------------
Traceback (most recent call last):
File "D:/workspace/Python/NLP/torch_light-master/LSTM-CNNs-CRF/train.py", line 148, in
loss = train()
File "D:/workspace/Python/NLP/torch_light-master/LSTM-CNNs-CRF/train.py", line 130, in train
loss, _ = model(word, char, label)
File "D:\Program\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "D:\workspace\Python\NLP\torch_light-master\LSTM-CNNs-CRF\model.py", line 187, in forward
char_feats = self.cnn(chars)
File "D:\Program\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "D:\workspace\Python\NLP\torch_light-master\LSTM-CNNs-CRF\model.py", line 163, in forward
encode = self.char_ebd(encode).unsqueeze(1)
File "D:\Program\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "D:\Program\Anaconda3\lib\site-packages\torch\nn\modules\sparse.py", line 108, in forward
self.norm_type, self.scale_grad_by_freq, self.sparse)
File "D:\Program\Anaconda3\lib\site-packages\torch\nn\functional.py", line 1076, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got CUDAIntTensor instead (while checking arguments for embedding)`

Image-Cap: Assertion Error

Hi,

@ne7ermore : So I get the following error when I try to load the actor model to the GPU (i.e. in train.py). Any help would be much appreciated.
Traceback (most recent call last): File "train.py", line 114, in <module> actor = actor.cuda() File "/home/user/anaconda2/envs/py35/lib/python3.5/site-packages/torch/nn/modules/module.py", line 216, in cuda return self._apply(lambda t: t.cuda(device)) File "/home/user/anaconda2/envs/py35/lib/python3.5/site-packages/torch/nn/modules/module.py", line 146, in _apply module._apply(fn) File "/home/user/anaconda2/envs/py35/lib/python3.5/site-packages/torch/nn/modules/rnn.py", line 123, in _apply self.flatten_parameters() File "/home/user/anaconda2/envs/py35/lib/python3.5/site-packages/torch/nn/modules/rnn.py", line 111, in flatten_parameters params = rnn.get_parameters(fn, handle, fn.weight_buf) File "/home/user/anaconda2/envs/py35/lib/python3.5/site-packages/torch/backends/cudnn/rnn.py", line 165, in get_parameters assert filter_dim_a.prod() == filter_dim_a[0] AssertionError

I checked to make sure that the library versions match with the requirements.

报错

您好，LSTM-CNNS-CRF中，运行train.py，报错RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.IntTensor instead (while checking arguments for embedding)。显示是model（word，char，label）的问题

Why only one image while training image captioning ?

def encode(self, imgs):
        if self.training:
            enc = self.enc(imgs)[0]
        else:
            enc = self.enc(imgs)
        enc = self.enc_out(enc)
        return enc

https://github.com/ne7ermore/torch-light/blob/master/Image-Cap/model.py

Confused by this conv1d operation

Hi, I'm reading this code for study and it helps me a lot.
I'm confused by this line:

torch-light/BERT/model.py

Line 74 in 254c133

nn.Conv1d(d_model, d_ff, 1),

from the source paper of BERT, I've not found any description that BERT use a conv1d layer in transformer instead of linear transformation.

And from http://nlp.seas.harvard.edu/2018/04/03/attention.html#position-wise-feed-forward-networks, this is implement by a mlp.

Can anyone kindly help me with this problem?

A query regarding Image-Cap

https://github.com/ne7ermore/torch-light/tree/master/Image-Cap
Hi @ne7ermore,
I found this code very educational. Thank you. But, I do not have a GPU. When I run train.py on a CPU, the estimated time of training is about 3-4 weeks. So, could you please share the trained model. It would be of great help. My email id - [email protected].
Thank You

人机对战时电脑计算的特别慢是因为没训练到位吗？还是电脑太渣

大佬您好，请问下为什么从150次开始损失一直在2.5左右，降不下来，您当初损失降得很顺利吗

invalid index of a 0-dim tensor

Hi ne7ermore,
Thank you a lot for your code sharing. I was running your retrieval chatbot code train.py and encountered the following error:

Traceback (most recent call last):
File "train.py", line 141, in
loss, corrects, acc, size = evaluate()
File "train.py", line 104, in evaluate
eval_loss += loss.data[0]
IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number

I searched for the reason and solution, and found that it was a Pytorch version issue, in version > 0.5 you have to change loss.data[0] to loss.item() as indicated in the error message. I just post this issue in order to remind others.
P.S. It seemed that loss.data.item() also worked.

Question regard Deep SRL module

Hi, I was going through your Deep SRL code which is the implementation of the paper Deep Semantic Role Labeling: What Works and What’s Next and i couldn't locate the places where you have implemented the viterbi decoding or the BIO and SRL constraints which are mentioned in the paper. Could you please help me in case i am missing something in the code or have these not been implemented in the code ?
Also, can this implementation handle multiple predicates in the same sentence?

How to add CRF layer after biLSTM output layer?

It seems that in biLSTM-CRF model only biLSTM is trained without a CRF layer added to the output layer of biLSTM. CRF is only called after the probability of each tag for each word is predicted by loading a trained biLSTM model (in predict.py). So, the CRF only works in predicting stage rather than in training stage?

You may add 'python3 corpus.py' before 'python3 train.py'

There was a data lacking problem after I ran train.py directly, then I ran corpus.py before training to solve that.

数据在哪下载

bert中的corpus语料数据找不到了，想知道数据在哪下载

Problems while running facial-beauty-prediction

Hi, where can I download the images and verification in the following paths:

/Users/nevermore/code/fuel/SCUT-FBP5500_v2/Images
 
/Users/nevermore/code/fuel/SCUT-FBP5500_v2/train_test_files/5_folders_cross_validations_files/cross_validation_1

facial-beauty-prediction/data/images
facial-beauty-prediction/data/validation

what's the meaning of computing the new_lr with emb_dim？

https://github.com/ne7ermore/torch_light/blob/6ec234c1e4db26404c8ef32c51c0050628c9bf64/vae-nlg/optim.py#L24