weinyn / dietclassifier-pytorch Goto Github PK

How to evaluate a new domain question dataset? During training process, datasets was split into training and evaluating parts. And I want to only evaluate a new testing datasets using a well-trained model. How to do that?

I have tried methods below:
using wrapper predict function to get intent and entity predict results, and get metrics. But I have to predict results using my own batch scripts, it's really time-consuming.

Is there any good ways to evaluting a new dataset?

i got this error when training model. but i don;t know how to fix this. please help

tensors is None
the type of tensors is <class 'NoneType'>
Traceback (most recent call last):
File "train.py", line 65, in
trainer.train()
File "/data1/Semantic_team/chatbot/DIET/DIETClassifier-pytorch/src/models/trainer.py", line 77, in train
self.trainer.train()
File "/data1/anaconda3/envs/diet_py36/lib/python3.6/site-packages/transformers/trainer.py", line 935, in train
self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
File "/data1/anaconda3/envs/diet_py36/lib/python3.6/site-packages/transformers/trainer.py", line 1004, in _maybe_log_save_evaluate
metrics = self.evaluate()
File "/data1/anaconda3/envs/diet_py36/lib/python3.6/site-packages/transformers/trainer.py", line 1449, in evaluate
metric_key_prefix=metric_key_prefix,
File "/data1/anaconda3/envs/diet_py36/lib/python3.6/site-packages/transformers/trainer.py", line 1566, in prediction_loop
loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
File "/data1/anaconda3/envs/diet_py36/lib/python3.6/site-packages/transformers/trainer.py", line 1695, in prediction_step
logits = nested_detach(logits)
File "/data1/anaconda3/envs/diet_py36/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 106, in nested_detach
return type(tensors)(nested_detach(t) for t in tensors)
File "/data1/anaconda3/envs/diet_py36/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 106, in
return type(tensors)(nested_detach(t) for t in tensors)
File "/data1/anaconda3/envs/diet_py36/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 107, in nested_detach
return tensors.detach()
AttributeError: 'NoneType' object has no attribute 'detach'

Hi, When I add more examples the prediction is not good. It was too bad prediction. Did you have any idea about it?

tks for your information, I don't know the cause of this error, so I added a little note for this issue in the readme

Originally posted by @WeiNyn in #5 (comment)

When I added some new intents and entities in nlu.yml and ran wrapper.py triggering train_model(). this arises with error below.

/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [0,0,0] Assertion t >= 0 && t < n_classes failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [1,0,0] Assertion t >= 0 && t < n_classes failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [2,0,0] Assertion t >= 0 && t < n_classes failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [3,0,0] Assertion t >= 0 && t < n_classes failed.
Traceback (most recent call last):
File "src/classification/wrapper.py", line 243, in
wrapper.train_model()
File "src/classification/wrapper.py", line 227, in train_model
trainer.train()
File "/mnt/e/intellemo/official/chat-nlp-aiapi/src/classification/trainer.py", line 52, in train
self.trainer.train()
File "/mnt/e/intellemo/official/chat-nlp-aiapi/algosrc/lib/python3.8/site-packages/transformers/trainer.py", line 1280, in train
tr_loss += self.training_step(model, inputs)
File "/mnt/e/intellemo/official/chat-nlp-aiapi/algosrc/lib/python3.8/site-packages/transformers/trainer.py", line 1773, in training_step
loss = self.compute_loss(model, inputs)
File "/mnt/e/intellemo/official/chat-nlp-aiapi/algosrc/lib/python3.8/site-packages/transformers/trainer.py", line 1805, in compute_loss
outputs = model(**inputs)
File "/mnt/e/intellemo/official/chat-nlp-aiapi/algosrc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/mnt/e/intellemo/official/chat-nlp-aiapi/src/classification/classifier.py", line 143, in forward
entities_loss = entities_loss_fct(active_logits, active_labels)
File "/mnt/e/intellemo/official/chat-nlp-aiapi/algosrc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/mnt/e/intellemo/official/chat-nlp-aiapi/algosrc/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 1120, in forward
return F.cross_entropy(input, target, weight=self.weight,
File "/mnt/e/intellemo/official/chat-nlp-aiapi/algosrc/lib/python3.8/site-packages/torch/nn/functional.py", line 2824, in cross_entropy
return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: CUDA error: device-side assert triggered
0%| | 2/22600 [00:01<4:19:44, 1.45it/s]

Is that something that i need to increase the number of labels?

Does this PyTorch implementation include every component described in the original paper?

DIET architecture
http://bl.ocks.org/koaning/raw/f40ca790612a03067caca2bde81e7aaf/

Here is the parameters defined in RASA's implementation:
https://github.com/RasaHQ/rasa/blob/main/rasa/nlu/classifiers/diet_classifier.py#L147

I don't see any of them available in this PyTorch implementation.

https://drive.google.com/drive/folders/1cAucUHO0FP_I-_atSpbyRwKEiflPPN7v?usp=sharing