georgian-io / multimodal-toolkit Goto Github PK

Hi,

I have a question about inference.
I tried running inference at the end of the notebook

import torch

model_inputs = test_dataset[0]
print(model_inputs.keys())
print(model_inputs['numerical_feats'].size())
with torch.no_grad():
    _, logits, classifier_outputs = model(
        model_inputs['input_ids'],
        token_type_ids=model_inputs['token_type_ids'],
        cat_feats=model_inputs['cat_feats'],
        numerical_feats=model_inputs['numerical_feats']
    )

But I get an error:

ValueError: Wrong shape for input_ids (shape torch.Size([156])) or attention_mask (shape torch.Size([156]))

for numerical_feats which size is torch.Size([3]). What am I doing wrong? Could you provide a more detailed example of how to run inference and what and if there is additional preprocessing to be done?

There is a minor code bug in attention_on_cat_and_numerical_feats combine method.

        if numerical_feats.shape[1] != 0:
            if self.numerical_feat_dim > self.text_out_dim:
                numerical_feats = self.num_mlp(numerical_feats)
            w_num = torch.mm(numerical_feats, self.weight_num)
            g_num = (torch.cat([w_text, **w_cat**], dim=-1) * self.weight_a).sum(dim=1).unsqueeze(0).T

w_cat should be w_num

While calling multimodal_transformers.data.load_data, I am getting this error.
Apparently, it has been renamed to get_feature_names_out.

Presently we have code to handle the devices in this file. It isn't clear as to why this is needed however. Further investigation is needed as the Trainer class does this already (https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py)

While trying to tune my model I found that somewhere in load_data categorical features have object numpy type which can't be turned to torch tensor in TorchTabularTextDataset.

On another note, I think it's hard to productionize models trained with your library, because of preprocessing. They work only for on all subset of data (train, validation, test) or only for one of each which can and will cause errors, because each subset can have different categories. I feel like loading data should be changed - CategoricalFeatures (it should also be saved so you can use it later) should be fitted on training_data and then only transform validation and test (and assign unknown categories some default other category).

Also it would be awesome to be able to create pipelines like in transformers or sklearn for quick experimentation.

As the title says, there is a mistake in tabular_combiner.py.
Line 409:
g_num = (torch.cat([w_text, w_cat], dim=-1) * self.weight_a).sum(dim=1).unsqueeze(0).T

should change to:
g_num = (torch.cat([w_text, w_num], dim=-1) * self.weight_a).sum(dim=1).unsqueeze(0).T

because it is in a numerical part which should use w_num instead of w_cat

1. Why do we need to keep initializing ``cat_cols'' and num_cols'' when we want to inspect how the model performs with text input only?
2. When categorical tabular features is not present (text and numerical features only), how do we initialize load_data_from_folder() function?
   I checked that the function requires the inputs to have at least one categorical features --> raising an error when discarding categorical initialization.

I have trained my multimodel with distibert as a base model but the time of Inferencing
I tried the below script

with torch.no_grad():
    _, logits, classifier_outputs = model(
        torch.tensor(input_val['input_ids']),
        attention_mask=torch.tensor(input_val['attention_mask']),
        cat_feats=pre['cat_feats'],
        numerical_feats=pre['numerical_feats']
    )

I got the following error after that.
IndexError Traceback (most recent call last)
Input In [145], in <cell line: 1>()
1 with torch.no_grad():
----> 2 _, logits, classifier_outputs = model(
3 torch.tensor(input_val['input_ids']),
4 attention_mask=torch.tensor(input_val['attention_mask']),
5 cat_feats=pre['cat_feats'],
6 numerical_feats=pre['numerical_feats']
7 )

File ~/cat_multimodel_env/lib/python3.8/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
1106 # If we don't have any hooks, we want to skip the rest of the logic in
1107 # this function, and just call forward.
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1109 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110 return forward_call(*input, **kwargs)
1111 # Do not call functions when jit is used
1112 full_backward_hooks, non_full_backward_hooks = [], []

File ~/cat_multimodel_env/lib/python3.8/site-packages/multimodal_transformers/model/tabular_transformers.py:314, in DistilBertWithTabular.forward(self, input_ids, attention_mask, head_mask, inputs_embeds, labels, output_attentions, output_hidden_states, class_weights, cat_feats, numerical_feats)
277 @add_start_docstrings_to_callable(DISTILBERT_INPUTS_DOCSTRING.format("(batch_size, sequence_length)"))
278 def forward(
279 self,
(...)
289 numerical_feats=None
290 ):
291 r"""
292 class_weights (:obj:torch.FloatTensor of shape :obj:(tabular_config.num_labels,),optional, defaults to :obj:None):
293 Class weights to be used for cross entropy loss function for classification task
(...)
311 combining module's output
312 """
--> 314 distilbert_output = self.distilbert(
315 input_ids,
316 attention_mask=attention_mask,
317 head_mask=head_mask,
318 inputs_embeds=inputs_embeds,
319 output_attentions=output_attentions,
320 output_hidden_states=output_hidden_states,
321 )
322 hidden_state = distilbert_output[0] # (bs, seq_len, dim)
323 pooled_output = hidden_state[:, 0] # (bs, dim)

File ~/cat_multimodel_env/lib/python3.8/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
1106 # If we don't have any hooks, we want to skip the rest of the logic in
1107 # this function, and just call forward.
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1109 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110 return forward_call(*input, **kwargs)
1111 # Do not call functions when jit is used
1112 full_backward_hooks, non_full_backward_hooks = [], []

File ~/cat_multimodel_env/lib/python3.8/site-packages/transformers/modeling_distilbert.py:488, in DistilBertModel.forward(self, input_ids, attention_mask, head_mask, inputs_embeds, output_attentions, output_hidden_states, return_dict)
485 head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)
487 if inputs_embeds is None:
--> 488 inputs_embeds = self.embeddings(input_ids) # (bs, seq_length, dim)
489 return self.transformer(
490 x=inputs_embeds,
491 attn_mask=attention_mask,
(...)
495 return_dict=return_dict,
496 )

File ~/cat_multimodel_env/lib/python3.8/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
1106 # If we don't have any hooks, we want to skip the rest of the logic in
1107 # this function, and just call forward.
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1109 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110 return forward_call(*input, **kwargs)
1111 # Do not call functions when jit is used
1112 full_backward_hooks, non_full_backward_hooks = [], []

File ~/cat_multimodel_env/lib/python3.8/site-packages/transformers/modeling_distilbert.py:108, in Embeddings.forward(self, input_ids)
96 def forward(self, input_ids):
97 """
98 Parameters
99 ----------
(...)
106 The embedded tokens (plus position embeddings, no token_type embeddings)
107 """
--> 108 seq_length = input_ids.size(1)
109 position_ids = torch.arange(seq_length, dtype=torch.long, device=input_ids.device) # (max_seq_length)
110 position_ids = position_ids.unsqueeze(0).expand_as(input_ids) # (bs, max_seq_length)

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

To resolve this i tried to reshape the input with the following code.

pre['input_ids']=pre['input_ids'].reshape(-1,len(pre['input_ids'])).to("cuda:0")
pre['attention_mask']=pre['attention_mask'].reshape(-1,len(pre['attention_mask'])).to("cuda:0")
pre['numerical_feats']=pre['numerical_feats'].reshape(-1,len(pre['numerical_feats'])).to("cuda:0")
pre['cat_feats']=pre['cat_feats'].reshape(-1,len(pre['cat_feats'])).to("cuda:0")

Then I got this error

RuntimeError Traceback (most recent call last)
Input In [94], in <cell line: 1>()
----> 1 model(input_ids=pre['input_ids'],attention_mask=pre['attention_mask'],cat_feats=pre['cat_feats'],numerical_feats=pre['numerical_feats'])

File ~/cat_multimodel_env/lib/python3.8/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
1106 # If we don't have any hooks, we want to skip the rest of the logic in
1107 # this function, and just call forward.
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1109 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110 return forward_call(*input, **kwargs)
1111 # Do not call functions when jit is used
1112 full_backward_hooks, non_full_backward_hooks = [], []

File ~/cat_multimodel_env/lib/python3.8/site-packages/multimodal_transformers/model/tabular_transformers.py:327, in DistilBertWithTabular.forward(self, input_ids, attention_mask, head_mask, inputs_embeds, labels, output_attentions, output_hidden_states, class_weights, cat_feats, numerical_feats)
325 pooled_output = nn.ReLU()(pooled_output) # (bs, dim)
326 text_feats = self.dropout(pooled_output)
--> 327 combined_feats = self.tabular_combiner(text_feats,
328 cat_feats,
329 numerical_feats)
330 loss, logits, classifier_layer_outputs = hf_loss_func(combined_feats,
331 self.tabular_classifier,
332 labels,
333 self.num_labels,
334 class_weights)
335 return loss, logits, classifier_layer_outputs

File ~/cat_multimodel_env/lib/python3.8/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
1106 # If we don't have any hooks, we want to skip the rest of the logic in
1107 # this function, and just call forward.
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1109 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110 return forward_call(*input, **kwargs)
1111 # Do not call functions when jit is used
1112 full_backward_hooks, non_full_backward_hooks = [], []

File ~/cat_multimodel_env/lib/python3.8/site-packages/multimodal_transformers/model/tabular_combiner.py:370, in TabularFeatCombiner.forward(self, text_feats, cat_feats, numerical_feats)
368 elif self.combine_feat_method == 'mlp_on_concatenated_cat_and_numerical_feats_then_concat':
369 tabular_feats = torch.cat((cat_feats, numerical_feats), dim=1)
--> 370 tabular_feats = self.cat_and_numerical_mlp(tabular_feats)
371 combined_feats = torch.cat((text_feats, tabular_feats), dim=1)
372 elif self.combine_feat_method == 'individual_mlps_on_cat_and_numerical_feats_then_concat':

File ~/cat_multimodel_env/lib/python3.8/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
1106 # If we don't have any hooks, we want to skip the rest of the logic in
1107 # this function, and just call forward.
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1109 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110 return forward_call(*input, **kwargs)
1111 # Do not call functions when jit is used
1112 full_backward_hooks, non_full_backward_hooks = [], []

File ~/cat_multimodel_env/lib/python3.8/site-packages/multimodal_transformers/model/layer_utils.py:52, in MLP.forward(self, x)
50 input = layer_inputs[-1]
51 if layer == self.layers[-1]:
---> 52 layer_inputs.append(layer(input))
53 else:
54 if self.bn:

File ~/cat_multimodel_env/lib/python3.8/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
1106 # If we don't have any hooks, we want to skip the rest of the logic in
1107 # this function, and just call forward.
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1109 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110 return forward_call(*input, **kwargs)
1111 # Do not call functions when jit is used
1112 full_backward_hooks, non_full_backward_hooks = [], []

File ~/cat_multimodel_env/lib/python3.8/site-packages/torch/nn/modules/linear.py:103, in Linear.forward(self, input)
102 def forward(self, input: Tensor) -> Tensor:
--> 103 return F.linear(input, self.weight, self.bias)

RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x19 and 4x1).

Kindly help me with this issue.

Hello, if I want to verify the performance of combination methods on Chinese tasks, does the pretrained Bert model support it? Or I should use self-trained Bert model?

I think the latest version of numpy cause this bug since downgrading numpy to version 1.24.1 seem to fix the issue.

It gives me classification error. Like one of the class completely blanks out. It does not happen in case of Bert Base.

     precision    recall  f1-score   support

         Yes     0.0000    0.0000    0.0000        63
          No     0.5000    1.0000    0.6667        63

In layer_utils.calc_mlp_dims(), should the line "dim = dim // division" instead be "dim = dim - dim // division"? Based on the documentation, that 'division' is the factor by which successive MLP layer sizes are reduced, I'd expect 'division=4' to mean that the next layer size in the MLP is reduced by a factor of 0.75, not 0.25. I also feel like this interpretation is more practical, as reducing it by a default factor of 4 seems too steep a reduction (one can easily end up with a single-layer MLP most of the time). Open to understanding more about how this design choice was made though, if you actually did intend the code as written.

Can numerical feature in table be multidimensional arrays, or only a signl number is allowed?

When running the sample Colab file, I receive an error when evaluate_during_training is set to True. When I comment out evaluate_during_training, the code runs but the calc_classification_metrics are not generated. I did find this message on HuggingFace which seems to be related to the issue with evaluate_during training -- huggingface/transformers#8792.

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-46-3435b262f1ae> in <module>()
----> 1 trainer.train()

5 frames
/usr/local/lib/python3.6/dist-packages/multimodal_transformers/data/tabular_torch_dataset.py in __getitem__(self, idx)
     49         item['labels'] = torch.tensor(self.labels[idx]) if self.labels is not None  else None
     50         item['cat_feats'] = torch.tensor(self.cat_feats[idx]).float() \
---> 51             if self.cat_feats is not None else torch.zeros(0)
     52         item['numerical_feats'] = torch.tensor(self.numerical_feats[idx]).float()\
     53             if self.numerical_feats is not None else torch.zeros(0)

TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.

any help with this error

Steps to reproduce:
Notebook: https://github.com/georgian-io/Multimodal-Toolkit/tree/master/notebooks
Run this notebook on evuluation phase you will see this error:

There is some issue with the way the numpy is handling the argmax() function.
With axis = 0 the error is

ValueError: could not broadcast input array from shape (2348,768) into shape (2348,)

With axis = 1 the error is

AxisError: axis 1 is out of bounds for array of dimension 1

The shape of the numpy array is (2,) so I am not sure how to resolve this issue.

------ Full evaluation error on train.evaluation() ----------

---------------------------------------------------------------------------
AxisError                                 Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_15804\2732109216.py in <module>
----> 1 trainer.evaluate()

~\anaconda3\envs\py-win-37\lib\site-packages\transformers\trainer.py in evaluate(self, eval_dataset, ignore_keys, metric_key_prefix)
   2937             prediction_loss_only=True if self.compute_metrics is None else None,
   2938             ignore_keys=ignore_keys,
-> 2939             metric_key_prefix=metric_key_prefix,
   2940         )
   2941 

~\anaconda3\envs\py-win-37\lib\site-packages\transformers\trainer.py in evaluation_loop(self, dataloader, description, prediction_loss_only, ignore_keys, metric_key_prefix)
   3218                 )
   3219             else:
-> 3220                 metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
   3221         else:
   3222             metrics = {}

~\AppData\Local\Temp\ipykernel_15804\165556549.py in calc_classification_metrics(p)
     22 #   pred_scores = softmax(p.predictions)
     23 
---> 24   pred_labels = np.argmax(p.predictions, axis=1)
     25   print('\nNot Executed\n')
     26   pred_scores = softmax(p.predictions, axis=1)[:, 1]

<__array_function__ internals> in argmax(*args, **kwargs)

~\anaconda3\envs\py-win-37\lib\site-packages\numpy\core\fromnumeric.py in argmax(a, axis, out)
   1193 
   1194     """
-> 1195     return _wrapfunc(a, 'argmax', axis=axis, out=out)
   1196 
   1197 

~\anaconda3\envs\py-win-37\lib\site-packages\numpy\core\fromnumeric.py in _wrapfunc(obj, method, *args, **kwds)
     52     bound = getattr(obj, method, None)
     53     if bound is None:
---> 54         return _wrapit(obj, method, *args, **kwds)
     55 
     56     try:

~\anaconda3\envs\py-win-37\lib\site-packages\numpy\core\fromnumeric.py in _wrapit(obj, method, *args, **kwds)
     41     except AttributeError:
     42         wrap = None
---> 43     result = getattr(asarray(obj), method)(*args, **kwds)
     44     if wrap:
     45         if not isinstance(result, mu.ndarray):

AxisError: axis 1 is out of bounds for array of dimension 1

First, I would like to say thank you for this amazing library!

I managed to train a model for my purposes, but I stumbled upon issues with using the trained model for further inference.
I'm fairly new to Huggingface's Transformers framework and I could not find any helpful resources on how I can easily use a customized model (such as these multimodal models) for inference in their framework. I have tried to bisect the script in main.py, to obtain only the parts of the code that is needed for inference, but it only got me so far.

I also saw that others are also having issues with using the output models from this library for further prediction, so I think it would be beneficial for the whole community of this library if some guidelines were provided in this matter. If this takes more than your current capacity can afford, I would be more than grateful, if you could give me some pointers on where I should start to be able to proceed with my work.

Thank you!

At present all models return return loss, logits, classifier_layer_outputs.

Transformer models generally return loss, logits, hidden_states, attentions. Maybe we realign with this in the future?

Ref: https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_outputs.py#L665-L699

Number of trainable parameters = 110197407

0%|          | 0/18140 [00:00<?, ?it/s]Didn't find an RNG file, if you are resuming a training that was launched in a distributed fashion, reproducibility is not guaranteed.
Traceback (most recent call last):
File "main.py", line 305, in
main()
File "main.py", line 202, in main
trainer.train(
File "/root/miniconda3/lib/python3.8/site-packages/transformers/trainer.py", line 1543, in train
return inner_training_loop(
File "/root/miniconda3/lib/python3.8/site-packages/transformers/trainer.py", line 1791, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "/root/miniconda3/lib/python3.8/site-packages/transformers/trainer.py", line 2539, in training_step
loss = self.compute_loss(model, inputs)
File "/root/miniconda3/lib/python3.8/site-packages/transformers/trainer.py", line 2571, in compute_loss
outputs = model(**inputs)
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 171, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 181, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 89, in parallel_apply
output.reraise()
File "/root/miniconda3/lib/python3.8/site-packages/torch/_utils.py", line 644, in reraise
raise exception
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 64, in _worker
output = module(*input, **kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/workspace/bin/multimodal_transformers/model/tabular_transformers.py", line 114, in forward
outputs = self.bert(
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 1012, in forward
embedding_output = self.embeddings(
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 236, in forward
embeddings += position_embeddings
RuntimeError: The size of tensor a (1487) must match the size of tensor b (512) at non-singleton dimension 1

Hello,

The notebook was running fine 2 months ago, but no longer after the huggingface update.
I think there is a problem with the load_data_from_folder function, because I get the following error message when training:
AxisError: axis 1 is out of bounds for array of dimension 1 (see pic.).

Thanks for helping!

Describe the bug
When trying to run any of the 2 shared notebooks in the repository the execution fails during imports.

The error message being the same both cases:

RuntimeError                              Traceback (most recent call last)
RuntimeError: module compiled against API version 0xf but this version of numpy is 0xe
RuntimeError                              Traceback (most recent call last)
RuntimeError: module compiled against API version 0xf but this version of numpy is 0xe
ImportError                               Traceback (most recent call last)
ImportError: numpy.core._multiarray_umath failed to import
ImportError                               Traceback (most recent call last)
ImportError: numpy.core.umath failed to import
RuntimeError                              Traceback (most recent call last)
RuntimeError: module compiled against API version 0xf but this version of numpy is 0xe
ImportError                               Traceback (most recent call last)
ImportError: numpy.core._multiarray_umath failed to import
ImportError                               Traceback (most recent call last)
ImportError: numpy.core.umath failed to import
RuntimeError                              Traceback (most recent call last)
RuntimeError: module compiled against API version 0xf but this version of numpy is 0xe
ImportError                               Traceback (most recent call last)
ImportError: numpy.core._multiarray_umath failed to import
ImportError                               Traceback (most recent call last)
ImportError: numpy.core.umath failed to import
TypeError                                 Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/transformers/utils/import_utils.py in _get_module(self, module_name)
   1145         try:
-> 1146             return importlib.import_module("." + module_name, self.__name__)
   1147         except Exception as e:

27 frames
TypeError: Unable to convert function return value to a Python type! The signature was
	() -> handle

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/transformers/utils/import_utils.py in _get_module(self, module_name)
   1146             return importlib.import_module("." + module_name, self.__name__)
   1147         except Exception as e:
-> 1148             raise RuntimeError(
   1149                 f"Failed to import {self.__name__}.{module_name} because of the following error (look up to see its"
   1150                 f" traceback):\n{e}"

RuntimeError: Failed to import transformers.trainer because of the following error (look up to see its traceback):
Unable to convert function return value to a Python type! The signature was
	() -> handle

When installing the module I also receive an error from pip. Maybe this leads to the error occurring during module imports:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. arviz 0.15.1 requires scipy>=1.8.0, but you have scipy 1.7.3 which is incompatible. google-colab 1.0.0 requires pandas==1.5.3, but you have pandas 1.3.5 which is incompatible. tensorflow 2.12.0 requires numpy<1.24,>=1.22, but you have numpy 1.21.6 which is incompatible.

To Reproduce
Steps to reproduce the behavior:

1. Copy example notebook from this repository.
2. Run cell containing !pip install multimodal-transformers
3. Run cell at section "All other imports are here:"
4. See error

Expected behavior
The modules should be imported successfully.

Desktop:

• OS: Windows 11
• Python: 3.10.12
• Environment: Colab

Is this possible, I'm having some troubles with the installation

  ERROR: Command errored out with exit status 1:
   command: /home/guillem.garcia/.conda/envs/fake/bin/python /home/guillem.garcia/.conda/envs/fake/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpbzk1y5ws
       cwd: /tmp/pip-install-sdckw8up/tokenizers_39bb85f7486c4db689f764bdf07dd5c8
  Complete output (38 lines):
  /tmp/pip-build-env-uvhtxigq/overlay/lib/python3.9/site-packages/setuptools/dist.py:461: UserWarning: Normalizing '0.8.1.rc2' to '0.8.1rc2'
    warnings.warn(tmpl.format(**locals()))
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib
  creating build/lib/tokenizers
  copying tokenizers/__init__.py -> build/lib/tokenizers
  creating build/lib/tokenizers/models
  copying tokenizers/models/__init__.py -> build/lib/tokenizers/models
  creating build/lib/tokenizers/decoders
  copying tokenizers/decoders/__init__.py -> build/lib/tokenizers/decoders
  creating build/lib/tokenizers/normalizers
  copying tokenizers/normalizers/__init__.py -> build/lib/tokenizers/normalizers
  creating build/lib/tokenizers/pre_tokenizers
  copying tokenizers/pre_tokenizers/__init__.py -> build/lib/tokenizers/pre_tokenizers
  creating build/lib/tokenizers/processors
  copying tokenizers/processors/__init__.py -> build/lib/tokenizers/processors
  creating build/lib/tokenizers/trainers
  copying tokenizers/trainers/__init__.py -> build/lib/tokenizers/trainers
  creating build/lib/tokenizers/implementations
  copying tokenizers/implementations/__init__.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/byte_level_bpe.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/base_tokenizer.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/sentencepiece_bpe.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/char_level_bpe.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/bert_wordpiece.py -> build/lib/tokenizers/implementations
  copying tokenizers/__init__.pyi -> build/lib/tokenizers
  copying tokenizers/models/__init__.pyi -> build/lib/tokenizers/models
  copying tokenizers/decoders/__init__.pyi -> build/lib/tokenizers/decoders
  copying tokenizers/normalizers/__init__.pyi -> build/lib/tokenizers/normalizers
  copying tokenizers/pre_tokenizers/__init__.pyi -> build/lib/tokenizers/pre_tokenizers
  copying tokenizers/processors/__init__.pyi -> build/lib/tokenizers/processors
  copying tokenizers/trainers/__init__.pyi -> build/lib/tokenizers/trainers
  running build_ext
  running build_rust
  error: Can not find Rust compiler
  ----------------------------------------
  ERROR: Failed building wheel for tokenizers

I cannot install rust in that computer because I am not admin, but when I install more modern versions of transformers, it works

Thank you for the library, it made the coding for the project I am currently working much easier. I was wondering if it is possible to set the weights for the imbalanced classification? Thank you!

when the config is set as follows, there is an error.

File "main.py", line 167, in main
if training_args.do_train:
File "/Users/sharon/Library/Python/3.8/lib/python/site-packages/transformers/trainer.py", line 707, in train
tr_loss += self.training_step(model, inputs)
File "/Users/sharon/Library/Python/3.8/lib/python/site-packages/transformers/trainer.py", line 994, in training_step
outputs = model(**inputs)
File "/Users/sharon/Library/Python/3.8/lib/python/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/Users/sharon/Desktop/Models/Multimodal-Toolkit-master/multimodal_transformers/model/tabular_transformers.py", line 419, in forward
combined_feats = self.tabular_combiner(pooled_output,
File "/Users/sharon/Library/Python/3.8/lib/python/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/Users/sharon/Desktop/Models/Multimodal-Toolkit-master/multimodal_transformers/model/tabular_combiner.py", line 408, in forward
numerical_feats = self.num_mlp(numerical_feats)
File "/Users/sharon/Library/Python/3.8/lib/python/site-packages/torch/nn/modules/module.py", line 1177, in getattr
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'TabularFeatCombiner' object has no attribute 'num_mlp'

I think there is a typo in file tabular_combiner.py:268, self.cat_mlp should be self.num_mlp

Hi,
I am trying to install the package with "pip install multimodal-transformers" on linux and get the following error:

Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
Complete output (21 lines):
/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/setuptools/dist.py:774: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
% (opt, underscore_opt)
Traceback (most recent call last):
File "", line 1, in
File "/storage/tmp/pip-req-build-8dvkr2p7/setup.py", line 37, in
'Programming Language :: Python :: 3.7',
File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/setuptools/init.py", line 87, in setup
return distutils.core.setup(**attrs)
File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 147, in setup
_setup_distribution = dist = klass(attrs)
File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/setuptools/dist.py", line 480, in init
for k, v in attrs.items()
File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 280, in init
self.finalize_options()
File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/setuptools/dist.py", line 899, in finalize_options
for ep in sorted(loaded, key=by_order):
File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/setuptools/dist.py", line 898, in
loaded = map(lambda e: e.load(), filtered)
File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/setuptools/_vendor/importlib_metadata/init.py", line 196, in load
return functools.reduce(getattr, attrs, module)
AttributeError: type object 'Distribution' has no attribute '_finalize_feature_opts'
----------------------------------------

hi!
Can I expect the package to work on other BERT models (not listed in the ReadMe) -- e.g., longformer

thank you!

Hi,
I have tried using numerical_transformer_method (str, optional) : box_cox and quantile_normal.
I get the predictions in a different scale as the original.

How to inverse transform the predictions to original labels?

i tried to reinstall numpy but still got error:

module 'numpy.linalg._umath_linalg' has no attribute '_ilp64'

This can help leverage hardware optimisations and significantly speed up inference: https://huggingface.co/transformers/serialization.html

It seems this would be only possible with transformers>=4.9.0 (see https://huggingface.co/transformers/serialization.html#configuration-based-approach)), so this issue seems related to #3.

I have an issue when inferring with text features only.

I loaded my test data as follows:

test_dataset = load_data(
    data_df=test,
    tokenizer=tokenizer,
    text_cols=['text'],
    label_col='labels',
    label_list=data_args.column_info['label_list'],
    categorical_cols=None,
    numerical_cols=None,
    sep_text_token_str=tokenizer.sep_token,
)

I used the following setup during training combine_feat_method='gating_on_cat_and_num_feats_then_sum' and I use the Hugging Face trainer as suggested in the demo notebook.

I got the error in the tabular_combiner.py file during inference as follows:

File /usr/local/lib/python3.9/dist-packages/multimodal_transformers/model/tabular_combiner.py:441, in TabularFeatCombiner.forward(self, text_feats, cat_feats, numerical_feats)
    438     g_mult_num = 0
    440 H = g_mult_cat + g_mult_num + self.h_bias
--> 441 norm = torch.norm(text_feats, dim=1) / torch.norm(H, dim=1)
    442 alpha = torch.clamp(norm * self.tabular_config.gating_beta, min=0, max=1)
    443 combined_feats = text_feats + alpha[:, None] * H

File /usr/local/lib/python3.9/dist-packages/torch/functional.py:1501, in norm(input, p, dim, keepdim, out, dtype)
   1499 if p == "fro" and (dim is None or isinstance(dim, int) or len(dim) <= 2):
   1500     if out is None:
-> 1501         return torch.linalg.vector_norm(input, 2, _dim, keepdim, dtype=dtype)
   1502     else:
   1503         return torch.linalg.vector_norm(input, 2, _dim, keepdim, dtype=dtype, out=out)

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Appreciate your help in advance.
Thanks

Hi, when I tried to pip install it, I encountered problems. @codeKgu

  Created wheel for multimodal_transformers: filename=multimodal_transformers-0.1.4a0-py3-none-any.whl size=20633 sha256=38e7d2cec34c74fd0449e91c591ce8427e66e540bfba2969607951e07bb5ac16
  Stored in directory: /Users/caffrey/Library/Caches/pip/wheels/10/ca/98/03f280d4b1c8bb8ead2f4eb5df5b74bd9736ca209972990112
  Building wheel for tokenizers (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for tokenizers (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [51 lines of output]
      /private/var/folders/x0/z03_gg6j6_ggrj9yck7c6f6c0000gn/T/pip-build-env-9mte3dpz/overlay/lib/python3.9/site-packages/setuptools/dist.py:530: UserWarning: Normalizing '0.8.1.rc2' to '0.8.1rc2'
        warnings.warn(tmpl.format(**locals()))
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-11.0-arm64-cpython-39
      creating build/lib.macosx-11.0-arm64-cpython-39/tokenizers
      copying tokenizers/__init__.py -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers
      creating build/lib.macosx-11.0-arm64-cpython-39/tokenizers/models
      copying tokenizers/models/__init__.py -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers/models
      creating build/lib.macosx-11.0-arm64-cpython-39/tokenizers/decoders
      copying tokenizers/decoders/__init__.py -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers/decoders
      creating build/lib.macosx-11.0-arm64-cpython-39/tokenizers/normalizers
      copying tokenizers/normalizers/__init__.py -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers/normalizers
      creating build/lib.macosx-11.0-arm64-cpython-39/tokenizers/pre_tokenizers
      copying tokenizers/pre_tokenizers/__init__.py -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers/pre_tokenizers
      creating build/lib.macosx-11.0-arm64-cpython-39/tokenizers/processors
      copying tokenizers/processors/__init__.py -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers/processors
      creating build/lib.macosx-11.0-arm64-cpython-39/tokenizers/trainers
      copying tokenizers/trainers/__init__.py -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers/trainers
      creating build/lib.macosx-11.0-arm64-cpython-39/tokenizers/implementations
      copying tokenizers/implementations/byte_level_bpe.py -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers/implementations
      copying tokenizers/implementations/sentencepiece_bpe.py -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers/implementations
      copying tokenizers/implementations/base_tokenizer.py -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers/implementations
      copying tokenizers/implementations/__init__.py -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers/implementations
      copying tokenizers/implementations/char_level_bpe.py -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers/implementations
      copying tokenizers/implementations/bert_wordpiece.py -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers/implementations
      copying tokenizers/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers
      copying tokenizers/models/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers/models
      copying tokenizers/decoders/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers/decoders
      copying tokenizers/normalizers/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers/normalizers
      copying tokenizers/pre_tokenizers/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers/pre_tokenizers
      copying tokenizers/processors/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers/processors
      copying tokenizers/trainers/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-39/tokenizers/trainers
      running build_ext
      running build_rust
      info: syncing channel updates for 'nightly-2020-05-14-aarch64-apple-darwin'
      info: latest update on 2020-05-14, rust version 1.45.0-nightly (75e1463c5 2020-05-13)
      error: target 'aarch64-apple-darwin' not found in channel.  Perhaps check https://doc.rust-lang.org/nightly/rustc/platform-support.html for available targets
      error: can't find Rust compiler
      
      If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.
      
      To update pip, run:
      
          pip install --upgrade pip
      
      and then retry package installation.
      
      If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
  Building wheel for sacremoses (setup.py) ... done
  Created wheel for sacremoses: filename=sacremoses-0.0.53-py3-none-any.whl size=895260 sha256=e0c1c85b58c441c00e8acb472af863785045a148042632a128949c7ed94e6ba2
  Stored in directory: /Users/caffrey/Library/Caches/pip/wheels/12/1c/3d/46cf06718d63a32ff798a89594b61e7f345ab6b36d909ce033
Successfully built multimodal_transformers sacremoses
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

Hi I am trying to install this on my Apple M1 Pro. I am getting the following error. Tried installing rustc but it did not help. Any workaround is appreciated.

running build_ext
      running build_rust
      info: syncing channel updates for 'nightly-2020-05-14-aarch64-apple-darwin'
      info: latest update on 2020-05-14, rust version 1.45.0-nightly (75e1463c5 2020-05-13)
      error: target 'aarch64-apple-darwin' not found in channel.  Perhaps check https://doc.rust-lang.org/nightly/rustc/platform-support.html for available targets
      error: can't find Rust compiler
      
      If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.
      
      To update pip, run:
      
          pip install --upgrade pip
      
      and then retry package installation.
      
      If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

Since you mentioned you're considering enhancing load_data(), I might also try to expose to the user different methods for imputation of missing numeric data. Currently in data_utils.load_num_feats() this defaults to median imputation, but this can be a poor choice if the reason the data is missing is due to real differences in the data generating process (ie, NULL data actually followed a different process than non-NULL data, and is meaningfully distinct from non-NULL data). In that case, one might instead want to encode the missing data with a distinct value from the non-NULL distribution prior to modeling.

Hi, thanks for assembling this project. I was wondering if there were any plans to incorporate multi-label classification. The labels API on the forward pass would be ideally changed from a single dim tensor (batch_size) of correct class indexes to a 2 dim tensor of (batch_size, n_labels) with binary indicators to remain consistent. Thanks again

In the model/tabular_combiner.py script, everytime MLP is initialized the bn parameter is set to True, ignoring the value of self.numerical_bn (the parameter normally responsible for determining whether batch normalization is done or not). This causes most feature combination methods to end up using Batch Normalization even if the numerical_bn parameter was set to False in TabularConfig.
An easy fix is to simply do bn=self.numerical_bn each time MLP is called, so that the parameter is accounted for instead of using True everytime.

Hi there,

When I tried to load the train/val/test set csv file that I splitted with load_data_from_folder in multimodal_transformers.data, the returned train_dataset/val_dataset/test_dataset will give me a strange length, which is totally not related to the original length of the csv file.
for example, the train_df.shape = (105195,25), while the train_dataset.cat_feats.shape = (131495,38).

For spliting dataset, I tried train_test_split and np.split, but they both gave me the same issue with loading.

But if I followed the exact same code in the notebook for splitting datasets, load_data_from_folder would work well. At the same time, if I modify one column, such as match the number with words from [0,2,0...] to [A,B,A...], it also cannot load in the correct way.

Does anyone have any suggestions?

According to the Example usage in the documentation: https://multimodal-toolkit.readthedocs.io/en/latest/notes/introduction.html
the TabularConfig class has an init parameter called use_num_bn (in the example, it is set to False) for deciding whether to use
Batch Normalization. In the actual TabularConfig class however there is no such parameter, instead there is a numerical_bn parameter with a default value of True. There is also a kwargs parameter.

Right now, if someone follows the example code, they will mistakenly -and probably unknowingly- end up using Batch Normalization, because the use_num_bn parameter will not actually do anything (it will just be absorbed as a kwarg), and instead the default True value of numerical_bn will take effect.

The TabularConfig class's documentation is correct, only the Example section has this mistake. I advise updating the Example code with the numerical_bn parameter instead to avoid misunderstandings.

Dear team, 

I am doing several experiments with the toolkit. Thank you very much for the good work. While trying XLNet with the combination module "weighted_feature_sum_on_transformer_cat_and_numerical_feats". I got the following error:

230             self.act_func = create_act(self.mlp_act)
231             self.layer_norm = nn.LayerNorm(self.text_out_dim)

--> 232 self.final_dropout = nn.Dropout(tabular_config.hidden_dropout_prob)
233 self.final_out_dim = self.text_out_dim
234
AttributeError: 'TabularConfig' object has no attribute 'hidden_dropout_prob'

I think in model initialization this configuration is missing. Please update the code.
tabular_config.hidden_dropout_prob = hf_model_config.hidden_dropout_prob

Describe the bug
When trying to run main.py with my own config JSON file, as it is advised in the README, the execution is aborted with an error
AttributeError: 'OurTrainingArguments' object has no attribute 'deepspeed_plugin'.

To Reproduce
Steps to reproduce the behavior:

1. Create new virtualenv (thus we get a clean slate).
2. Clone repository containing main.py from GitHub.
3. Install the module with pip: pip install multimodal-transformers
4. Pass an appropriate JSON file to the scrip and run main.py:
   python /content/Multimodal-Toolkit/main.py /content/data/config.json
5. See error

Expected behavior
The selected model should be trained with the configuration given in the JSON.

Desktop:

• OS: Windows 11
• Python: 3.10.12
• Environment: Colab

In data_utils.CategoricalFeatures._one_hot(), could you let the handle_unknown argument to sklearn's OneHotEncoder be passed through to the user, so that they have the option to specify handle_unknown='ignore'? As-is, the code and example notebook becomes problematic in the common case where train versus validation or test sets experience different distinct levels for a categorical variable. In this case, we cannot score the model trained on the train set on the test or validation set because the different number of levels in the categorical variable will cause an error. This happens when you have rare categorical levels.

Another option is to use load_data() on the entire modeling dataset and only then split it into train-test-val later, but this seems not straightforward, at least if you have pre-existing indices for the train-test-val sets (given you are trying to split a pytorch Dataset).

Adding support for using an LLM instead of the current models would be nice to have.

I believe there is a typo under docs/source/notes/introduction.rst on line 98.

model_inputs['num_feats'] = numerical_feat

should be

model_inputs['numerical_feats'] = numerical_feat

if we are not explicitly specifying the features in the forward pass. The forward pass defined for the Tabular models takes a numerical_feats parameter, not num_feats. So in the example given, it would compute the forward pass with None for the numeric features.

Longer snippet from introduction.rst below:

model_inputs['cat_feats'] = categorical_feat
model_inputs['num_feats'] = numerical_feat
model_inputs['labels'] = labels

loss, logits, layer_outs = model(**model_inputs)

There is a bug in the method "attention_on_cat_and_numerical_feats", where if you have both categorical and numerical features, the MLP defined as self.cat_mlp gets overwritten instead of initializing self.num_mlp.

Hi there,

I have been using the load_data function to create my datasets for HuggingFace's Trainer class, which has worked really well so far. However, I am not sure how I should approach inferencing without a label column which load_data requires. Would really appreciate some advice as I am rather new to this. Thank you!

I found that the multimodal-toolkit do not have embedding,it just use onehot encoding to deal with category,which leading to the too high dimension and caused memory error.If any one can help??

I arbitrarily decided to go from 0.1.4-alpha to 0.2-alpha for the latest release. Would like some suggestions on the best way to go about with versioning for future releases.

Thanks for great library, a question, where should the date column go to?

I was trying to reproduce the example notebook
https://colab.research.google.com/github/georgianpartners/Multimodal-Toolkit/blob/master/notebooks/text_w_tabular_classification.ipynb#scrollTo=ABT1hK9cRsuk
and got the error

RuntimeError Traceback (most recent call last)
in

~/anaconda3/lib/python3.7/site-packages/transformers/trainer.py in train(self, model_path, trial)
750
751 if self.args.evaluate_during_training and self.global_step % self.args.eval_steps == 0:
--> 752 metrics = self.evaluate()
753 self._report_to_hp_search(trial, epoch, metrics)
754

~/anaconda3/lib/python3.7/site-packages/transformers/trainer.py in evaluate(self, eval_dataset)
1155 eval_dataloader = self.get_eval_dataloader(eval_dataset)
1156
-> 1157 output = self.prediction_loop(eval_dataloader, description="Evaluation")
1158
1159 self.log(output.metrics)

~/anaconda3/lib/python3.7/site-packages/transformers/trainer.py in prediction_loop(self, dataloader, description, prediction_loss_only)
1236 samples_count = 0
1237 for inputs in tqdm(dataloader, desc=description, disable=disable_tqdm):
-> 1238 loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only)
1239 batch_size = inputs[list(inputs.keys())[0]].shape[0]
1240 samples_count += batch_size

~/anaconda3/lib/python3.7/site-packages/transformers/trainer.py in prediction_step(self, model, inputs, prediction_loss_only)
1327
1328 with torch.no_grad():
-> 1329 outputs = model(**inputs)
1330 if has_labels:
1331 loss, logits = outputs[:2]

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
725 result = self._slow_forward(*input, **kwargs)
726 else:
--> 727 result = self.forward(*input, **kwargs)
728 for hook in itertools.chain(
729 _global_forward_hooks.values(),

~/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
159 return self.module(*inputs[0], **kwargs[0])
160 replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
--> 161 outputs = self.parallel_apply(replicas, inputs, kwargs)
162 return self.gather(outputs, self.output_device)
163

~/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py in parallel_apply(self, replicas, inputs, kwargs)
169
170 def parallel_apply(self, replicas, inputs, kwargs):
--> 171 return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
172
173 def gather(self, outputs, output_device):

~/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py in parallel_apply(modules, inputs, kwargs_tup, devices)
84 output = results[i]
85 if isinstance(output, ExceptionWrapper):
---> 86 output.reraise()
87 outputs.append(output)
88 return outputs

~/anaconda3/lib/python3.7/site-packages/torch/_utils.py in reraise(self)
426 # have message field
427 raise self.exc_type(message=msg)
--> 428 raise self.exc_type(msg)
429
430

RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
output = module(*input, **kwargs)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/multimodal_transformers/model/tabular_transformers.py", line 116, in forward
numerical_feats)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/multimodal_transformers/model/tabular_combiner.py", line 426, in forward
cat_feats = self.cat_layer(cat_feats)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/multimodal_transformers/model/layer_utils.py", line 52, in forward
layer_inputs.append(layer(input))
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 93, in forward
return F.linear(input, self.weight, self.bias)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 1690, in linear
ret = torch.addmm(bias, input, weight.t())
RuntimeError: mat1 dim 1 must match mat2 dim 0

I did the following modifications to the notebook which I don't think would cause the error:

Change 1: Use load_data to create train, val and test datasets

# Get Datasets
train_dataset = load_data(
    df_train,
    data_args.column_info['text_cols'],
    tokenizer,
    label_col=data_args.column_info['label_col'],
    label_list=data_args.column_info['label_list'],
    categorical_cols=data_args.column_info['cat_cols'],
    numerical_cols=data_args.column_info['num_cols'],
    sep_text_token_str=tokenizer.sep_token,
)

val_dataset = load_data(
    df_val,
    data_args.column_info['text_cols'],
    tokenizer,
    label_col=data_args.column_info['label_col'],
    label_list=data_args.column_info['label_list'],
    categorical_cols=data_args.column_info['cat_cols'],
    numerical_cols=data_args.column_info['num_cols'],
    sep_text_token_str=tokenizer.sep_token,
)

test_dataset  = load_data(
    df_test,
    data_args.column_info['text_cols'],
    tokenizer,
    label_col=data_args.column_info['label_col'],
    label_list=data_args.column_info['label_list'],
    categorical_cols=data_args.column_info['cat_cols'],
    numerical_cols=data_args.column_info['num_cols'],
    sep_text_token_str=tokenizer.sep_token,
)

change 2: didn't use cache_dir since Colab Example didn't specify it and the default is None

tokenizer = AutoTokenizer.from_pretrained(
    model_args.model_name_or_path
)

config = AutoConfig.from_pretrained(model_args.model_name_or_path)

Hi

can you share the code related to regression with price prediction on melbounre airbnb dataset?

In https://github.com/georgian-io/Multimodal-Toolkit/blob/master/multimodal_transformers/model/tabular_transformers.py#L10, the statements
from transformers.modeling_bert import BERT_INPUTS_DOCSTRING (and others)
lead to an error with the most recent transformers package, and should be changed to:
from transformers.models.bert.modeling_bert import BERT_INPUTS_DOCSTRING etc