Unable to use neuron sdk to compile GPT2 model about aws-neuron-sdk

[mrnikwaws]

Hi Damiox,

Thanks for your question. In torch.neuron.trace we use torch.jit.trace to generate a graph of operators that get compiled for neuron hardware. The following modification to your code shows the pytorch operators which are not supported. These are prefixed by aten:: in the nodes of the generated graph

from transformers.tokenization_gpt2 import GPT2Tokenizer
from transformers.modeling_gpt2 import GPT2LMHeadModel
import torch
import torch_neuron

# loading gpt2 medium model
tokenizer = GPT2Tokenizer.from_pretrained('gpt2', pad_token='<|endoftext|>')
model = GPT2LMHeadModel.from_pretrained('gpt2',torchscript=True)
model.eval()

# generating example input
tokens = [tokenizer.encode(t) for t in ['I like to drink coke']]
tensors = torch.LongTensor(tokens)

# using neuron sdk to compile the model
model_jit = torch.jit.trace(model, example_inputs=[tensors])

#print( model_jit.graph )

## Get the operators in the model
operators_in_model = set()

for node in model_jit.graph.nodes():
    if node.kind().startswith("aten"):
        operators_in_model.add( node.kind() )

## Get the supported operations in the current version of torch-neuron
supported_operators = set(torch.neuron.get_supported_operations())

print("The following operations are currently supported in torch-neuron")

for op in supported_operators:
    print(op)

not_supported_operators = operators_in_model - supported_operators

print()
print("The following operations are currently not supported in torch-neuron for this model")
for op in not_supported_operators:
    print(op)

#model_neuron = torch.neuron.test_trace(model, example_inputs=[tensors])

There is more information in the pytorch github repo here: https://github.com/pytorch/pytorch/wiki/PyTorch-IR if you are interested.

I'll also find out more about what we have tested for GPT2 on tensorflow-neuron and get back to you here.

[Damiox]

Alternatively, I also tried to re-save the pytorch model to run a clean test, so I loaded my model and re-save it as follows:

from transformers.tokenization_gpt2 import GPT2Tokenizer
from transformers.modeling_gpt2 import GPT2LMHeadModel
import torch

# loading gpt2 medium model
tokenizer = GPT2Tokenizer.from_pretrained('gpt2', pad_token='<|endoftext|>')
model = GPT2LMHeadModel.from_pretrained('data/GPT2-345M/', torchscript=True)
model.eval()

# generating input
tokens = [tokenizer.encode(t) for t in ['I like to drink coke']]
tensors = torch.LongTensor(tokens)
print(tensors)

# re-saving model
torch.save(model, 'gpt2-serial.pt2')

Then I tried to use that gpt2-serial.gpt2 model with neuron-sdk similarly to the steps detailed in https://github.com/aws/aws-neuron-sdk/blob/master/docs/pytorch-neuron/tutorial-compile-infer.md :

import torch
import torch_neuron


# loading gpt2 medium model
model = torch.load('gpt2-serial.pt2')
model.eval()

# generating example input
tensors = torch.LongTensor([[40,  588,  284, 4144,  763,  365]])
print(tensors)

# using neuron sdk to compile the model
model_neuron = torch.neuron.trace(model, example_inputs=[tensors])

model_neuron.save('gpt2-serial-neuron.pt2')

And ran into the same issue:

Traceback (most recent call last):
  File "neuron-gpt2-next.py", line 17, in <module>
    model_neuron = torch.neuron.trace(model, example_inputs=[tensors])
  File "/home/prod/neuron/lib/python3.5/site-packages/torch_neuron/decorators.py", line 150, in trace
    transform_torch_graph_to_tensorflow( func, example_inputs, args, kwargs )
  File "/home/prod/neuron/lib/python3.5/site-packages/torch_neuron/decorators.py", line 294, in transform_torch_graph_to_tensorflow
    tensor_outputs = _resolve_func(node)(op, *tensor_inputs)
TypeError: arange() takes from 2 to 6 positional arguments but 8 were given

[mrnikwaws]

Hi Damiox,

Thanks for reporting this issue. Right now, Neuron-torch does not support all of the required operators for GPT2. This error message should be improved and we have opened an internal issue to track it.

I did a quick test which bypasses that failure, and there are other operators that need to be added to the Neuron-torch supported set. Please keep your eyes open for future announcements on operator support and any explicit release notes on GPT2

[Damiox]

Hi. Thanks for the feedback. Could you please elaborate more on which "operators" are not supported and if there's anything that can be fixed in the https://github.com/huggingface/transformers repo to make it work? Additionally, I would like to ask if you have a benchmark available to share with me so I can check the improvements between inf1.xl and g4dn.xl machines. At which scale there's a 3x throughput increase by using inf1 machines? Thanks Damian Nardelli El El lun, 2 mar. 2020 a la(s) 20:59, mrnikwaws <[email protected]> escribió:

…

Hi Damiox, Thanks for reporting this issue. Right now, Neuron-torch does not support all of the required operators for GPT2. This error message should be improved and we have opened an internal issue to track it. I did a quick test which bypasses that failure, and there are other operators that need to be added to the Neuron-torch supported set. Please keep your eyes open for future announcements on operator support and any explicit release notes on GPT2 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#88>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAESN7GD6YDDRUI62I2SJWTRFRB4XANCNFSM4K7ASFTA> .

[Damiox]

Could you please clarify if GPT2+Tensorflow should work with neuron sdk? I'm not sure about it.... Thanks Damian Nardelli

…

On Tue, Mar 3, 2020 at 9:46 AM Damian Nardelli ***@***.***> wrote: Hi. Thanks for the feedback. Could you please elaborate more on which "operators" are not supported and if there's anything that can be fixed in the https://github.com/huggingface/transformers repo to make it work? Additionally, I would like to ask if you have a benchmark available to share with me so I can check the improvements between inf1.xl and g4dn.xl machines. At which scale there's a 3x throughput increase by using inf1 machines? Thanks Damian Nardelli El El lun, 2 mar. 2020 a la(s) 20:59, mrnikwaws ***@***.***> escribió: > Hi Damiox, > > Thanks for reporting this issue. Right now, Neuron-torch does not support > all of the required operators for GPT2. This error message should be > improved and we have opened an internal issue to track it. > > I did a quick test which bypasses that failure, and there are other > operators that need to be added to the Neuron-torch supported set. Please > keep your eyes open for future announcements on operator support and any > explicit release notes on GPT2 > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#88>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAESN7GD6YDDRUI62I2SJWTRFRB4XANCNFSM4K7ASFTA> > . >

[Damiox]

Would the GPT2 architecture work in Neuron with Tensorflow? Also do you have any benchmark to take a look about the performance improvements by using inf1 machines?

[jeffhataws]

We are still working on TensorFlow 2.0 which is requirement for GPT2. For a sample of Inferentia capabilities, please take a look at our ResNet50 example (https://github.com/aws/aws-neuron-sdk/blob/master/docs/technotes/performance-tuning.md) and Bert example (https://github.com/aws/aws-neuron-sdk/tree/master/src/examples/tensorflow/bert_demo).

[aws-taylor]

Hello Damian,

Is there anything else with which we can help?

-Taylor

[Damiox]

I'm waiting for the new required operators to be available in neuron-sdk. Will this be announced? In the meantime I'm not using this framework

[aws-taylor]

Hello Damian,

I would encourage you to follow our PyTorch release notes for release announcements. You can also watch our roadmap.

-Taylor

[aws-taylor]

Hello Damian,

It appears your immediate questions have been addressed. Please feel free to re-open if you have any further questions.

Regards,
Taylor

[Damiox]

Hey @aws-taylor I see there has been a new release from aws-neuron-sdk ...
I was trying to check whether the unsupported pytorch operators were already supported by neuron-sdk. I can't find details in https://github.com/aws/aws-neuron-sdk/projects/2 to track the progress of this ticket.

[aws-zejdaj]

Resolved, we tested GPT-2 so the fix is coming in the next neuron-cc release. Please reopen if any issues . Thanks for your patience.