Can't run inference on a batch larger then 1 (resnet-50 tutorial) about aws-neuron-sdk

[mrnikwaws]

Hi ebochkov,

Thanks for the feedback. Unfortunately this is a known issue the team is working on: https://github.com/aws/aws-neuron-sdk/blob/master/docs/technotes/performance-tuning.md#compiling-for-batching-optimization (note 2). At the moment only the float32 options is available for PyTorch, leading to this restriction.

I'll comment back on this issue with any updates.

[aws-taylor]

Hello ebochkov,

I just wanted to let you know that we have not forgotten about you. While we do not have any fixes yet, we are hard at work with some performance and correctness fixes that we expect to improve our batching performance. We'll continue to track this issue and update as we have results.

-Taylor

[aws-zejdaj]

Still working on Pytorch batching support

[mrnikwaws]

Hi ebochkov,

ResNet-50 is now running with batch sizes of greater than 1! In my testing I see the best results when I run compilation and inference using batch 5 for 224 x 224 images, but you should feel free to experiment. The team is working on updating some of the tutorial material for torch-neuron, but I wanted to let you know so that you can rerun your previous tests if you want to.

You can change the existing "compile-infer" tutorial for compilation by using:

batch_size = 5
image = torch.zeros([batch_size, 3, 224, 224], dtype=torch.float32)

in place of

image = torch.zeros([1, 3, 224, 224], dtype=torch.float32)

for compilation, and for inference change:

image = torch.tensor(image.numpy()[np.newaxis, ...])

## Load model
model_neuron = torch.jit.load( 'resnet50_neuron.pt' )

## Predict
results = model_neuron( image )

to

image = torch.tensor(image.numpy()[np.newaxis, ...])
batch_size = 5

## Create a "batch" image with enough images to go on each of the four cores
batch_image = image

for i in range(batch_size - 1):
    batch_image = torch.cat( [batch_image, image], 0 )

## Since the first inference also loads the model to the chip let's exclude it 
## from timing
results = model_neuron( batch_image )

This will simply make 5 copies of the image in the batch to test. Note that the tensor sizes for compilation and inference must match.

I am going to keep this issue open for now until I hear back from you or post a link to the updated tutorial information.

[mrnikwaws]

The tutorial for ResNet-50 is now updated with the guidance requested here - closing this issue. Please re-open if you are still having problems.