Converter for pytorch models that have multiple inputs about tinyneuralnetwork

[peterjc123]

@bayleef1 We do support models with multiple inputs. You need to pass the inputs as a tuple. e.g.

input_1 = torch.randn(1, 3, 224, 224)
input_2 = torch.randn(1, 3, 224, 224)
converter = TFLiteConverter(model, (input_1, input_2), ...)
converter.convert()

[bayleef1]

OK, I successfully converted pytorch model with multiple inputs and outputs that exclude lstm state parameters.
When I tried to update lstm state by adding parameters in function 'forward(self, lstm_h, lstm_c, ...)' and 'output, (lstm_h, lstm_c) = self.lstm(input, (lstm_h, lstm_c))', I got the error message:
assert len(missing_outputs) == 0, f'Some output nodes are missing: {missing_outputs}'
AssertionError: Some output nodes are missing: ['hx1.1', 'hx2.1']
I wonder if the converter could solve this properly when both model inputs and outputs include lstm state.

[peterjc123]

@bayleef1 Yes, it's due to a limitation that you cannot read or write variables in the TFLite graph. So you need to export the model without adding them to inputs or outputs. But you can still use them through the variable mechanism.
For example, consider the following subgraph.

You are free to access them through set_tensor and get_tensor, given you've found out the locations of the state tensors.

import tensorflow as tf
import numpy as np

interpreter = tf.lite.Interpreter(model_path='xxx.tflite')
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
tensor_details = interpreter.get_tensor_details()
tensor_shapes = {d['index']: d['shape'] for d in tensor_details}

# set inputs
for i in range(len(input_details)):
    interpreter.set_tensor(input_details[i]['index'], np.random.random(input_details[i]['shape']))

# set states
state_tensors = [3, 21]
for i in state_tensors:
    interpreter.set_tensor(i, np.zeros(tensor_shapes[i]))

# invoke
interpreter.invoke()

# get outputs
outputs = []
for i in range(len(output_details)):
    outputs.append(interpreter.get_tensor(output_details[i]['index']))

# get states
states = []
for i in state_tensors:
    states.append(interpreter.get_tensor(i))

[peterjc123]

https://github.com/alibaba/TinyNeuralNetwork/blob/main/docs/FAQ.md#how-to-convert-a-model-with-lstm
https://github.com/alibaba/TinyNeuralNetwork/blob/main/docs/FAQ_zh-CN.md#%E7%AE%97%E5%AD%90%E4%B8%8D%E6%94%AF%E6%8C%81%E5%A6%82%E4%BD%95%E5%A4%84%E7%90%86