yolov8-seg output transpose (0,3,1,2) about onnx2tf

[PINTO0309]

All you have to do is break up the model before the final layer, Transpose.

Read the README so thoroughly that it has holes in it.

https://github.com/PINTO0309/onnx2tf#2-run-test

# Split the model at the middle position for debugging
# Specify the output name of the OP
$ onnx2tf -i resnet18-v1-7.onnx -onimc resnetv15_stage2_conv1_fwd resnetv15_stage2_conv2_fwd

[PINTO0309]

Simply extrapolate Transpose to the back of the final layer.

https://github.com/PINTO0309/onnx2tf#parameter-replacement

https://github.com/PINTO0309/onnx2tf/tree/main/json_samples

[ChenLiangChong]

Hello, it seems like my previous response did not address my issue.

In the context of YOLOv8, the final output tensor shape is (1, 84, 8400) for my project. In YOLOv8-seg, the original output shape is (1, 116, 8400) and (1, 32, 160, 160).

The YOLOv8 official implementation also handles this special case in ultralytics/ultralytics/nn/autobackend.py, line 444.

            # TF segment fixes: export is reversed vs ONNX export and protos are transposed
            if len(y) == 2:  # segment with (det, proto) output order reversed
                if len(y[1].shape) != 4:
                    y = list(reversed(y))  # should be y = (1, 116, 8400), (1, 160, 160, 32)
                y[1] = np.transpose(y[1], (0, 3, 1, 2))  # should be y = (1, 116, 8400), (1, 32, 160, 160)
            y = [x if isinstance(x, np.ndarray) else x.numpy() for x in y]

In the model_zoo tflite provided by me, the shapes are (1, 116, 8400) and (1, 160, 160, 32), but in ONNX, it's (1, 116, 8400) and (1, 32, 160, 160). This means that the mask part is automatically transposed during conversion.

I am wondering if it's possible to add a transpose operation just before the last layer to handle this special case. When deploying, I want to output the original shapes (1, 116, 8400) and (1, 32, 160, 160) for efficiency. I have ten outputs for reg, cls, ms, and seg, respectively:

(1, 80, 80, 80) (1, 80, 40, 40) (1, 80, 20, 20)
(1, 1, 4, 6400) (1, 1, 4, 1600) (1, 1, 4, 400)
(1, 32, 80, 80) (1, 32, 40, 40) (1, 32, 20, 20)
(1, 32, 160, 160)

However, after converting to TFLite, the shapes become:

(1, 80, 80, 80) (1, 40, 40, 80) (1, 20, 20, 80)
(1, 6400, 4, 1) (1, 1600, 4, 1) (1, 400, 4, 1)
(1, 80, 80, 32) (1, 40, 40, 32) (1, 20, 20, 32)
(1, 160, 160, 32)

[ChenLiangChong]

Hello, I have already attempted to extrapolate the Transpose operation to the back of the final layer.
However I encountered some problems, and the output of the onnx model compared to the TFLite output doesn't seem to indicate that the operation was successful.

I think the reg3 in onnx and Identity_4296 in tflite is the same output.
but i still got (1,1,4,400) and (1,4,400,1)

I suspect there might be an issue with how I wrote my JSON file. The JSON file I used is shown below.

{
  "format_version": 1,
  "operations": [
    {
      "op_name": "reg1",
      "param_target": "outputs",
      "param_name": "reg1",
      "post_process_transpose_perm": [0, 3, 2, 1]
    },
	    {
      "op_name": "cls1",
      "param_target": "outputs",
      "param_name": "cls1",
      "post_process_transpose_perm": [0, 3, 2, 1]
    },
	    {
      "op_name": "reg2",
      "param_target": "outputs",
      "param_name": "reg2",
      "post_process_transpose_perm": [0, 3, 2, 1]
    },
	    {
      "op_name": "cls2",
      "param_target": "outputs",
      "param_name": "cls2",
      "post_process_transpose_perm": [0, 3, 2, 1]
    },
	    {
      "op_name": "reg3",
      "param_target": "outputs",
      "param_name": "reg3",
      "post_process_transpose_perm": [0, 3, 2, 1]
    },
	    {
      "op_name": "cls3",
      "param_target": "outputs",
      "param_name": "cls3",
      "post_process_transpose_perm": [0, 3, 2, 1]
    },
	    {
      "op_name": "mc1",
      "param_target": "outputs",
      "param_name": "mc1",
      "post_process_transpose_perm": [0, 3, 2, 1]
    },
	    {
      "op_name": "mc2",
      "param_target": "outputs",
      "param_name": "mc2",
      "post_process_transpose_perm": [0, 3, 2, 1]
    },
		    {
      "op_name": "mc3",
	  "param_target": "outputs",
      "param_name": "mc3",
      "post_process_transpose_perm": [0, 3, 2, 1]
    },
		    {
      "op_name": "seg",
      "param_target": "outputs",
      "param_name": "seg",
      "post_process_transpose_perm": [0, 3, 2, 1]
    }
  ]
}

Can you help me confirm what adjustments are needed? Thank you.

[ChenLiangChong]

I attempted to add a transpose layer at the end of an ONNX model, but after the conversion, that layer was directly removed.