Help converting and using custom models about onnxstream

[vitoplantamura]

Hey, you're not too far from the end result!! The problem is that OnnxStream expects inputs to have "fixed" shapes. More details here: https://onnxruntime.ai/docs/tutorials/mobile/helpers/make-dynamic-shape-fixed.html Onnx Simplifier has an argument that allows you to make an input shape "fixed" (--overwrite-input-shape). Personally I prefer to use the utility explained in the previous link and then run Onnx Simplifier. You can use Netron to see the name of your model inputs and their shape. The SD 1.5 UNET has 3 inputs, with the following shapes: samples -> (1,4,64,64) timestep -> (1) encoder_hidden_states -> (1,77,768) Once the input shapes are fixed, you can run Onnx Simplifier and finally my onnx2txt! Give it a try and let me know if everything is clear or if you have any problems! Thanks, Vito

[GaelicThunder]

Vito, you mad lad, it worked!
I had a 128Gb swap external SSD that i used for swap for generating bigger images on my raspberry pi project , which is nothing fancy like this project!
I was thinking, maybe i can help and write a "guide" on this? Maybe it helps others (i saw some question popping of in the issues)

[vitoplantamura]

hahaha, great! yes please if you can: it's an interesting topic, I imagine for other people too :-) Of course I will put a link to your guide in the project's README. Thanks! Vito

[GaelicThunder]

Oh that is out of my knowledge to be completely honest... We might as well ask Vito if he knows how to fix it, if another model doesnt work

[elliot-sawyer]

All good, part of the fun of learning new stuff! Thanks heaps for your input

[GaelicThunder]

Thanks a lot for the kind explanation!
I tried to follow the explanations, so i tried some commands:

python -m onnxruntime.tools.make_dynamic_shape_fixed --input_name x --input_shape 1,4,64,64 model.onnx model_fixed1.onnx

python -m onnxruntime.tools.make_dynamic_shape_fixed --input_name context --input_shape 1,77,768 mode_fixed1.onnx model_fixed2.onnx (there is no encoder_hidden_states, only x, timesteps and context in my model, is it not 1.5??? Maybe its 1.4???)

python -m onnxruntime.tools.make_dynamic_shape_fixed --input_name timesteps --input_shape 1 model_fixed2.onnx model_fixed3.onnx

then i run Onnx Simplifier > go on the notebook and i get the usual bunch of files.

Now i have the Converted folder with inside model.txt and a lot on .bins
If i try to use it, i get the same error from before:

#20
I tried to fix it like it was suggested in the issue, but i end up having much more errors like that:
=== ERROR === read_file: unable to open file (/home/d00l1n/Desktop/Converted_A0/text_encoder_fp32/model.txt).

=== ERROR === read_file: unable to open file (/home/d00l1n/Desktop/Converted_A0/unet_fp16/model.txt).
I tried to fix them in the same way as the issue number 20 and it kinda worked?

Then i tried to put the converted files into the unet folder but i get this:
=== ERROR === Model::parse_tensor_string: invalid shape.

I moved my converted model into the unet_fp16, and used the other folders from the normal SD 1.5 in the Windows release of OnnxStream.
Im sure im doing a mess somewhere.

The command i use is this btw:
./sd --models-path ./Converted/ --prompt "space landscape" --steps 28 --rpi

Thanks again for the help and the patience!

[vitoplantamura]

The procedure you followed is correct. The fact that the names of the inputs differ from those I indicated depends on how the UNET model was exported (through the torch.onnx.export API). But it is a "cosmetic" difference. In order to understand what is happening, can you post the output of onnx2txt? Thanks, Vito

[GaelicThunder]

Sure, here is the output!
Unsqueeze -> 532
Cast -> 286
Mul -> 313
Cos -> 1
Sin -> 1
Concat -> 261
Gemm -> 24
Sigmoid -> 47
Conv -> 98
Reshape -> 441
InstanceNormalization -> 61
Add -> 265
Transpose -> 160
LayerNormalization -> 48
MatMul -> 158
Split -> 1
Einsum -> 64
Softmax -> 32
Shape -> 213
Gather -> 417
Div -> 125
Slice -> 32
Erf -> 16
Resize -> 3
TOTAL -> 3599

[vitoplantamura]

The presence of the "Shape" operator indicates that Onnx Simplifier is not doing its job. In my experience, this is not caused by Onnx Simplifier, but more likely by Onnx's Shape Inference. In any case, there remains only one alternative: re-export the model by modifying the parameters of torch.onnx.export. In your case, locate this file on your computer: https://github.com/AUTOMATIC1111/stable-diffusion-webui-tensorrt/blob/master/export_onnx.py and make sure that: - opset_version is 14 - remove dynamic_axes Once this is done, you can try again with Onnx Simplifier and my onnx2txt. If you decide to proceed, please send me the final output of onnx2txt. Thanks, Vito

[vitoplantamura]

just a question: is this model published on Hugging Face? So I can take a look at it too Thanks, Vito

[GaelicThunder]

Sorry for the late answer, i didn't have time.
So, i tried with the export_onnx.py modify, but i run into 1 problem: my laptop which has 6Gb of Vram and (this time) couldnt convert it (without the edit it can make it, idk why).
So i tried with my Desktop, but i got a Radeon VII so i had to make the extension run on CPU.

The script worked and converted it, but this time i didnt had onlt the .onnx but i got a .data aswell.
Owever, Onnx Simplifier and onnx2txt showed the "Shape" operator.

Unsqueeze -> 513 Mul -> 357 Cos -> 1 Sin -> 1 Concat -> 253 Gemm -> 24 Sigmoid -> 47 Conv -> 98 Reshape -> 441 InstanceNormalization -> 61 Add -> 361 Transpose -> 160 ReduceMean -> 96 Sub -> 48 Pow -> 48 Sqrt -> 48 Div -> 170 MatMul -> 156 Split -> 2 Einsum -> 64 Softmax -> 32 Shape -> 194 Gather -> 402 Slice -> 32 Erf -> 16 Resize -> 3 TOTAL -> 3628

So i retried converting using export_onnx with forcing the
x = torch.randn(1, 4, 16, 16).to(devices.device, devices.dtype)
timesteps = torch.zeros((1,)).to(devices.device, devices.dtype) + 500
context = torch.randn(1, 77, 768).to(devices.device, devices.dtype)
to the right values. (idk if was a good thing, but i wanted to test it myself before going back asking for help!)
BUT i got the same result, for some reason...

Now, im gonna upload the model so you can take a look.
Thanks a lot, really, you are helping me so much!

[vitoplantamura]

hi, An important thing that unfortunately came to mind only now: I always use Linux (Ubuntu) when I run Onnx Simplifier. I remember it causing a lot of problems on Windows. Windows WSL should also be fine. I noticed some Einsums among the operations performed by the SD implementation of AUTO1111. Einsum is not yet supported by OnnxStream. So I think it is more useful to try to export the ONNX from the Hugging Face implementation of SD, which is the one I have always used. I tried this code and it exports your model correctly: #--------------- from diffusers import StableDiffusionPipeline import torch pipe = StableDiffusionPipeline.from_single_file( " https://huggingface.co/GaelicThunder/CustomModel/blob/main/Model.safetensors " ) dummy_input = ( torch.randn(1, 4, 64, 64), torch.randn(1), torch.randn(1, 77, 768)) input_names = [ "sample", "timestep", "encoder_hidden_states" ] output_names = [ "out_sample" ] torch.onnx.export(pipe.unet, dummy_input, "/home/vito/Downloads/unet_temp.onnx", verbose=False, input_names=input_names, output_names=output_names, opset_version=14, do_constant_folding=True, export_params=True) #--------------- After running this code (in a Notebook for example), you can run Onnx Simplifier (in Linux) and onnx2txt (also in Windows if you want). Let me know! Thanks, Vito

[GaelicThunder]

Sorry it took me this much time to answer!
I was trying to get make it work on my Desktop Radeon VII but i got nothing if problems. Im waiting for a GPU on Colab (google not giving one to me for some time now) and ill get back to you!
Agai, sorry for the 5 day late reply!

[vitoplantamura]

I don't think you will be able to run the code in my previous message or (especially) Onnx Simplifier in Google Colab's 12GB of RAM :-) IIRC Onnx Simplifier required 75GB of RAM for SD 1.5's UNET! The best solution is Windows WSL or a virtual machine with Linux (VirtualBox), adding 100GB of swap space. I understand that this is a convoluted procedure but unfortunately I don't see any alternatives :-( Thanks, Vito

[GaelicThunder]

I did a pull request #36, if that is not ok i will do a separate guide for you to link! Thanks again!

[elliot-sawyer]

@GaelicThunder, you provide an example on how to run the onnx2txt.ipynb script to convert a safetensors model to an onnx format suitable for a Pi4? I've been following along with the dreamshaper_8 model and while it does process to a dreamshaper_8.onnx file and a bunch of bias's and weights in a folder, it won't run on the pi because it's missing tokenizer/vocab.txt. onnx2txt seems to be the missing link there, but I'm not sure how to use the ipynb file to process it.

If you're not using Dreamshaper 8, it'd be good to see an example of how you got this to work using a model other than the base ones. Thanks!

[GaelicThunder]

Hi @elliot-sawyer, sorry im a little lost, are you able to run onnx2txt? For the missing tokenizer/vocab.txt , have you extracted the base SD 1.5 from here? After you extract the base 1.5 SD weights and overwrite them with the one you get from onxx2txt. This should fix that error ( i say should cause the expert is Vito, not me, im just an enthusiast eheh )

[elliot-sawyer]

I'm not sure how to run onnx2txt at all or even sure what it does. I'm just trying to use different models other than the base on my pi - the base SD 1.5 example is working perfectly :)

Thanks anyway for your help!

[GaelicThunder]

Oh my, now i get it! Ok so, from the instructions:
This code can run a model defined in the path_to_model_folder/model.txt: (all the model operations are defined in the model.txt text file; OnnxStream expects to find all the weights files in that same folder, as a series of .bin files)/ The model.txt file contains all the model operations in ASCII format, as exported from the original ONNX file. Each line corresponds to an operation: for example this line represents a convolution in a quantized model: In order to export the model.txt file and its weights (as a series of .bin files) from an ONNX file for use in OnnxStream, a notebook (with a single cell) is provided (onnx2txt.ipynb).
Basically, you can open the ipynb on Colab or wherever you run your Notebook and change:

ONNX_FILENAME = "<ONNX_FILENAME.onnx>"
DEST_FOLDER = "<DEST_FOLDER>"

this to what suits your case.

The programm will output a bunch of files that you will have to put in the folder as i said earlier, on top of the other 1.5 SD.

Let me know how it goes, and tell me if you managed to do the Onnx Simplifier part, since you didn't talk about it!

[elliot-sawyer]

Sweet! I'll another go with OnnxSimplifier. I did try it and if I recall it spit out some warnings, but a ton of files were generated. I believe the only thing missing was the vocab.txt file, which prevented it from running on the Pi. I wasn't very thorough recording my progress, so I'll have another crack and report back.

When you wrote these instructions, did you try it with a custom model other than the base SD models? If I try to follow the same one you used, I might be able to eventually spot some differences.

[GaelicThunder]

Yes i tried it with a custom model i made (since the normal SD1.5 Vito used was already trasformed and working and i wanted my custom anime s**t), was a merged one with some modifications, i think the link is down now tho, sorry! It should be working with any other SD1.5 model tho.
Also, i said OnnxSimplifier, howerver remember to follow the main guide as it can use up to ~100Gb of Swap Ram!

[elliot-sawyer]

I had a go with it this evening and here's what I found:

1. Trying to convert https://civitai.com/models/4384/dreamshaper, tried both v7 and v8
2. Got an error: 'aten::scaled_dot_product_attention' to ONNX opset version 14 is not supported. Fixed with this snippet and changed to version 17

The option B steps do not work, it fails on the first command: onnx.onnx_cpp2py_export.checker.ValidationError: The model does not have an ir_version set properly.

The onnx_simplifier command shown in the readme doesn't work, for me through some searching I found this instead: python3 -m onnxsim dreamshaper8.onnx dreamshaper8_simplified.onnx

I get the same error when running that command:

Simplifying...
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/path/to/local/lib/python3.10/site-packages/onnxsim/__main__.py", line 5, in <module>
    main()
  File "/path/to/local/lib/python3.10/site-packages/onnxsim/onnx_simplifier.py", line 489, in main
    model_opt, check_ok = simplify(
  File "/path/to/.local/lib/python3.10/site-packages/onnxsim/onnx_simplifier.py", line 199, in simplify
    model_opt_bytes = C.simplify(
onnx.onnx_cpp2py_export.checker.ValidationError: The model does not have an ir_version set properly.

I might try with another model entirely to see if that makes a difference

[elliot-sawyer]

@vitoplantamura I've tried a few different models, the provided python script all seem to return the same error consistently: onnx.onnx_cpp2py_export.checker.ValidationError: The model does not have an ir_version set properly. Any feedback is very appreciated!

[thexa4]

I had this happen too, it was caused by the model being saved using external data. Using this simplifier worked: https://github.com/luchangli03/onnxsim_large_model

[noah003]

have you solved your problems? @elliot-sawyer