Comments (3)
tianleiwu
commented on July 30, 2024
what do you mean gets stuck (or could you share the outputs of the above script that is not normal)?
It is likely the first inference run will take longer due to cuDNN convolution algo tuning and resource allocation, the remaining runs shall be faster.
from onnxruntime.
quarrying
commented on July 30, 2024
what do you mean
gets stuck(or could you share the outputs of the above script that is not normal)? It is likely the first inference run will take longer due to cuDNN convolution algo tuning and resource allocation, the remaining runs shall be faster.
The output is as follows:
2024-07-19 17:58:51.796326681 [I:onnxruntime:, inference_session.cc:174 ConstructorCommon] Creating and using per session threadpools since use_per_session_threads_ is true
2024-07-19 17:58:53.146773527 [I:onnxruntime:, inference_session.cc:840 Initialize] Initializing session.
2024-07-19 17:58:53.151257663 [I:onnxruntime:, reshape_fusion.cc:37 ApplyImpl] Total fused reshape node count: 0
2024-07-19 17:58:53.154013361 [I:onnxruntime:, reshape_fusion.cc:37 ApplyImpl] Total fused reshape node count: 0
2024-07-19 17:58:53.162242333 [V:onnxruntime:, inference_session.cc:679 TransformGraph] Node placements
2024-07-19 17:58:53.162261331 [V:onnxruntime:, inference_session.cc:681 TransformGraph] All nodes have been placed on [CUDAExecutionProvider].
2024-07-19 17:58:53.166021272 [V:onnxruntime:, session_state.cc:71 CreateGraphInfo] SaveMLValueNameIndexMapping
2024-07-19 17:58:53.166334752 [V:onnxruntime:, session_state.cc:116 CreateGraphInfo] Done saving OrtValue mappings.
2024-07-19 17:58:55.055747308 [I:onnxruntime:, finalize_session_state.cc:173 SaveInitializedTensors] Saving initialized tensors.
2024-07-19 17:58:55.269780199 [I:onnxruntime:, finalize_session_state.cc:225 SaveInitializedTensors] Done saving initialized tensors
2024-07-19 17:58:55.289089454 [I:onnxruntime:, inference_session.cc:954 Initialize] Session successfully initialized.
2024-07-19 17:58:55.344849 starts
2024-07-19 17:58:55.350650700 [I:onnxruntime:, sequential_executor.cc:150 Execute] Begin execution
2024-07-19 17:58:55.783378 elapsed 0.43862199783325195
2024-07-19 17:58:55.860368 starts
2024-07-19 17:58:55.869268259 [I:onnxruntime:, sequential_executor.cc:150 Execute] Begin execution
2024-07-19 17:58:55.922098 elapsed 0.06176280975341797
2024-07-19 17:58:55.984943 starts
2024-07-19 17:58:55.989988341 [I:onnxruntime:, sequential_executor.cc:150 Execute] Begin execution
2024-07-19 17:58:56.042832 elapsed 0.05792117118835449
2024-07-19 17:58:56.097794 starts
2024-07-19 17:58:56.102932070 [I:onnxruntime:, sequential_executor.cc:150 Execute] Begin execution
2024-07-19 17:58:56.154957 elapsed 0.057192087173461914
2024-07-19 17:58:56.209661 starts
2024-07-19 17:58:56.214824273 [I:onnxruntime:, sequential_executor.cc:150 Execute] Begin execution
2024-07-19 17:58:56.267699 elapsed 0.058066606521606445
2024-07-19 17:58:56.322444 starts
2024-07-19 17:58:56.327602975 [I:onnxruntime:, sequential_executor.cc:150 Execute] Begin execution
2024-07-19 17:58:56.379816 elapsed 0.05740189552307129
2024-07-19 17:58:56.434758 starts
2024-07-19 17:58:56.439920970 [I:onnxruntime:, sequential_executor.cc:150 Execute] Begin execution
2024-07-19 17:58:56.492616 elapsed 0.05788826942443848
2024-07-19 17:58:56.548453 starts
2024-07-19 17:58:56.553534271 [I:onnxruntime:, sequential_executor.cc:150 Execute] Begin execution
2024-07-19 17:58:56.605954 elapsed 0.05753040313720703
2024-07-19 17:58:56.662649 starts
2024-07-19 17:58:56.667794682 [I:onnxruntime:, sequential_executor.cc:150 Execute] Begin execution
2024-07-19 17:58:56.719645 elapsed 0.05702567100524902
2024-07-19 17:58:56.775130 starts
2024-07-19 17:58:56.780257027 [I:onnxruntime:, sequential_executor.cc:150 Execute] Begin execution
2024-07-19 17:58:56.833278 elapsed 0.05822920799255371
2024-07-19 17:58:56.891268 starts
2024-07-19 17:58:56.896471060 [I:onnxruntime:, sequential_executor.cc:150 Execute] Begin execution
2024-07-19 17:58:56.948649 elapsed 0.057410240173339844
2024-07-19 17:58:57.003492 starts
2024-07-19 17:58:57.008576873 [I:onnxruntime:, sequential_executor.cc:150 Execute] Begin execution
The program may stop at any inference time, which could be the first time, the second time, or any other time.
from onnxruntime.
tianleiwu
commented on July 30, 2024
1.4 is too old.
Could you upgrade to onnxruntime-gpu 1.18.1 and cuda 11.8, cudnn 8.9?
from onnxruntime.
Related Issues (20)
- Multi-threaded GPU inferencing failing with whisper-small: Non-zero status code returned while running DecoderMaskedMultiHeadAttention node HOT 4
- TensorRT EP failed to create engine from network. HOT 5
- Custom Op Library does not work for CUDA HOT 2
- How to do multithreaded infer with onnxruntime HOT 1
- CUDA provider fallback to CPU is not working when CUDA_PATH environment variable exists
- using TensorRT EP by nuget HOT 5
- Unable to append DML Provider HOT 1
- EP Error /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:123 HOT 1
- FAIL : LoadLibrary failed with error 126 "" when trying to load "C:\code\Blueprint.Net.Server\bin\Debug\net8.0-windows10.0.22621.0\runtimes\win-x64\native\onnxruntime_providers_cuda.dll" ” HOT 2
- [Build] Cross compilation of the onnxruntime 1.5.1 for ARMv7 32bit target for gcc 4.9.2 HOT 6
- Quantization failed! The onnxruntime.quantization.quantize_dynamic seems didn't convert to the qint8 .onnx file successfully HOT 1
- [Build] ADD_LIBRARY cannot create target "memory" because another target with the same name already exists between xnnpack and absl HOT 1
- Create Custom Node in CUDA
- [Feature Request] Memory Commit Savings. Possible total memory savings. Allow fully optimized model to be serialized to disk and used as-is without large heap allocs HOT 1
- [Web] Error: Tensor's size(512) does not match data length(1024)
- Incorrect NaN handling for Min and Max operators on CPU with a single element input HOT 2
- TensorRT EP's inference results are abnormal. HOT 2
- [Build] Missing DirectML build in 1.18.1 HOT 1
- Activate thread pool will cause crash. HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from onnxruntime.