Comments (11)
Hi Wallace00V,
Thanks for reaching out! Yes, you can by serializing the TensorRT engine and executing using the TensorRT C++ runtime.
Assume you optimized your model by calling
model_trt = torch2trt(model, [data])
The model_trt
is a TRTModule
, which contains an attribute engine
which is the TensorRT engine. You can serialize this engine and save it to the path model_trt.engine
disk by calling
with open('model_trt.engine', 'wb') as f:
f.write(model_trt.engine.serialize())
This serialized engine may be used with the TensorRT C++ runtime as described in the TensorRT Developer Guide.
If you're unfamiliar with using TensorRT directly, please let me know and I'd be happy to help.
Best,
John
from torch2trt.
@jaybdub
thx for your answer, but i am unfamiliar with TensorRT. Please, how to load the 'model_trt.engine' in TensorRT C++ runtime?
from torch2trt.
@jaybdub all right, i figured it out just like the following:
std::stringstream gieModelStream;
gieModelStream.seekg(0, gieModelStream.beg);
std::ifstream cache( "model_trt.engine" );
gieModelStream << cache.rdbuf();
cache.close();
IRuntime* runtime = createInferRuntime(gLogger);
assert(runtime != nullptr);
gieModelStream.seekg(0, std::ios::end);
const int modelSize = gieModelStream.tellg();
gieModelStream.seekg(0, std::ios::beg);
void* modelMem = malloc(modelSize);
gieModelStream.read((char*)modelMem, modelSize);
nvinfer1::ICudaEngine* engine = runtime->deserializeCudaEngine(modelMem, modelSize, NULL); assert(engine != nullptr);
free(modelMem);
from torch2trt.
@jaybdub
Hi.
I got an error while using TensorRT C++ to deserialize the engine, which has some interpolate layers.
[TensorRT] ERROR: Cannot deserialize plugin interpolate
Segmentation fault
How to fix that?
from torch2trt.
I have the same error that is occured in deserializing the engine in C++
ERROR: Cannot deserialize plugin interpolate.
from torch2trt.
have the same error that is occured in deserializing the engine in C++. Do you solve the problem? @TheLittleBee @donghoonsagong
from torch2trt.
same problem , may be need a pluginFactory, have you solve the problem?
@donghoonsagong @Raneee @TheLittleBee @Wallace00V @jaybdub
from torch2trt.
same problem I believe: [TensorRT] ERROR: getPluginCreator could not find plugin interpolatetorch2trt version 1 namespace torch2trt
from torch2trt.
@skyler1253 @xieydd @Raneee @donghoonsagong @TheLittleBee @Wallace00V @jaybdub
Same problem. I can deserialize it in Python but failed in Cpp. Have you any ideas?
from torch2trt.
@skyler1253 @xieydd @Raneee @donghoonsagong @TheLittleBee @Wallace00V @jaybdub @crazybob23
same problem, do you solve this problem?
from torch2trt.
@skyler1253 @xieydd @Raneee @donghoonsagong @TheLittleBee @Wallace00V @jaybdub @crazybob23 @zbz-cool
same problem, do you solve this problem?
from torch2trt.
Related Issues (20)
- AttributeError: 'tensorrt_bindings.tensorrt.INetworkDefinition' object has no attribute 'add_convolution'. Did you mean: 'add_convolution_nd'? HOT 3
- Segmentation fault on nn.Linear HOT 1
- AttributeError: 'DataParallel' object has no attribute 'img_size' HOT 2
- There are questions about the parameters needed for forward propagation in model transformation HOT 2
- _trt.shape overflow when converting from pytorch to TRT HOT 5
- How to save a model to TRT file? Not covered in the docs HOT 1
- OSS-Fuzz Integration
- SyntaxWarning of dataset.py:61 HOT 1
- Unable to convert SigLIP text transformer due to missing model input when exporting model to onnx
- Is that available on orin nano? HOT 2
- ONNX export mode is set to TrainingMode.EVAL, but operator 'instance_norm' is set to train=True HOT 2
- AttributeError when using torch.nn.functional.pad
- Failed to 'sudo python3 setup.py install --plugins'
- ROOT CAUSE? `InstanceNorm3D` with tensorRT result consistency with nn.Instance_norm3d
- ROOT CAUSE?when used `nn.Transpose3d `, the shape of weight of transpose3d was consistency with nn.Transpose3D in torch.
- Unsupported method! HOT 1
- apiVersion: apps/v1 kind: Deployment metadata: name: titan-dataguard spec: replicas: 3 selector: matchLabels: app: titan-dataguard template: metadata: labels: app: titan-dataguard spec: containers: - name: titan-dataguard image: your_docker_image ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: titan-dataguard spec: selector: app: titan-dataguard ports: - protocol: TCP port: 80 targetPort: 80 type: LoadBalancer
- Add ProfilingVerbosity Option to Engine Build Configuration
- when use own defined convertor, why int tensor input has no _trt attribute? HOT 2
- Unsupported methods encountered issue, `torch.Tensor.to(torch.device("cuda"))`. Will torch.Tensor.to for devices be implemented? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from torch2trt.