Comments (14)
hi,me too
I am also curious the internal of visiond. why not open source?
tks
from openpilot.
From objdump on visiond (running from an nvidia TX2):
Dynamic Section:
NEEDED libgnustl_shared.so
NEEDED libavformat.so.57
NEEDED libavcodec.so.57
NEEDED libswscale.so.4
NEEDED libavutil.so.55
NEEDED libz.so.1
NEEDED libOmxVenc.so
NEEDED libOmxCore.so
NEEDED libgsl.so
NEEDED libCB.so
NEEDED libOpenCL.so
NEEDED libcutils.so
NEEDED libm.so
NEEDED liblog.so
NEEDED libdl.so
NEEDED libc.so
from openpilot.
@geohot Is it possible for Comma to release a visiond skeleton that does the following:
- Reads data from the camera
- Has OpenCV libraries loaded
- Returns no accel change, no brake change and no steering change.
i.e. visiond with no neural networks, the community can write other agents just using opencl.
from openpilot.
strings
says it uses at least Qualcomm FastCV Library.
from openpilot.
They apparently use opencv.
https://github.com/commaai/openpilot/blob/master/selfdrive/visiond/visiond
Q: there a better forum for questions? Does anyone know where to discuss?
-Sorry to barge in.
from openpilot.
- The sun is very bright during the day. At night the picture is a little blurry but who cares.
- visiond uses an internal framework to run the neural net. Tensorflow is way too slow on phones.
from openpilot.
You can already do this with the released visiond. See how UI reads the image from visiond, it's open source.
As for linking OpenCV, that should just work the same as on a desktop on NEOS.
from openpilot.
The problem is that if I run visiond just to get the camera feed from it I am not sure I would have enough processing power to try to reproduce it. As I understand things it would continue to run the network, save the videos, etc.
from openpilot.
If all you plan to use is OpenCV none of that will have any effect, since OpenCV is CPU only. Network and videos are done off CPU.
from openpilot.
I was thinking of using opencl on the adreno GPU for now ...
... But after your comment I realized that I will have a full Nvidia GTX 1070 on the back of the car, so I can develop for tensorflow during the next few months and in the near future hopefully we all switch to the oneplus [1] that comes with a Snapdragon 835 for tensorflow support on CPU, GPU and DSP [2]
[1] https://oneplus4.net/oneplus-5-release-date/
[2] https://www.qualcomm.com/news/snapdragon/2017/01/09/tensorflow-machine-learning-now-optimized-snapdragon-835-and-hexagon-682
from openpilot.
To answer the original poster and maybe close the issue.
-
If the car on the road is very fast, how can the visiond take a sharp photograph in a very short time?
George answered. To add to it, due to how visiond works, sharp pictures are not needed, you can see in this other repo how neural networks are even used to increase resolution on low res pictures: https://github.com/alexjc/neural-enhance -
Which deep learning framework does the visiond use now ? I want to optimize the backend with arm cpu, the tensorflow is too heavy.
visiond uses a custom backend, I believe they are using OpenCL + OpenCV, and possibly a reverse engineered camera driver. They can train on the cloud with a lot of gpu power, and then port the right network and weights with a lot of dot products, trigonometrical operations, sum, etc on a simple c program to be run on the GPU. See a very basic example in python here: http://www.wildml.com/2015/09/implementing-a-neural-network-from-scratch/
from openpilot.
Issue #4 mention the API.
from openpilot.
visiond is now open source.
from openpilot.
Cool. I found it in https://github.com/commaai/openpilot/tree/devel/selfdrive/visiond and will definitely give it a go now. :)
from openpilot.
Related Issues (20)
- Make CI jobs pass 100/100 times
- Switch to pytest HOT 4
- process replay: torqued doesn't receive all high-freq messages HOT 1
- process replay: support decimated logs with FrequencyBasedRcvCallback
- torqued: filter out high longitudinal acceleration
- Build failure on macOS HOT 3
- Pytest and Scons in Devcontainer fails
- torqued: misses some carState frames
- test LogReader scenarios for auto_source function HOT 1
- `CANPacker` for Nissan produces inconsistent `desired_angle` with Openpilot when it is large
- First bootlog doesn't have any git info
- Mismatch in allowable steering angle in Openpilot and panda for Tesla: Openpilot clip angle to current angle of the car to avoid EPS fault HOT 5
- Make `replay` tool run in realtime on a comma 3/3X
- comma 3X: no panda due to invalid CR3 config HOT 5
- lower thumbnail resolution & frequency and remove from camerad HOT 1
- New car support docs HOT 2
- Trigger Jenkins on external PRs HOT 1
- C3 Screen Too Bright @ Night HOT 3
- Expose jenkins and testing closet publicly
- params learner unstable during dynamic driving
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openpilot.