Safety, Ethics and openpilot about openpilot

[geohot]

I appreciate you engaging about these topics. Would prefer you did so with a real world identity and without a throwaway account though.

This is not a product for end users. As stated in the README, this is open source software for research purposes only. See https://github.com/CPFL/Autoware and https://github.com/PolySync/OSCC for examples of similar software.

Re: failure modes of closed source cars, this is a concern we have. I would love to see pressure on auto manufacturers to open this information up.

Re: visiond being closed, I address this in the presentation. Consider it a QM, https://en.wikipedia.org/wiki/Automotive_Safety_Integrity_Level#QM part. I believe the Mobileye EyeQ3, the chip Tesla uses, is QM as well. All code actually controlling the car is open source, and that's where the safety checks lie.

Re: the legal status of openpilot, comma.ai does not warranty this software for any purpose. People downloading this code are responsible for using it in a safe and legal way in whatever jurisdiction they may lie. We are not selling or shipping a product, and you are responsible for using this software safely and responsibly.

Re: some individuals will cause a hazard. This can be said about many things from kitchen knives to poorly assembled furniture. As with the two previous examples, which to date have killed way more people than self driving car testing, I believe most people will use this responsibly.

Bottom line, cars are dangerous. Inaction is still an action.

[toxicgumbo]

I wasn't the original writer, but your response is a good indicator that you're distancing yourself and Comma.ai from responsibility. I hear and read increasing attempts to disconnect from accountability and insinuate that anyone who gets hurt brings it upon themselves. If anyone is being vague or unclear, I'll say it plainly: You are naive, reckless, and will kill people through misleading them. Rather then defend yourself, try actually responding to what people are asking for. Build a subgroup. Asking for pull requests is simply a way to delay, deflect focus, and demand that others clean up the Proprietary Pilot mess. Put up a cash bounty if you want to play that way.

You trumpet this science experiment as if it's superior to everything on the market (except perhaps Tesla). You use it illegally on public roads. You avoid government safety legislation. You keep referring to your device as "open source" when only a subset is (don't fool yourself into thinking QM applies here). You build up enthusiasm and market yourself freely and widely in the press, badmouthing those who are working aggressively to use and develop safety features, redundancy, and functionality at the risk of visible progress. That's a cowardly shot for you to take because the bulk of people aren't keeping up with what these companies are doing. Meanwhile, you're claiming that a crappy cellphone and rehashed CAN code for a narrow range of pre-existing driver-assist hardware replaces all of this. People are get excited and shouldn't be.

People kill themselves with kitchen knives and poorly assembled furniture. It's a poor comparison. Those aren't autonomous or rely on artificial decision making designed by someone else.

Bottom line, you will kill people with a product the government pleaded you not to distribute and that a growing number of people are seeing for what it is: a cobbled together hack of Instructables.

[geohot]

Huh? Proprietary Pilot? Cash bounty? Don't fool yourself into thinking QM applies here? Kitchen knives kill people but it's okay because they aren't autonomous? We use it illegally on public roads? (citation needed) People are get excited and shouldn't be?

The government asked us to delay sales of a product. We did exactly that. As most people seem to understand, this is research code only. Happy to engage constructively, but If this conversation becomes toxic I will close this issue.

[toxicgumbo]

Citation very publicized. I don't work for you. It's out there amongst all the other publicity you've been embracing. There's plenty of lockerroom talk about how great, perfect, and revolutionary this project is, but little to none about safety and its use only as research.

I read the NHTSA response. Why apply for advancement of a product based on code considered "research"? For attention? For credibility? To attract developers?

George, if your project means something, stand behind it in all aspects. Publish how it won't get anyone killed and you're behind that initiative 100%. Insurers, laws, and automakers identify the autonomous car now as the "driver" and responsibly party. That means Comma.ai has/will have legal ties to consequences. You posted two other projects where safety is deliberately written into the design. It's discussed regularly on the mailing lists and publicly. Do the same. Make it (and your users) a priority.

It's good that you're willing to discuss things constructively despite a rephrase of words and threat of closing this issue (fortunately, it's being read and is archived). I and others have been promoting constructive ideas and I've spoken honestly and openly. I haven't read anything constructive by you yet, particularly in these areas which seem most concerning to many of us. Mostly I see an unwillingness to talk genuinely about the issues.

If closing this issue is an easy way to move forward and live in the fog of fanboys and avoid criticism, no worries. Others will speak up.

[geohot]

This is not an autonomous car. This is a repository on GitHub with free software that implements a driver assistance system. We discuss safety @ https://github.com/commaai/openpilot/blob/master/SAFETY.md

I'm closing this issue, if you have specific actionable concerns please open issues for them.

[geohot]

To the original poster, both the constant updating and car failure modes issues are things we'd like to address. Suggestions? We already had planned to make it more clear when the code updates, and we'll be probably adding a release branch for stability. Re: car failure modes, we've done our best to reverse engineer error codes from the car, but I agree a lot more work needs to be done in this area.

[paramaggarwal]

The original poster and follow-up criticism in this thread are not constructive. They cite fear and uncertainty but no concrete suggestions on how to deal with it.

Safety Considerations:
Human drivers are unsafe and unreliable. Having an additional assist system by definition can only help improve this. Please understand the difference between safer and safe.

Open-source Strategy:
Trying to dictate what parts of the stack someone should open is like asking for a free lunch. Yes, some parts are not open. But it's the same strategy as a Pixel phone vs. the Android OSSP or the Chrome browser vs. Chromium project.

Education and Research:
Personally I have learnt a lot by going through this project - I do not intend to assemble one, but it has been educational to me.

Use on Public Roads:
A discussion about development of drive-by-wire cars is different from driving these development/research cars on the road. If someone chooses to use someone else's code on the road and put everyone at risk - that is their own fault.

This is the same as driving by hand - by choosing to drive the car yourself rather than let someone more experienced do it - you are putting yourself and others at risk. By choosing to drive the car if you are not feeling well - is putting yourself and others at risk. Giving the wheel to someone whose driving you don't trust - is putting yourself and others at risk. Hence an individual makes a number of decisions while taking the wheel, and use of someone else's software in your own car on public roads is the responsibility and decision of the driver, not this project.

[geohot]

Re: constructive criticism. We are always looking for ways to make openpilot, and driving in general, safer. We believe that self driving car researchers are much safer than certain other groups of people legally allowed on public roads, namely teenagers, old people, and "buzzed" people.

[toxicgumbo]

The following two comments seem to have gone missing:

26#issuecomment-266348650

Marius Slavescu [email protected]
to Mention, commaai/openpi., me
@toxicgumbo unfortunetly this whole project seems to be very limited, with core components closed source, and the founder very unfriendly :-( just look how he closes most of the issues.

If you'll like to contribute to an open initiative not tied to any company profits, that is looking to tackle and solve all problems related with Self Driving Cars, check out this new organization:

http://github.com/ossdc

Join the discussion here:

http://ossdc.org/join

26#issuecomment-266363034

Param Aggarwal [email protected]
to commaai/openpi., me, Mention
I fail to understand the criticism towards this project here. It's free code and you do whatever you like with it. One could either make good things from it or bad. It's for research and exploration and not a product.

By definition of open-source and MIT license, there is no liability of any sort on the creators of this project. Everyone in the industry can learn and take the good parts and ignore the bad parts.

Can we please work towards making cars safer together? And not try to shoot down new projects for lacking complete code release or lack enhanced safety checks. It's not a product, it's not mature.

Two important words in this thread: constructive and research.

The original poster and follow-up criticism in this thread are not constructive. They cite fear and uncertainty but no concrete suggestions on how to deal with it.

Constructive: Constructive is what one makes of it. If criticism leads to a dismissal of concerns and closed tickets, then there is nothing constructive going on. If my tone is somehow offensive to others here, my apologies. However, expressing concerns does not require expressing concrete suggestions. There is genuine fear to be recognized here--people are writing code around closed systems anticipating the movement of 3-ton vehicles in the company of the public--and "how to deal with it" is exactly what should be an ongoing discussion (from day 1). By putting this code in the public and claiming a reliability level on par or exceeding systems known to be some of the safest around, the public is inherently less safe. Concrete suggestion: Active and ongoing public discussion of security and safety concerns ranging from foggy weather, to CAN injection, to external electrical interference, to tire blowouts, device hardlocks, optics failures, HMI intrusion (deliberate or otherwise), non-responsive driver, unexpected response from subsystems, etc.

Research: The use of this word, here, in this project, is debatable; at best it's research with a soft R. CAN packet recording/replay in "a garage in San Francisco" and programming around that (while hoping not to trigger undocumented and unknown proprietary module code) isn't research, it's more akin to hackaday.io jailbreak project.

And there-in is the major issue, particularly with issues of safety. Misleading claims such as "It's about on par with Tesla Autopilot at launch, and better than all other manufacturers" create a false sense of security with those who see the examples in action with real people. Moreover, users, testers, and developers cannot possibly understand the entirety of the codebase and must, on some level, put trust in the environment and code maintainers/developers having an emphasis on safety. This is impossible, however, since we're talking about a best-attempt of reverse engineering on several vehicle platforms using propriety devices and no relationships with those product manufacturers or access to complete module code being explored through CAN sniffing.

As a reminder to folks, this public repository exists because the project was deemed unsafe--in a very clear and seemingly restrained statement--by the NHTSA ("dedicated to achieving the highest standards of excellence in motor vehicle and highway safety").

And here's a choice quote from The Verge:

Last year I met with most of the CEOs of the big auto companies. A lot of them are just hopeless, all adamantly convinced they must own self-driving technology, in house.

There are hundreds of vehicle manufacturers the world over and Mr. Hotz met with most? And they're hopeless because, in reality, they have numerous business, engineering, and IP relationships involving hundred of employees and didn't want to contract with or purchase the IP of Comma.ai? Perhaps they know what they're looking at.

Either way, open sourcing components of this software means it moves ahead, gains traction from free press, picks up some free developers, and attempts to send a big middle-finger to the NHTSA and those who weren't interested. Short-lived gains.

This is where there's a difference between research and Research and why when someone like Musk says he's unconcerned, I believe him.

research here is small group of siloed people who are attempting to stand on the shoulders of giants and claim they are now on par with major OEMs and businesses (groups that do millions of simulations, tests, safety checks, data analysis, and use a variety of redundant sensors). Where are the relationships with the companies making the driver-assist equipment? Where are the NDA whitepapers? Where's the access to source code and full behavioral processes? Where are the PhDs, engineers, refereed papers, conference proceedings and everything that goes with doing Research? Can a few people make a major contribution to the scene? Sure, but don't claim this is safe when you're working with a shoddy cell camera behind a windshield in optimal lighting using basic understanding of the equipment you're controlling. All these bold claims in writing and in the press are for brand inflation and are misleading.

Research elsewhere in the field, particularly where Mr. Hotz has called accomplished companies and research groups who develop safe and proven technologies, "losers". The two projects mentioned above, Autoware and Polysync, actually do have their hands in Research. Autoware is largely tied to the University of Tokyo and has established connections with trained researchers, engineers, driver-assist manufacturers, and has partnerships with OEMs. Polysync has been in use for some time in autonomous and driver-assist vehicle research and also has partnerships with the big loser brands and OEMs. They have very different lifestyles, but are taken very seriously because they don't exist in a siloed development environment, the relationships are made, and are reviewed and tested by experts. For a good example of an automotive project, look at Automotive Grade Linux. It's a different beast, but an open group that is fully transparent, doesn't go around trashing everyone in the industry, considers safety a top issue (as CAN injection/interference and ECU control/confusion are a reality), and has a working relationship with OEMs and academics.

is is the same as driving - by choosing to drive the car yourself rather than let someone more experienced do it - you are putting yourself and others at risk.

It's a bit generalizing to suggest here and in other words that driverless cars are safer than humans, particularly at this stage of the game. If openpilot is to gain traction, it will need to leave the jailbreak scene and not burn bridges with equipment manufacturers getting there. As is, it's at a state where it puts everyone interested at risk. Even the occasional code comments expressing uncertainty suggest that. "Open source" won't be a cover or excuse, either. All eyes will look for someone to blame and they'll turn to Comma.ai, the developers, and possibly the equipment manufacturers who had no involvement. It will come down to "this downloadable software killed a family of 5" and there won't be the maneuvering or fine-and-forget-it freedom that Tesla had.

Unless you try to pin the blame on the original "OpenPilot" group: :-)
https://en.wikipedia.org/wiki/OpenPilot

All this aside, I firmly believe this project will be a cash cow in one form or another, whether it should be or not. That might be through support, brand identity and sale, contract hiring, licensing of more mature code (debatable unless not based on reverse engineering), talks, general awareness and insight, etc. It's easy for me to replace "Comma.ai" with "Megasquirt"--a company that people still think in 2016 is open source and that has built itself on the back of free developers, breaking engines through features over safety, giving a talk that starts off with a slide that says the users are to blame for problems, closing issues, banning people from IRC and their forum for speaking up. I've worked closely with demagogues, cult of personalities, sociopaths, and arrogance--all have their place is product design, leadership, and deployment, but even those personalities knew safety inherent in the code and for users is the first design priority.

We believe that self driving car researchers are much safer than certain other groups of people legally allowed on public roads, namely teenagers, old people, and "buzzed" people.

Concrete suggestion: Assume they're not, get to know OEMs, work together, then program and test, test, test, accordingly.

[toxicgumbo]

Despite my concerns, keep up the momentum and best wishes. Looks like a lot of fun in the right setting and circumstances.

[paramaggarwal]

(Responding, as I feel that someone needs to be vocal against the "regulate-it" argument.)

1. I appreciate your response and am able to see your perspective. It would be amazing if everything was the way you describe it.
2. I'm sure if they had 50 more people, he would assign 24 to legal, 24 to safety and continue advancing the state of technology with the rest two. The point being that just like education and health, if transportation is also going to be an ultra-regulated field - then we are going to be hurting innovation and hurting garage startups from ever being given a chance.
3. The way I understand it, they are looking for after-market and OEM manufacturers outside the US when no one in the US took interest. Also, they personally deem the AI layer as proprietary and everything else is commodity - hence the open source move. Once they get a partner and are able to work towards an integrated project, they will be able to do the kind of intensive safety-focused tests and enhancements as you mention.

I'm not completely sure what we are discussing here. If one has any ideas on how to make the code more robust and safe, they can open a PR. Also, I think his comparison to other systems was in terms of functionality/performance.

[geohot]

Off topic posts will be deleted. If you have specific actionable concerns regarding openpilot, please open issues for them. This is not the correct place to promote your comma.ai competitor, invite me to come speak, discuss if certain groups have a monopoly on the word "research", or debate whether Megasquirt is the devil. Your suggestion, "Get to know OEMs" has nothing to do with openpilot, an open source driver assistance system. This is not the comma.ai forum, this is a piece of free software.

Topic locked.