daben / asild-carnd-capstone-project

Final project of the Udacity Self-Driving Car Nanodegree: Programming a Real Self-Driving Car

Udacity Self-Driving Car Nanodegree: Term 3

Capstone Project: System Integration

Team: ASILD

Team Members

• David Moriano (lead) : [email protected]
• Benoit Miroux : [email protected]
• Pierre Merriaux : [email protected]
• Sean Devlin : [email protected]
• Tim Abrahamsen : [email protected]

Introduction

This is the final capstone project for Udacity's Self-Driving Car Nanodegree. It combines previous lessons on waypoint following, PID control, image recognition, and ROS to navigate a self-driving car around a track, while stopping for red lights.

Concepts

Concepts explored in this project:

• ROS
• Image recognition and classification
• Waypoint following
• PID control

Viewing the Project

You must have Ubuntu 16.04 or 14.04 installed to run the project. To run the project on the simulator, follow the instructions here to download and install ROS and the Udacity simulator, and to build and run the project.

Source code for ROS nodes is located in ./ros/src.

Write-Up

The starter code for this project includes a framework for ROS nodes for perception, planning, and control of a self-driving car. The diagram for the whole system is below:

The work of this project focuses on three ROS nodes:

• A waypoint updater (./ros/src/waypoint_updater) that feeds a list of next waypoints to a waypoint follower, and determines the target speed (necessary for maintaining desired velocity and stopping for red lights).

• A controller (./ros/src/twist_controller) that takes in the target linear and angular velocities from the waypoint follower, and calculates steering angle and throttle/brake values using a PID controller.

• A traffic light detector (./ros/src/tl_detector) that takes in images from a forward-facing camera, determines if there is a traffic light in the image, and if so determines the color of the traffic light.

Waypoint Updater

The waypoint_updater node subscribes to the /base_waypoints topic, which contains an array of waypoints for the entire track. The waypoint_updater node finds for the closest waypoint in front of the car, and publishes an array of the next 200 waypoints to the /final_waypoints topic, which is used by the waypoint_follower node to determine the target linear and angular velocity for the car.

The array of waypoints published to /final_waypoints includes the x and y coordinates of each waypoint, as well as their target linear velocities. This is necessary for keeping the car in the lane, as well as maintaining the desired velocity and stopping for red lights.

The waypoint_updater node also subscribes to the /traffic_waypoint topic, which contains the nearest waypoint to the stop line of a red light, if one is found. It uses this information to command appropriate target velocities for the car to stop for red lights (and to go for green lights).

Twist Controller

The /twist_controller node subscribes to the /current_velocity and /twist_cmd topics, which give the actual and target velocities (linear and angular) of the car, respectively.

It calculates the desired throttle, braking, and steering using a PID controller and a physical model of the car (which includes parameters such as mass, wheel radius, etc), and publishes the steering, throttle, and brake commands to the vehicle/steering_cmd, vehicle/throttle_cmd, and vehicle/brake_cmd topics, respectively, which command the actuators in the car.

Traffic Light Detector

The tl_detector node subscribes to the following topics:

• /vehicle/traffic_lights: An array of waypoints for known traffic lights on the track
• /camera_info: Calibration parameters for the camera
• /image_color: A color image taken by the camera
• /current_pose: Position and orientation of the car
• /base_waypoints: The array of all waypoints for the track

The tl_detector node uses a convolutional neural network to find traffic lights in image published in /image_color and determine the color of the light (see below).

Red light

Green light

The tl_detector node then publishes color of the light and the waypoint nearest the stop line for the light, which are used by the waypoint_updater node to determine target velocities to either stop at a light or go through it.

asild-carnd-capstone-project's Issues

Incorrect handling of velocity units in waypoint_loader

• In line 54, self.velocity * 0.27778, seems that the right constant is (miles/hour to meters/second).
• In line 68, vel = math.sqrt(2 * MAX_DECEL * dist) * 3.6, seems that the 3.6 should be removed.

The traffic light detector publish rate is coupled to the waypoints loader rate

We should decouple both so we can get TLD soon enough to plan the stop. One possibility is to remove the self.waypoints = None in tl_detector.py. The impact of this should be investigated. We have to take into account the cost of running the detector.

Speed targets are in kmph

Caleb: The speed in waypoint loader is in kmph. You should use this variable to set the speed. This does mean that the current setting would result in Carla running at 10 kmph. We created some confusion by asking everyone to target 10 mph when the launch file had 10 kmph. Sorry about that. I'll add some comments/classroom docs to make this clearer.

WaypointLoader publish frequency is too high for average systems

WaypointLoader is publishing a big list of waypoints at 40 Hz. This consumes too much cpu. Because this waypoints don't change, we can reduce this frequency to be able run the nodes in an average system.

Wrong integrator in twist_controller/pid.py

The implementation of the PID controller in twist_controller is wrong. The integrator is updated in the wrong place.

DBWNode doesn't publish twist commands

DBWNode must publish the appropriate throttle, brake, and steering commands in their respective topics: /vehicle/xxxx_cmd.

This node depends on an updated list of waypoints in /final_waypoints, see #1.

The DBW status is published in /vehicle/dbw_enabled. When disabled, the PID controllers should be reset to avoid the accumulation of the error.

The simulator never disables the DBW module and it only emits publish the status once at the start.

The type of each command defines is range as per the message definition in Dataspeed DBW:

https://bitbucket.org/DataspeedInc/dbw_mkz_ros/src/996cd2152159/dbw_mkz_msgs/msg/?at=default

bridge.py is the responsible of translating the dbw range to the simulator range.

Incorrect units for the yaw angle

• waypoint_loader.py doesn't acknowledge that the loaded yaw angle is in degrees.
• bridge.py should pass yaw angles in degrees to create_light.

Traffic Light Detection not implemented

The TLD node must publish the index of the waypoint in the /base_waypoints list for the nearest upcoming red light to the topic /traffic_waypoint.

This index will be used by the waypoints updater to stop the car.

The actual position of the traffic lights for the simulator track is given in the traffic_light_config.py.

To solve this issue we have to:

1. Find the waypoint of the nearest upcoming traffic light.
   • A precise definition of nearest will be needed.
2. Locate the traffic light in the camera image
   • This can be done using the camera geometry and the known position of the traffic lights in the world or using a generic traffic light detector like SSD.
   • Note that detection has to run at least at 10Hz.
3. Classify the state of the traffic light (RED/GREEN/YELLOW or RED/NO-RED).
   • This can be integrated in the detector or it can be done using a separated classifier.

The simulator provides the ground truth in /vehicle/traffic_lights, but the TLD must work also in the Udacity's test track.

This issue doesn't depend of other issues.

The bridge with the simulator queue the messages in an unbounded queue

An unbounded queue can accumulate old messages in certain situations of low performance affecting the ability to control the simulator in real time. Because the only messages sent to the simulator are real time commands, it should be safe to replace the queue with a dictionary.

TrafficLightDetector using the ground truth provided from the simulator

A TLD that posts the position of the upcoming traffic light and uses the ground truth of the simulator to get its state would decouple the development of the WaypointUpdater and TwistControllers from the final TLD.

A README suitable for submission is missing

WaypointUpdater doesn't publish waypoints

WaypointUpdater must subscribe to the topics /base_waypoints and /current_pose and publish a list of waypoints to /final_waypoints.

• /base_waypoints publishes all the waypoints in the track.
• /current_pose publishes the current pose of the ego vehicle.

waypoint_updater should start publishing a list of waypoints ahead of the car:

• The first waypoint in the list published to /final_waypoints should be the first waypoint that is currently ahead of the car.
• The total number of waypoints ahead of the vehicle that should be included in the /final_waypoints list is provided by the LOOKAHEAD_WPS variable in waypoint_updater.py.

For this issue we will ignore the traffic lights.

Comments:

• The term ahead of the car has to be defined precisely.
• I think we can assume that the track is a loop.

WaypointUpdater doesn't adjust the speed for upcoming red traffic lights

WaypointUpdater must:

• subscribe to /traffic_waypoint which gives the index in /base_waypoints of an upcoming red light,
• adjust the target of speed of the /final_waypoints to stop the car at the right place.

Requirements:

• Acceleration should not exceed 10 m/s^2
• jerk should not exceed 10 m/s^3.
• For code submitted for testing on Carla, the top speed of the vehicle must be 10 mph.

This issue depends on #1, but can be implemented without the TLD using the ground truth provided by simulator in the topic /vehicle/traffic_lights.

tl_detector requires a version of tensorflow not available in Carla

Carla has installed tensorflow==1.0.0. Our current detector requires at least 1.1.0.