Python + OpenCV Neural Network + Haar-Cascade Classifiers
Objective
Modify a RC car to handle three tasks: self-driving on the track, stop sign and traffic light detection, and front collision avoidance.
System Design
The system consists of three subsystems: input unit (camera, ultrasonic sensor), processing unit (computer) and RC car control unit.
Input Unit
A Raspberry Pi board (model B+), attached with a pi camera module and an HC-SR04 ultrasonic sensor is used to collect input data. Two client programs run on Raspberry Pi for streaming color video and ultrasonic sensor data to the computer via local Wi-Fi connection. In order to achieve low latency video streaming, video is scaled down to QVGA (320×240) resolution.
Processing Unit
The processing unit (computer) handles multiple tasks: receiving data from Raspberry Pi, neural network training and prediction(steering), object detection(stop sign and traffic light), distance measurement(monocular vision), and sending instructions to Arduino through USB connection.
TCP Server
A multithread TCP server program runs on the computer to receive streamed image frames and ultrasonic data from the Raspberry Pi. Image frames are converted to gray scale and are decoded into numpy arrays.
Neural Network
One advantage of using neural network is that once the network is trained, it only needs to load trained parameters afterwards, thus prediction can be very fast. Only lower half of the input image is used for training and prediction purposes. There are 38,400 (320×120) nodes in the input layer and 32 nodes in the hidden layer. The number of nodes in the hidden layer is chosen fairly arbitrary. There are four nodes in the output layer where each node corresponds to the steering control instructions: left, right, forward and reverse respectively (though reverse is not used anywhere in this project, it’s still included in the output layer).
Below shows the training data collection process. First each frame is cropped and converted to a numpy array. Then the train image is paired with train label (human input). Finally, all paired image data and labels are saved into a npz file. The neural network is trained in OpenCV using back propagation method. Once training is done, weights are saved into a xml file. To generate predictions, the same neural network is constructed and loaded with the trained xml file.
Object Detection
This project adapted the shape-based approach and used Haar feature-based cascade classifiers for object detection. Since each object requires its own classifier and follows the same process in training and detection, this project only focused on stop sign and traffic light detection.
OpenCV provides a trainer as well as detector. Positive samples (contain target object) were acquired using a cell phone, and were cropped that only desired object is visible. Negative samples (without target object), on the other hand, were collected randomly. In particular, traffic light positive samples contains equal number of red traffic lights and green traffic light. The same negative sample dataset was used for both stop sign and traffic light training. Below shows some positive and negative samples used in this project.
To recognize different states of the traffic light(red, green), some image processing is needed beyond detection. Flowchart below summarizes the traffic light recognition process.
Firstly, trained cascade classifier is used to detect traffic light. The bounding box is considered as a region of interest (ROI). Secondly, Gaussian blur is applied inside the ROI to reduce noises. Thirdly, find the brightest point in the ROI. Finally, red or green states are determined simply based on the position of the brightest spot in the ROI.
Distance Measurement
Raspberry Pi can only support one pi camera module. Using two USB web cameras will bring extra weight to the RC car and also seems unpractical. Therefore, monocular vision method is chosen.
This project adapted a geometry model of detecting distance to an object using monocular vision method proposed by Chu, Ji, Guo, Li and Wang (2004).
P is a point on the target object; d is the distance from optical center to the point P. Based on the geometry relationship above, formula (1) shows how to calculate the distance d. In the formula (1), f is the focal length of the camera; ∂ is camera tilt angle; h is optical center height; (x0, y0) refers to the intersection point of image plane and optical axis; (x, y) refers to projection of point P on the image plane. Suppose O1 (u0,v0) is the camera coordinate of intersection point of optical axis and image plane, also suppose the physical dimension of a pixel corresponding to x-axis and y-axis on the image plane are dx and dy. Then:
v is the camera coordinates on y-axis and can be returned from the object detection process. All other parameters are camera’s intrinsic parameters that can be retrieved from camera matrix.
OpenCV provides functions for camera calibration. Camera matrix for the 5MP pi camera is returned after calibration. Ideally, a_x and a_y have the same value. Variance of these two values will result in non-square pixels in the image. The matrix below indicates that the fixed focal length lens on pi camera provides a reasonably good result in handling distortion aspect. Here is an interesting article discussing the focal length of pi camera with stock lens and its equivalent to 35mm camera .
The matrix returns values in pixels and h is measured in centimeters. By applying formula (3), the physical distance d is calculated in centimeters.
RC Car Control Unit
The RC car used in this project has an on/off switch type controller. When a button is pressed, the resistance between the relevant chip pin and ground is zero. Thus, an Arduino board is used to simulate button-press actions. Four Arduino pins are chosen to connect four chip pins on the controller, corresponding to forward, reverse, left and right actions respectively. Arduino pins sending LOW signal indicates grounding the chip pins of the controller; on the other hand sending HIGH signal indicates the resistance between chip pins and ground remain unchanged. The Arduino is connected to the computer via USB. The computer outputs commands to Arduino using serial interface, and then the Arduino reads the commands and writes out LOW or HIGH signals, simulating button-press actions to drive the RC car.
Results
Prediction on the testing samples returns an accuracy of 85% compared to the accuracy of 96% that the training samples returns. In actual driving situation, predictions are generated about 10 times a second (streaming rate roughly 10 frames/s).
Haar features by nature are rotation sensitive. In this project, however, rotation is not a concern as both the stop sign and the traffic light are fixed objects, which is also a general case in real world environment.
For distance measurement aspect, the ultrasonic sensor is only used to determine the distance to an obstacle in front of the RC car and provides accurate results when taking proper sensing angle and surface condition into considerations. On the other hand, Pi camera provides “good enough” measurement results. In fact, as long as we know the corresponding number to the actual distance, we know when to stop the RC car. Experimental results of detecting distance using pi camera are shown as below:
In this project, the accuracy of distance measurement using monocular vision approach could be influenced by the following factors: (1) errors in actual values measurement, (2) object bounding box variations in detecting process, (3) errors in camera calibration process, (4) nonlinear relationship between distance and camera coordinate: the further the distance, the more rapid change of camera coordinate, thus the greater the error.
Overall, the RC car could successfully navigate on the track with the ability to avoid front collision, and respond to stop sign and traffic light accordingly.
Thank you very much for the demo
LikeLiked by 1 person
How do u stream colour video over TCP/IP, i tried using vnc server and putty, the code [camera.preview()] is not working, but when i connect my pi to monitor over HDMI cable it works. Can you share me the code please.
LikeLiked by 3 people
Thank you , excellent work
LikeLiked by 1 person
would you care to share the code please ?
LikeLike
This is fantastic, well done
LikeLike
What a fun project, thanks for the write up, looks like something I’d like to try when I’ve got a few weekends spare.
Cheers
LikeLiked by 1 person
Very good work. Do you intend to release the code.
LikeLiked by 1 person
Nice work ! I was thinking about doing something similar :).
https://linkitoneblog.wordpress.com/2015/07/15/linkit-one-rover-preview/
LikeLiked by 1 person
Would you consider sharing you code you are using, I am just a hobbiest, but very interested in learning this technology.
LikeLike
Well Done zhengludwig,keep innovating
LikeLike
Really good job man… Keep going
LikeLike
cool stuff
LikeLike
Great work! Here is the link to the repo, it is not mentioned in this post (yet):
https://github.com/hamuchiwa/AutoRCCar
LikeLiked by 1 person
Could you please put the link of the haar cascade xml files for traffic light and stop sign?
Thanks
LikeLiked by 1 person
Very nice. Congratulations.
Cheers,
LikeLike
This is impressive… Keep up the great work!!!!!
LikeLike
awesome dude …great work
LikeLike
Awesome great work
LikeLike
so cool,could I make one like yours?
LikeLike
code please
LikeLike
https://github.com/hamuchiwa/AutoRCCar
LikeLike
So sweet!
Can you list the parts for the car setup?
Below is what I gathered so far:
Raspberry Pi board (model B+)
pi camera module
HC-SR04 ultrasonic sensor
What type of RC car? How does it work/receive commands to move?
LikeLike
Just found the list on your about page!
Hope this is what you are looking for:
Hardware
RC Car
Arduino Uno
Raspberry Pi and camera module
HC-SR04 Sensor
Laptop
Router
Miscellaneous – cables, resistors and etc.
Software
Programming languages – Python, C
PyCharm (IDE)
Arduino Software (IDE)
Picamera
PiSerial
Pygame
OpenCV-Python
Numpy
LikeLike
For what task do you use c here?
Sent from Yahoo Mail on Android
LikeLike
The codes that have been uploaded on github are enough to recreate the project isn’tit?
Sent from Yahoo Mail on Android
LikeLike
I am interested in building the car and writing my own software to have it drive autonomously.
Did you follow (or have any guides) to putting the car together with all of the components?
LikeLike
Hi Zheng, very nice work. I wonder how did your label the training data. Can you get the trainging label from reading a RC controller?
LikeLike
Hi Alex, whenever you press the arrow key, you label the image.
LikeLike
Hi Zheng, thanks. So you are also using arduino to read from the rc controller? I am using a slightly different rc car and controller like below. Do you have any good tutorial for hacking rc controller? http://www.ebay.com/itm/Traxxas-Slash-XL-5-2WD-RTR-w-TQ-2-4GHz-Short-Course-Electric-RC-Truck-58024-/201048261083?
LikeLike
Hi Alex, Arduino is connected to the controller. It sends signal to the controller.
Take a look at this: http://thelivingpearl.com/2013/01/04/drive-a-lamborghini-with-your-keyboard/
By the way, that’s a nice RC car
LikeLike
Thanks and I will try that!
LikeLike
Are you Chinese? Do you mind chatting over wechat or facebook, etc as I might have some questions regarding building a similar car. Thanks. you can email me at alexxucui@gmail.com.
LikeLike
cool my wechat id is : alexxucui
LikeLike
Hello Zheng!
I’m interested in recreating your self-driving car project but also adding in the option of interrupting the program and controlling the rc yourself(via computer) and then maybe returning to the program, do you have any ideas. Do you think it is possible to modify or add to your code to make that happen?
LikeLike
Hi, I think that can be achieved by adding a couple of lines. For example, press a key to switch mode.
LikeLike
Zheng, This is wonderful work ! I’m planning to start working on this project (just for fun !). Can you suggest what RC car should I use? Also, how should I power the pi? I have a USB power bank.
Thanks in Advance
LikeLike
Hi, a simple RC car is good enough (mine is pretty simple). USB power bank works great. Best of luck~!
LikeLike
Hi, This is awesome project, can you suggest how the neural communication is achieved for receiving the data from ultrasonic to arduino and data send to rc car from arduino????
LikeLike
Arduino doesn’t get data from the sensor directly, there is a computer in between Raspberry pi and Arduino.
LikeLike
what type of RC cars we have to use in this project, either with servo or without servo??
LikeLike
Hi, it’s just a simple rc car. I didn’t use the arduino or raspberry pi to control the rc car directly.
LikeLike
Hello Zheng .
I am interested in recreating your project but I want to implement the whole project on pi itself without streaming the video to another computer .
Is it possible ?
LikeLike
See the discussion relates to your question from here
LikeLike
Hello, Zheng!
Cool project. I would like to get few inputs from you as I’m working on a similar project.
LikeLike
Please check out the Github page, hope it answers your questions.
LikeLike
How do you control the rc car ….?
LikeLike
Keyboard -> Arduino -> Remote controller -> RC car
LikeLike
Thanks for the reply…..Can you also suggest the how does neural network is achived . I read your document i am not clear but to my knowledge it works by reading the first and last raw array values of the image captured by camera and also suggest how the path is teached to the car using camera ……??????
LikeLike
Hey Zheng!
Thanks for the help, have you uploaded the object recognition code as well?
LikeLike
Code is on Github, OpenCV provides functions for object detection.
LikeLike
Hello Zheng
Can you upload the images of the traffic light when it shows red ?
LikeLike
Can you also suggest the how does neural network is achived . I read your document i am not clear but to my knowledge it works by reading the first and last raw array values of the image captured by camera and also suggest how the path is teached to the car using camera ……??????
LikeLike
Hlo……i am currently trying your project …..In mpl_training.py i got the following error please help to fix .i’m using python 3.5 os-ubuntu16.04
num_iter = model.train(train, train_labels ,params = params)
TypeError: only length-1 arrays can be converted to Python scalars
………reply soon thank you
LikeLike
So far is there anyone succeeded in making it to work (following Zheng’s project)?
LikeLike
Here is one: https://www.youtube.com/watch?v=KrvvmLEQk8o
LikeLike