Introduction to Udacity Self-Driving Car Simulator

Udacity recently made its self-driving car simulator source code available on their GitHub which was originally built to teach their Self-Driving Car Engineer Nanodegree students.

Now, anybody can take advantage of the useful tool to train your machine learning models to clone driving behavior.

In this article, I’ll explain:

What You Can Do with it?

You can manually drive a car to generate training data, or your machine learning model can autonomously drive for testing.

In the main screen of the simulator, you can choose a scene and a mode.

Simulator Main Screen
Simulator Main Screen

First, you choose a scene by clicking one of the scene pictures. In the above, the lake side scene (the left) is selected.

Next, you choose a mode: Training Mode or Autonomous Mode. As soon as you click one of the mode buttons, a car appears at the start position.

Simulator Train Mode
Simulator Train Mode

Training Mode

In the training mode, you drive the car manually to record the driving behavior. You can use the recorded images to train your machine learning model.

To drive the car, use the following keys:

If you like using mouse to direct the car, you can do so by dragging:

To start a recording your driving behavior, press R on your keyboard.

You can press R again to stop the recording.

Finally, use ESC to exit the training mode.

You can see the driving instructions any time by clicking CONTROLS button in the top right of the main screen.

Autonomous Mode

In the autonomous mode, you are testing your machine learning model to see how well your model can drive the car without dropping off the road / falling into the lake.

Although the driving in the above video may not be the best ever possible, it is quite rewarding to watch a trained model autonomously driving the car without going off the road.

Technically, the simulator is acting as a server where your program can connect to and receive a stream of image frames from.

Simulator (Server) <-> Model (Client)
Simulator (Server) <-> Model (Client)

For example, your Python program can use a machine learning model to process the road images to predict the best driving instructions, and send them back to the server.

Each driving instruction contains a steering angle and an acceleration throttle, which changes the car’s direction and the speed (via acceleration). As this happens, your program will receive new image frames at real time.

How to Set Up the Simulator

You’ll need the following:


Unity is a game development environment in which Udacity self-driving car simulator is built.

Unity Editor
Unity Editor

You can install it as follows:


The Udacity self-driving car simulator uses Git LFS to manage large files.


The above image is from The below is a quote from the site:

Git Large File Storage (LFS) replaces large files such as audio samples, videos, datasets, and graphics with text pointers inside Git, while storing the file contents on a remote server like or GitHub Enterprise.

For Mac, you can use brew to install it:

brew install git-lfs

Once installed, you need to enable it for Git on your machine.

git lfs install

For Windows and Linux, please see for details.

Note: I’m assuming you already have the git command in your environment. If not, please follow the instruction to install git before installing git-lft.

Udacity Self-Driving Car Simulator Github

This Github project contains a Unity project for building the simulator.

Udacity Self-Driving Car Simulator GitHub
Udacity Self-Driving Car Simulator GitHub
git clone

That’s it for preparation.

Quick Start

Let’s build the self driving car simulator.

In Unity,

Unity Build Settings
Unity Build Settings

Modifying the Scenes in the Simulator

If you want to modify the scenes in the simulator, you’ll need to deep dive into the Unity projects and rebuild the project to generate a new executable file.

Unity Editor
Unity Editor

For this, I suggest that you first read the in the Self-Driving Car Simulator GitHub to learn the basics of how to navigate in the Unity project.

For more details on Unity itself, please visit

Deep Learning Driver Example

You can try a predefined model to see how a deep learning model drives autonomously.

git clone
cd car-behavioral-cloning
conda env create -f environments.xml
source activate car-behavioral-cloning
Simulator Initial Screen
Simulator Initial Screen
Simulator Main Screen
Simulator Main Screen
Simulator - Lake Track
Simulator — Lake Track
python model.h5

You can choose a different image quality in the simulator startup screen to see how it affects the car’s driving.

The has more details of the Deep Learning model. You can modify the model parameters and train it to generate different model files using the saved train data from the Training Mode.

Enjoy !


Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store