Subido por joscullo


Original Article
A reproducible
educational plan to teach
mini autonomous race
car programming
International Journal of Electrical Engineering
& Education
2020, Vol. 57(4) 340–360
! The Author(s) 2020
Article reuse guidelines:
DOI: 10.1177/0020720920907879
Süleyman Eken1 , Muhammed Şara2,
Yusuf Satılmış2, Münir Karslı2,
Muhammet Furkan Tufan2,
Houssem Menhour2 and Ahmet Sayar2
As autonomous cars and complex features of them grow in popularity, ensuring that
analyses and capabilities are reproducible and repeatable has taken on importance in
education plans too. This paper describes a reproducible research plan on mini autonomous race car programming. This educational plan is designed and implemented as
part of a summer internship program at Kocaeli University and it consists of theoretical
courses and laboratory assignments. A literate programming approach with the Python
language is used for programming the race car. To assess the educational program’s
impact on the learning process and to evaluate the acceptance and satisfaction level of
students, they answered an electronic questionnaire after finishing the program.
According to students’ feedback, the reproducible educational program is useful for
learning and consolidating new concepts of mini autonomous car programming.
Reproducible research in robotics, mini autonomous car, robot operating system, deep
learning, Jupyter Notebook, electronic questionnaire
Department of Information Systems Engineering, Kocaeli University, _Izmit, Kocaeli, Turkey
Department of Computer Engineering, Kocaeli University, _Izmit, Kocaeli, Turkey
Corresponding author:
Süleyman Eken, Department of Information Systems Engineering, Kocaeli University, _Izmit 41001, Kocaeli,
Email: [email protected]
Eken et al.
Reproducibility is considered an attainable minimum standard for evaluating scientific claims when fully independent replication of a study with different data is
not feasible. This standard requires researches to make both their data and computer code available to their peers, but it still fails to achieve full replication, since
the same data and not independently collected data are analyzed again.
Nevertheless, this standard while limited still allows for assessing the quality of
scientific claims by verifying the original data and codes. It managed to fill the gap
between full replication and none at all; after all, the reproducibility of a study can
lie on a scale of possibilities rather than just the two mentioned extreme points.
Reproducible research has been a recurring subject appearing in high-impact journals. The state of reproducibility has been studied thoroughly in some domains,
e.g., in computer systems research1 or bioinformatics.2 Some studies have covered
the field of robotics research as well. Compared with other fields, robotics is special
in that it aims to enable an embodied agent to interact with the world physically.
Research on robotics has been concerned with creating efficient models that
meet requirements like body dynamics rigidity, or multiple-view geometry that
determines an action output based on the perceptual input, or predict the perceptual output based on a certain action. This, while working fine in a controlled setup
with well-defined variables, made it a difficult challenge for robots to move out of
these environments to the real world with its unpredictability and the wider range
of tasks and situations. Robots are significantly and quickly improving regarding
how they perceive the world around them, they can now sense the world in new
unprecedented ways: see, hear, read, touch, etc.3–5 The ability to deal with these
new inputs and learn from them pushes the capabilities of robots steps ahead. Even
though researchers have studied and applied various machine learning approaches
on this task, this often comes with a caveat represented by some assumptions or
simplifications in the mapping between perception and action and vice versa.6–8
This, however, can be alleviated, thanks to the availability of big data and the
improved machine learning approaches to benefit from it.9
Unlike computational science disciplines, robotics research faces practical challenges when it comes to replicating or reproducing results for objective assessment
and comparison. The difficulty of perfectly controlling environmental variables,
poor understanding of comparable evaluation metrics or finding goal similarity
across different domains, determining what information is needed to allow result
replication, they all contribute to making this challenging. To make matters worse,
we are yet to build a solid theoretical foundation for experimental replicability in
robotics, hindering research progress and technology transfer. Though it is worth
noting that recent years have seen significant movement forward in this regard,
with workshops10,11 and summer schools12 covering the design, planning, experimental methodology, and benchmarking of reproducible research in robotics.
IEEE Robotics and Automation Magazine also introduced a new type of articles,
R-article. It includes a description and access information to data sets and the
International Journal of Electrical Engineering & Education 57(4)
source code or its executable files on CodeOcean,13 as well as hardware (HW)
description and identifier,14 with details on the tests that have been run.
Teaching and learning autonomous vehicle programming can be a difficult task,
and adding robotic vehicle design and construction to the mix only makes the
learning curve even steeper. This paper presents a reproducible research plan to
teach autonomy using state-of-the-art sensors and embedded computers for mobile
robotics built upon a 1/10-scale racing car platform. A combination of theoretical
courses and laboratory assignments has been given to undergraduate students
using this methodology as part of a summer internship program at Kocaeli
University. First of all, we covered the theoretical courses of this program, this
includes subjects such as basics of commanding the vehicle actuators (steering and
drive motor), robot operating system (ROS) fundamentals, leveraging data from
the Light detection and ranging (Lidar), the Inertial Measurement Unit (IMU) and
the wheel odometer, basics of control systems, basics of blob detection in images,
fundamentals of robot perception including visual servoing. Then, students have to
implement what they learned in the laboratory exercises. The 10 laboratory experiments are designed in a way to provide the students with hands-on experience with
the mini autonomous vehicle programming in this research plan. These experiments offer a fitting laboratory solution to bridge the gap between the theoretical
knowledge acquired from the courses and the real-time application. The contributions of this paper can be summarized with:
• We have presented modern scientific computational environments for programming mini autonomous vehicles using a literate programming approach with the
Python programming language due to its popularity within the scientific computing community. The environment makes it possible to combine algorithmic
details and explanations with the implementation code in one file.
• We have evaluated the reproducible educational plan based on students’
The rest of this paper is organized as follows: section “Related work” presents
related work about courses and robotics competitions with autonomous vehicles
and similar platforms for teaching this subject. Section “Used platform” describes
the open-source HW and software of the platform. Section “Educational
approach” introduces the plan including theoretical lessons and laboratory assignments. Section “Evaluation and Discussion” gives students’ feedback on our
reproducible educational program and discussion of the results. Finally, the last
section presents the “conclusion” of this work.
Related work
The use of autonomous vehicles is becoming a common choice as a tool for teaching several related subjects, including robotics, electronics,15 programming, and
complex control. Carnegie Mellon’s Robot Autonomy course pioneered this
Eken et al.
approach in 2002 and set the framework for using creativity to spark the students’
interest in learning.16,17 There is also LEGO MindStorms NXT, a low-cost programmable robotics kit that has seen far reach out for it managed to advance the
ability of teachers and learners of all ages to produce autonomous vehicles in an
easy and timely manner, thanks to its canned libraries and a powerful graphical
programming language.18
Competitions and camps also have shown incredible results, setting them as
effective tools in motivating students to extend their knowledge and improve
their ability to program a robotic vehicle to run autonomously.19,20 The FIRST
Robotics Challenge, the FIRST Lego League,21 BEST Robotics,22 National
Robotics Challenge (NRC),23 and EARLY Robotics24 are a few examples.
These programs are geared toward precollege students and cover a wide range
of choices of robotics HW for students to build and experiment with. There are
many other examples25,26 of successful approaches to teaching the subject.
During this program, we took advantage of the flexibility and extensibility of
the Jupyter notebook web application in our development environment, this
approach facilitates both the educational and the reproducible research workflow.
In addition to those benefits, the use of the popular Python programming language
minimizes the effort required to get up and running with the environment and get a
hold of its syntax and semantics. Moreover, the availability of free and opensource packages for this programming language is another reason for choosing
it; they can be easily integrated into the environment, extending it to suit whatever
the developer needs.
The platforms used to teach autonomous control are as varied as the curricula
for teaching; this variation covers a wide range of capabilities and price points.27
They also vary in how robust they are, from children toys to industrial equipment
passing by hobbyist levels. Most notable examples at the lower price end include
the use of Lego kits,21,28,29 kits based on the iRobot Create fitted with either a
Gumstix-embedded computer30 or a laptop.31 These platforms serve the important
purpose of lowering the entry bar for as many students as possible, especially those
with limited budgets. However, the majority of them lack state-of-the-art sensors
and computers that power and make many contemporary robotic systems possible.
The platform we opted to use on the other hand includes the state-of-the-art HW
utilized in real-world robotic vehicles today. Detailed information about the platform is given in the following section.
Used platform
In this section, we will describe the HW configuration of the car kit and its features. Racecar, or Rapid Autonomous Complex–Environment Competing
Ackermann-steering Robot, was initially developed for a competition by MIT in
2015, then updated in 2016 to be used for teaching robotics.32 The kit uses a 1/10
scale Traxxas RC Rally Car racing car as its backbone (see Figure 1(a)), accompanied with VESC,33 an open-source electronic speed controller. And the Nvidia
International Journal of Electrical Engineering & Education 57(4)
Figure 1. Hardware parts of used car kit. (a) Racecar mini autonomous developer kit (b) and its
hardware components.
Jetson TX2 developer board as its main processor, which comes with 256 CUDA
cores. The following are a list of some advantages associated to open source
systems: lesser HW costs, no vendor lock-in, simple license management, abundant
support, scaling and consolidating, low-cost and large amounts of available information on the Internet.
As for sensors, the kit includes three types: a stereo camera, a Lidar, and an
IMU. ZED stereo camera34 allows for the use of standard stereo matching
approaches to acquire depth information from two images by two different cameras. The Stereolabs Software Development Kit (SDK) implements a semi-global
matching algorithm that works on Graphics Processing Unit (GPU)-based computers, which makes it suitable for our platform with Jetson TX2. 3D mapping
Eken et al.
Table 1. ZED camera video modes.
Video mode
Frames per second
Output resolution
4416 1242
3840 1080
2560 720
1344 376
also can be done using the same algorithm. The camera is capable of working in
the following video modes. The camera is capable of working in the following
video modes as tabulated in Table 1.
The second sensor is the Scanse Sweep Lidar.35 It can operate at a distance of 40
m away collecting 1000 samples per second. The Sweep has a head rotating clockwise while emitting a laser beam to collect data in a single plane. Finally, we have
Sparkfun 9 degrees of freedom (9-DoF) Razor IMU M036 as the IMU. It has a
SAMD21 microprocessor paired with an MPU-9250 9-DoF sensor, making it a
small and flexible IMU. This means we can program it for various tasks: monitoring and logging motion, transmitting Euler angles over a serial port, or acting as
a step-counting pedometer. The 9-DoF Razor’s MPU-9250 has a triple 3-axis
sensors configuration: an accelerometer to measure linear acceleration, a gyroscope to measure angular rotation, and a magnetometer to measure magnetic
field vectors around it.
The parts can be placed on two Laser cut or 3D printed plates. One plate is
dedicated to carrying the Nvidia Jetson TX2 dev kit, while the rest of the parts can
be distributed between the second plate and the car chassis (see Figure 1(b) for
components). The main computer runs ROS on top of the Ubuntu operating
system. ROS environment serves the purpose of making our software stack modular, i.e. separate software modules can be developed for different tasks, such as
the computer vision system, the autonomous driving system, or the motors control
Educational approach
Within the 2018 summer internship program, a group of 15 undergraduate computer engineering students was selected to be taught complex robot tasks on the
racecar mini autonomous developer kit. The applied reproducible educational
program includes theoretical lectures and laboratory assignments that allow students to practice their theoretical knowledge and get hands-on experience. A literate programming approach (via Jupyter notebooks) with the Python
programming language is used for programming the vehicle.
Originally introduced in 2011 as IPython notebooks, Jupyter notebooks37 is an
open-source interactive web-based application streamlining and simplifying literate
programming.38,39 Jupyter notebooks supported only Python at first, as the early
International Journal of Electrical Engineering & Education 57(4)
Figure 2. Thresholding example of Jupyter Notebook from assignment labs.
name implies, then extended the support to a large number of programming languages, by adding the appropriate Jupyter kernels. The kernels communicate with
the system using Jupyter’s communication protocol, the so-called Jupyter messaging protocol. A notebook is a JavaScript Object Notation (JSON) document,
which is parsed and displayed in a human-friendly format (see Figures 2 and 3).
Each laboratory assignment notebook is a document containing the description of
an image-analysis workflow using equations, text, the executable implementation
of the workflow, and figures and tables generated by running the code. The code is
split into logical sections called “code cells” or “code chunks” that can be executed
interactively, providing a natural setting for modifying the code and experimenting
with parameter values and seeing their effect on algorithm performance. Another
benefit of running individual code chunks is that one can choose to run some parts
only once, while others can be repeatedly executed. For example, while a dataloading cell only needs to run once, an algorithmic cell could run several times to
compare different approaches to solving the same problem before settling on the
best one.
Eken et al.
Figure 3. Segmentation example of Jupyter Notebook from assignment labs.
Theoretical lectures
This subsection introduces the seven theoretical lectures. Each lecture includes a
brief description of the learning objectives. Table 2 shows the recommended road
map for instructors, with each assignment and its appropriate lecture.
LEC0—ROS fundamentals. This lecture covers the first basics of commanding the
vehicle actuators (steering and motor). The students get familiar with how to
obtain data from the Lidar, the IMU, and the wheel odometer (through the
VESC). They also learn how to work with ROS40 and its components such as
nodes, topics, messages, services, etc. ROS is a Berkeley Software Distribution
(BSD)-licensed software for controlling robotic components on a PC. An ROS
consists of a number of independent nodes that communicate with each other
using a publish/subscribe messaging pattern.41 For example, a node might implement a particular sensor’s driver, and publishes its acquired data in a stream of
International Journal of Electrical Engineering & Education 57(4)
Figure 4. Examples from the data set. (a) Sample 1, (b) sample 2, (c) sample 3, (d) sample 4,
(e) sample 5, and (f) sample 6.
messages. These messages are then available for consumption by all other nodes
that subscribe to that stream, including filters, loggers, and also higher level systems such as pathfinding, collision detection, etc. Our software modules running
on the Jetson42 are structured as ROS nodes. The nodes for the LIDAR and other
sensors are part of the software package given to students. Students integrate
existing software modules (drivers for reading sensor data, and their custom algorithms) to quickly develop a complete autonomous system.
LEC1—Control systems theory. Many engineered systems have an input affecting the
value of some output. If we are interested in regulating the values of the output, as
we frequently are, we become interested in algorithms to adjust the inputs accordingly. This will be facilitated by making feedback about the current state of the
output available to the control algorithm, though it is also possible to try to
“blindly” adjust the input. Regulation algorithms and their implementations are
called control systems. The difficulty of the control problem depends on certain
factors including the physical complexity of the system, its time dynamics and
those of the desired output trajectory, the resolution and latency of the available
Eken et al.
Figure 5. An example from dataset and its XML file. (a) A sample of parking traffic sign (b) and
its annotated XML file.
Table 2. Syllabus for teaching mini autonomous race car programming.
Laboratory assignment(s)
LEC0—ROS fundamentals
LEC1—Control Systems Theory
LEC2—Introduction to Computer
Vision and OpenCV
LEC3—Deep Learning
Fundamentals and Tools
LEC4—Object Detection and
LEC5—Visual Servoing
Testing and deployment of all
assignments on the mini
autonomous vehicle
LAB0—ROS Development with Python
LAB1—Bang-Bang and Safety Controllers
LAB2—OpenCV Basics
LAB3—Streaming Video with OpenCV
LAB4—Semantic Segmentation
LAB5—Blob Detection using the ZED Camera
LAB6—End-to-End Learning for Steering
Autonomous Vehicle
LAB7—Traffic Sign Classification and Localization
LAB8—Parking in front of a Visual Marker
LAB9—Histogram Filters and 2D Kalman Filter
feedback sensors, etc.43 We study a collection of common and relatively straightforward control techniques such as feed-forward control, Bang–Bang control,
Proportional–Integral–Derivative (PID) control, differential drive velocity
International Journal of Electrical Engineering & Education 57(4)
LEC2—Introduction to computer vision and OpenCV. An important part of the human
driving experience is keeping an eye on the environment, as the safety of the driver
and that of many other people depends on that. We specifically try to spot any
obstacles such as pedestrians and other cars and evade them. Similarly, autonomous vehicles must leverage their sensors and intelligence to perform the same task
of detecting obstacles as it reinforces the car’s understanding of its environment.
Other vehicles on the road represent one of the most important obstacles to deal
with, as they are most likely the biggest and the most frequently encountered
objects in our lane or neighboring ones. Within this lecture, students start by
learning basic operations on image processing with OpenCV45 such as accessing
pixel values and modifying them, accessing image properties, setting region of
Image (ROI), splitting and merging images, blurring, finding contours, line, and
shape detection. Then they go on learning more advanced concepts applicable to
autonomous driving.
LEC3—Deep learning fundamentals and tools. In 2005, Sebastian Thrum and his team
won the DARPA Grand Challenge46 using machine learning, which ignited a huge
interest in machine learning and deep learning for developing autonomous
vehicles. Today, continuous advances in deep learning technology keep pushing
the autonomous vehicle industry forward. Within this lecture, students learn how
to train a deep neural network to correctly classify newly introduced images,
deploy their models into applications, find ways to improve the performance of
these applications, deploy end-to-end deep learning models for autonomous driving. They also get familiar with frameworks like Tensorflow,47 the most famous
library used in production for deep learning models and Keras48 which is a highlevel API built on TensorFlow.
LEC4—Object detection and classification. Perception, which is the task of environment
sensing including object classification, detection, 3D position estimation, and
simultaneous localization and mapping (SLAM),49 is as important to autonomous
vehicles as it is to human drivers. Without it, the vehicle would not be able to
navigate through any kind of obstacles. And that brings us to the subtask of object
detection, for which we need accurate and reliable algorithms to deploy on the
autonomous vehicle. Within this lecture, students learn OverFeat,50 VGG16,51 and
YOLO52 models.
LEC5—Visual servoing. Visual servoing is the use of cameras and a computer vision
system to control the motion of a robot. Students leverage what they learned about
OpenCV and ROS so far combined with image-based visual servoing to program
the racecar to park in front of solid-color objects, like orange cones or specific
traffic signs.
LEC6—Localization. Navigation is a key and challenging task for mobile robots. To
perform navigation successfully, four subtasks are required: perception,
Eken et al.
localization, cognition, and motion control. Localization is the robot’s ability to
determine its position relative to the environment, and it is the area that has
received the most research attention in the past decade, allowing for significant
advances to be made. The use of localization techniques enables a robot to find two
basic pieces of information: what are the robot’s current location and orientation
within its environment? This lecture also covers subjects ranging from histogram
filters to N-dimensional Kalman Filters as applied in localization.
Laboratory assignments
This section introduces the 10 laboratory experiments53 accompanying the
LAB0—ROS development with Python. Typically, the entire software functionality of a
robot is divided using ROS into different modules, each running over one or
multiple nodes. This laboratory assignment covers how to write a simple ROS
application consisting of a publisher and subscriber node in python and getting
familiar with concepts like topics and messages. Node is the ROS term for an
executable that is connected to the ROS network. Topics are named buses over
which nodes exchange messages. A message is a simple data structure, comprising
typed fields such as Bool, Int32, String, etc. In this laboratory, students will implement a publisher ROS node that generates a random number between 0 and 10
and write a subscriber ROS node that will square the generated number and
display the result.
LAB1—Bang-bang and safety controllers. The LIDAR sensor provides a cloud of distance measurements in a polar form to the system. Using this data, a system can
map the environment around it and navigate it accordingly. In this lab, students
are asked to implement a Bang-Bang safety controller as an “emergency stop”
node using ROS and Python. The node will receive the LIDAR data through
the LIDAR sensor topic. If the distances measurements right in front of the car
are below a specified threshold, the node will recognize that there is an obstacle
and publish a stop command to the SAFETY topic. The car, in turn, will stop as
soon as it receives that command preventing the collision.
LAB2—OpenCV basics. In this lab students learn how to use OpenCV for basic
operations such as reading an image from a file, displaying an image on the
screen, gray-scale conversion, mirroring, resizing, cropping, image segmentation,
and shuffling. Also, they use the MatPlotLib library54 to display images.
LAB3—Streaming video with OpenCV. In this lab, students learn how to send a continuous stream of images with ROS as a topic. Our goal here is learning how to
distribute sensor data from one node to several other nodes that need the data.
International Journal of Electrical Engineering & Education 57(4)
As is the case with the camera in our project, and the several modules that will be
using its pictures to perform different tasks.
LAB4—Semantic segmentation. Image semantic segmentation is the partitioning of an
image into semantically meaningful areas or classes. In a sense, it is similar to
image classification but applied at a pixel level. Semantic segmentation has been
used in a wide range of applications ranging from robotics to augmented reality,
including in autonomous driving. For the latter application, a scene could be segmented into streets, traffic signs, street markings, cars, pedestrians, or sidewalks,
all of which are important data for an autonomous vehicle. In this laboratory,
students will train a SegNet55 on the CamVid data set56 for road scenes, and then
try it on the data from the car’s camera.
LAB5—Blob detection using the ZED camera. The goal of this lab is to learn how to use
the stereo camera with the onboard GPU to influence the vehicle’s control. This is
possible thanks to computer vision algorithms implemented in the OpenCV library
which gives a robot a sense of perception in its field of view. Using this perception
to locate targets, called “blobs,” and track them, the students are asked to follow
and ultimately make decisions based on the position of these blobs. The task of
blob detection can be summarized with the following computer vision operations:
• The first step is converting the frames from RGB (red, green, blue) into HSV
(hue, saturation, value) color space. The differences can be understood from the
names; while RGB stores each pixel as a combination of red, green, and blue
values, HSV describes them as a combination of hue (angle around the color
wheel), saturation (how “pure” the color is), and value (how vibrant the color
• A threshold is applied to the image to filter the desired blob out with a certain
color range. Filtering red for example is achieved by keeping only pixels with
values between HSV (2.95%, 47%) and HSV (20, 100%, 100%).
• The image is eroded to remove noise and smooth edges.
• Outlines around the blobs are highlighted using The OpenCV findContours
• Smaller contours usually result from noise, so we drop them and keep only the
one with the largest area.
• A polygon encapsulating the resulting contour is approximated using OpenCV’s
approxPolyDP. If it fits the shape of the desired blob, we keep it, otherwise, we
drop it.
• The blob is described as a single position in terms of the center of the final
LAB6—End-to-end learning for steering autonomous vehicle. Vehicle steering, as simple as
it seems, is one of the most challenging aspects of an Artificial Intelligence (AI)
robotic system. That is why researchers have focused on training deep neural
Eken et al.
Table 3. Sample from data set.
File name
network models on visual data from front cameras paired with the steering
angles.57 In this lab, students deploy different end-to-end deep learning models
on the Racecar and evaluate their performance. Experimental analysis shows that
different models successfully manage to steer the vehicle autonomously. The pipeline for this lab can be listed as follows:
• Building data set: The vehicle’s speed and steering angle are controlled with a
Logitech F710 joystick, an ROS module subscribes to the events of the joystick
recording both these values as well as images taken from the ZED which are
then stored in a directory as shown in Table 3.
• Splitting data set into training and test samples.
• Python generator is used to generate data for training on the fly, rather than
storing everything in memory. Also, data augmentation techniques such as flipping, brightness change, and shifts are applied.
• Deploying different end-to-end models after training and steering the vehicle
based on data from the front view camera.
LAB7—Traffic sign classification and localization. Recognizing traffic signs58 is another
crucial task of autonomous driving. In this lab assignment, students are asked to
achieve this using a real-time object detection and classification algorithm, namely
Tiny-YOLO.59 Students implement an ROS node with their trained Tiny-YOLO
model. The workflow of traffic signs detection can be broken up into two steps: (i)
locating possible signs in a large image and (ii) classifying the cropped traffic signs
if it is indeed relevant and recognized by the trained model or dismissing it if not.
In this lab, students cover in detail the different steps needed to create a pipeline,
which enables localization and classification of traffic signs. The pipeline itself
looks as follows:
• Building data set: same as in LAB6 (see Figure 4(a) to (f)).
• Image annotation: we labeled the frames using an image annotation tool called
LabelImg.60 The traffic signs are first encompassed in a rectangular bounding
box then the resulting area was labeled according to the sign’s class.
Annotations are stored in an XML file following the same format used by
ImageNet and called PASCAL VOC. Each entry stores the coordinates,
International Journal of Electrical Engineering & Education 57(4)
width, height, and class label of its respective image (see Figure 5(a) and (b) for
an example).
Data normalization: Learning is more stable by reducing variability across the
training data, the network will generalize better to novel data, and the inputs
will fall within the useful range of your nonlinearities.
Data augmentation: Image augmentation is a necessary step to improve the
performance of the model and mitigate the downside of having a small training
data set for our deep neural network. This is achievable by synthesizing training
images using one or more processing methods such as random rotation, brightness change, and axis flips. imgaug61 is a powerful library implementing image
augmentation techniques for machine learning.
Getting Google Colab ready to use: Google Colaboratory, or shortly Colab,62 is
a free cloud service offered by Google for machine learning education and
research. It offers dedicated virtual machines equipped with NVIDIA Tesla
K80 cards for GPU acceleration and comes with an online Jupyter notebook
environment canceling the need for any setup on a local machine.
Image classification using different pretrained models: In this part, students
learn transfer learning and fine-tuning. Darknet-19 was used as an example
pretrained model.
Measuring network performance: Students put the model to the test with their
testing dataset and evaluate various performance metrics like accuracy, loss,
precision, etc.
LAB8—Parking in front of a visual marker. The goal of this lab is binding the vehicle’s
vision directly to its control. Allowing it to reliably navigate toward a visual
marker. The visual marker is determined by blob detection which is covered in
LAB5, what this lab adds on top of that is the ability to track the blob’s center
along the horizontal axis and issue the appropriate Ackermann steering commands
to direct the vehicle toward it.
LAB9—Histogram filters and 2D Kalman filter. For this lab, students deal with filtering,
which is a general method for estimating unknown variables from noisy observations over time. In particular, they learn the Bayes Filter and some of its variants:
The Histogram Filter and the Kalman Filter. They can apply the algorithms to the
robot localization problem.
Evaluation of educational program and discussion of results
To quantitatively assess the educational program’s impact on the above learning
process and get a tangible idea on the students’ satisfaction level, an electronic
survey was conducted at the last session of the summer internship program, in
which students had to grade their satisfaction in 14 questions form according to
the five-level Likert-type scale: strongly disagree, disagree, neutral, agree, and
strongly agree. Thus, they were able to express their opinion about how useful
Eken et al.
Table 4. Students’ survey questions.
Learning value
LAB0 enhanced my ability to understand the theoretical ROS lecture in a new way.
LAB1 enhanced my ability to understand the theoretical feedback control systems
lecture in a new way.
LABs 2-5 enhanced my ability to understand the theoretical image processing lecture
in a new way.
LAB6 and LAB7 enhanced my ability to understand the theoretical deep learning
lecture in a new way.
LAB8 enhanced my ability to understand the theoretical visual servoing lecture in a
new way.
LAB9 enhanced my ability to understand the theoretical localization lecture in a
new way.
Value added
The reproducible educational plan helped me to integrate different technologies into
working systems solutions.
I can learn similar topics in this way without such a training plan.
I can prepare such a reproducible educational study for other lessons.
Design and other issues
The ideas and concepts within the reproducible educational plan were presented
clearly and easy to follow.
The time allocated to the course was enough.
I was interested in this topic before I started to study it.
I am interested in this topic after having studied it.
I recommend this educational program to a fellow student.
the program was in reinforcing and improving their knowledge of the field. This
feedback from students is important because it gives instructors a good idea on
which characteristics of their teaching approach should be modified or not.
Allowing each new session to be an improvement over the previous one.
Moreover, the interaction between students and educators ensures the effectiveness
of the course, the maintenance of the quality of university education and permits
them to reflect upon their experiences among others. Questionnaire items are structured in three subscales: “Learning value,” “Value added,” and “Design and other
issues.” Following is an explanation of the aim of each item:
• Learning value encompasses the questions that try to probe students’ perceptions of how effectively the reproducible educational program is designed for the
purpose of teaching mini autonomous race car programming.
• Value added tries to assess the ability to learn and prepare similar systems and
how much of this ability is thanks to this program.
• Design and other issues focus on students’ judgment of the clarity of the course
and their interest in the program.
International Journal of Electrical Engineering & Education 57(4)
Figure 6. Student responses for the reproducible educational plan survey (number of
students ¼ 15). (a) Learning value (Q1–Q6) (b) value added (Q7–Q9) and (c) design and other
issues (Q10–Q14).
The full questionnaire is presented in Table 4. Answers are collected anonymously and the survey data are analyzed using Excel. The first six items make up
the Learning value subscale. The subsequent three items make up the value added
subscale. While the last five items relate to the design and other issues subscale.
Figure 6(a) to (c) represents the responses of the 15 students who took part in the
survey, the high percentage of “strongly agree” and “agree” answers indicates the
success of this reproducible educational program for mini autonomous car programming. About the learning value, 60% of students agree or strongly agree,
showing an important positive attitude toward the improvement of their abilities
in topics such as ROS, feedback control systems, image processing basics, and
visual servoing. Forty-seven percent of students agree or strongly agree that they
can integrate different technologies, but a relatively large amount of students are
neutral toward preparing such a reproducible educational notebook. Most of
the students answered positively regarding the educational design and other
issues section.
A reproducible educational program specifically focused on the teaching of mini
autonomous car programming has been presented in this paper. The proposed
Eken et al.
training program includes different lectures and laboratory assignments. Data
from students’ feedback indicate the success in achieving the learning objectives
of the program. The student response was satisfactory, showing a high degree of
interest in the proposed approach as well as satisfaction with it. We will increase
the number of students in the upcoming years for this reproducible open-source
robotic education program.
We plan to add a vehicle2vehicle communication module for better handling of
closed roads and congested traffic scenarios. The locations of traffic accidents and
jams are expected to be broadcast by other vehicles allowing the receivers to avoid
such roads and plan alternative routes.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Kocaeli University
Scientific Research and Development Support Program (BAP) in Turkey under project
number 2017/067HD. We would also like to thank OBSS for their support and
OpenZeka for their training under MARC program.
Süleyman Eken
1. Collberg C and Proebsting TA. Repeatability in computer systems research. Commun
ACM 2016; 59: 62–69.
2. Hothorn T and Leisch F. Case studies in reproducibility. Brief Bioinformat 2011; 12:
3. Ruvolo P. Dude, where’s my robot?: A localization challenge for undergraduate robotics. Proc AAAI ConfArtifIntell 31: 4798–4802.
4. Bihl T, Jenkins T, Cox C, et al. From lab to internship and back again: learning autonomous systems through creating a research and development ecosystem. In: Proc AAAI
Conf Artif Intell 33: 9635–9643.
5. Casser V, Pirk S, Mahjourian R, et al. Depth prediction without the sensors: leveraging
structure for unsupervised learning from monocular videos. In: Proc AAAI Conf Artif
Intell 33: 8001–8008.
6. Chi L and Mu Y. Deep steering: learning end-to-end driving model from spatial and
temporal visual cues. arXiv preprint arXiv:170803798 2017.
7. Fridman L, Jenik B and Reimer B. Arguing machines: perception control system redundancy and edge case discovery in real-world autonomous driving. arXiv preprint
arXiv:171004459 2017.
International Journal of Electrical Engineering & Education 57(4)
8. Snider JM, et al. Automatic steering methods for autonomous automobile path tracking.
Pittsburgh, PA: Robotics Institute, 2009.
9. Bohg J, Ciocarlie M, Civera J, et al. Big data on robotics. Big Data 2016; 4:
10. Workshop at ICRA,
icra2017workshoprrr (accessed 10 February 2019).
11. Workshop at ICRA in Brisbane, (accessed 10 February 2019).
12. IEEE RAS summer school on deep learning for robot vision, http://robotvision2019. (accessed 10 February 2019).
13. Code ocean, (accessed 10 February 2019).
14. Bonsignorio F. A new kind of article for reproducible research in intelligent robotics
[from the field]. IEEE Robot Automat Mag 2017; 24: 178–182.
15. Dahnoun N. Teaching electronics to first-year non-electrical engineering students. Int J
Electr Eng Educ 2017; 54: 178–186.
16. Nourbakhsh I, Crowley K, Wilkinson K, et al. The educational impact of the robotic
autonomy mobile robotics course. Technical Report CMU-RI-TR-03-29. Pittsburgh,
PA: Carnegie Mellon University, 2003.
17. Nourbakhsh IR, Crowley K, Bhave A, et al. The robotic autonomy mobile robotics
course: robot design, curriculum design and educational assessment. Auton Robots 2005;
18: 103–127.
18. de Gabriel JMG, Mandow A, Fernandez-Lozano J, et al. Using lego nxt mobile robots
with labview for undergraduate courses on mechatronics. IEEE Trans Educ 2011; 54:
19. Calnon M, Gifford CM and Agah A. Robotics competitions in the classroom: enriching
graduate-level education in computer science and engineering. Global J Eng Educ 2012;
14: 6–13.
20. Bousaba NA, Conrad CMHJM and Cecchi V. Keys to success in the IEEE hardware
competition. In: Proceedings of the American Society for Engineering Education Annual
Conference, pp.1–18.
21. Chris C. Learning with first LEGO league. In: McBride R and Searson M (eds)
Proceedings of society for information technology & teacher education international conference 2013. New Orleans, LA: Association for the Advancement of Computing in
Education (AACE), pp. 5118–5124.
22. Fike H, Barnhart P, Brevik CE, et al. Using a robotics competition to teach about and
stimulate enthusiasm for Earth science and other STEM topics. In EGU General Assembly
Conference Abstracts, EGU General Assembly Conference Abstracts 18: EPSC2016–10.
23. National robotics challenge, (accessed 10 February 2019).
24. Engineering and robotics learned young (early), (accessed
10 February 2019).
25. Self-driving fundamentals: Featuring Apollo,
ing-car-fundamentals-featuring-apollo–ud0419 (accessed 10 February 2019).
26. Mit 6.s094: Deep learning for self-driving cars, (accessed
10 February 2019).
27. McLurkin J, Rykowski J, John M, et al. Using multi-robot systems for engineering
education: teaching and outreach with large numbers of an advanced, low-cost robot.
IEEE Trans Educ 2013; 56: 24–33.
Eken et al.
28. Danahy E, Wang E, Brockman J, et al. Lego-based robotics in higher education:
15 years of student creativity. Int J Adv Robot Syst 2014; 11: 27.
29. Afari E and Khine M. Robotics as an educational tool: Impact of Lego mindstorms.
International Journal of Information and Education Technology 2017; 7: 437–442.
30. Crenshaw TL and Beyer S. Upbot: a testbed for cyber-physical systems. In: Proceedings
of the 3rd international conference on cyber security experimentation and test. CSET’10.
Berkeley, CA: USENIX Association, pp. 1–8.
31. Kelly J, Binney J, Pereira A, et al. Just add wheels: leveraging commodity laptop hardware for robotics and AI education. In: AAAI 2008 AI Education Colloquium. Chicago,
IL. Also appears in AAAI Technical Report WS-08-02. Menlo Park, CA: AAAI Press,
pp. 50–55.
32. Karaman S, Anders A, Boulet M, et al. Project-based, collaborative, algorithmic robotics for high school students: programming self-driving race cars at MIT. In: 2017 IEEE
integrated STEM education conference (ISEC). pp. 195–203. IEEE.
33. VESC—open source esc, (accessed
10 February 2019).
34. Stereo labs, (accessed 10 February 2019).
35. Scanse, (accessed 10 February 2019).
36. Sparkfun, (accessed 10 February
37. Jupyter notebook, (accessed 10 February 2019).
38. Kluyver T, Ragan-Kelley B, Perez F, et al. Jupyter notebooks—a publishing format for
reproducible computational workflows. In: Loizides F and Scmidt B (eds) Positioning
and power in academic publishing: players, agents and agendas. Amsterdam: IOS Press,
pp. 87–90.
39. Shen H. Interactive notebooks: sharing the code. Nature 2014; 515: 151–152.
40. Ros, (accessed 10 February 2019).
41. Kul S, Eken S and Sayar A. Distributed and collaborative real-time vehicle detection
and classification over the video streams. Int J Adv Robot Syst 2017; 14:
42. Jetson tx2 module, (accessed
10 February 2019).
43. Twigg P, Ponnapalli P and Fowler M. Workshop problem-solving for improved student
engagement and increased learning in engineering control. Int J Electr Eng Educ 2018;
55: 120–129.
44. Skruch P, et al. Control systems in semi and fully automated cars. In Mitkowski W,
Kacprzyk J and Oprzedkiewicz K (eds) Trends in advanced intelligent control, optimization and automation. Cham: Springer International Publishing, pp. 155–167. ISBN
45. Opencv, (accessed 10 February 2019).
46. Darpa grand challenge,
(accessed 10 February 2019).
47. Tensorflow, (accessed 10 February 2019).
48. Keras, (accessed 10 February 2019).
49. Montemerlo M, Thrun S, Koller D, et al. Fastslam: a factored solution to the simultaneous localization and mapping problem. In: In Proceedings of the AAAI national conference on artificial intelligence. Menlo Park, CA: AAAI, pp. 593–598.
International Journal of Electrical Engineering & Education 57(4)
50. Sermanet P, Eigen D, Zhang X, et al. Overfeat: integrated recognition, localization and
detection using convolutional networks, (2013).
51. Simonyan K and Zisserman A. Very deep convolutional networks for large-scale image
recognition. CoRR 2014; abs/1409.1556.
52. Redmon J, Divvala S, Girshick R, et al. You only look once: unified, real-time object
detection. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR),
pp. 779–788.
53. Racecar-training, (accessed 10 February
54. Matplotlib, (accessed 10 February 2019).
55. Segnet, (accessed 10 February 2019).
56. Motion-based segmentation and recognition dataset,
projects/VideoRec/CamVid/ (accessed 10 February 2019).
57. Karsli M, Satilmiş Y, Şara M, et al. End-to-end learning model design for steering
autonomous vehicle. In: 2018 26th Signal processing and communications applications
conference (SIU), pp. 1–4. IEEE.
58. Sat Ilm Iş Y, Tufan F, Şara M, et al. Cnn based traffic sign recognition for mini
autonomous vehicles. In: Swiatek
J, Borzemski L and Wilimowska Z (eds)
Information systems architecture and technology: proceedings of 39th international conference on information systems architecture and technology—ISAT 2018. Cham: Springer
International Publishing, pp. 85–94. ISBN 978-3-319-99996-8.
59. Redmon J and Farhadi A. Yolo9000: better, faster, stronger. In: 2017 IEEE conference
on computer vision and pattern recognition (CVPR), pp. 7263–7271.
60. labelimg, (accessed 10 February 2019).
61. imgaug, (accessed 10 February 2019).
62. Colab, (accessed 10 February 2019).