Motivation

In 2011 the German Traffic Sign Recognition Benchmark yielded the satisfying result that image sections containing traffic signs can be reliably recognized with state-of-the-art classification algorithms. With this in mind we gladly present today the German Traffic Sign Detection Benchmark (GTSDB).

In spite of strong advances in image processing research detection due to several studies on this topic, traffic sign detection is still a challenging real-world problem of high industrial relevance. A detailed comparison of different detector types and processing aproaches is, however, missing.

Traffic sign detection is a search problem in natural (outdoor) images. A useful detector must, therefore, be able to cope with rotation, different lighting conditions, perspective changes, occlusion and all kinds of weather conditions. During the creation of our database we took special care on a diverse and representative compilation of single image frames.

Humans are capable of detecting the large variety of existing road signs with close to perfect reliability in experimental setups. This does obviously not apply to real-world driving, where the drivers attention is regularly drawn to different tasks and situations.

Competition task

The competition task is a detection problem in natural traffic scenes. Participating algorithms need to pinpoint the location of given categories of traffic signs (prohibitory, mandatory or danger). This can and should be done for different parametrizations of the algorithm in order to receive different values for precision (i.e. the percentage of detection results that are actually traffic signs) and recall (i.e. the percentage of given traffic signs that were actually found). The performance is computed by an area-under-curve measure for the detector's precision-recall plot on the test dataset.

Although the problem domain of advanced driver assistance systems implies constraints on the runtime of employed algorithms, this competition will not take processing times into account, as this puts too much emphasis on technical aspects like implementation issues and choice of programming language. We want to keep technical barriers as low as possible in order to encourage as many people as possible to participate in the proposed competition.

We will provide example code concerning reading of images, writing results, and testing own implementations. In addition, there are many publicly available resources that can be used in order to access state-of-the-art methods useful for the competition, e. g., the OpenCV library for computer vision and the Shark library for machine learning.

Evaluation procedures

The competition begins with the public availability of the training data set, corresponding ground truth data, and additional technical documentation on the competition website (see Dataset and Schedule). Potential participants are invited to develop, train and test their solutions based on these data sets.

We will provide results of established algorithms as a baseline. The chosen algorithms exemplify several currently competing approaches on the problem of traffic sign detection.

Shortly before the competition ends, the test set, without ground truth, will be published on the competition website. Participants can compute results for this dataset in several differently parametrized runs and submit results online using the convenient web interface. The performance is directly evaluated and the results are displayed in a precision-recall curve as the computed total performance is shown in a leaderboard. This way participants can compare their performance immediately.

Ground-truth for the test set will be made available after the competition is closed to allow participants to access the full database.

Availability of datasets

The training data will be made publicly available on December 1, 2012. The test set will be made available on February 18, 2013. The submission website will be open until the IJCNN's paper submission deadline.

The German Traffic Sign Detection Benchmark is a single-image detection assessment for researchers with interest in the field of computer vision, pattern recognition and image-based driver assistance. It is introduced on the IEEE International Joint Conference on Neural Networks 2013. It features ...

• a single-image detection problem

• 900 images (devided in 600 training images and 300 evaluation images)

• division into three categories that suit the properties of various detection approaches with different properties

• an online evaluation system with immediate analysis and ranking of the submitted results

Image format

• The images contain zero to six traffic signs. However, even if there is a traffic sign located in the image it may not belong to the competition relevant categories (prohibitive, danger, mandatory).

• Images are stored in PPM format

• The sizes of the traffic signs in the images vary from 16x16 to 128x128

• Traffic signs may appear in every perspective and under every lighting condition

Annotation format

Annotations are provided in CSV files. Fields are seperated by a semicolon (;). They contain the following information:

• Filename: Filename of the image the annotations apply for

• Traffic sign's region of interest (ROI) in the image

• leftmost image column of the ROI

• upmost image row of the ROI

• rightmost image column of the ROI

• downmost image row of the ROI

• ID providing the traffic sign's class

You can download source code that will help you to read and compare the annotation files with your detector's results. For an explanation of the class IDs we refer to the ReadMe.txt in the download package.