With the rapid developments in the computer vision field, a lot has changed with regards to the possibilities of this technique. There have been numerous cases of real time applications which have been developed and put into practice. Some of these examples include: self-driving cars, AI-driven crop machines on farms, detecting weeds and removing them and even the cashier-less Amazon-Go stores.
The past couple of months, Xomnia data scientist Romano Vacca worked together with an internal team of Nederlandse Spoorwegen (NS) during a Proof of Concept (POC), in which they tried to answer the question:
Can state of the art computer vision techniques recognise train signals, as well as distinguish the different colours in real time during a train ride, in such a way that this information may assists the cabin driver in its decision making
The Setup
For this project, one specific train on a specific track was chosen. This track was almost in a straight line, with one big curve in the middle. Furthermore, most of the track was a single track, which made the scope a bit easier.
Moreover, a decision had to be made on a position to place the camera, taking into account certain rules such as it should not be in the eyesight of the cabin driver. The camera was connected to a small computer, together with gps and internet connection. The data was sent in realtime to the cloud using an IoT service. Dedicating time and looking for the best way to set it up was seen as very valuable.
The Data
Having a camera record all day can lead up to quite a lot of data. There was no wifi connection possible to send the data to the cloud. This meant we had to use a SIM card for data transfer. While using a Sim card can be convenient, it is also an expensive way of transmitting gigabytes of data. To minimise the amount of data, while still having the right quality and quantity, we decided to only record 16 hours a day, from 7 in the morning to 11 at night on weekdays. This way, we would have bright and dark images to have the model learn to detect the signals in all the possible scenarios. Other scenarios that were taken into account were sunny days, rainy days, foggy days and other weather conditions that can occur.
The Model
Looking at the requirements of a model that has to provide realtime information about certain objects, YoloV3 was chosen. This object detection algorithm is known as one of the fastest, while still having a good accuracy. Detecting signals on the track was not that difficult since traffic lights are one of the objects most sophisticated models are pre-trained on. So, having the weights of the model adapt to train signals which are a little bit different than regular traffics, did not cost a lot of time.
Next to the algorithm for the signals, some business rules also needed to be taken into account. An example: If there are two signals next to each other, which one is relevant for the train? If these two signals have the same status e.g. green, then there is no problem, however if one is green and the other is red, this could have some serious consequences. To tackle this problem, using some basic computer vision technique such as canny edge detection, a method was created that detected the train track the train was on. With this knowledge, the signal closest to the track can be determined to the one that holds the appropriate information.
The initial goal was to demo this in real time, having the model provide information to the cabin driver. Unfortunately, due to the COVID-19 circumstances, this couldn’t happen.
As for the next steps, the advanced analytics team of the NS will continue working on this project. They focus on improving the accuracy as well as taking more objects into account so even better advice could be given to the cabin driver.