Artificial intelligence in the automotive industry
Blackbox neural network: Interactive visualization improves understanding of decision-making processes in autonomous vehicles
In self-driving cars, reliable Convolutional Neuronal Networks (CNN) are essential. With their help, artificial intelligence (AI) is supposed to automatically recognize other traffic participants. However, the more autonomously the car drives, the greater the demands on the safety of the algorithms. In order to protect the human life, a deep understanding of the inner-processes of these neural networks is necessary. However, a CNN operates as a black box per se, so the complex decision paths are difficult to understand and are making it hard to assess any safety risks. These challenges can be solved by appropriate visualization. For this purpose, ARRK Engineering GmbH has developed a validation tool for analyzing the decision-making processes. Thanks to its interactive visualization, the program allows a deeper insight into each layer of a CNN. All weights of the neurons can be manually adjusted to see their impact on the final object recognition. Furthermore, the influences of different confounding factors as well as certain training methods can be easily detected and thus the CNN can be optimized in a later step.
Interactive visualization of decision-making processes
The interaction of the individual neurons in the numerous layers of a CNN is extremely complex. Each layer and each neuron perform special tasks in the recognition of an object - for example, a rough sorting according to shapes or the filtering of certain colors. However, each step contributes in different manner to the success of correct object recognition and, can in some cases even worsen the result. This complexity leads to the fact that the importance of individual neurons for the decision has been inscrutable so far. Therefore, ARRK Engineering has developed an interactive and user-friendly graphical interface to visualize these paths. "In this way, the decision-making process of a CNN can be visually represented," Diviš said. "In addition, the relevance of certain neurons to the final decision can be increased, decreased or even eliminated. In real time, the tool immediately determines the impact of these changed parameters after each adjustment has been made. Thus, the importance of certain neurons and their task can be more easily identified and understood. "The streaming of the data can be paused at any time for stress-free and convenient analysis. During this stop, the individual elements can then be examined in more detail via intuitive operation.
For this visual baking, the experts at ARRK Engineering chose the cross-platform programming interface OpenGL to ensure the greatest possible flexibility. This means that the software can be used universally on any device - be it PC, cell phone or tablet. "We also placed particular emphasis on optimizing the calculation and the subsequent graphical display," explains Diviš. "Therefore, frame per seconds (FPS) in particular were checked in our final benchmark tests. Within this framework, we were able to determine that even when processing a video and using a webcam, the frame rate was stable at around 5 FPS - even when visualizing 90 percent of all possible feature maps, which is roughly equivalent to 10,000 textures. "Despite the large amount of graphical information and data, no FPS fluctuation are thus to be expected.
Analysis of the critical and anti-critical neurons
For teaching the CNN within ARRK Engineering's visualization tool, the deep learning APIs TensorFlow and Keras are used as a basis, serving as a flexible implementation of all classes and functions in Python. Other external libraries can also be easily connected. Once the neural network has been sufficiently trained, the analysis of critical and anti-critical neurons can begin - the Neurons' Criticality Analysis.
Increased safety through deeper understanding of CNN's
With the visualization tool, ARRK Engineering enables a graphical validation of neural networks. Thanks to the software in conjunction with the NCA, further steps can now reduce safety risks in autonomous driving through additional mechanisms for plausibility checks. "Our goal is to minimize or better partition the number of critical neurons so that we can rely on robust image recognition of the AI," Diviš sums up. "So we're already looking forward to feedback from the field, as this will allow us to further optimize the tool." At the moment, for example, ARRK Engineering is already working on adding object detection to the classification. Here, however, there is still the challenge that, in addition to the classification, the coordinates of the objects must also be visualized in a clear manner.