[ICCPS’22] Best Paper: interpretable Detection of Distribution Shifts in Learning Enabled Cyber-Physical Systems

Post by: Souradeep Dutta, Yahan Yang, Ramneet Kaur, Insup Lee
09 Dec 2022
0

Introduction

Autonomous systems with learning-enabled components (LECs) rely on deep neural networks in order to achieve high performance for various applications. It is well known that neural networks are vulnerable to distribution shifts (e.g., weather changes and adversarial perturbations). This vulnerability raises the safety and robustness concerns of learning-enabled cyber-physical systems (CPS) in the real world. For instance, in an advanced emergency braking system (AEBS), the perception-based LEC takes images from the camera as inputs to determine the distance between the car and its front object. A reinforcement learning (RL) controller uses the estimated distances as inputs to decide whether the vehicle needs to brake to maintain a safe distance from the front object. Suppose the training data of the camera only had cars as front objects and achieved high performance. In reality, when the leading vehicle changes from a car to a bike, it introduces a distribution shift. This leads to an incorrect distance estimation and finally leads to a crash with the front object. This scenario indicates the necessity of detecting distribution shifts for a cyber-physical system. The current state-of-the-art out-of-distribution (OOD) detection methods achieve promising performance on high-dimensional data like images. However, the black-box nature of those methods cannot provide a reason for their OOD decisions and may lead to a high false alarm rate. The users need more information to trust the decision.   

To overcome these challenges of OOD detection in learning-enabled cyber-physical systems, we propose a memory-based detector to provide a real-time and interpretable method to perceive anomalous inputs for LECs. 

The memory-based OOD detector could detect deviations (obstacle changes from car to bike) from in-distribution data.

Scenario: Autonomous Driving 

We consider an autonomous driving scenario here. Previous work has proposed a physical attack that can mislead autonomous cars by painting lines on the road. When a car is at the right turn corner, the car recognizes this as a left turn because of the painted lines. It goes off the road and hits the fence. This adversarial perturbation essentially causes a distribution shift from training data and leads to a crash. (In our experiments, we simulated the clean and adversarial autonomous driving scenarios using CARLA).  

Left: Clean images of a right turn corner. Middle: Painted lines drive the car to turn left and hit the fence. Right: The memory-OOD detector perceives the adversarial painted lines as the deviation from training distribution.

Using our memory system, we can efficiently compress the training data into a small number of memories and detect all crashes beforehand with a low false alarm rate. The running time of the algorithm is well within the sampling period of the system, implying that our OOD detection method is real-time. Moreover, compared to the previous OOD detection methods, the memory-based OOD detector is able to provide interpretability of the decision. In particular, our algorithm is capable of generating a heatmap to visualize the attributes responsible for OOD-ness. For a sample labeled as OOD, we find the memory with the smallest distance and highlight the difference between the out-of-distribution data and the memory at the pixel level. As shown in this adversarial attack case, our algorithm judges the OOD-ness of this sample based on pixels that include the adversarial lines in the image. This provides a reasonable interpretation to the user to trust the detector’s decision.   We additionally evaluate our method on OOD-ness due to weather change which is also a challenging scenario for autonomous driving. More details can be found in our paper. 

Scenario: Driving with LiDAR inputs

Left: Here is a setting where the car should take a right turn on an L-shaped track. Middle: The dots show the distance estimates as provided by the sensor in a real environment. It matches well with the position of the obstacles when there is no reflection from the wall. Right: Due to reflection from the left wall, it gives a false impression of no obstacle to the left of the car.

Neural Networks with LiDAR inputs also suffer from distribution shifts relative to the training data. The OOD data incurred can lead to severe accidents in practice. Consider a car from the F1/10 Autonomous Racing Competition. It navigates square tracks using only LiDAR measurements to judge obstacles and make general orientation decisions. The neural network controller is trained with simulated data where the environment’s surface is assumed to be non-reflective since it is challenging to model the reflectiveness of the environment. However, a missing LiDAR array happens when the car navigates in the environment with reflective components. In the specific case shown in the figure, the reflection from the wall causes the LiDAR rays to return to the sensor much later than expected. This causes the car to believe that no obstacle is on the left. 

Previous work shows that the number of crashes increases when an F1/10 car operates in highly reflective surroundings. The hypothesis is that the distribution shifts from LiDAR experiments (missing LiDAR rays) may lead to a crash. The question is if we can detect distribution shifts, can we alarm the system before the actual crash? Our experiments show that our OOD detector could protect cars from crashing 80% of the time and could alarm the system around 12 timesteps before the actual crash happens.  

Memory-based OOD detector

Overview of the memory-based OOD detector

Memory serves as an essential part of cognitive development at the early childhood stage. Inspired by the human memory system, we introduce a  system to explicitly store prototypical training data points as memories. At training time, we cluster the training data into several groups using a medoid-based clustering technique. The medoids selected for each cluster are collected as memories. The distance metric for building the clusters is the structural similarity index metric (SSIM), which measures the similarity between images based on the human perception system. For a given test datapoint, our detector computes the similarity distance between the test datapoint and each memory in the memory system. Then, we apply an abstract kernel density estimation method to estimate the probability of a test datapoint being drawn from a distribution captured by the memory system. Data points that are not admitted by the memory system are labeled as OOD samples. Since the system deals with time-series data in most scenarios, we additionally adopt a shifting window-based approach to threshold the number of OOD samples in a window to flag distribution shifts and raise the alarm.  

Conclusion

Detecting anomalous inputs for learning-enabled components is an essential ingredient for the safety and reliability of cyber-physical systems. In this work, we propose a simple and interpretable out-of-distribution data detection method and show its promising performance in various autonomous driving scenarios. Our memory-based OOD detector shows the potential of interpretable OOD detection for LECs in CPS.

Authors: Yahan Yang, Ramneet Kaur, Souradeep Dutta, and Insup Lee are with the University of Pennsylvania, Dept of Computer and Information Science. This paper is the winner of the Best Paper Award at the International Conference on Cyber-Physical Systems (ICCPS) 2022.

Disclaimer: Any views or opinions represented in this blog are personal, belong solely to the blog post authors and do not represent those of ACM SIGBED or its parent organization, ACM.

Tags: