We highlight a dangerous pitfall in the state-of-the-art evaluation methodology of deep learning algorithms, as applied in several CPS and IoT application spaces, where collecting data from physical experiments is difficult. The article is inspired by the real experiences of the authors. An extended version appears in the IoT-AE Workshop in conjunction with MILCOM 2022 [1]. Few would disagree today…

Picture credit: Pixabay Machine learning is en vogue, being applied to many classes of problems. One of them is cybersecurity, where ML is used to find vulnerabilities in code, simulate attacks, and detect when an intruder has breached a system's defenses. Ignoring that intrusion detection is an admission of defeat (it comes into play when your system is already compromised!)…