The Need for Empirical Survey-based Research

I have worked in research for a decade and a half, and I am privileged to have been a part of different research communities during this time. As a researcher, I was born in the design automation community, but I got increasingly involved in the real-time community as I grew up. This was very interesting for a young researcher and I learned a lot from this, because each community has its own perspective, practices, strengths, and weaknesses.

Once I joined ESI (TNO) back in 2016 and started working in applied research, I started reading papers and attending conferences in the modelling community. There, I came across empirical survey-based research that systematically investigated industry trends, needs and practices, and that studied adoption and perceived benefits and drawbacks of different technologies and methodologies. I immediately found this line of work incredibly useful as it elevated my understanding of what happened in industry from a collection of anecdotes based on conversations with a few people in a few companies to something that could capture the experience of hundreds of people across industrial domains. This was very helpful as it allowed me to get a perspective on how the research that was conducted in the field fit into the industrial reality of the men and women that are ultimately supposed to use the results. From an academic perspective, I also had the feeling that this line of work provided all the citations I needed for the introductions of my papers, as it helped me position my own work on modelling in a broader industrial reality.

Empirical research is an established research direction in social science, but also in technical fields, such as software engineering and to a lesser extent system engineering. However, there was no work like this in the area of real-time systems. I decided to change this and pitched the idea to Rob Davis, Mitra Nasri, and Geoffrey Nelissen and Sebastian Altmeyer during a meeting in Amsterdam in May 2019. To my delight and surprise, they all enthusiastically decided to work on this together, effective immediately.

Performing survey-based research may seem deceptively simple at first. After all, all you have to do is to make a survey with questions, ask people to fill it out, and analyze the results. It turns out each of these steps involve substantial challenges to ensure validity and quality of the final results. Luckily, there are well-established methods that provide guidance along the way. We had long discussions to precisely define the goals for our work, to identify the target population and decide how to invite a sufficiently large representative sample from it to take the survey without accidentally including people with the wrong expertise. Of course, we also needed to select the right questions to ask and decide how to phrase them to avoid misunderstandings. The latter is surprisingly difficult when the questions are sent to people with different backgrounds, working in different domains with their own definitions and jargon. It is hence a fine balance between being precise and concise.  Results had to be collected and analyzed, and the right statistical methods and tools selected to allow the survey results to be generalized to the wider target population that our sample represented. This involved a lot of work, but we always had the feeling this was one of the most important papers we had worked on.

Empirical Research Session at RTSS

It took us more than a year to get from the pitch in Amsterdam to a paper that was ready for submission. Another six month later,  it is my great pleasure to announce that our paper “An Empirical Survey-based Study into Industry Practice in Real-time Systems” has appeared at the 41st IEEE Real-Time Systems Symposium (RTSS). The paper presents results and observations from a survey of 120 industry practitioners in the field of real-time embedded systems. The survey provides insights into the characteristics of the systems being developed today and identifies important trends for the future. The survey aims to inform both academics and practitioners, helping to avoid divergence between industry practice and fundamental academic research. We could not fit everything we had to say into this paper, so there is also a supporting technical note entitled “A Comprehensive Survey of Industry Practice in Real-time Systems“. A separate session was dedicated to this work on the last day of RTSS 2020. The session began with a 25 minute paper presentation, which is available here:

The paper presentation was followed by a panel discussion involving three industry practitioners from the three main industrial domains covered by the survey: Marcelo Lopez Ruiz (Microsoft), representing the consumer electronics industry, Simon Schliecker (Volkswagen), representing automotive, and Stephen Law (Rolls-Royce), providing an avionics perspective. The panel discussed four key questions relating to the survey results. This included what important characteristics of real-time systems highlighted in the survey results are the most relevant with respect to their domains, as well as if there were any characteristics that were missing in the survey. There was also a discussion about  the most relevant trends in real-time systems development now, and looking ahead over the next 10 years. The panelists concluded by giving recommendations to the academic community, such as what areas to work more or less on, and what assumptions (not to)  make going forward. The opening statements from the panelists related to the four questions was pre-recorded and followed by a live discussion. The session finished after one hour, before there was time to take questions from the audience. A separate Zoom room was created for this purpose and to allow the interaction to continue, which it did for another hour! We were very pleased with the interest in this paper and in the session.

The pre-recorded part of the panel is available here:

Emerging Research Direction

I hope that this work is the first of many empirical research papers in real-time systems. There are many ways to continue with this line of work. First of all, others need to replicate our results to validate that they hold for different samples of the target population. For this purpose, we will be happy to transfer the survey we made on SurveyMonkey, such that it can be reused. Secondly, our survey was very broad and covers real-time systems across many application domains. More specific questions could be obtained if the focus was on a single domain, although the main challenge will be finding enough representative participants with a narrow focus. Thirdly, surveys are only one way of conducting empirical research. Another method sometimes used in software engineering is to use interviews, allowing more in-depth questions to be asked. However, the drawback of this method is that it is more time consuming to interview are large number of participants and to encode and analyze the results.

This direction in real-time system research is just emerging and we hope it will grow and become a well-established part of the research conducted in the community. This would help us better understand the industry we are trying to serve and help us close the gap between academic research and industry practice. A first important step is that this direction is recognized by all main conferences and journals in the area of real-time systems and explicitly included in the call for papers. You can play an important part here by helping us communicate the value of empirical research to others in our community and beyond.

Author bio: Benny Akesson received his MSc degree at Lund Institute of Technology, Sweden in 2005 and a PhD from Eindhoven University of Technology, the Netherlands in 2010. Since then, he has been employed as a Researcher at Eindhoven University of Technology, Czech Technical University in Prague, and CISTER/INESC TEC Research Unit in Porto.  Currently, he is working as a Senior Research Fellow at ESI (TNO) in Eindhoven. Since 2019, he is also Endowed Professor at the University of Amsterdam, where he holds the Chair of Design Methodologies for Cyber-physical systems. Prof. Akesson’s research interests include fundamental as well as applied aspects of model-based engineering and real-time systems, areas in which he has published more than 70 peer-reviewed publications and two books.