EMSOFT 2020 PC Chair’s Report
EMSOFT, the ACM SIGBED International Conference on Embedded Software, is one of the three core conferences of the annual Embedded Systems Week (ESWEEK). The other two conferences are CASES, the International Conference on Compilers, Architecture and Synthesis, and CODES+ISSS, the International Conference on Hardware/Software Codesign and System Synthesis. The 16th edition of ESWEEK and the 19th edition of EMSOFT were held virtually this year.
A report about ESWEEK 2020 in general and the experiences of the General Chairs, Tulika Mitra and Andreas Gerstlauer, with the virtual conference format is available here. In this post, we focus on recounting the preparation for and running of EMSOFT specifically from our perspective as program chairs.
As program chairs, we were responsible for forming the program committee, running the review process, deciding on the technical program, and facilitating the presentation sessions. We worked together with the EMSOFT steering committee, chaired in 2020 by Christoph Kirsch and Wang Yi, and the ESWEEK organizing committee, chaired in 2020 by Tulika Mitra and Andreas Gerstlauer.
The articles are published in the IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems Volume 39, Issue 11.
Program Committee Selection
As usual, preparation begin almost immediately after the previous conference (EMSOFT 2019 in New York City). We shortlisted potential program committee members based mainly on the following criteria.
- Expertise in the core conference topics including formal methods, hardware architectures, hybrid systems modeling and verification, memory technologies, machine learning and neural networks, operating systems, programming languages, and real-time systems and scheduling. We sought to match expertise with the number of submissions in a given topic in the previous year.
- Publication and other involvement at EMSOFT. We especially tried to invite people who had presented articles at EMSOFT but who had never, or at least not recently, served on the program committee.
- Previous participation in the program committee. We sought a balance between inviting previous program committee members, and thus ensuring continuity and retaining valuable expertise, and seeking to renew the committee and introduce new perspectives.
- Geographical, institutional, and gender diversity.
The final program committee comprised 58 members and two chairs. 25 members came from institutions in the United States, 23 from institutions in the Middle-East and Europe, and 12 from institutions in Asia. 16 members (27%) had not served on the 2019 program committee. There were 10 women (17%) and 50 men (83%).
We would have liked to include more new members and also to improve the ratio of women to men, but not all our invitations were accepted and these constraints were difficult to solve.
We are very grateful to our colleagues who accepted to serve on the program committee in 2020 and who did their best to review, discuss, and select articles.
The deadline for abstracts coincided with pandemic lockdowns in many parts of the world and we were initially worried that we would not receive many submissions. We finally received, however, a record 216 abstracts, from which 138 journal-track papers were submitted. This compares with 107 submissions in 2019 and 122 in 2020. 50 submissions were received from the Asia-Pacific region, 43 from European countries, and 43 from North America.
The problem became, rather, how to handle the unexpected reviewing load, especially given that most program committee members were having to adapt quickly to online teaching and working from home. We decided to limit the number of reviews per member to 9 articles. This meant that 43% of papers received 3 reviews and 55% received 4 reviews, the remainder received 5 reviews. Review allocation involves solving a complex scheduling problem between bids, conflicts, paper topics, and reviewer expertise. Note that the text of the abstract is often critical in deciding which reviewers are allocated. Concise, factual statements of content and topics help to ensure that a submission is read and evaluated by the most appropriate experts!
Overall 494 reviews were completed of which 12% rated an article as “unfixable, not novel, out of scope”), 48% as “below the bar”, 35% as “above the bar”, and 5% as “award quality”. The reviewing period was followed by one week of online discussions. In many cases, these discussions were decisive, with reviewers convincing one another to change their score or opinion. When an article generated significant discussion, we requested that one of the participants draft a “meta-review” summarizing the different perspectives and reasons of the final decision.
After the discussions, 45 articles (32%) were accepted for the second round of reviewing. The authors of these articles worked to address reviewer comments and respond directly to questions and remarks. There was then a second week of online PC discussion with less articles to consider, but with harder questions to resolve. It is always unpleasant to reject papers in the second round after authors have gone to additional trouble and the decision was not taken lightly. Overall, 35 articles (25%) were accepted for presentation at the conference and appearance in the associated journal edition. This is a typical acceptance rate for EMSOFT.
The authors of one of the articles rejected in the first round wished to rebut some of the criticisms made in the reports. They noted that unlike EMSOFT, conferences like the ACM Conference on Computer and Communications Security (CCS) include a rebuttal period. We decided in this case, based on the reviews and discussion, to allow a rebuttal and requested the reviewers to reevaluate the article. While time-consuming for all concerned, this process was useful if ultimately unsuccessful.
We thank everyone who submitted an article to EMSOFT this year, regardless of whether it was accepted or not. We hope that the reviewer comments were helpful in any case.
EMSOFT also includes a Work-in-Progress track for shorter submissions with preliminary results. This year we received 25 submissions: 15 from Europe, 7 from Asia, and 3 from the United States. They were reviewed by a separate 9 member committee and 10 (40%) were accepted.
The Virtual Event
ESWEEK was originally supposed to be held this year in Shanghai. When it became clear that this would not be possible, it was relocated to Hamburg, and, finally, as the pandemic developed, became an online virtual event. We were very lucky that these uncertainties were managed so well by Tulika, Andreas, and the rest of the ESWEEK organizing committee, allowing us to focus on reviewing and the technical program.
The accepted articles were grouped into seven technical sessions each graciously chaired by a representative of the PC.
The sessions followed the form agreed upon by the ESWEEK Organizing Committee. For each journal-track article a 25 minute prerecorded video was made available to registered attendees prior to the event, and a 2 minute lightning talk was presented in a live session followed by a total of 20 minutes of questions and answers per session. Regrettably, Work-in-Progress articles were not presented in a live session as is usually the case. These strict timing constraints arose from the decision to run the conference in a four-hour window at the intersection of day or evening timezones in Asia, Europe, and North America. While it seems that the pre-recorded videos were little watched prior to the live sessions, there are plans to make them permanently available.
We prepared for the virtual event by running practice sessions with all presenters and chairs in the days leading up to ESWEEK. This allowed us to sort out various technical issues and familiarize the participants with the format. We had a lot of help both before and during the event, in particular from Aviral Shrivastava, the Virtual Conference Chair, Hoeseok Yang, the Web Chair, and our two student volunteers, Alexander Cathis and Somdip Dey, not to mention the overall management by Tulika and Andreas.
The actual live sessions over the Zoom videoconferencing software went well given the circumstances. There were between 40 and 50 participants connected for most sessions, and split roughly in half for the two sessions that ran in parallel. It was great to see and hear the presenters, and to have some questions from participants and chairs, but there is no denying that it was not as good as a physical event. In particular, we did not manage to recreate anything like the sociable atmosphere that usually characterizes EMSOFT. Some of us especially missed discussing science over coffee and beer with colleagues!
Two awards were presented at EMSOFT this year under the coordination of Jürgen Teich, the ESWEEK Awards Chair.
For the best paper award, three candidates were selected based on reviews and discussions. They were carefully considered by an award committee comprising the chairs and four members of the program committee. A decision was very difficult to reach, but the honour was finally awarded to Efficient Feasibility Analysis for Graph-based Real-Time Task Systems by Jinghao Sun, Rongxiao Shi, Kexuan Wang, Nan Guan, and Zhishan Guo.
The two runners-up were EM-Fuzz: Augmented Firmware Fuzzing via Memory Checking Instrumentation by Jian Gao, Yiwen Xu, Yu Jiang, Zhe Liu, Wanli Chang, Xun Jiao, and Jiaguang Sun; and Reachability analysis of linear hybrid systems via block decomposition by Sergiy Bogomolov, Marcelo Forets, Goran Frehse, Kostiantyn Potomkin, and Christian Schilling.
The Test-of-Time award was decided by a committee of three respected members of the EMSOFT community. They decided to give the award to the very deserving Interface Theories for Component-Based Design by Luca de Alfaro and Thomas A. Henzinger in EMSOFT 2001. Both Luca and Thomas “attended” the conference to accept the award and comment on the work described in their article.
Each EMSOFT PC chair serves for two years before joining the steering committee. Thus one of us (Linh) remains to chair EMSOFT 2021 together with David Broman, who recently accepted to become co-chair and then chair, while the other (Tim) is freed of the responsibility.
Below are the tentative dates for EMSOFT 2021 (and ESWEEK 2021):
- Abstract Submission: April 2, 2021
- Full Paper Submission: April 9, 2021 (firm)
- Notification of Acceptance: July 5, 2021
- Paper Submission: June 4, 2021 (firm)
- Notification of Acceptance: July 5, 2021
Proposals for ESWEEK
- Workshop Proposals: April 16, 2021
- Tutorial Proposals: April 30, 2021
- Special Session Proposals: April 30, 2021
We strongly encourage the community to submit papers to EMSOFT 2021!
Authors: Timothy Bourke is a Research Scientist at Inria Paris. His research is centered on rigorous approaches to modelling, programming, and verifying embedded control systems and their physical environments. He focuses particularly on applying interactive theorem provers and synchronous languages to practical problems. Prior to his current position, Bourke worked in the seL4 verification group in Sydney, and also in the ICT industry for several years. Bourke has a PhD from the University of NSW and a Bachelor of Computer Engineering from the University of Wollongong. He teaches at the École normale supérieure and the École polytechnique.
Linh Thi Xuan Phan is an Associate Professor of Computer Science at the University of Pennsylvania. She works in real-time, embedded, cyber-physical systems (CPS) and distributed systems. Her research focuses on theoretical foundations and systems techniques for safety, performance, and security guarantees. Recently, she has been working on predictable, real-time cloud platforms for CPS, IoT and NFV applications, methods for detecting and defending CPS and distributed systems against DDoS and Byzantine attacks, and provenance-based diagnosis techniques for data centers. She currently serves as the Secretary-Treasurer for ACM SIGBED.
Disclaimer: Any views or opinions represented in this blog are personal, belong solely to the blog post authors and do not represent those of ACM SIGBED or its parent organization, ACM.