“The Conference is Dead! Long live the Conference!” AKA The Case for Changing the Current Conference Submission Model in Real-time and Embedded Systems | Part I
tl;dr single, annual, deadlines; lack of revision history for resubmissions; delays between paper submission and availability to the community; large load on reviewers; environmental costs of physical PC meetings — all problems in the current conference model in use in the real-time and cyber-physical systems communities.
We have all been there. That one review which cost us a paper acceptance and months, even years, of effort goes down the drain. Maybe the reviewer “didn’t get it”. Or “if only the reviewers had asked and we could have clarified!”. Or any number of reasons why a paper was rejected. Maybe it was the tight deadline that forced us to submit a paper before it was really complete — “if only we have another day, week, nay a month!” 1. The conference-heavy publication mechanisms in Computer Science have become a source of frustration for both authors as well as reviewers. The latter have to deal with the pressure of reviewing an ever increasing number of papers, albeit within shorter windows, thus potentially reducing the quality of reviews and feedback. “Why can’t these authors ever write their motivation, implementation and results clearly and solve the halting problem at the same time? Is that too much to ask?” The very advantage of the conference publishing mechanism, i.e. shorter turnaround times, now often works to being a disadvantage. Entire social media groups and message boards abound with jokes about “Reviewer Number 2” for these and related reasons.
As Computer Science is one of the top majors for incoming freshmen, the graduate school enrollment is also increasing. Combined with the increasing number of (advertised) faculty positions in the last few years, it seems obvious that there will be a lot of research in this area in the coming years, translating to an increased number of paper submissions. Along with that comes an increased review load for program committee (PC) members. Even in 2013, it was estimated that Computer Scientists review as many as 100 papers per year! If anything, this number has only increased in the intervening years. The increased pressure to publish leading to a higher workload for reviewers will introduce noise and variance into the reviewing system often leading to frustration for both, authors as well as reviewers. We must introspect about our current conference publication models and perhaps overhaul the system. The net result could be less pressure for everyone involved, thus leading to better quality papers in general.
The current method practised in the real-time community is a “one-shot” model, i.e. the authors have one deadline per conference per year. The organizers and reviewers for the conference then have a fixed amount of time (usually 1-2 months) to review, discuss and finalize the program for the conference. The papers are usually made available to the attendees during the actual conference and then to the rest of the community shortly thereafter. This model, that has seemingly worked for decades, has shown to have numerous problems of late. I will lay out each of them in the following paragraphs.
1. A rush to submit papers by a firm, once-a-year deadline forces authors to send in papers that are probably not polished enough. Hence, they’re either written in a hurry or could have included more (and often better) results or insights. The alternative is to either wait for a whole year or submit to a different conference which means sending it to a venue that may not be the best fit for the work.
Typically, if the reviewers do their work correctly, such papers may get rejected. Now the authors are in a similar quandary as before — either revise and submit the paper during the next cycle (one year later) or send it (revised or otherwise) to a different venue.
“But we have two top tier conferences, RTSS and RTAS so why not submit to the other one if your paper is rejected?” 2
2. Lack of revision history and inconsistency of reviews between different program committees means that a paper that has received feedback, and has qualitatively been improved by addressing all reviewer comments, may still be rejected by the next venue (or even the same venue a year later). Research shows that while reviewers will often agree on the best and worst papers under submission, the large number of papers that fall in the middle have a lot of variance in acceptance/rejection decisions. Thus, a paper can bounce around for many years and take up a significant amount of human effort before it is either published or dropped as being hopeless by the authors.
3. The often six months+ delays in the submission of a paper and its availability means that it takes longer for the scientific results of the paper to take hold and even longer for it to garner citations. Of course this is assuming that the paper is accepted on the first attempt.
4. Depending on the size of the PC and conference, typically reviewers are asked to provide feedback on anywhere from 10-15 papers at once. While they may have 1-2 months to read and write the reviews, it is often the case that many PC members wait until close to the deadline to submit their reviews. I have been told by multiple reviewers that the papers were read on the flight over to the PC meeting (in itself a problem) and some reviews were only completed during the PC meeting itself! 3. This can result in rushed low quality reviews. But having to review such a large number of papers all at once also leads to increased pressure and workload for the PC members.
5. The requirements that all PC members must meet in person results in a large carbon footprint that could have been avoided if either there were fewer papers to discuss at any given point in time or the meetings were conducted over video/teleconferencing 4. While this may seem to be a tertiary issue to what is being discussed here, I believe that as scientists, it behoves us to be more responsible citizens of the planet.
1 Such stringent “in person” requirements also disadvantage many people, e.g. those with small children (and disproportionately women as a result), people with disabilities and those from countries from where international travel is either very hard (say due to visa requirements) or expensive.
So what is the solution then? Are we to do away with conferences entirely? What about the networking and mentorship opportunities that PC meetings provide? Maybe the authors should reflect carefully before submitting a paper that they think is “not ready”? Fortunately, there is a solution that addresses all of this while increasing the quality of papers! Tune in to Part II of this article next week for details.
1 The author is of the belief that at least a few of their rejected papers would have been “award worthy”, if not for the time crunch under which they were submitted.
2 Sadly, not everyone in the community believes that RTSS and RTAS are equivalent. Note that EMSOFT and ECRTS are also excellent top-calibre conferences in the field but I don’t mention them here since I’m unaware of their internal workings.
3 Some of these issues may be alleviated in RTSS/RTAS of late due to the large gap between the deadlines for review submission and actual PC meeting. But the issue still remains for conference reviews in the field and computer science in general.
4 Though this might become the norm in a post-COVID world.
Author bio: Sibin is a researcher in the areas of systems, security and resiliency. He is currently Researching Assistant Professors at the University of Illinois at Urbana-Champaign [UIUC]. The real-time, embedded and cyber-physical systems communities have been his home for nearly 2 decades. While not trying to break systems or indoctrinating the next generation of students, he also likes to break (err, try to improve on) “systems” like conferences, publication mechanisms and so on. These posts stem from that. Thanks and I’ll see you at my TED talk.
Disclaimer: Any views or opinions represented in this blog are personal, belong solely to the blog post authors and do not represent those of ACM SIGBED or its parent organization, ACM.