“The Conference is Dead! Long live the Conference!” AKA The Case for Changing the Current Conference Submission Model in Real-time and Embedded Systems | Part II

tl;dr proposing a rolling deadline and a “revise and resubmit” mechanism for real-time  conferences; PC commitments/chairs for one full year; starting from 2021.

In Part I of this series on introspecting on the conference model in our community, I highlighted some systemic issues that warrant deeper discussion. In this second part of this series, I lay out a series of proposals and a new model that we can discuss, as a community, and potentially adopt. 

I am proposing that the real-time community move towards a VLDB-style multi-deadline, revise and resubmit model for its conferences to address most of these issues. What this would look like: 

  • Multiple deadlines per year for the conference/community with the frequency of deadlines to be determined. 
  • PC chair(s) would serve for one full year 
  • The PC members would also serve for one full year
  • Review cycles start as soon as one of the deadlines passes. For instance, if the deadline is the first of every month, then the papers are assigned to the respective reviewers right away and they must submit reviews within some prefixed time window (say, 3 weeks/ 2 months)
  • Decisions for the paper would be one of:
    • Accept (akin to a minor revision in journals)
    • Revise and resubmit (similar to a major revision) — authors are provided detailed feedback and given a firm deadline (say one month). At the end of this time period, the same reviewers will decide whether the revised version is satisfactory for the paper to be accepted.
    • Reject — this paper cannot be submitted to the conference for one year. Authors of rejected papers should also be able to submit a rebuttal that the PC and chair(s) can read and decide whether to move it to accepted/revise status. 
  • Accepted/revision papers have one to two months to make their changes.
  • An accepted paper is immediately archived on IEEE or ACM. This will result in a faster dissemination of the results. 
  • There are two ways to tie in this rolling deadline model to the two major real-time conferences:
    • Have separate rolling deadlines/committee/organizers for each OR
    • Use the same organizers, committee members and rolling deadlines for both. For instance, any papers accepted by August 1 of a calendar year will appear in RTSS while any paper accepted by January 1 will appear in RTAS that year. This will ensure consistency for the entire year and elevates both excellent conferences to be top tier ones. 
  • Ability for authors to include comments and revision history if a previously rejected paper is resubmitted. 
Collaboration
Collaboration by Real-Time Conferences/community is required for success.

To start with, the rolling deadline model could be applied for one conference (say RTAS). Once all the problems have been resolved, both conferences can move to the new model.  

The main advantage of this proposed model? Better quality papers! Authors no longer need to rush while submitting papers. They can wait for a little while longer and submit the paper when they are ready! This will substantially improve the quality of the papers being submitted to the conference and improve the overall research output of the community. 

This model also reduces the load on reviewers at any point in time by spreading it out over multiple cycles. For instance, instead of reviewing 10-15 papers at once, I could do two or three every review cycle. This will lead to better quality reviews, once again improving the paper quality! There will also be consistency across papers since the program committee will serve for the entire year. Hence, they will be able to calibrate papers submitted in a particular sub area throughout their review periods. There is also a lower workload for the chair(s) at any point in time. 

The power imbalance is reduced since authors can now push back and provide rebuttals. These rebuttals can be longer and more detailed and sent to the same reviewers. In addition, PC chairs (or associate chairs) can be actively involved in “reviewing the reviews” and the rebuttals. Hence, any misunderstandings or fallacies during the review cycle can be easily addressed. In addition, the ability to mark the paper as “revise and resubmit” provides an opportunity for authors to fix problems pointed out by the reviewers or improve the paper with better writing/more results. Have I already mentioned that this would improve the quality of papers overall, especially the accepted papers? 

By allowing authors of rejected papers to attach prior reviews and revision information with subsequent submissions will reduce the variance between different program committees. Reviewers can now look at the paper holistically and, allow me to suggest a radical approach here, accept the paper if the revision has addressed all of the suggestions from the prior submission! This prevents a paper from bouncing around endlessly and, to reiterate, improve the paper quality significantly. Of course, the committee can reject the paper again if the authors haven’t addressed the prior comments in a satisfactory manner or introduced new flaws while attempting to do so.

The rolling deadline model reduces the travel burden for PC meetings since they will be held more often (once a month?). Hence they can be conducted via videoconferencing and for shorter amounts of time (say 1-2 hours each), thus providing more opportunities for researchers to interact and/or mentor younger colleagues. Such a method will also reduce the inequities faced by those disadvantaged by the current models as mentioned earlier. I believe that the community will significantly benefit as a result. 

I would also like to reiterate the advantage of archiving the paper online as soon as it is accepted and properly formatted. The scientific community gains earlier access to research results and the papers also start garnering citations earlier. A win-win situation if ever there was one. The conferences will now be about the presentations. Since the papers have been made available much earlier, both the audience as well as the authors will be better prepared thus leading to better conference presentations and interactions.

Of course no model is without some disadvantages and I will try to list them out and address them to the best of my ability here:

  1. Increased load on reviewers/organizers. Instead of a single review window, PC members must deal with multiple deadlines. While this is true, the review load each time is lower (perhaps 2-3 papers). This may be acceptable and easier to manage. I personally find it easier to find time to read two or three papers at a stretch than 10 — 15. If we increase the size of the program committee (since they serve for a full year) this load can be reduced even further.
  2. Additional work for reviewers since they have to re-evaluate accepted and “revise and resubmit” papers. I believe that this is a price we must pay as a community especially since it leads to better quality papers and a better experience for authors.
  3. The qualitative difference between RTSS and RTAS (in the minds of some community members) can be a problem. For instance, if someone wants their paper to be published in RTSS but it was accepted in November (recall that the paper would appear in RTAS) then they may be upset. There are multiple ways to deal with this: 
    1. The authors know the guidelines ahead of time and it is their responsibility to wait and submit it at the right time of year.  
    2. PC chair(s) decide whether the papers belong to RTSS/RTAS and can move a paper around regardless of when it was accepted. This could create controversy and also leads to arbitrary decisions which we must try to avoid.
    3. The community comes together and decides that both RTSS and RTAS are of  the same caliber!1  So authors should be happy regardless of which conference “publishes” it. In fact CSRankings already considers the two to be equivalent
  4. VLDB reported an increase in the total number of submissions. It is unclear whether this will affect us negatively but it will be good for the community and encourage a more broader participation. 
Rotating Deadlines are a Good idea!
Rolling Deadlines are good.

One final reason in favor of rolling deadlines is that many CS other communities are moving towards similar models. VLDB was one of the first to do so. Major conferences in the security community, viz. IEEE S&P (the conference formerly known as Oakland), ACM CCS, USENIX Security and NDSS have all moved to the rolling submission model.2 The main privacy conference, PETS, has also done so. Other major systems conferences, such as NSDI, SIGMETRICS and MobiComm, have adopted similar models. EuroSys, while not moving to a rolling deadline model yet, has recently announced that they will include author responses and revision options. So there is a chance that our community might be left behind in what is quickly becoming a “best practice”. In fact, we have the advantage of being a relatively small community (compared to these others) and, hence, nimble. So moving to a new model may be easier for us. We also have the hindsight at learning from the challenges faced, and mistakes made, by these other communities. 

In conclusion, I believe that this more flexible, modern conference submission process will be beneficial to the real-time community. It will likely increase the quality of papers as well as reviews, reduce the per-instance load on the program committee, increase participation and bring balance to the force (ok maybe only bring a better balance between the reviewers and authors). 

p.s.: Some further details:

  1. When to start. I would say that we should start the rolling deadlines on June 1, 2021 (in time for RTAS 2022). So RTSS 2021 would remain unchanged. Hence, any papers submitted between June — December 1, 2021 would be eligible for acceptance to RTAS 2022. Any papers submitted January 1, 2022 — July 1, 2022 would be considered for RTSS 2022. 
  2. The larger program committee can have associate chairs (or area chairs). Hence these area chairs can channel papers in their area to the appropriate subset of reviewers and track their progress, reviews and revision closely. This, if done right, can also improve the quality of reviews since the area chairs can flag problematic reviews (incorrect ones, badly written ones, etc.) and either ask the reviewer to redo them or assign the paper to an additional reviewer. 
  3. The associate/area chairs can be given the titles similar to associated editors/sub-editors of journals. This might help with their tenure cases, perhaps. 

p.p.s.: I have created a Google Doc to collect feedback on these ideas. It can be a place for the community to come together and discuss this issue. Another way to do this could be via a public github repository, akin to the method followed by the IEEE S&P community. In fact the security community has done an excellent job documenting not just their journey and discussions through the switch to rolling deadlines but also their steady-state processes as  well as the roles for program committees. I would encourage anyone interested in this process to read them as they are quite instructive.

___
Footnotes:

1 Since the community and organizers for both conferences have significant (if not complete) overlap, this should be relatively straightforward.

2 They all have different frequencies for their rolling deadlines.

___

Author bio: Sibin is a researcher in the areas of systems, security and resiliency. He is currently Researching Assistant Professors at the University of Illinois at Urbana-Champaign [UIUC]. The real-time, embedded and cyber-physical systems communities have been his home for nearly 2 decades. While not trying to break systems or indoctrinating the next generation of students, he also likes to break (err, try to improve on) “systems” like conferences, publication mechanisms and so on. These posts stem from that. Thanks and I’ll see you at my TED talk.

DisclaimerAny views or opinions represented in this blog are personal, belong solely to the blog post authors and do not represent those of ACM SIGBED or its parent organization, ACM.

Leave a comment