Cultivating the Ground for Effective Rebuttals in Networking and Systems Venues
Published:
I recently had the pleasure of serving as a TPC member for The Web Conf 2024 (Systems and Infrastructure for Web, Mobile, and WoT track). When I read the CFP, I was particularly excited to see the rebuttal process, a prospect I had not been exposed to before — neither as an author nor as a reviewer.
Unlike other communities, rebuttals in Networking and Systems venues are rare. Sigcomm dabbled with it on multiple occasions but never built consensus, MobiSys/Mobicom seem to be experimenting with it but I’m not aware if the likes of NSDI, OSDI etc. have utilized rebuttals with any reasonable permanency.
Thus, prior to WWW, my view of rebuttals could at best be descried as vicarious, mainly consisting of misgivings passed on to me either by fellow PhD students (back in grad school) or faculty who had direct experience. Needless to say, I deemed it to be of limited benefit if not completely futile.
When WWW’s rebuttal phase commenced, I eagerly awaited author responses. However, much to my disappointment, the rebuttals did little to change any reviewer scores or the eventual fate of the papers (the ones one my pile) and just like that my vicarious misgivings became first-hand.
Since then, I have reflected multiple times on why rebuttals are ineffective in Networking and Systems venues. In my view, one potential reason is lack of formal training with rebuttals — for authors in writing them and for reviewers in evaluating them.
Barring generic web advice on writing rebuttals, which is available aplenty, I believe rebuttals need stronger institutionalization, especially in Networking/Systems community. We seem to be stuck in a cycle where we don’t do rebuttals often and thus the ones we do lack execution quality and are judged poorly.
The reluctance for rebuttals is therefore understandable and this inertia will likely continue till the ground is sufficiently prepared. Thus I make the following three suggestions which focus on learning, understanding and attaining experience with rebuttals in pedagogical settings.
Augmenting seminar courses. Faculty often deliver seminar courses where students typically take turns to present papers and write critiques/commentaries. In these courses presenters can be additionally tasked to act as a corresponding author by including a rebuttal to critiques/commentaries in their presentation, answering questions in the process and generating debate.
Shadow PCs should contain shadow authors. Networking/Systems conferences often run Shadow PCs which are typically serviced by PhD students. These Shadow PCs should consider two changes (i) include rebuttals even if not included in the main conference and (ii) assign shadow author roles to participants (in addition to reviewer roles) where the shadow authors write rebuttal for one or more paper(s). A competition for best review and rebuttal can incentivize interest and eventually embellish students’ resumes.
Parliamentary debate style reading groups. Most research groups run weekly reading groups, these groups should consider dividing the students in for and against groups for the given paper(s) and debate them for an outcome. This may be a better use of time instead of simply showing up to these meetings.
I believe training students to write rebuttals and encouraging them to revisit their judgements can prepare the ground for rebuttals to be more organically accepted in Networking/Systems conferences by PCs of the future, thus the community should look to plug this gap. Towards this, I’d be greatly interested in knowing if anyone has looked to implement these or other similar ideas in their sphere of influence.
As a parting thought, I feel we are at times guilty of over indexing on imparting area expertise to PhD students but lose sight of more durable goals of a PhD degree – the skills to read, write, think, present and judge.
And besides, in this age of LLMs, who is interested in Networking and Systems expertise anyway 🙂