CSEE&T 2020 will employ a full double-blind review process for research track papers. The papers submitted must not reveal the authors’ identities. Identities will only be revealed to the reviewers when final decisions have been made. The Organizers reserve the right to desk-reject without review any submission that does not adhere to the instructions outlined below.
This document draws on other guides from ASE 2017, the CAV 2016 FAQ, David Walker’s double-blind reviewing FAQ from POPL 2015, and includes content from the ICSE 2018 and ICSA 2019 site. Some sections have been copied verbatim.
Author Checklist
Authors must make every effort to honor the double-blind review process. In case of questions, please contact the PC Chairs. Please adhere to the following checklist:
- Omit authors’ names and institutions from your title page.
- When you cite your own work, refer to it in the third person. For example, if your name is Smith and you have worked on automated bug repair, instead of saying “We extend our work on information retrieval for finding and removing earwigs [0],” you should say “We extend Smith’s [0] work on information retrieval for finding and removing earwigs.”
- There may be cases in which the current submission is clear follow up of one of your previous work, and despite what recommended in the previous point, reviewers will clearly associate authorship of such a previous work to the current submission. In this case, you may decide to anonymize the reference itself at submission time. For example: “based on previous results [10]” .. where the reference is reported as “[10] Anonymous Authors. Omitted per double blind reviewing.” In doing so, however, please make sure that your new submission is self-contained and its content can be reviewed and understood without accessing the previous paper.
- Do not include acknowledgements of people, grants, organizations, etc. that would give away your identity. You may, of course, add these acknowledgements in the camera-ready version.
- In general, aim to reduce the risk of accidental unblinding. For example, if you use an identifiable naming convention for your work, such as a project name, use a different name for your submission, which you may indicate has been changed for the purposes of double-blind reviewing. This includes names that may unblind individual authors and their institutions. For example, if your project is called “GoogleDeveloperHelper,” which makes it clear the work was done at Google, for the submission version, use the name “DeveloperHelper” or “BigCompanyDeveloperHelper” instead.
- Avoid revealing the institution affiliations of authors or at which the work was performed. For example, if the evaluation includes a user study conducted with undergraduates from the CS 101 class that you teach, you might say “The study participants consist of 200 students in an introductory CS course.” You can of course add the institutional information in the camera-ready. Similar suggestions apply for work conducted in specific organizations (e.g., industrial studies). In such cases, avoid to mention the organization’s name. Instead, you may just refer the organization as “Org” or “Company,” etc. When appropriate and when this does not help too much in revealing the company’s name, you might mention the context (e.g., financial organization, video game development company, etc.).
- Avoid linking directly to code repositories or tool deployments which can reveal your identity. You may post anonymized links (with a warning that following said link may reveal authors’ identities), include links to anonymized code or deployments. When creating such repositories, a good practice can be asking somebody in your team to test the anonymization of the repository and of its content. In case anonymization is difficult to be achieved and you still want to provide availability of data/tools, you can simply state that you will link to the code or deployment in the camera-ready version.
Frequently Asked Questions
Q: What exactly do I have to do to anonymize my paper?
A: Your job is not to make your identity undiscoverable but simply to make it possible for our reviewers to evaluate your submission without having to know who you are. The specific guidelines are simple: omit authors’ names from your title page (or list them as “omitted for submission”), and when you cite your own work, refer to it in the third person.
Q: I would like to provide supplementary material for consideration, e.g., the code of my implementation or proofs of theorems. How do I do this?
A: On the submission site there will be an option to submit supplementary material along with your main paper. This supplementary material need not be anonymized; it will only be revealed to reviewers after they have submitted their review of your paper and learned your identity. Reviewers are under no obligation to look at this material. The submission itself is the object of review and so it should strive to convince the reader of at least the plausibility of reported results; supplemental material only serves to confirm, in more detail, the idea argued in the paper. Of course, reviewers are free to change their review upon viewing supplemental material (or for any other reason). For those authors who wish to supplement, we encourage them to mention the supplement in the body of the paper so reviewers know to look for it, if necessary. E.g., “The proof of Lemma 1 is included in the non-anonymous supplemental material submitted with this paper.”
Q: Is there a way for me to submit anonymous supplemental material which could be considered by a reviewer before she submits her review (rather than potentially non-anonymous material that can only be viewed afterward) ?
A: There is no official channel for doing this: the submission site only accepts potentially non-anonymous material. That said, authors have been known to release a TR, code, etc. via an anonymous hosting service, and to include a URL to that material in the paper. However, we discourage authors from using such tactics except for materials that cannot, for some reason, be uploaded to the official site (e.g., a live demo). We emphasize that authors should strive to make their paper as convincing as possible within the submission page limit, in case reviewers choose not to access supplemental material. Also, see the next question.
Q: Can I supplement my submission using a URL that links to auxiliary materials instead of submitting such materials to the EasyChair system directly?
A In general, we discourage authors from providing supplementary materials via links to external web sites. It is possible to change the linked items after the submission deadline has passed, and, to be fair to all authors, we would like to be sure reviewers evaluate materials that have been completed prior to the submission deadline. Having said that, it is appropriate to link to items, such as an online demo, that can’t easily be submitted. Needless to say, attempting to discover the reviewers for your paper by tracking visitors to such a demo site would be a breach of academic integrity. Supplementary items such as PDFs should always be uploaded to EasyChair.
Q: I am building on my own past work on the WizWoz system. Do I need to rename this system in my paper for purposes of anonymity, so as to remove the implied connection between my authorship of past work on this system and my present submission?
A: No. The relationship between systems and authors changes over time, so there will be at least some doubt about authorship. Increasing this doubt by changing the system name would help with anonymity, but it would compromise the research process. In particular, changing the name requires explaining a lot about the system again because you can’t just refer to the existing papers, which use the proper name. Not citing these papers runs the risk of the reviewers who know about the existing system thinking you are replicating earlier work. It is also confusing for the reviewers to read about the paper under Name X and then have the name be changed to Name Y. Will all the reviewers go and re-read the final version with the correct name? If not, they have the wrong name in their heads, which could be harmful in the long run.
Q: I am submitting a paper that extends my own work that previously appeared at a workshop. Should I anonymize any reference to that prior work?
A: No. But we recommend you do not use the same title for your new submission, so that it is clearly distinguished from the prior paper. In general there is rarely a good reason to anonymize a citation. One possibility is for work that is tightly related to the present submission and is also under review. But such works may often be non-anonymous. When in doubt, contact the Program Chairs.
Q: Am I allowed to post my (non-blinded) paper on my web page? Can I advertise the unblinded version of my paper on mailing lists or send it to colleagues? May I give a talk about my work while it is under review?
A: As far as the authors’ publicity actions are concerned, a paper under double-blind review is largely the same as a paper under regular (single-blind) review. Double-blind reviewing should not hinder the usual communication of results.
Q: Will the fact that this conference is double-blind have an impact on handling conflicts-of interest?
A: Authors will enter their conflicts of interest with the PC in the conference software system when they register their papers. This information will be used to avoid conflicts of interest in reviewing.
A conflict of interest is defined as a situation in which a reviewer can be viewed as being able to benefit personally in the process of reviewing a paper. For example, if areviewer is considering a paper written by a member of their own group, a current student, their advisor, their close collaborators (including recent coauthors) then the outcome of the review process can have direct benefit to the reviewer’s own status.
As an author, you should list PC members which you believe have a conflict with you. Tihs conference will apply the ACM SIGSOFT conflict guidelines. Also, please identify institutions with which you are affiliated; all employees or affiliates of these institutions will also be considered conflicted.
If a possible reviewer does not meet the above criteria, please do not identify him/her as conflicted. Doing so could be viewed as an attempt to prevent a qualified, but possibly skeptical reviewer from reviewing your paper. If you nevertheless believe that a reviewer who does not meet the above criteria is conflicted, you may identify the person and send a note to the Program Chairs.
Q: How should I refer to my own papers that are under consideration somewhere else?
A: If the work being cited is in press, it may be cited as “Anonymous.” We would remind authors that submissions to this conference must be original and not under consideration elsewhere.
For reviewers
Q: What should I do if I if I learn the authors’ identity?
A: If at any point you feel that the authors’ actions are largely aimed at ensuring that potential reviewers know their identity, you should contact the Program Chairs. Otherwise you should not treat double-blind reviewing differently from regular blind reviewing. In particular, you should refrain from seeking out information on the authors’ identity, but if you discover it accidentally this will not automatically disqualify you as a reviewer. Use your best judgment.
Q: The authors have provided a URL to supplemental material. I would like to see the material but I worry they will snoop my IP address and learn my identity. What should I do?
A:You can use VPN, the Tor browser, or proxies such as HideMe, to anonymously access URLs for supplemental material.
Q: How do we handle potential conflicts of interest since I cannot see the author names?
A: The conference review system will ask that you identify conflicts of interest when you get an account on the submission system. Please see the Author FAQ above to decide how to identify conflicts. Feel free to also identify additional authors whose papers you feel you could not review fairly for reasons other than those given (e.g., strong personal friendship). The PC Chairs will be responsible for conflict management but can only work with the information provided.
More information about bias in merit reviewing
(Taken verbatim from ICSE2018 webpage)
For those who are interested in motivations for double blind reviewing, a very well argued, referenced and evidenced article in favour of double blind review processes for Software Engineering conferences can be found in the blog post by Claire Le Goues: https://www.cs.cmu.edu/~clegoues/double-blind.html.
And here is a list of double-blind resources from Robert Feldt: http://www.robertfeldt.net/advice/double_blind_reviewing/.
And a more formal study of the subject by Moritz Beller and Alberto Bacchelli: https://peerj.com/preprints/1757/