Reviewing Guidelines

ISMAR is a high­ quality conference with a competitive submission process. For example, in 2016, ISMAR only accepted 25.2% of submitted papers. ISMAR has a rigorous reviewing process that is similar to the processes used by ACM SIGGRAPH and ACM UIST. Every submitted paper and poster is subjected to this process.

As a reviewer, your work is of vital importance to the conference and we rely on your expertise and good work to generate an outstanding program. The key points while reviewing are:

• The main criterion is a judgment of the degree to which the submitted paper contributes substantial new research or scholarship to the field of Mixed and Augmented Reality.
• Reviewers rate the paper using a 6-point ranking scale, and provide a written evaluation. The written evaluation needs to support the reasoning behind the numeric ranking.
• After the review period, there is a discussion period, where all reviewers can see the other reviews. Reviewers may adjust their evaluations. Please use this period to seek a consensus evaluation.

First off: Reviewer Profile

First, log into the Precision Conference System (PCS), and complete a profile describing your expertise in the different topic areas for submissions. List all topics where you feel qualified. Be honest, and don’t just list all areas of interest, but the ones that you feel confident about reviewing. This will help the program committee members to judge if you are the best person to do a specific review.

In your profile you can also add comments about your previous experience as a reviewer: indicate for which ISMAR conferences you have reviewed, as well as any other related conferences or journals. If it’s your first time reviewing, the program committee will be able to see this and take it into account as well.

What to look for:

The main criterion for accepting a paper is the degree to which the submitted paper contributes substantial new research or scholarship to the field of Mixed and Augmented Reality. A good submission should also be technically correct and adhere to the standards of academic writing. Minor errors and flaws can be corrected and should not be reason to reject a paper. Be a little forgiving of poor English; it would be too bad if a truly novel paper was rejected because of poor English that could be corrected before final submission.

Numeric ratings and written evaluation

Submissions are rated in the review form on a numeric scale from 1 to 6, where 1 is “Definitely reject”, and 6 is “Definitely accept”. Be as precise in your rating of a submission as possible. Moreover, your written evaluation must support the numeric rating. The written evaluation should always indicate why the specific rating was given; if a submission should be accepted, be sure to state the reasons for it; if a submission should be rejected, again list the arguments against it. For example, your review may be discounted if you rate a paper highly but only describe its flaws, or give a very brief review.

Paper length

You should consider paper quality versus length according to a contribution-per-page judgment. For example, a paper with eight pages should have more contribution than one with four pages. Please evaluate the paper based on the presented length and form. Do not discount a shorter paper, because the authors could have shown more applications or evaluated against another concept or system. Similarly, if possible, do not recommend shortening a long submission, unless you can clearly articulate how shortening the paper would make it stronger. If your judgment is that the paper cannot be accepted in its submitted form, then please explain your reasoning.

Papers vs. Posters

Posters, as in previous years, are considered to be the first presentations of novel ideas without necessity for substantial implementation or testing, or novel applications without significant new technical advancement. The main criteria for judging poster submissions are novelty and usefulness to the ISMAR community, as well as the estimated interest within the ISMAR community to discuss the ideas with the author, in order to help the authors further mature the work.

2-page ISMAR poster abstracts are not considered archival publications, so a paper that builds on related work previously presented as an ISMAR poster is fine (and encouraged!), but the paper should cite the poster. This is true even though the 2-page poster abstract appears in the IEEE Digital Library. In particular, do not consider the 2-page poster abstract to count against the novelty of the paper!

“Flawed” papers

Reviewers should be careful to not immediately dismiss a submission that appears to have a flaw (e.g., in a user study), but otherwise make a contribution. It is possible that such a paper would be accepted if the issues were addressed in some way during the shepherding process, e.g., by removing the associated prose, or adding prose (e.g., in the conclusions of the paper) that acknowledges the issue.

Discussion period and second review cycle

Following the first review period, the program committee member assigned to the submission will open a discussion about the evaluation to arrive at a consensus decision on the submission. This step is important to make the final decision more comprehensible to the authors. Please follow the invitation and participate in the discussion. You will be able to read the other reviews, post messages and revise and update your own review. There will be a second round of reviews for papers which are to be discussed at the PC meeting. Each of those papers will receive two additional reviews. The program committee member will usually also let you know their recommendation.

Presentation of the review

In writing your review, please be polite to the authors. Even if you rate a paper poorly, you can critique it in a positive voice. As part of polite reviewing practice, you should always state what is good about a paper first, followed by your criticisms. If possible, you should offer suggestions for improvement along with your criticism. Think about the type of review feedback you would like to receive on a rejected paper you submitted.

In addition, in order to thrive as a community, ISMAR needs new people to attend, bringing new energy and new ideas. If a student or person from another technical community, who does not yet know how to write a successful ISMAR paper, has a paper rejected with very harsh reviews, the may well give up on ISMAR as a community and take their efforts elsewhere. This is the psychology of the paper author: if a paper is accepted, then harsh reviews are fine (“that reviewer didn’t like it, but so what, it got in anyway!”), but if a paper is rejected, then harsh reviews carry enormous weight. Therefore, in order for ISMAR to continue to thrive, we need for reviewers to be especially careful when writing reviews of papers that are likely to be rejected.

Timely reviews

Please finish your review on time. After all reviews are completed, the discussion period will commence. Then, the senior reviewer, who invited you to review the paper, will write a meta­-review, which summarizes the reviews and provides the rationale for the suggested paper decision. All of these tasks have tight time limits, and rely on reviews being completed by the deadline.