General Reviewing Guideline
All Robotics and Machine Learning papers should be solid scientific papers, regardless of their specific area. Your review should be constructive, thorough, and polite.
We judge the merit of a paper based on six criteria: Relevance; Significance; Technical soundness; Novelty; Quality of Evaluation; and Clarity. For each criterion you have to assign a score between 1 and 10. You should also provide detailed comments justifying your evaluation along with suggestions for improving the paper. Furthermore, please provide specific information on which issues you would like the authors to address in their rebuttal.
Below we provide a detailed explanation of the different criteria and scores. While all criteria are important, we want to encourage highly original and novel papers, and therefore would like you to pay special attention to the "Novelty" criterion.
Here is some guidance on the meaning of the criteria and their numerical scores.
Relevance
What is the relevance of this paper to an Robot Learning audience?
1: Not relevant, 4: Moderately relevant, 7: Relevant to researchers in subarea only, 10: Relevant to general Robot Learning
Significance
Are the results important? Are other people (practitioners or researchers) likely to use these ideas or build on them? Does the paper address a difficult problem in a better way than previous research? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions on existing data, or a unique theoretical or pragmatic approach?
1: Not significant, 4: Moderately significant, 7: Significant, 10: Highly significant
Technical soundness
Is the paper technically sound? Are the concepts correct and accurate?
1: Has major errors, 4: Has minor errors, 7: Technically sound, 10: Major technical contribution
Novelty
Are the problems or approaches novel? Is this a novel combination of familiar techniques? Is it clear how this work differs from previous contributions? Is related work adequately referenced?
1: Not novel, 4: Moderately novel, 7: Novel, 10: Novel and innovative; will open up new areas of research
Quality of Evaluation
Are claims well-supported by theoretical analysis or experimental results? How convincing is the evidence in support of the conclusions? Are the authors careful (and honest) about evaluating both the strengths and weaknesses of the work? Is the evaluation appropriate for the contribution?
1: Not convincing, 4: Moderately convincing, 7: Convincing, 10: Very convincing
Clarity
Is the paper clearly written? Is it well-organized? (If not, feel free to make suggestions to improve the manuscript.) Does it adequately inform the reader? (A well written paper should provide enough information for the expert reader to reproduce its results.)
1: Poor, 4: Satisfactory, 7: Good, 10: Excellent
Overall Score
- Trivial or wrong or known: Clearly below Robot Learning threshold, I assume no further discussion is needed.
- A strong rejection: I will strongly argue for rejection.
- A clear rejection: I argue for rejection.
- An OK paper, but not good enough. A rejection: This should be rejected, although I would not be upset if it were accepted.
- Marginally below the acceptance threshold: I tend to think it should be rejected it, but having it in the program would not be that bad.
- Marginally above the acceptance threshold: I tend to think it should be accepted, but leaving it out of the program would be no great loss.
- Good paper, accept: It should probably be accepted, although I would not be upset if it were rejected.
- Top 50% of accepted papers: a very good paper, a clear accept. I advocate acceptance of this paper.
- Top 15% of accepted papers: an excellent paper, a strong accept. I advocate and will fight for acceptance.
- Top 5% of accepted Robot Learning papers: a seminal paper for the ages. Clearly an outstanding paper. I assume no further discussion is needed.
Confidence score
1: The reviewer's evaluation is an educated guess and it is quite likely that the reviewer did not understand central parts of the paper. Either the paper is not in the reviewer's area, or it was extremely difficult to understand
4: The reviewer is fairly confident that the evaluation is correct. It is possible that the reviewer did not understand certain parts of the paper, or that the reviewer was unfamiliar with a piece of relevant literature. Mathematics and other details were not carefully checked.
7: The reviewer is confident but not absolutely certain that the evaluation is correct. It is unlikely but conceivable that the reviewer did not understand certain parts of the paper, or that the reviewer was unfamiliar with a piece of relevant literature.
10: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature.