We believe that the peer review process would benefit from structure and that a rigorous review can be achieved using a standardized instrument. The Rubriq scorecard is the cornerstone of a transparent peer review process that allows manuscripts to be uniformly evaluated outside the lens of one specific journal.
What is the main purpose of the scorecard?
By translating the essential elements of a well-constructed, well-communicated research story into quantifiable metrics, the Rubriq scorecard provides clear feedback to authors while generating thorough reviews that can be transferred from journal to journal. The ability to map scores from various sections or subsections of the scorecard to individual journals facilitates data-driven manuscript-journal matching, which helps authors quickly identify the highest impact, best fitting journal for their paper.
How does the scorecard work?
The scorecard is broken into three sections: Quality of Research, Quality of Presentation, and Novelty and Interest; these are each broken into subsections. At the heart of the scorecard is a set of rubrics for each subsection designed to help calibrate reviewers around the key items essential to a paper following standard scientific methodology. The rubric itself consists of a series of items that may be missing or inadequate in the manuscript. As reviewers select items from these lists, suggested scores are generated based on the importance of the item. Reviewers may adjust the subsection scores according to their judgement.
What's an R-Score?
The R-Score is the overall score for a paper that combines all three of the sections. Overall section scores are calculated from the reviewer input, weighted according to the importance of each subsection, and the scores from all three reviewers are averaged. The upper limit of the overall score is determined by the novelty and interest score. A well-executed study with limited interest may have high quality scores, but a low overall score because the novelty and interest value is low.
How has the scorecard been validated?
Our scorecard has already gone through several rounds of construction, review, testing and re-construction. Some highlights of the development are detailed in our white paper, available for download below. It shows the process of our scorecard development and validation to date. We are currently continuing cycles of testing and feedback, and expect that our beta release will provide additional input as we prepare for the final version for our live release (Phase Three).
Click to download our current white paper about “The Science of the Scorecard”
Click here to see a quick summary of our previous version changes
- Scorecards from 20 different biomedical journals were analysed to determine the essential items to include in the evaluation
- The items were grouped into three sections: Quality of Research, Quality of Presentation, and Scientific Importance (later changed to Novelty and Interest)
- Items (now called subsections) were rated as poor, weak, modest, strong, or very strong
- Descriptors were added to each of the subsections to help calibrate reviewers
- Impact Assessment condensed into a single Novelty & Interest rating, with 9 levels (poor to exceptional)
- After face validation test 1 (internal panel of Managing Reviewers)
- Descriptors rewritten to remove subjective statements and provide a more uniform gradation from low to high quality
- New format condensed the descriptors into items to be checked for each subsection
- Value categories (poor, good, etc.) were replaced by a visual-analog scale (VAS) for each subsection
- Subsection and section weightings were applied based on reviewer survey
- Overall score algorithm was established
- A deduction system was created, where reviewers could select applicable statements about the weaknesses of each subsection
- Each statement was assigned a deduction corresponding to the severity of the statement
- Deductions were taken sequentially from a maximum score of 10 to produce a suggested starting score based on the reviewer's input
- Reviewers were able to adjust the suggested score as they saw fit
- Face validation test 2 (panel of AJE Managing Editors with PhDs in biomedical fields)
- Statements were added in the areas of data repetition, background information, and consistency of data in text and tables
- Novelty and Interest ratings changed from discrete values (1-9) to a 0-10 Visual-Analog scale, anchored with descriptions at the high and low ends
- Refined text, reduced repetition, and addressed consistency of language around the items in the rubric
- Face validation test 3 (external panel of reviewers)
- A new subsection, Title, abstract, and Introduction, was added; subsection weightings were adjusted accordingly
- Items were refined
- “Conclusions” was moved to “Quality of Presentation” section
- Novelty and Interest section was broken into separate rubrics for “Novelty” and “Interest”
- Patent application underway
More information about our scorecard development will be posted when it is available.