Automated Review uses Artificial Intelligence to support the analysis of responses submitted in assessments. This feature helps standardize grading criteria, reducing review time and increasing the accuracy of the final score.
Where to Find It
You set up automated review when creating or editing a question of the following types: Free text, Audio, Video, File, Whiteboard, Conversational interview, Free coding, Project or Remote IDE.
Access: Library → Questions → New Question → Automated Review
How to Activate
Within the question editor, locate the Automated Review section.
Click on Add Criterion.
Fill in the required information:
Name: Define a title for the criterion (e.g., "General Architecture and Scope").
Description: Detail what the AI should evaluate (e.g., "Check if the system delivers a global real-time clock...").
Weight: Choose a relevance score from 1 to 5, depending on the impact on the final score.
If the criterion is purely observational (no impact on the score), toggle the Extra Criterion option.
Click Save.
Grade with AI
In the automated review section, right above the criteria, you'll find the Grade with AI option. It comes on by default: every question with review criteria is graded by the AI.
Turn it off when you'd rather have your team grade the answers to that question. The answers then go to manual review, and the criteria stay on the question to guide whoever grades them.
When you add the question to an assessment, the option comes along with its library value, like the weight and the duration. To change it for that assessment only, open the question in the Sections list, click Edit in its settings and use Grade with AI. The change applies to that assessment only and leaves the library question as it is.
After the talent submits
Once the talent finishes the assessment, the AI grades the answers and the score updates on its own. Until then, the question shows as Processing in the individual result, and the results list shows how many questions are processing next to the score. If the AI can't grade an answer, such as a recording with no audio, the question goes to Manual review. See Understanding the results of the evaluations.
Best Practices
Use clear and objective descriptions: The more detailed the criterion, the better the AI results.
Combine technical and behavioral criteria: (e.g., Code Quality + Adherence to the Proposal).
Review weights: Balance the impact of each item to reflect its importance in the total score.
Score Calculation
The final score for the question is calculated based on the configured criteria and their respective weights. Each criterion receives an individual grade assigned by the AI (from 0 to 100%).
The calculation follows this formula:
$$\text{Final Score} = \frac{\sum (\text{criterion grade} \times \text{criterion weight})}{\sum \text{weights of all criteria}}$$
Example:
Criterion | Weight | AI Grade (%) | Weighted Result |
General Architecture | 2 | 80 | 160 |
Documentation | 1 | 100 | 100 |
Main Functionality | 3 | 60 | 180 |
Total | 6 |
| 440 |
Calculation: $440 \div 6 = 73.3\%$
In this example, the final score for the question would be 73.3%.
Note: Criteria marked as Extra are not included in the calculation—they are only displayed as observations for the human reviewer.