AI-Only Review Mode – Member Help Guide

1. Introduction

AI-Only Review Mode provides a streamlined review workflow for simple challenges that are well suited for AI-based evaluation.

Once the submission phase ends, submissions enter an AI Review Phase, where AI Reviewers evaluate submissions and generate scores and feedback. Members can review the AI feedback and provide comments during this phase.

After the AI Review Phase, submissions enter an Approval Phase, during which copilots review member feedback and make score adjustments when appropriate.

At the end of the Approval Phase, winners are announced based on the final review scores.

2. What This Means for Each User

AI-Only Review Mode introduces a simplified review workflow. Let’s look at what this means for each user type.

2.1 For Copilots

Configuring AI-Only Review Mode

When creating a challenge, select AI-Only Review Mode in the review configuration.

All other AI Review configuration options are similar to those available in the standard AI Review workflow.

Since submissions are evaluated through the AI Review process, manual reviewers do not need to be configured for challenges using AI-Only Review Mode.

AI Review Phase

Once the submission phase ends, the challenge enters the AI Review Phase.

Once the submission phase ends, the challenge enters the AI Review Phase, which remains open for 24 hours.

During this phase, AI Reviewers evaluate submissions and generate scores and feedback.

As a copilot, you can:

• View AI review scores and feedback for all submissions

• Monitor feedback submitted by competitors on scorecard questions

• Track submissions that may require attention during the Approval Phase

No action is required during the AI Review Phase. Any score updates or adjustments can be made during the Approval Phase after member feedback has been collected.

Approval Phase

After the 24-hour AI Review Phase concludes, the challenge enters the Approval Phase, which remains open for 12 hours.

During this phase, copilots can:

• Review feedback submitted by competitors

• Review the associated AI scores and comments

• Update AI review scores when appropriate

This provides a human-in-the-loop validation process, ensuring that AI-generated results can be reviewed and adjusted when necessary.

Finalizing Results

At the end of the 12-hour Approval Phase:

• Winners are announced automatically based on the latest review scores

• Any score updates made during the Approval Phase are included in the final results

• If no updates are made, winners are announced based on the original AI Review scores

2.2 For Competitors (Submitters)

Note: As in the standard AI Review workflow, AI scores and feedback are available during the submission phase, allowing you to improve and resubmit your submission before submissions close.

AI Review Phase

After the submission phase ends, the challenge enters the AI Review Phase.

Once AI Review is complete, you can view:

• Aggregated score

• Individual workflow scores

• Detailed feedback for each workflow

You can:

• Click on individual workflow scores to view detailed feedback and comments

• Hover over the information icon next to the aggregated score to understand how it is calculated

Providing Feedback on AI Reviews

You can provide feedback on the AI Review for individual scorecard questions.

To provide feedback:

• Open the detailed feedback view for an AI workflow

• Navigate to the relevant scorecard question

• Use the thumbs up or thumbs down icons to rate the AI review

• Click Reply below the AI review comment to provide detailed feedback

• Submit your feedback

Your feedback is visible to the copilot and may be considered when reviewing and updating AI-generated scores.

Approval Phase

After the 24-hour AI Review Phase concludes, the challenge enters a 12-hour Approval Phase.

During this phase:

• Copilots review feedback submitted during the AI Review Phase

• Copilots may update AI-generated scores when appropriate

No action is required from competitors during the Approval Phase. However, any score adjustments made by copilots during this period will be reflected in the final results.

Winner Announcement

Once the 12-hour Approval Phase concludes:

• Final scores are locked

• Winners are determined based on the latest available scores

• Results are announced automatically

If score changes were made during the AI Review Phase, the updated scores will be used. Otherwise, the original AI Review scores will determine the final rankings.