Judging

When you run multiple agents on the same task, judge agents can automatically evaluate the results and help you identify the best solution. Judges analyze code quality, correctness, and completeness—providing objective feedback that saves you time reviewing variants.

What Are Judge Tasks?

A judge task is a special task that evaluates other tasks in a group. Unlike regular tasks that modify your code, judges:

• Run after primary tasks complete
• Have read-only access to primary task results
• Analyze code quality, correctness, and completeness
• Produce evaluation notes and scoring
• Do not modify your repositories

Judge tasks appear in the task group alongside other variants.

How Judges Evaluate

When a judge task runs, it:

1. Reads all primary task results—patches, summaries, exit codes, logs
2. Analyzes the code changes each agent made
3. Reviews test results and error messages
4. Evaluates each variant on multiple dimensions
5. Generates a detailed report with scores and recommendations

Evaluation Dimensions

Judges score variants on:

• Correctness: Does the code solve the problem? Are edge cases handled? Do tests pass?
• Code quality: Is it readable, maintainable, and following good patterns?
• Completeness: Are all requirements addressed? Is anything missing?
• Performance: Is the implementation efficient? (when applicable)

Each dimension receives a score, and judges provide detailed notes explaining their reasoning.

Judges may use or add other dimensions based on the task context.

Automatic Judging

You can configure task groups to automatically launch judge tasks when primary agents finish.

Configuring Auto-Judge

When creating a task group, select which agents should serve as judges.

Multiple judges provide independent evaluations, reducing bias and increasing confidence in the results.

When Auto-Judge Launches

Judge tasks launch automatically when:

• All primary tasks have completed
• At least two variants finished successfully
• Multiple variants made file changes
• No follow-up instructions are pending

If conditions aren't met, auto-judge is skipped—but you can always launch judges manually.

Manual Judge Launch

You can launch judge tasks at any time:

1. Open the task group
2. Click the Judge button
3. Select which agents to use as judges
4. Judge tasks are created and queued

This is useful when auto-judge conditions weren't met, or when you want additional evaluation after making changes.

Review Mode

Judging compares variants against each other, so it needs at least two. When you select a single variant instead, the judge switches to review mode and evaluates that one solution on its own merits.

The difference is what comes back. A comparison judge names a winner; a review judge returns a verdict:

• Pass: the solution is correct and acceptable
• Needs Improvement: functional, but has issues
• Fail: incorrect or fundamentally flawed

To launch one, open the Judge panel, select exactly one variant, and choose your judges. The launch button changes to Launch Review. Verdicts appear on the judgment cards in the History tab.

Reviews are always launched by hand. Auto-judge only fires when at least two variants finished successfully, so it never produces a review.

A review judge still fills in improvement feedback, even when the verdict is Pass. Treat the verdict as the decision and the feedback as suggestions.

Judge Consensus

When multiple judges evaluate the same variants:

• Each judge scores independently
• Results can be compared side-by-side
• Consensus emerges when judges agree on the best variant
• Disagreements highlight areas worth closer review

If two out of three judges recommend the same variant, that's a strong signal. If judges disagree significantly, you may want to review their reasoning before deciding.

Using Judge Feedback

Judge feedback isn't just for picking a winner—it helps you improve the code.

Common Issues Judges Identify

• Test failures: Some tests aren't passing
• Edge cases: Boundary conditions not handled
• Error handling: Missing validation or exception handling
• Code style: Inconsistent naming or formatting
• Incomplete implementation: Features not fully implemented

Feedback Loops

After reviewing judge feedback:

1. Identify specific issues mentioned in the evaluation
2. Send follow-up instructions to the winning variant addressing those issues
3. The agent resumes and implements improvements
4. Optionally re-run judges to verify the improvements

This creates a refinement cycle where judges catch issues that agents then fix. See Providing Feedback for how to send judge feedback as a follow-up.

Keep Reviewing Until Pass

In review mode, CoderFlow can run that cycle for you instead of you repeating it by hand.

Tick Keep reviewing until Pass in the Judge panel before launching, and set a round limit. After each review, the feedback is sent to the variant automatically, the variant addresses it, and the same judges re-evaluate. That is one round.

The loop stops as soon as any of these is true:

• The review passes. With several judges, every one of them has to pass.
• The rounds run out. You set the limit when you launch, up to ten.
• The variant stopped changing code. If a turn produces no changes, the agent has effectively declined the feedback, and another round would only repeat itself.

The History tab shows the current round while the loop runs, with a Stop button, and the reason it stopped once it finishes. Group completion notifications are held until then, so you are notified once at the end rather than once per round.

With more than one judge, a round waits for all of them, then merges their feedback into a single follow-up so the variant addresses everything in one turn. A judge that fails to produce a verdict does not stop the loop as long as another one did.

Each round is one agent turn plus one run per judge, so the round limit is also your cost ceiling. Three is a reasonable default.

The loop is offered in review mode only. Comparison judging returns a winner rather than a verdict, so there is no pass condition to stop on.

Judges Don't Approve

Important: Judge tasks provide feedback and recommendations only. They do not:

• Automatically approve changes
• Commit or push code
• Mark tasks as winners

You make the final decision on winner selection and approval. Judges inform your decision—they don't make it for you.