> ## Documentation Index
> Fetch the complete documentation index at: https://docs.platform.toloka.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Quality control

> How the platform checks labeled data — the quality criteria each output is judged against, who reviews the work, the verdicts you get back, what a failed check triggers, and what review costs.

Quality control is on by default on labeling nodes. You define criteria for acceptable labels and choose a review path. Quorum also offers a [no-review mode that relies on offboarding rules](#offboarding-rules-on-quorum-nodes).

![Quality control: the criteria the agent writes feed a reviewer — an LLM or a person, chosen by the review path — which turns a labeled item into per-criterion verdicts; a failing verdict sends the item back to be redone or regenerated.](https://mintcdn.com/toloka-platform/e2y6apyB8pyO00uJ/images/explanation/quality-control/quality-flow.svg?fit=max\&auto=format\&n=e2y6apyB8pyO00uJ\&q=85\&s=f9084aac5f020df37c9e41b60e996ad5)

The nodes differ in who reviews the work and what a failed check sets in motion.

## Quality criteria

A quality criterion is a named check on one output field. It defines a pass condition and a fail condition. It can also include an instruction of up to 5,000 characters seen only by the LLM reviewer.

The agent writes the criteria from the task you describe, then you review them and either tell the agent what to change or edit them by hand on the node's **Quality Requirements** card.

A node cannot deploy until its criteria clear a floor enforced by [Self-check](https://docs.platform.toloka.ai/docs/data/concepts/self-check.md): at least one criterion, each with a name filled in, each targeting one of the node's own output fields, and each pass and fail condition at least 10 characters long. On a Generation node, whether experts or an LLM produce the labels, every output field must be covered by at least one criterion. The Quorum node relaxes this requirement, so you can check a subset of its outputs.

## Who reviews the work

The review path is who checks the labels against the criteria. It is a per-node choice, and the paths a node offers depend on its kind:

| Node | Review paths available |
| - | - |
| Expert labeling (Generation) | LLM review, human review, or both; or customer review (on request, see [Customer review](#customer-review)) |
| Quorum | LLM review, or none (offboarding rules instead) |
| Synthetic labeling | LLM review, or customer review (on request, see [Customer review](#customer-review)) |

LLM review is the default wherever it is available, and it grades every item. Human review, available on the expert-labeling node, checks a sample rather than every item: you set the share reviewed, from 10% to 100%, and the time a reviewer is given per item, at least 30 seconds or 3 seconds per quality criterion, whichever is greater. When both paths run together, the LLM grades every item and human reviewers double-check a sample. Synthetic labeling uses no experts, so it offers no human review. The Quorum node's no-review path is covered under [offboarding rules](#offboarding-rules-on-quorum-nodes).

## The review model

When an LLM reviews the work, it uses a review model. This per-node setting is separate from any model that generates synthetic labels. The platform has a short list of models, each with its own per-call price, and preselects a recommended one. The agent keeps that default when it builds the node. To use a different model, open the node and pick it under its **Quality Assurance** settings. On a Quorum node, you can also ask the agent to change the review model. On a Generation node, change it yourself under **Quality Assurance**. The choice covers all criteria on the node. The node's [exam](https://docs.platform.toloka.ai/docs/data/reference/nodes/generation.md#exam) uses the same model for the checks that need one; an exam answer compared directly against the golden answer needs no model. When the review mode names no model — human-only review, or a Quorum node with no reviewer — those exam checks use the recommended default.

## Verdicts

Every reviewed criterion returns a verdict with a written reason. The verdict is pass or fail unless the reviewer cannot reach either one. In that case, the verdict gives one of these reasons: the item could not be processed, the submitted data was not valid input, a field the check needed could not be read, the check itself failed, or the check was skipped.

The verdict's shape follows the output it covers:

* a plain output returns one verdict per criterion.
* an output that repeats within an item returns a list of verdicts, one per repetition.
* on a Quorum node, each criterion returns one verdict per expert, and a repeating output returns a list per expert.
* when LLM and human review both run on a node, each criterion returns one verdict from each, side by side.

Alongside the verdicts, a reviewed item carries a **review metadata field** with details about the review. This extra output field is named after the node and ends in `_qa_metadata`.

## When a check fails

A failing verdict can send the work back.

On expert-labeled nodes, a rejected submission is returned for correction, or reassigned, so the item gets another pass before it counts as done. The platform handles this rework automatically.

A task your team rejects under [customer review](#customer-review) is not redone: the item fails at the node.

Under synthetic labeling, a label that fails its check is generated again and rechecked, up to one further attempt (two attempts total). Those retries consume billable tokens like any other generation, and total token spend stays within the run's contracted maximum. See [Label with an LLM instead of experts](https://docs.platform.toloka.ai/docs/data/guides/synthetic-labeling.md) for how this shows up in the results.

## Offboarding rules on Quorum nodes

To use Quorum without a reviewer, choose **No QA, Offboarding Only**, set an overlap of at least 3, turn off the per-expert task cap, and add at least one offboarding rule on the **Offboarding** card. Quality is then enforced by offboarding experts who persistently disagree with the majority on a chosen single-label output field. Answers are not graded, rejected, or changed.

See [Offboarding rules](https://docs.platform.toloka.ai/docs/data/reference/nodes/quorum.md#offboarding-rules) for the settings.

## Customer review

> [!NOTE]
>
> Customer review isn't part of the standard setup. To turn it on for your team, contact support and describe the cases you plan to use it for. Until it's on, the mode doesn't appear under **Quality Assurance**, and the API, CLI, and SDK answer `403 The "customerReview" feature is not enabled for this team`.

Customer review is a review path on a Generation node in which your own team accepts or rejects every task the node produces. No LLM review or human review runs on that node. Quality criteria are still required, because a reject names the criteria the task fails. The Quorum node doesn't offer customer review.

To use it, choose **Customer review** on the node's **Quality Assurance** card. It works with expert labeling and synthetic labeling alike.

### What to expect

* Your team has 7 days to decide each task, counted from when the task is ready for review. A task nobody decides in that time is accepted automatically: the expert is paid and the task is billed.
* Items wait at the node until their review closes, so the run moves at your team's review pace.
* A rejected item fails at the node and isn't redone.

### How a review goes

* **Accept** is final: the expert is paid, the task is billed, and the item moves on to the next node.
* **Reject** needs the criteria the task fails and a comment that explains what is wrong. The platform then checks that reason against the task and the criteria you named, and the review takes no decisions until the check finishes. If the reason holds, the reject is final: the item fails at the node and nothing is billed for the task. If the platform can't confirm the reason, the task comes back to your team with the platform's explanation, and you can accept it or reject it again with a clearer reason.
* Each review allows up to 10 checked rejects. After that, accept the task or send a forced reject.
* A forced reject skips the check and is final at once. It is available only through the API and the CLI, and only after the platform has sent a reject of the same review back to you.
* A reject that is still being checked at the deadline closes when the check finishes: the reject is final if the reason holds, and otherwise the task is accepted automatically.

### Where to decide

In the run's **Items** tab, the **In review** filter shows the items that wait for your team. Open an item and switch to its **Review** tab to see the task as the expert saw it, the criteria, and earlier decisions, then click **Accept** or **Reject**. To work from code, use the [`toloka review`](https://docs.platform.toloka.ai/docs/cli/reference/review) commands or the SDK's review methods, starting with [`reviews.list`](https://docs.platform.toloka.ai/docs/sdk/reference/reviews/list).

### Output fields

A customer-review node writes no per-criterion verdict fields.

### What customer review costs

* An accepted task, whether your team accepted it or its 7 days ran out, bills the node's per-task price, or its generation tokens under synthetic labeling.
* A task whose reject is final bills nothing.
* The platform's check of a reject's reason isn't billed. The cost estimate has no review line for the node, and its maximum assumes every task is accepted.

## What quality control costs

Review costs depend on the path and how much reviewing happens.

| Path | What is billed | Main multipliers |
| - | - | - |
| LLM review | One model call per check, at the review model's per-call price | items × criteria × repetitions × experts |
| Human review | Each sampled review, at a per-review price | sample rate × expected reviews per item |
| Quorum offboarding rules | No separate charge | — |
| Customer review | No separate charge. An accepted task bills its labeling cost; a rejected one bills nothing | items accepted |

The human-review price uses the per-task price you set, multiplied by review time divided by labeling time, plus a 10% premium. An item sent back for rework is reviewed more than once, so the expected reviews per item can exceed one.

For what **Expected** and **Maximum** mean on these lines, see [Pipeline pricing](https://docs.platform.toloka.ai/docs/data/concepts/pricing.md). For the exact per-call and per-review figures, see the [Generation node](https://docs.platform.toloka.ai/docs/data/reference/nodes/generation.md) and [Quorum node](https://docs.platform.toloka.ai/docs/data/reference/nodes/quorum.md) references.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.