Quality control
How the platform checks labeled data — the quality criteria each output is judged against, who reviews the work, the verdicts you get back, what a failed check triggers, and what review costs.
Every labeling node ships with quality control turned on: without a check between the labeler and you, a wrong label reaches your dataset unnoticed, and you find out only after the run is paid for. It stays a choice, not an obligation — the Quorum node can drop the reviewer entirely and protect quality through expert agreement instead. The quality model is the same wherever labels are produced: the criteria state the conditions a good label must meet, and the node checks its labels against them before the results reach you. Each node’s reference page states only what differs from this shared model.
The nodes differ in who reviews the work and what a failed check sets in motion.
Quality criteria
Section titled “Quality criteria”A quality criterion is a named check one output is judged against. Each criterion targets a single output field and states two conditions plus an optional instruction of up to 5,000 characters, handed only to the LLM reviewer: what a passing label looks like and what a failing one looks like. Criteria are how you turn “label these well” into checks a reviewer can apply the same way every time.
The agent writes the criteria from the task you describe, then you review them and either tell the agent what to change or edit them by hand on the node’s Quality Requirements card. For example, you can accept a borderline case you would keep or reject one you would not. The agent provides the initial list.
A node cannot deploy until its criteria clear a floor enforced by Self-check: at least one criterion, each with a name filled in, each targeting one of the node’s own output fields, and each pass and fail condition at least 10 characters long. On a Generation node, whether experts or an LLM produce the labels, every output field must be covered by at least one criterion. The Quorum node relaxes this requirement, so you can check a subset of its outputs.
Who reviews the work
Section titled “Who reviews the work”The review path is who checks the labels against the criteria. It is a per-node choice, and the paths a node offers depend on its kind:
| Node | Review paths available |
|---|---|
| Expert labeling (Generation) | LLM review, human review, or both |
| Quorum | LLM review, or none (agreement-based control instead) |
| Synthetic labeling | LLM review only |
LLM review is the default wherever it is available, and it grades every item. Human review, available on the expert-labeling node, checks a sample rather than every item: you set the share reviewed, from 10% to 100%, and the time a reviewer is given per item, at least 10 seconds. When both paths run together, the LLM grades every item and human reviewers double-check a sample. Synthetic labeling is locked to LLM review because it uses no experts. The Quorum node’s no-review path is covered under agreement checks.
The review model
Section titled “The review model”When an LLM reviews the work, it uses a review model. This per-node setting is separate from any model that generates synthetic labels. The platform has a short list of models, each with its own per-call price, and preselects a recommended one. The agent keeps that default when it builds the node. To use a different model, open the node and pick it under its Quality Assurance settings. On a Quorum node you can also ask the agent to switch it; on a Generation node the review model is the one review setting the agent’s controls do not yet cover, so a change there is always made by hand. The choice covers all criteria on the node.
Verdicts
Section titled “Verdicts”Every reviewed criterion returns a verdict with a written reason. The verdict is pass or fail unless the reviewer cannot reach either one. In that case, the verdict gives one of these reasons: the item could not be processed, the submitted data was not valid input, a field the check needed could not be read, the check itself failed, or the check was skipped.
The verdict’s shape follows the output it covers:
- a plain output returns one verdict per criterion.
- an output that repeats within an item returns a list of verdicts, one per repetition.
- on a Quorum node, each criterion returns one verdict per expert, and a repeating output returns a list per expert.
- when LLM and human review both run on a node, each criterion returns one verdict from each, side by side.
Alongside the verdicts, a reviewed item carries a review metadata field with details about the review. This extra output field is named after the node and ends in _qa_metadata.
When a check fails
Section titled “When a check fails”A failing verdict can send the work back.
On expert-labeled nodes, a rejected submission is returned for correction, or reassigned, so the item gets another pass before it counts as done. The platform handles this rework automatically. A submission that stays rejected earns the expert no payment.
Under synthetic labeling, a label that fails its check is generated again and rechecked, up to two further attempts — a platform-set cap you do not configure. Those retries consume billable tokens like any other generation, and total token spend stays within the run’s contracted maximum. See Label with an LLM instead of experts for how this shows up in the results.
Agreement checks on Quorum nodes
Section titled “Agreement checks on Quorum nodes”The Quorum node can protect quality without a reviewer, by watching how often each expert agrees with the group. Under no review this is not optional: the node needs at least one offboarding rule to deploy, an overlap of at least 3, and the per-expert cap switched off. When an expert repeatedly lands in the minority on a chosen label, an offboarding rule removes them from the node. The rule acts only on experts. It grades no individual label, rejects nothing, and leaves the returned answers unchanged.
Agreement-based control is the Quorum node’s alternative to LLM review, so it has these couplings: it needs the no-review path, at least three experts per item, and the per-expert task cap turned off. The Quorum node reference covers the exact agreement thresholds and the setup.
What quality control costs
Section titled “What quality control costs”Review costs depend on the path and how much reviewing happens.
| Path | What is billed | Main multipliers |
|---|---|---|
| LLM review | One model call per check, at the review model’s per-call price | items × criteria × repetitions × experts |
| Human review | Each sampled review, at a per-review price | sample rate × expected reviews per item |
| Quorum agreement | No separate charge | — |
The human-review price comes from the expert payout scaled to the review time, plus a 10% premium. An item sent back for rework is reviewed more than once, so the expected reviews per item can exceed one.
For what Expected and Maximum mean on these lines, see Pricing and billing. For the exact per-call and per-review figures, see the Generation node and Quorum node references.