# Fine-tune an LLM with an agent This runbook guides an AI coding agent through supervised fine-tuning with the Toloka CLI. It applies to both new and existing workflows. ## Mission Help the user achieve the SFT outcome they asked for while preserving their task, data, model, quality, latency, deployment, and budget constraints. The requested outcome may be to inspect existing state, prepare or update a Training draft, calculate an estimate, start an approved run, monitor a run, or retrieve a completed Model Version. Do not create later-stage resources merely to complete the whole lifecycle. For prompt compression, explain the difference and route to `https://platform.toloka.ai/agents/gisting.md`. For labeled-data creation, route to `https://platform.toloka.ai/agents/data-labeling.md`. Never silently change the method. ## How to work 1. Read the request and current CLI, project, Training, Dataset, and run state before writing. 2. Treat the sections below as a dependency-aware reference, not a script. Skip steps already satisfied or irrelevant to the requested outcome, while preserving dependencies and safety gates. 3. Honor explicit choices first. If a requested project, source, Dataset, or configuration is incompatible, explain the evidence and tradeoff before proposing a substitute. 4. Make routine reversible choices and briefly explain them. Ask focused questions only for material ambiguity, missing authorization or inputs, conflicting constraints, or destructive or out-of-scope changes. 5. An inspection or estimate-only request ends after its answer. Start a Training Run only when the user requested a run and approved its exact ready estimate. ## Safety and platform invariants - Never ask the user to paste an API key into chat. The user enters it through `toloka login`. - Never invent a project ID, Dataset ID, Base Model ID, Model ID, Model Version ID, filter, price, or task requirement. Read them from CLI output. - Never edit `toloka.json`; the CLI owns that file. - Use project-owned Datasets containing representative, authorized training data. Keep credentials and production data out of source control and agent messages. - Preserve training and validation semantics. Keep validation examples out of training and avoid duplicate or near-duplicate examples across the roles. - Without a validation selection, the provider reserves 10% of the training rows as a holdout. - Never start a Training Run before the user sees the ready estimate and explicitly approves its exact `maximumCharge`. - A Training is an editable draft. Each Training Run is an immutable, billable execution; changing the draft does not mutate an existing run. - Do not pass `--new-model`. The first run creates the output Model from the Training name. - Use `--model-version` only for an exact existing LoRA adapter Model Version (one produced by fine-tuning); it replaces, rather than accompanies, `--base-model`. Full-weights versions are rejected. - Use absolute paths for estimate files. Commands can run in separate shells. - Keep waits bounded. A timeout means the run is still running, not failed or completed. - Report only IDs, prices, states, and behavior observed in command output. A completed run does not prove deployment or inference readiness. ## Domain model and full lifecycle - **SFT** adapts model weights from labeled chat conversations. Each row must follow the platform chat contract and end with the assistant response the model should learn to produce. - A **source** is either a pinned catalog Base Model or a project-owned immutable Model Version. An explicit Model Version bypasses Base Model selection. - A **Dataset selection** is a Dataset plus an optional top-level equality filter. Training and validation may use separate Datasets or disjoint selections from one Dataset. - Cost estimation validates frozen Dataset selections and the selected source before GPU work. - A successful paid run publishes one immutable Model Version. SFT output contains adapter weights; serving or deployment is a separate decision. A full create-and-run request has these dependencies: ```text working CLI and identity | v appropriate project binding | v compatible source + training/validation selections | v SFT Training draft --> ready estimate --> approve maximumCharge | v one Training Run | v observed Model Version ``` ## Prepare the environment when needed Reuse a compatible existing workspace. Do not force a fresh onboarding directory, reinstall dependencies, or rerun initialization when the current workflow is already prepared. When CLI state is unknown, run: ```sh toloka --version toloka dataset --help ``` The standalone CLI includes its runtime, so Training-only work needs no separate Bun or Node.js installation, package manifest, SDK, TypeScript, or test packages. This runbook requires a stable Toloka CLI release at version `0.2.0` or newer: `toloka --version` must print `0.2.0` or a newer version with no prerelease suffix, and `toloka dataset --help` must list `create`. If either command is unavailable, the version is older or prerelease, or the capability check fails, install the latest stable version: ```sh curl -fsSL https://platform.toloka.ai/cli/install.sh | sh ``` Use the install path printed by the command if the shell does not refresh its `PATH`. Only for a new workspace, use the directory supplied by the user or create a task-appropriate directory such as `toloka-fine-tuning`. Run `toloka init` here only to create the Agent Bootstrap used to install the workflow skill; its missing-SDK warning applies to pipeline authoring, not Training commands: ```sh mkdir -p toloka-fine-tuning && cd toloka-fine-tuning && pwd toloka init ``` The printed directory is ``. Use absolute paths for later local files. In an initialized workspace, keep its source root and inspect installed skills instead: ```sh toloka skill list ``` Install `toloka-fine-tuning` only when absent, or refresh it with `--force` when the installed copy is stale: ```sh toloka skill install toloka-fine-tuning toloka skill install toloka-fine-tuning --force ``` If the catalog does not contain the skill, update to the latest stable CLI with the installer above. Read `.agents/skills/toloka-fine-tuning/SKILL.md` before acting. ## Establish identity and project context Check authentication without exposing the token: ```sh toloka whoami ``` If it reports the intended reachable account, continue without another login. Read failures precisely: `Rejected:` means the platform refused the key; `Not reachable:` means the platform did not answer and the key was not evaluated; `Platform error:` means the platform failed. For a missing or rejected credential, tell the user to generate an API key at `https://platform.toloka.ai` under **Company** → **API keys**, then ask them to run `toloka login` in their own terminal and enter it at the masked prompt. Continue only after `toloka whoami` reports the intended account. Fix connectivity instead of requesting a new key for `Not reachable:`. Inspect project context before selecting or creating anything: ```sh toloka project current toloka project list --limit 20 ``` Choose the project from evidence: 1. If the user explicitly named a project, use that exact target. Register it with `toloka project add ` when necessary, or use `toloka project switch ` when it is already registered. 2. Otherwise, keep the current binding when its identity and resources are consistent with the task. 3. If no suitable project is bound and a mutation requires one, create a new task-named project. Do not create a project for an inspection that can be answered from an explicitly addressed resource. Do not silently choose an unrelated project or repoint an existing checkout. If the current binding conflicts with the request and the intended target is genuinely ambiguous, ask one focused project question. When an existing checkout is correctly bound to another project, keep that binding intact. Use an explicit `--project-id ` for isolated inspection commands or a separate source root for a mutable workflow instead of overwriting `toloka.json`. For a new project, derive a short kebab-case alias and name from the task, such as `support-triage-sft`: ```sh toloka project create --name "" [--team-id ] [--delivery-order-id ] ``` Without `--team-id`, the CLI uses the account's only writable team. If it reports `Missing team id.`, ask the user to choose from the listed teams. A project owned by an internal team (`toloka whoami` marks it `(internal)` on the `Teams:` line) needs a Sunday delivery order: run `toloka delivery-order list` and pass the only listed order with `--delivery-order-id `; ask when several viable orders remain. Verify the selected context: ```sh toloka project current toloka project url ``` When using a bound source root, read `toloka.json` and confirm its `projectId` matches `toloka project current`. Do not edit it. ## Inspect the requested state and inputs For an existing workflow, start with the resource the user referenced: ```sh toloka training list --json toloka training get --json toloka training runs --json toloka training status --json ``` Run only the commands needed to answer the request. Do not manufacture a new Training or run for inspection. ### Choose training and validation data Inspect project Datasets and candidate fields: ```sh toloka dataset list toloka dataset fields ``` Use the data and role assignments the user supplied. A typical compatible row is: ```json {"messages":[{"role":"user","content":"Classify this request"},{"role":"assistant","content":"billing"}]} ``` The final assistant message is the training target. Check that examples are representative, authorized, and consistent with the intended task. A separate validation selection gives a stable evaluation set; when none exists, omit validation flags and use the provider's 10% holdout rather than inventing validation data. When local JSONL files need uploading, organize Datasets by role, not automatically by file. `dataset upload` appends, so several compatible files for one role may share one Dataset: ```sh toloka dataset create toloka dataset upload toloka dataset create toloka dataset upload toloka dataset fields ``` Create and upload only the role Datasets that are needed. CLI uploads to an otherwise undeclared Dataset store each row whole. A `split` property nested inside that uploaded object is not a top-level Dataset field, so it cannot drive `--train-filter` or `--validation-filter`; use separate role Datasets or provider holdout. An existing Dataset with a real top-level split field may use disjoint filters such as `split=train` and `split=validation`. Never configure overlapping training and validation selections. ### Choose and verify the model source Honor an explicit Base Model or Model Version. If neither is specified, choose a suitable source from evidence rather than defaulting to the smallest or largest model. Consider: - upstream documentation and user-provided benchmarks for task, language, and domain fit; - complexity and quality target, data volume, and overfitting risk; - training budget and duration; - serving latency, memory, and deployment constraints; - relevant prior results from an existing project Model Version. A smaller model is useful as a cost and latency baseline when the quality target permits it, but size is not evidence of fitness by itself. Briefly state the choice, evidence, and tradeoffs. For a catalog Base Model, inspect the available identities, then verify the exact catalog ID and pinned upstream revision: ```sh toloka base-model list --json toloka base-model get --json ``` The catalog output identifies the upstream model and revision; consult upstream documentation or user evidence for context length, languages, and other capabilities it does not expose. For an existing Model Version source, do not select a Base Model as well. Inspect its project-owned Model and exact version: ```sh toloka model list --json toloka model get --json toloka model versions --json ``` Confirm the requested Model Version ID and revision, and record its `runtimeBase` and the owning Model's `baseModelId`. If those identities conflict, report the conflict rather than guessing or substituting. The platform rejects a Model Version outside the selected project. The ready estimate later provides the concrete compatibility check for the source type, underlying Base Model, SFT method, and data. ## Reuse, create, or update the Training draft If an existing Training already matches the requested method and inputs, reuse it. A new draft is appropriate only when the user requested one or no suitable draft exists. Use a descriptive name such as `support-triage-sft-1` unless the user supplied a name. SFT is the default method, so do not pass `--method`: ```sh toloka training create "" \ --base-model \ --dataset \ --validation-dataset \ --json ``` Omit `--validation-dataset` when using provider holdout. For disjoint top-level selections, add `--train-filter =` and `--validation-filter =`. For an exact existing Model Version source, replace `--base-model` with `--model-version `; the source must be a LoRA adapter Model Version (a full-weights version is rejected with "Training from a Model Version requires a LoRA adapter source"). The first run creates a new output Model, while the source Version remains the training parent. Preserve user-specified hyperparameters and choose unspecified values from task complexity, data volume, validation evidence, and deployment constraints. The current CLI uses explicit parameters rather than `regime`: `--lora-rank 8|16|32` (default 16), `--epochs 1..5` (default 3), `--learning-rate` within 0.00001–0.001 (default 0.00015), `--batch-size 8|16|32|64` (default 8, sequences per optimizer step across all GPUs), and `--max-sequence-length 64..16384` (default 12288 tokens per example). Rank 8 is useful for narrow tasks or small data; consider 32 when 16 underfits and sufficient data supports the extra capacity. Choose sequence length from actual examples within the supported limit. These defaults are starting points, not mandatory choices. `--gist-tokens` is gisting-only and rejected here. `training update ` takes the same flags except `--method`; omitted flags retain their current values. After creating or updating a draft, verify its normalized method, exact source, selections, filters, and configuration: ```sh toloka training get --json ``` Do not start a run yet. ## Validate inputs and estimate cost Create an absolute estimate-file path under ``, then run: ```sh toloka training cost --json > ``` The command freezes Dataset selections, waits for validation exports, and writes a ready estimate to the file. It is the concrete compatibility check for the selected source, SFT method, and data. If the estimate is blocked, the command exits 1, writes nothing to the file, and prints `Training cannot run with these inputs:` with one `- ` line per issue to stderr. Report every issue, correct its cause, and calculate a new estimate. Never reuse an estimate after the Training configuration changes. For a duration issue, use the reported row and token counts, epochs, GPU count, and wall time to compare supported options. `qwen3-32b` and `qwen3-235b` use 8 GPUs and may reduce wall time, but a model change also affects task fit, serving constraints, and cost. Reducing rows can weaken coverage; reducing epochs can weaken adaptation. Do not change the requested model, discard data, or alter epochs merely to clear the error. Explain the tradeoff, obtain direction when material, update the draft, and re-estimate. For a ready estimate, report: - Training ID and exact source; - training and validation selections and row counts; - token totals; - maximum GPU seconds; - `maximumCharge` and currency; - estimate fingerprint. If the user asked only for validation or an estimate, stop here. Otherwise present the approach and estimate together and ask the user to approve the exact `maximumCharge`. Any later source, data, filter, or hyperparameter change requires a new estimate and new approval. ## Start and monitor only the requested run After both a run request and explicit approval, start one run with the approved estimate: ```sh toloka training run \ --estimate-file \ --version-name \ --json ``` Both flags are optional at the CLI level. Pass `--estimate-file` here so a fresh unseen quote cannot replace the approved one. Without `--version-name`, the Model Version takes the Training name. Read the returned Training Run ID. Monitor only as far as useful for the user's request, using bounded commands: ```sh toloka training status --json toloka training logs toloka training metrics [--key ] [--resolution ] [--json] ``` Use `toloka training logs --follow` only with a bounded command timeout. Relevant states include `queued`, `running`, `stopping`, `blocked`, `completed`, `failed`, and `canceled`. Metrics report the loss and accuracy published so far; an unavailable explanation is not evidence that the run failed. If the run is blocked on funds or payment, report the exact message. Resume only after the user confirms the underlying billing issue is resolved: ```sh toloka training resume --json ``` Cancel only on explicit request: ```sh toloka training cancel --json ``` ## Verify and report the requested result For a completed run, read the Model Version ID and server-assigned revision from status output, then verify the Training's output Model: ```sh toloka training get --json toloka model get --json toloka model versions --json ``` Download the immutable Model Version only when requested: ```sh toloka model version download --out ``` Report the identifiers and evidence relevant to the user's request. For a completed lifecycle, include the Training ID, Training Run ID and terminal state, approved maximum charge and observed charged amount, output Model ID, Model Version ID and revision, whether logs were complete or truncated, and any unresolved block or timeout. Do not claim deployment or inference readiness. The durable Model Version is the training output; deployment is a separate workflow.