Compress a prompt with gisting

Learn a compact prompt representation from examples plus a prompt text file. A Training is an editable draft. A Training Run is one billed execution. The first run creates the output Model from the Training name and publishes an immutable Model Version.

Gisting is a different method from supervised fine-tuning. Always pass --method gisting and --prompt-file. For weight adaptation from labeled conversations, see Fine-tune a model.

Copy IDs from command output into the next command. Do not invent them.

Before you start

You need:

A supported standalone CLI system: macOS or glibc Linux on ARM64 or x64. Native Windows and musl Linux releases are not available yet. The CLI includes its runtime; see CLI installation.
A Toloka account on a team. You sign in through your browser; only automation without a browser needs an API key from the Business Console (Company → API keys → Generate API key). See Manage your team and balance.
A team with a funded balance. Cost estimation is free. Starting a Training Run charges up to the printed maximum charge.
Example conversations and a non-empty UTF-8 prompt file no larger than 256 KiB.

curl -fsSL https://platform.toloka.ai/cli/install.sh | sh
toloka version --json
toloka login
toloka whoami

Use the latest CLI release. This CLI-only workflow does not import the SDK and needs no separate runtime. For updates, PATH fixes, and migration, see Updates and migration from Bun global installs. There is no toloka update command; rerun the installer for an immediate update.

login asks how you want to sign in and opens your browser on the Toloka login page. whoami prints the platform URL, the account you signed in with, and Reachable: N project(s) visible.

Step 2 — Create or select a project

The Training, dataset, prompt file, billing, and resulting Model belong to one project.

toloka project list
toloka project create gisting --name "Support ticket gisting"
toloka project current

project create prints Created project pp.… as gisting. If you have more than one team, pass --team-id. To reuse an existing project:

toloka project add support-gisting pp.YOUR_PROJECT_ID
toloka project switch support-gisting

Step 3 — Put example conversations in a dataset

Each row is a chat. The last turn must be assistant, and at least one turn must be user. Save this file as support-tickets.jsonl:

{"messages":[{"role":"user","content":"Hi, how do I reset my password? I can't find the option anywhere."},{"role":"assistant","content":"question"}]}
{"messages":[{"role":"user","content":"This is the third time my order arrived damaged. I'm furious."},{"role":"assistant","content":"complaint"}]}
{"messages":[{"role":"user","content":"Just wanted to say your support team was incredibly helpful today, thank you!"},{"role":"assistant","content":"praise"}]}
{"messages":[{"role":"user","content":"What are your business hours on weekends?"},{"role":"assistant","content":"question"}]}
{"messages":[{"role":"user","content":"The app keeps crashing every time I open the checkout page. Please fix this."},{"role":"assistant","content":"complaint"}]}
{"messages":[{"role":"user","content":"Amazing product, it has saved me hours every week. Great job!"},{"role":"assistant","content":"praise"}]}
{"messages":[{"role":"user","content":"Can I change the shipping address after placing an order?"},{"role":"assistant","content":"question"}]}
{"messages":[{"role":"user","content":"I was charged twice for the same subscription and nobody is responding."},{"role":"assistant","content":"complaint"}]}
{"messages":[{"role":"user","content":"Do you offer student discounts?"},{"role":"assistant","content":"question"}]}
{"messages":[{"role":"user","content":"The new dashboard update is fantastic, much easier to use now."},{"role":"assistant","content":"praise"}]}

Do not split the file yourself. A split column on the dataset puts each row in a part: the Training learns on the train part, validates on the validate part, and never uses the test part, which stays held out so you can evaluate the model on it later. Step 6 creates that column for you.

Create the dataset, then append the rows:

toloka dataset create support-tickets

Then:

toloka dataset upload support-tickets ./support-tickets.jsonl
# Uploaded 10 rows to dataset support-tickets.

Step 4 — Write the prompt file

Save this as prompt.txt. The file must be non-empty UTF-8 and at most 256 KiB.

Classify each support ticket as question, complaint, or praise.
Reply with one of those three labels and nothing else.

Step 5 — Choose a base model

toloka base-model list

The catalog currently includes qwen3-4b, qwen3-8b, qwen3-32b, and qwen3-235b. For gisting, choose qwen3-4b or qwen3-8b: gisting publishes the full model, the larger tiers exceed the Model Version size limits, and creating a gisting Training with them is rejected. This page uses qwen3-8b. The printed ID is what --base-model accepts.

Step 6 — Create the gisting Training draft

Pass --method gisting, --prompt-file, and the dataset from Step 3:

toloka training create "Support ticket gisting" \
  --method gisting \
  --prompt-file ./prompt.txt \
  --base-model qwen3-8b \
  --dataset support-tickets
# Split support-tickets by column split: placed 10 rows, 0 kept their part.
# Froze slice Support ticket gisting (dsl.…) with 10 rows.
# Created training trg.…

--dataset puts every row that has no part yet into train, validate or test (80/10/10, in a new split column), freezes the rows into a slice named after the Training, and trains on that slice. Of the 10 rows, 8 train, 1 validates, and 1 is held out for testing. A slice never changes, so every run of this Training learns on the same rows, and you can always tell which rows a model learned from. To choose other shares or balance the parts by a label, run toloka dataset split support-tickets --ratios 70/10/20 --balance-by <label-column> first; to train on a slice you froze yourself, pass --slice <slice-id>. See Split a dataset and freeze a slice.

When the dataset already has a split column, for example from an earlier Training, --dataset keeps every row’s part, places only rows added since, and freezes a new slice for this Training:

# Split support-tickets by column split: placed 0 rows, 10 kept their part.
# Froze slice … (dsl.…) with 10 rows.

Confirm the draft:

toloka training get trg.YOUR_TRAINING_ID

The table shows Method as Prompt gisting, the Slice and Split column it learns on, and Output Model as - until the first run publishes one.

--gist-tokens, --epochs 1..5, --learning-rate, --batch-size 8|16|32|64, and --max-sequence-length 64..16384 set the gisting hyperparameters. Omit them to take the defaults: "1/8" of the prompt, 3 epochs, learning rate 0.03, batch size 8, and 3,072 tokens.

--gist-tokens takes an absolute count — 8, 32, or 64 — or a fraction of the prompt you are compressing: 1/4, 1/8, 1/16, or 1/32. A fraction fixes the compression ratio rather than the token count: the run divides the prompt’s token length by that number, rounds up, and clamps the result to 8 through 256. Use a fraction when the prompt is long or its length changes between drafts. training get reports the fraction you set; the count it resolved to is chosen inside the run. See Base models and settings.

Step 7 — Review the cost

toloka training cost trg.YOUR_TRAINING_ID

The command prints a table with training rows, tokens, maximum GPU seconds, and Maximum charge. When a split column now puts rows of the train part in a test part, a Left out as test row says how many; they stay out of training. The test part of the Training’s own split column is never in the train part, so it is not counted there. If the estimate is blocked, the command exits with code 1 and prints Training cannot run with these inputs: followed by one - <message> line per problem. Fix each and calculate again. With --json, a blocked estimate writes nothing to stdout.

Step 8 — Start the Training Run

toloka training run trg.YOUR_TRAINING_ID

The command calculates a fresh ready estimate and names the Model Version after the Training (Support ticket gisting). Copy the Training Run id (trn.…).

toloka training status trn.YOUR_RUN_ID
toloka training logs trn.YOUR_RUN_ID

Status values include queued, running, stopping, blocked, completed, failed, and canceled. If the run is blocked, restore billing and run toloka training resume trn.YOUR_RUN_ID. While the run is running, the Progress row shows the stage inside the run; queued or pending there means the run is waiting for GPU capacity and has not started training yet.

A run can train longer than the estimate predicted. It stays running, the charge stays capped at the maximum charge you approved, and the Model Version is published when training finishes. Do not start a second run because the first one is slower than estimated. A run that fails because of the platform is not charged: the whole hold returns to the team balance, even though training status still shows the GPU seconds it used. When its error says retryable: true, run toloka training run again with the same --estimate-file: a failed run frees its estimate, and the maximum charge stays the same.

Step 9 — Use the trained model

When status is completed, the table shows the Model Version. The Training now names its output Model:

toloka training get trg.YOUR_TRAINING_ID
toloka model versions mdl.YOUR_MODEL_ID

Download the files. Copy the Model Version id (mdv.…) from the table:

toloka model version download mdv.YOUR_VERSION_ID --out ./support-ticket-gisting
# Downloaded Model Version mdv.… revision 1 to /…/support-ticket-gisting

The folder holds a full model: config.json, the model-*.safetensors weight shards, and the tokenizer files, including the text files merges.txt and chat_template.jinja. The learned prompt is baked in as special tokens <|gist_0|>, <|gist_1|>, and so on, one per gist token the run trained. Serve the folder as-is, for example with vllm serve ./support-ticket-gisting.

Build each request the way the run trained: the gist tokens first, as plain text, then the conversation rendered with the model’s chat template. The conversation holds the same turns as your training rows without the final assistant turn, does not repeat the original prompt, and has thinking disabled. A chat endpoint wraps everything in message markers, so send the prompt you build to the completions endpoint:

from openai import OpenAI
from transformers import AutoTokenizer

model_dir = "./support-ticket-gisting"
tok = AutoTokenizer.from_pretrained(model_dir)
gist = sorted((t for t in tok.get_added_vocab() if t.startswith("<|gist_")), key=lambda t: int(t[7:-2]))
prompt = "".join(gist) + tok.apply_chat_template(
    [{"role": "user", "content": "My card was charged twice for one order."}],
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
response = client.completions.create(
    model=model_dir,
    prompt=prompt,
    max_tokens=1024,
    temperature=0,
    logit_bias={str(t): -100 for t in tok.convert_tokens_to_ids(gist)},
)
print(response.choices[0].text)

logit_bias keeps the model from generating gist tokens. If your serving stack accepts only chat messages, put the gist tokens alone in a system message at the start of the conversation. That differs from training by the system message markers, so check the answers on your validation rows before you rely on it.

You trained the model. To test it, deploy it. You can use our partner, Nebius.

If something fails

What you see	What it means	What to do
`Not logged in. Run toloka login once to sign in through your browser on this machine:`	No login is in effect.	Run `toloka login`, or set `TOLOKA_API_KEY`.
`No project selected. Run toloka project create for a new project, toloka project add for an existing project, or cd into a checkout.`	No project is selected.	`toloka project create`, `toloka project add`, or pass `--project-id`.
`Dataset "…" does not exist … no pipeline in source declares it.`	The dataset has to exist before rows can be uploaded.	Run `toloka dataset create <name>`, then upload again.
`The gisting prompt file is empty.` / `The gisting prompt file exceeds the 256 KiB limit.` / `The gisting prompt file must contain valid UTF-8 text.`	`--prompt-file` failed a local check.	Point at a non-empty UTF-8 file no larger than 256 KiB.
`--prompt-file is required for gisting.`	`--method gisting` was set without a prompt file.	Pass `--prompt-file`.
`--lora-rank applies to SFT; gisting takes --gist-tokens.`	`--lora-rank` was passed on a gisting Training.	Use `--gist-tokens` instead.
`--dataset or --slice is required.`	The Training names no rows to learn on.	Pass `--dataset <dataset>` to split it and freeze a slice, or `--slice <slice-id>`.
`Slice dsl.… freezes no split column`	The slice was frozen before the dataset had a split column.	Pass `--dataset <dataset>`, or run `toloka dataset split <dataset>`, take a new slice, and pass it.
`Choose exactly one of --base-model and --model-version.`	Both source flags were set, or neither was.	Pass one.
`Training from a Model Version requires a LoRA adapter source`	`--model-version` points at full weights. Gisting continues a LoRA adapter version (one produced by fine-tuning).	Pass a fine-tuned Model Version, or `--base-model`.
`Training cannot run with these inputs:`	The cost estimate is blocked. The command exits with code 1.	Read each `- <message>`, fix inputs, calculate again.
`This Training would take up to … hours … which exceeds the 6-hour limit.`	The estimated wall time is over the 6-hour limit.	Reduce rows or epochs. Gisting supports only `qwen3-4b` and `qwen3-8b`.
`A Model Version with this name already exists`	A previous run already used this Training name as the version name.	Pass `--version-name` with a new name.
`status` is `blocked`	Funds or payment stopped the run.	Restore billing, then `toloka training resume`.

Training — how Trainings, cost estimates, Training Runs, and Model Versions fit together
CLI — install, sign in, and project selection
Fine-tune a model — supervised fine-tuning on the same Training commands
Create and manage datasets — creating a dataset on the Datasets page
Pricing and billing — how the team balance is drawn down