Annotation Queues

Annotation Queues are a manual evaluation method which is built for domain experts to add scores and comments to traces, observations or sessions.

Why use Annotation Queues?

Manually explore application results and add scores and comments to them
Allow domain experts to add scores and comments to a subset of traces
Add corrected outputs to capture what the model should have generated
Align your LLM-as-a-Judge evaluation with human annotation

Set up step-by-step

Create a new Annotation Queue

Click on New Queue to create a new queue.
Select the Score Configs you want to use for this queue.
Set the Queue name and Description (optional).
Assign users to the queue (optional).

An Annotation Queue requires a score config that defines the scoring dimensions for the annotation tasks. See how to create and manage Score Configs for details.

Add Traces, Observations or Sessions to the Queue

Once you have created annotation queues, you can assign traces, observations or sessions to them.

To add multiple traces, sessions or observations to a queue:

Select Traces, Observations or Sessions via the checkboxes.
Click on the "Actions" dropdown menu
Click on Add to queue to add the selected traces, sessions or observations to the queue.
Select the queue you want to add the traces, sessions or observations to.

To add single traces, sessions or observations:

Click on the Annotate dropdown
Select the queue you want to add the trace, session or observation to

Process Annotation Queue

You will see an annotation task for each item in the queue.

On the Annotate Card add scores on the defined dimensions
Click on Complete + next to move to the next annotation task or finish the queue

Review experiment results with a QA team

Use a queue to hand experiment results to domain experts who need to review answers, check supporting sources, and explain failures.

Engineer setup

Run your application on the test cases, or publish saved check results. Include the question, relevant source material, reference answer, and case ID.
Compare the experiment runs and add the relevant observations to the annotation queue through the Annotate dropdown. For version comparisons, include both versions and identify the baseline and candidate in the queue description.
Configure a score such as answer_correct and describe what counts as a pass. Include the release policy and review instructions in the queue description; do not ask reviewers to infer them from a numeric score.
Have an organization Admin or Owner invite reviewers with a role that allows scoring, such as Member. A Viewer can read results but cannot annotate. Assign reviewers to the queue if needed; queue assignment does not grant project access. Project-specific roles depend on your plan.

Reviewer steps

Open the queue and read the question, supporting source, reference answer, and application output. Inspect linked observations when you need more context.
Record the configured score. Add a comment explaining the failure and the supporting evidence. If the automated check is wrong, say so rather than marking the application wrong to match it.
Add a corrected output when you can supply the answer the application should have returned. Complete the item and move to the next one.

After review, the engineer checks proposed corrections and adds accepted cases to the dataset, using the reviewed answer as the expected output. A correction does not automatically update the dataset or approve a release. Keep the original run and case IDs for provenance, approve the new dataset version, and re-run both application versions if inputs or expected outputs changed. If only the grader changed, re-score both versions with that definition.

Keyboard shortcuts

Processing a queue is fully keyboard-driven, so you can score items without reaching for the mouse. The shortcuts follow a navigate-then-edit model and are suppressed while you are typing in a text field or while a dialog or dropdown is open. Press ? in the queue to open the in-app cheatsheet.

Key	Action
`→` / `←`	Next / previous item in the queue
`↑` / `↓`	Move between score fields (wraps at the ends)
`1`–`9`	Select the Nth option on a categorical or boolean field
`Enter`	Commit the current value, or open a dropdown
`Esc`	Leave a text field and return to field navigation
`Cmd`/`Ctrl` + `Enter`	Complete the item and advance to the next
`?`	Open the keyboard shortcuts cheatsheet

Number badges appear on categorical options when a field is focused, and the ⌘↵ / Ctrl↵ hint is shown on the Mark Completed button. The score-field shortcuts (↑/↓, 1–9, Enter, Esc) also work in the inline Annotate drawer on trace, observation, and session pages; the queue navigation keys (→/←, Cmd/Ctrl + Enter) apply when processing a queue.

Manage Annotation Queues via API

You can manage annotation queues via the API. This allows for scaling and automating your annotation workflows or using Langfuse as the backbone for a custom vibe coded annotation tool.

Was this page helpful?

On this page