Skip to content

All lessons

Area 04 // EvalsLesson 70 / 14014 min read

Human labeling and user feedback loops

Human labeling is people with clear guidelines judging AI outputs to create trusted ground truth, and user feedback loops collect signals from real users (thumbs, edits, complaints) and route them back into fixes, eval sets and training data.

Think of it like this

Analogy

A restaurant learns in two ways. Food critics (human labelers) visit with a scoring guide and taste carefully; their reviews are trusted. Customer comment cards and leftovers (user feedback) arrive constantly: noisy, biased towards complainers, but they reveal problems critics never see. A good restaurant reads the comment cards every week, investigates, fixes recipes, and adds tricky dishes to the critics' tasting menu.

The problem it solves

Automated evals and LLM judges need ground truth to be calibrated against, and that ground truth comes from people. But people disagree unless they share clear criteria. Meanwhile, real users constantly reveal problems, yet most teams collect thumbs-down clicks into a database nobody reads. Both sources are wasted unless they're organised and fed back into the system.

How it works: human labeling

  1. Decide what to label: answer quality per criterion, relevance of retrieved chunks, safety, the correct category, the ideal answer.
  2. Write annotation guidelines: definitions, a decision procedure, and examples of each label, especially borderline ones.
  3. Pick labelers with domain knowledge for expert tasks (lawyers for legal, clinicians for medical); general crowds only for simple judgements.
  4. Pilot: two or more people label the same 30–50 items.
  5. Measure inter-annotator agreement (percentage and Cohen's kappa). Discuss disagreements and update the guidelines; repeat until agreement is good.
  6. Label at scale with spot-checks and some overlapping items to keep measuring agreement.
  7. Use the labels: golden datasets, judge calibration, error analysis, and sometimes fine-tuning data.
  8. Make labeling efficient: good tools (annotation queues in tracing platforms), pre-filled judge suggestions for humans to confirm or correct, keyboard shortcuts.

How it works: user feedback loops

  1. Collect explicit feedback: thumbs up/down, star ratings, "report a problem", optional comments.
  2. Collect implicit feedback: copying the answer, accepting a suggestion, editing a draft before sending (and how much), retrying or rephrasing, abandoning, escalating to a human.
  3. Attach feedback to traces so every signal links to the exact input, context, prompt version and output (tracing).
  4. Triage regularly: review thumbs-down and escalations, run error analysis, find patterns.
  5. Fix and protect: fix the cause, add representative cases to the golden dataset, and measure with evals.
  6. Track feedback per version (prompt, model) to spot regressions in production.
  7. Remember the bias: few users vote, unhappy users vote more, and thumbs-up often means "fast" rather than "correct". Feedback finds problems; evals and sampled reviews measure quality.
Closing the loop: every signal ends up as a fix and a test.

See it in code

Install
pip install scikit-learn
Needs
Python 3.10+

Measuring agreement between two labelers, and turning raw thumbs-up/down events into a per-version signal.

labels_and_feedback.py
from collections import defaultdict
from sklearn.metrics import cohen_kappa_score
 
# HUMAN LABELING: two reviewers label the same 10 answers as good (1) or bad (0).
alice = [1, 1, 0, 1, 1, 0, 1, 0, 1, 1]
bob   = [1, 1, 0, 1, 0, 0, 1, 0, 1, 0]
print(f"inter-annotator agreement: {sum(a == b for a, b in zip(alice, bob)) / 10:.0%}, "
      f"kappa {cohen_kappa_score(alice, bob):.2f}")
disagreements = [i for i, (a, b) in enumerate(zip(alice, bob)) if a != b]
print("send to a third reviewer / clarify the guidelines for items:", disagreements)
 
# USER FEEDBACK LOOP: thumbs up/down events from production, grouped by prompt version.
events = [("v7", "up"), ("v7", "down"), ("v7", "up"), ("v7", "down"), ("v7", "down"),
          ("v8", "up"), ("v8", "up"), ("v8", "down"), ("v8", "up"), ("v8", "up")]
votes = defaultdict(lambda: {"up": 0, "down": 0})
for version, vote in events:
    votes[version][vote] += 1
for version, v in votes.items():
    print(f"{version}: {v['up'] / (v['up'] + v['down']):.0%} thumbs up ({v['up'] + v['down']} votes)")
print("next: review the thumbs-down conversations and add them to the golden dataset")
Output
inter-annotator agreement: 80%, kappa 0.60
send to a third reviewer / clarify the guidelines for items: [4, 9]
v7: 40% thumbs up (5 votes)
v8: 80% thumbs up (5 votes)
next: review the thumbs-down conversations and add them to the golden dataset

80% raw agreement but kappa only 0.60: once chance agreement is removed, the two reviewers agree only moderately, so the guidelines need work. And with five votes per version, "v8 is twice as good" is not yet a conclusion, just a reason to look closer.

A real example

Example

An email-drafting assistant that learned from edits

An email assistant drafts replies that users edit before sending. The team logged an "edit distance" for each draft: how much the user changed. Drafts that were heavily rewritten clustered around two topics: pricing questions (the assistant was vague) and German-speaking customers (the tone was too informal).

Two support leads labelled 300 of those drafts with written guidelines (kappa 0.74 after one round of clarification). The labelled cases became golden-set entries and calibration data for the team's judge. After fixes, average edit distance on pricing emails fell by half, and the judge, now calibrated on those labels, watches for regressions.

The easy way

Start here

Add a thumbs up/down to your AI feature, store it with the trace ID, and spend 30 minutes a week reading the thumbs-down ones. For labeling, write a one-page guideline with five examples, have two people label the same 30 items, and compare.

When you grow: the production setup

Teams run annotation queues inside their observability platform, use LLM pre-labels with human confirmation, pay for expert reviewers on high-stakes domains, track agreement continuously, capture rich implicit signals (edits, retries, escalations), report feedback per version and segment, and maintain a documented pipeline from feedback to fix to golden case to eval.

Trade-offs

What you gain

  • Human labels are the trusted ground truth everything else is calibrated against.
  • User feedback reveals real-world problems evals don't cover.
  • Closing the loop makes the system steadily better.

What it costs you

  • Expert labeling is slow and expensive.
  • User feedback is sparse, noisy and biased.
  • Privacy: feedback and traces contain user data that must be protected.

When not to use it

  • Don't use raw thumbs-up rate as a release gate; it's too noisy and biased.

Common mistakes

  • Labeling without written guidelines or agreement checks.
  • Using non-experts for expert judgements.
  • Collecting feedback nobody reads.
  • Feedback not linked to traces, so you can't see what went wrong.
  • Treating thumbs-up rate as the quality metric.

Interview questions

Try answering out loud first, then open the model answer.

How do you run a human labeling effort for LLM evaluation?

Model answer

Define the labels and write guidelines with definitions and borderline examples; choose labelers with the right expertise; pilot with overlapping items and measure inter-annotator agreement (kappa); resolve disagreements and refine guidelines until agreement is acceptable; label at scale with ongoing overlap and spot checks; use efficient tooling; and feed the labels into golden datasets and judge calibration.

How would you design a user feedback loop for an AI assistant?

Model answer

Capture explicit feedback (thumbs, ratings, comments) and implicit signals (retries, edits, copies, abandonment, escalations), each linked to the full trace and version. Review negative signals regularly with error analysis, prioritise patterns, fix root causes, add representative cases to the golden dataset, verify with evals, and monitor feedback trends per release, remembering the signals' biases.

What is Cohen's kappa and why use it instead of raw agreement?

Model answer

Cohen's kappa measures agreement between two raters corrected for the agreement expected by chance, ranging from below 0 (worse than chance) to 1 (perfect). With imbalanced labels, two raters can agree often by chance alone, so raw agreement overstates consistency; kappa gives a more honest view of whether the guidelines produce reliable labels.

Quiz

Pick an answer to see whether it’s right and why.

  1. 01Why do human labelers need written guidelines?
  2. 02What does low inter-annotator agreement usually indicate?
  3. 03Which is an implicit feedback signal?
  4. 04Why shouldn't thumbs-up rate be your only quality metric?
  5. 05What should happen to a thumbs-down conversation?

0 of 5 answered

Key terms

Human labeling
People judging or annotating AI outputs, following written guidelines, to create trusted ground truth.
Annotation guidelines
Written definitions and examples that tell labelers how to judge each case consistently.
Inter-annotator agreement
How often different labelers give the same label to the same item; low agreement means unclear guidelines.
User feedback loop
Collecting user signals (thumbs, edits, retries, complaints), finding the causes and feeding fixes and new test cases back into the system.
Cohen's kappa
A measure of agreement between two raters that corrects for agreement expected by chance; 1 is perfect.

Before and after

Read first

Read next

Progress is saved in this browser only.