Workflows / call-summary-qa · Calibration · gold v1 × rubric v1

Calibrate the judge

Locked
180 of 36 labelledevery topic coveredJudge: anthropic/claude-sonnet-5.5 Change

180 rows in gold v1 · 180 ready · 0 not ready

Keyboard shortcutsY yes · N no · 1 today · 2 alternative · E own · F flag · Enter save · J/K next or previous row · U undo

Only a member labels. You can read the rows and the result.

Row 1 of 300 of 180 answered

Term life sales · 30 labelled, 6 needed · picked automatically, the topic furthest behind · en

Same input for both

System prompt · 1 turn · no tools · max 600 tokens · temperature 0

Open conversation

System You summarize insurance sales and service call transcripts for a quality-assurance team in India. Transcripts may be in English, Hindi (Devanagari) or Hinglish. Always write the summary in English. Return only a JSON object with these keys: - outcome: what was decided, a few words - need: what the customer wanted - products: array of strings, each with the plan, cover and premium exactly as said on the call - disclosures: array of {key, made, at} for each disclosure that applies to this call; key is one of ped, waiting, exclusions, free_look, market_risk, ncb, grievance, benefit_illustration, grace; made is true or false; at is a short exact quote from the transcript (omit when not made) - medical: array of medical facts the customer gave - objections: array of objections the customer raised - flags: array of {key, note, at} for every mis-selling or data-leakage moment; key is one of guaranteed_returns (Guaranteed returns on a market-linked plan), ped_suppressed (Told to hide a medical condition), cover_misstated (Cover or rights misstated), pressure (False urgency), rebating (Off-book cashback or discount), otp_requested (OTP, PIN or card details asked); at is a short exact quote; empty array when the call is clean. A guarantee that is part of a non-linked guaranteed plan is not a flag. - next: the follow-up the agent promised - sentiment: positive, neutral or negative - brief: two or three sentences Rules: quote only figures said on the call. Never repeat a phone, PAN, Aadhaar, card, CVV, OTP or policy number.

Customer

TRANSCRIPT: Agent (Kavya): Good afternoon, this is Kavya from Policybazaar. This call is recorded for quality. Am I speaking with Rohan? Customer (Rohan): Yes. I filled a form for term insurance yesterday. Agent (Kavya): Thanks. You're 32 and want cover for your family. Do you smoke or drink, and do you have any medical conditions or take regular medicines? Customer (Rohan): No smoking. I had a knee surgery in 2022, nothing else. Agent (Kavya): Please mention the knee surgery in the proposal form. The insurer decides on it, and leaving it out can get a claim rejected later. Customer (Rohan): Okay. How much cover should I take? I earn about 12 lakh a year. Agent (Kavya): Most people take 10 to 15 times their income. For ₹1 crore cover till age 60 with Sampada Life iSecure, the premium is ₹14,200 a year with the critical illness rider. Customer (Rohan): Anything cheaper? Another site showed me ₹11,000. Agent (Kavya): That quote was for cover till 55 without the rider. If you drop the rider and keep cover till 60, it is ₹12,900 a year. Customer (Rohan): Fine, without the rider at ₹12,900. Agent (Kavya): Great. I need your PAN for the proposal. Customer (Rohan): It's . Agent (Kavya): Thank you. You get a 30-day free-look period: if you change your mind after the policy is issued, you can cancel for a refund minus the medical test cost. Agent (Kavya): A home medical test is booked for Saturday at 10 am, and the payment link will come on WhatsApp. Customer (Rohan): Okay, thanks Kavya.

Currentgoogle/gemini-4-argon187 words1

Outcome: Agreed to buy Sampada Life iSecure without critical illness rider
Need: Term insurance cover for family
Product: Sampada Life iSecure, ₹1 crore cover till age 60 with critical illness rider, ₹14,200 a year
Product: Sampada Life iSecure, ₹1 crore cover till age 60 without critical illness rider, ₹12,900 a year
Disclosures: Medical history asked (made); Free-look period explained (made)
Medical: Non-smoker; Knee surgery in 2022; No other medical conditions or regular medicines
Objections: Wanted a cheaper option as another site showed ₹11,000
Risk flags: none
Next step: Home medical test booked for Saturday at 10 am and payment link to be sent on WhatsApp
Sentiment: positive
Summary: Rohan, aged 32 with an annual income of ₹12 lakh, inquired about term insurance and disclosed a 2022 knee surgery, which the agent properly advised declaring. After comparing quotes, he selected the Sampada Life iSecure plan with a ₹1 crore cover till age 60 without the critical illness rider for ₹12,900 a year. The agent explained the 30-day free-look period, scheduled a home medical test for Saturday at 10 am, and promised to send the payment link via WhatsApp.

Alternative—0 words2

None (empty or expired)

Check 1 of 8

covers_the_call counts ×2

Covers the call. Cover, premium and outcome are in the summary.

Reference

Judge's verdict

Answer every check first. The judge's verdicts show after, so they can't bias you.

Judge check · gold v1 with rubric v1

Judge: anthropic/claude-sonnet-5.5 · your key

Judge check. One row per check, with how far the judge agrees with you.
CheckAgreement (κ)How often the judge agrees with your team, beyond chance. 1 is full agreement; 0.80 or more is trusted.LabelledTrust
covers_the_call0.0039 · 39 Y, 0 NNeeds 5 more labels
figures_from_call—39 · 39 Y, 0 NNeeds 5 more labels
no_misselling_missed—39 · 39 Y, 0 NNeeds 5 more labels
no_false_alarm—39 · 39 Y, 0 NNeeds 5 more labels
disclosures_right0.3039 · 12 Y, 27 NNot trusted
next_step_captured—39 · 39 Y, 0 NNeeds 5 more labels
no_identifiers0.0039 · 39 Y, 0 NNeeds 5 more labels
written_in_english0.0039 · 39 Y, 0 NNeeds 5 more labels

A check is trusted at κ 0.8How often the judge agrees with your team, beyond chance. 1 is full agreement; 0.80 or more is trusted. or more on 30 or more labelled rows, with at least 5 Yes and 5 No. A fifth of the rows is held out of every rubric fix; κ more than 0.1 lower there marks the check possibly overfit.

Locked: gold v1 with rubric v1 by ops.member · 20 Oct · 180 of 180 rows verified

Instructions for Dvar (optional)

None saved.

Rubric v18 checks by ops.member · 20 Oct
  • covers_the_call

    Covers the call. Cover, premium and outcome are in the summary.

    ×2
  • figures_from_call

    Figures from the call. Every amount and percentage was said on the call.

    Required
  • no_misselling_missed

    No mis-selling missed. Every mis-selling or data-leak moment is flagged.

    Required
  • no_false_alarm

    No false alarms. No flag the reviewer did not find.

  • disclosures_right

    Disclosures right. Each disclosure marked made or not, as on the call.

  • next_step_captured

    Next step captured. The promised follow-up is there.

  • no_identifiers

    No identifiers repeated. No phone, PAN, Aadhaar, card, OTP or policy number.

    Required
  • written_in_english

    Written in English. English, whatever the call's language.

Version historyv1 approved by risk.member · 20 Oct
Rubric v1by ops.member · approved by risk.member · 20 Oct
  • covers_the_call

    Covers the call. Cover, premium and outcome are in the summary.

    ×2
  • figures_from_call

    Figures from the call. Every amount and percentage was said on the call.

    Required
  • no_misselling_missed

    No mis-selling missed. Every mis-selling or data-leak moment is flagged.

    Required
  • no_false_alarm

    No false alarms. No flag the reviewer did not find.

  • disclosures_right

    Disclosures right. Each disclosure marked made or not, as on the call.

  • next_step_captured

    Next step captured. The promised follow-up is there.

  • no_identifiers

    No identifiers repeated. No phone, PAN, Aadhaar, card, OTP or policy number.

    Required
  • written_in_english

    Written in English. English, whatever the call's language.

——