| Section | Measured (real model calls) | Computed live | Written for the demo, or assumed |
|---|---|---|---|
| Call QA and the scorecard | Every summary, from a real model call at temperature 0, one prompt and one token limit for all four summarizers (Today's model (stand-in): dvar:glm-5.3-flash; Closed-source model: google/gemini-4-argon; Open model: dvar:gemma-4-31b-it; Open model + examples: dvar:gemma-4-31b-it + 8 examples); its ₹ from the call's usage (OpenRouter's reported cost for the closed-source model, Dvar's list price for the others) and its latency. Runs A-20261022-173201, A-20261022-172612, 2026-10-22. | The eight checks on every summary, the evidence links, pass counts, the release gate, cost per minute (measured ₹ over the calls' minutes) and the live summary's grading. | The 18 calls and their reviewed summaries (the answer key). Today's model is a stand-in, the cheapest small Dvar model that returned a summary on 17 of 18 calls; Policybazaar's own summarizer was not measured. "Open model + examples" is not fine-tuned. |
| Advisor Copilot | Each sample question's answer from the small model (dvar:gemma-4-31b-it) and from the closed-source model (google/gemini-4-argon), with ₹ and time per answer. Run D-20261022-175119, 2026-10-22. | Retrieval over the wording, the six runtime checks, the routing to the closed-source model, the session's cost. | The made-up products' wording (regulator lines follow IRDAI's public rules) and the 12 questions. The guarantee and misstate checks were fixed after the first run flagged two correct refusals; that run is kept as D-20261022-173428. |
| Claims documents | What each model read from images of the five documents and the confidence it gave (dvar:gemma-4-31b-it small, google/gemini-4-argon large), with ₹ and time. Run E-20261022-174420, 2026-10-22. | Schemas, formats, dates, totals, the patient against the policy, the made-up policy's limits, routing, the review queue and scoring against the answer key. | The five specimen documents (PAN, masked Aadhaar and cheque drawn to the real layouts; a fictitious hospital's discharge summary and bill) and their answer key. Clean drawings, not scans. |
| Evals: Eval Console | 180 calls (the 18 with 10 customer names each) summarized by the same four summarizers, 3 repeats each on the Dvar models and 1 on the closed-source model, with tokens, ₹ and latency; an LLM judge on a sample of 40. Runs B-20261022-174852 and C-20261022-182846. | Scores, intervals, topic minimums, the four gates, judge agreement and kappa, computed as the eval engine computes them. Each summary graded by the demo's own check code. | The quality bar (0.85) and the latency bar for gate G4 (15 s) are assumptions. |
| Model upgrade | Cost per minute of call per model, from the scorecard's measured ₹. | Scaling by your sliders; gate results; today's misses. | The 17 lakh minutes a day default is from PB Fintech's Q1 FY27 deck. The passes per call are yours to set. That today's model costs what its stand-in costs on Dvar is an assumption. |
All measurement calls together: 1,981 model calls, ₹270.43, between 2026-10-22 and 2026-10-22, including smoke tests, retries and one run stopped and redone. Every call is a line in results/ledger.jsonl, and every output in results/ by run id.
In a pilot
- • Today's model measured first, on 200 to 500 calls your QA team has already reviewed, masked, inside your environment.
- • Checks and rubric agreed with your QA and compliance teams; the judge calibrated against their labels before it scores anything.
- • Candidate models scored on the same calls; a candidate fine-tuned on the reviewed misses, through the same gate.
- • Costs from your own GPU and traffic figures, reported per analysed call.