| Model | Quality | p99 latency | Cost per request | vs today | Result |
|---|---|---|---|---|---|
| Gemini 4 Argon Current modelopenrouter/google/gemini-4-argon | 81.1% replayed on 180 gold rows | 24,557 ms benchmark replies on the same messages | ₹0.52 benchmark replies on the same messages | — | Today |
| openai_compatible/dvar:gemma-4-31b-it | 34.4% replayed on 180 gold rows | 31,207 ms replayed on 180 gold rows | ₹0.0412 replayed on 180 gold rows | −92% | Fails 4 checks |
dvar:gemma-4-31b-it is fully right on 34.4% of gold rows, under the 85.0% quality bar; fails a critical check (an invented figure, a missed mis-selling flag or a repeated identifier) on 17 gold rows. The release rule allows none; falls below the minimum on Term life sales: 0.00 vs 0.85 (30 gold rows); Health sales and porting: 0.40 vs 0.85 (30 gold rows); Savings and ULIP sales: 0.40 vs 0.85 (30 gold rows); Motor renewals: 0.47 vs 0.85 (30 gold rows); Claims support: 0.33 vs 0.85 (30 gold rows); Policy service: 0.47 vs 0.85 (30 gold rows); is too slow: p99 31,207 ms replayed on 180 gold rows, over the 15,000 ms limit. Weakest topic: Term life sales 0.00 | |||||
| openai_compatible/dvar:gemma-4-31b-it_8-examples | 71.7% replayed on 180 gold rows | 21,383 ms replayed on 180 gold rows | ₹0.0696 replayed on 180 gold rows | −87% | Fails 3 checks |
dvar:gemma-4-31b-it_8-examples is fully right on 71.7% of gold rows, under the 85.0% quality bar; falls below the minimum on Term life sales: 0.67 vs 0.85 (30 gold rows); Health sales and porting: 0.67 vs 0.85 (30 gold rows); Claims support: 0.33 vs 0.85 (30 gold rows); Policy service: 0.77 vs 0.85 (30 gold rows). Every other gated topic is at or above its minimum; is too slow: p99 21,383 ms replayed on 180 gold rows, over the 15,000 ms limit. Weakest topic: Claims support 0.33 | |||||
No candidate here is fine-tuned. “+ 8 examples” is the open model with 8 reviewed summaries of other calls in its prompt. A model tuned on reviewed, masked calls is the next step, through this same gate: Fine-tune. | |||||
No candidate passes every check. Open a model's row for its reasons.
Prices are per request, at each provider's list price.
Judge: 0 of 8 checks trusted.
| Topic | Rows | Today · Gemini 4 Argon | Candidate · dvar:gemma-4-31b-it | Candidate · dvar:gemma-4-31b-it_8-examples | ||
|---|---|---|---|---|---|---|
| Term life sales | 30 | 0.33 | 0.00 | −0.33 | 0.67 | +0.34 |
| Health sales and porting | 30 | 1.00 | 0.40 | −0.60 | 0.67 | −0.33 |
| Savings and ULIP sales | 30 | 0.97 | 0.40 | −0.57 | 0.87 | −0.10 |
| Motor renewals | 30 | 1.00 | 0.47 | −0.53 | 1.00 | 0.00 |
| Claims support | 30 | 0.63 | 0.33 | −0.30 | 0.33 | −0.30 |
| Policy service | 30 | 0.93 | 0.47 | −0.46 | 0.77 | −0.16 |
| All topics, weighted by traffic | 180 | 0.81 | 0.34 | −0.47 | 0.72 | −0.09 |