dvar

Four summarizers, one answer key, eight checks

18 calls, each graded by the same 8 code checks against a reviewed answer key.

Today's model (stand-in)

Stand-in: dvar:glm-5.3-flash

11/18

calls pass every check

1 call with a critical miss or no summary

₹0.067 a call · ₹0.0119 per minutep50 8.4 s · slowest 15.9 s

Closed-source model

google/gemini-4-argon

15/18

calls pass every check

No critical miss

₹0.522 a call · ₹0.0931 per minutep50 7.4 s · slowest 10.0 s

Open model

dvar:gemma-4-31b-it

6/18

calls pass every check

2 calls with a critical miss or no summary

₹0.041 a call · ₹0.0073 per minutep50 26.5 s · slowest 29.3 s

Open model + examples

dvar:gemma-4-31b-it + 8 examples in the prompt

14/18

calls pass every check

No critical miss

₹0.070 a call · ₹0.0125 per minutep50 16.9 s · slowest 28.1 s

Closed-source model: can replace today's model

No critical miss; at least as good as today.

Open model: blocked

  • No summary on 2 calls.
  • Passes 6 calls, fewer than today's 11.

Open model + examples: can replace today's model

No critical miss; at least as good as today.

Cost at your volume →