Quality thresholds need attention: per_class.invalid.recall, consistency.
Explore evaluation →Evaluations
Model quality of the pipeline stages: latest gate per suite, then each suite's thresholds, trend and history. scorecard.json
Suites passing1of 2 latest evaluations Needs attention1Suites below their quality gate Recorded evaluations6Explore the history of each suite Cases in latest runs65Labelled cases across suites
Model quality
Evaluations
Latest result per suite. A passing gate means its required thresholds were met.
About these results & retention
Offline evals of the LLM pipeline stages against labelled datasets (docs, ADR-0027). Last 10 live runs per suite; runs that failed on error_rate (provider outage) are left out. Updated 2026-11-01T22:13:24.172Z. Pipeline runs, traces and health: pipeline views.
validation · Metrics, thresholds, trend & history
validation ✕ Fail per_class.invalid.recall, consistency
Last run 2026-10-19 · run 39260535854-validation · model groq:openai/gpt-oss-120b · 35 cases × 3 repeats
verdict_match
0.8571▼ 0.0667
score_in_range
0.9444▲ 0.0185
blocker_match
0.5455▼ 0.2121
suggested_ac_count
0.9905▼ 0.0095
valid recall
1=
invalid recall
0.7826▼ 0.1015
error_rate
0=
consistency
0.8857
latency p95
22.4 s
tokens in / out
274.7k / 20.3k
Thresholds
| Metric | Gate | Value | Status |
|---|---|---|---|
scores.verdict_match.mean | ≥ 0.8 | 0.8571 | ✓ pass |
per_class.invalid.recall | ≥ 0.8 | 0.7826 | ✕ fail |
per_class.valid.recall | ≥ 0.8 | 1 | ✓ pass |
consistency | ≥ 0.9 | 0.8857 | ✕ fail |
error_rate | ≤ 0.05 | 0 | ✓ pass |
Trend
verdict_matchscore_in_rangesuggested_ac_countblocker_match
History
| Date | Run | Model | Dataset | verdict_match | score_in_range | blocker_match | suggested_ac_count | error_rate | consistency | Gate |
|---|---|---|---|---|---|---|---|---|---|---|
| 2026-10-19 | 39260535854-validation | groq:openai/gpt-oss-120b | ec41af4f | 0.8571 | 0.9444 | 0.5455 | 0.9905 | 0 | 0.8857 | ✕ |
| 2026-10-16 | 38847780079-validation | groq:openai/gpt-oss-120b | ec41af4f | 0.9238 | 0.9259 | 0.7576 | 1 | 0 | 0.9429 | ✓ |
| 2026-10-12 | 38271632735-validation | groq:openai/gpt-oss-120b | c5608e3a | 0.8667 | 0.9259 | 0.5758 | 0.9905 | 0 | 0.9714 | ✕ |
| 2026-10-04 | 37173049724 | groq:openai/gpt-oss-120b | f37c25c0 | 1 | 1 | — | 1 | 0 | 1 | ✓ |
review · Metrics, thresholds, trend & history
review ✓ Pass
Last run 2026-10-17 · run 39000563980-review · model anthropic:claude-opus-5-5 · 30 cases × 1 repeats
verdict_match
0.9655▲ 0.0919
flags_issue
1=
no_false_alarm
0.9286▲ 0.1905
request_changes recall
1=
approve recall
0.9▲ 0.1
withheld recall
1▲ 0.4167
error_rate
0=
consistency
—
latency p95
37.4 s
tokens in / out
123.6k / 10.9k
Thresholds
| Metric | Gate | Value | Status |
|---|---|---|---|
scores.verdict_match.mean | ≥ 0.75 | 0.9655 | ✓ pass |
per_class.request_changes.recall | ≥ 0.8 | 1 | ✓ pass |
per_class.approve.recall | ≥ 0.6 | 0.9 | ✓ pass |
consistency | ≥ 0.8 | — | – not measured |
error_rate | ≤ 0.05 | 0 | ✓ pass |
Trend
verdict_matchflags_issueno_false_alarm
History
| Date | Run | Model | Dataset | verdict_match | flags_issue | no_false_alarm | error_rate | consistency | Gate |
|---|---|---|---|---|---|---|---|---|---|
| 2026-10-17 | 39000563980-review | anthropic:claude-opus-5-5 | dcff1773 | 0.9655 | 1 | 0.9286 | 0 | — | ✓ |
| 2026-10-17 | 38994398984-review | anthropic:claude-opus-5-5 | dcff1773 | 0.8736 | 1 | 0.7381 | 0 | 0.9655 | ✓ |