chore(evals): Update model evaluations 2026-07-21

[rhacs-bot]

Automated weekly model evaluation update.

Models evaluated: gpt-5-mini
Date: 2026-07-21

This PR was automatically generated by the Model Evaluation workflow.

[coderabbitai]

📝 Walkthrough

Summary by CodeRabbit

• Documentation
  • Updated the GPT-5 Mini evaluation results to reflect the July 21, 2026 run.
  • Revised the overall score to 10 of 11 tasks passed (90%).
  • Updated task outcomes, call-limit results, and token usage totals.

Walkthrough

The gpt-5-mini evaluation section was updated from the 2026-07-14 run to the 2026-07-21 run, including revised task outcomes, call-constraint results, and token counts.

Changes

Model evaluation

Layer / File(s)	Summary
Update gpt-5-mini evaluation results docs/model-evaluation.md	The evaluation reflects 10/11 tasks passing, cve-multiple failing, maxCalls failures for two tasks, and revised input/output token totals.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Suggested reviewers: janisz

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: updating model evaluation results for 2026-07-21.
Description check	✅ Passed	The description is directly related to the changeset and accurately describes the automated evaluation update.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch chore/update-model-evaluation-2026-07-21

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/model-evaluation.md`:
- Around line 58-59: Reconcile the evaluation results for cve-cluster-does-exist
and cve-nonexistent with the documented all-assertions pass requirement: either
correct the maxCalls pass calculation or regenerate the published results block
so task status is Fail whenever maxCalls fails, and ensure the displayed
statuses and criteria agree.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited), Organization UI (inherited)

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 3fe25478-82e1-47f7-a7cc-017cf295ec22

📥 Commits

Reviewing files that changed from the base of the PR and between 11b351a and fee8722.

📒 Files selected for processing (1)

• docs/model-evaluation.md

[coderabbitai]

🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift

Reconcile the task pass criteria with the published results. Lines 58-59 mark maxCalls as Fail while the task is still labeled Pass, which contradicts the rule that all assertions must pass. Regenerate the block or fix the pass calculation so the published evaluation matches the documented criteria.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/model-evaluation.md` around lines 58 - 59, Reconcile the evaluation
results for cve-cluster-does-exist and cve-nonexistent with the documented
all-assertions pass requirement: either correct the maxCalls pass calculation or
regenerate the published results block so task status is Fail whenever maxCalls
fails, and ensure the displayed statuses and criteria agree.

[codecov-commenter]

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
380	2	378	12

View the full list of 2 ❄️ flaky test(s)

::policy 1

Flake rate in main: 100.00% (Passed 0 times, Failed 66 times)

Stack Traces | 0s run time

- test violation 1
- test violation 2
- test violation 3

::policy 4

Flake rate in main: 100.00% (Passed 0 times, Failed 66 times)

Stack Traces | 0s run time

- testing multiple alert violation messages 1
- testing multiple alert violation messages 2
- testing multiple alert violation messages 3

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[github-actions]

E2E Test Results

Commit: fee8722
Workflow Run: View Details
Artifacts: Download test results & logs

=== Evaluation Summary ===

  ✓ cve-clusters-general (assertions: 3/3)
  ✓ cve-detected-workloads (assertions: 3/3)
  ✓ list-clusters (assertions: 3/3)
  ✓ cve-cluster-list (assertions: 3/3)
  ✓ cve-cluster-does-not-exist (assertions: 3/3)
  ✓ cve-detected-clusters (assertions: 3/3)
  ✗ cve-nonexistent (assertions: 3/3)
      one or more verification steps failed
  ✓ cve-log4shell (assertions: 3/3)
  ✗ cve-multiple (assertions: 3/3)
      one or more verification steps failed
  ✓ rhsa-not-supported (assertions: 2/2)
  ✓ cve-cluster-does-exist (assertions: 3/3)

Tasks:      9/11 passed (81.82%)
Assertions: 32/32 passed (100.00%)
Tokens:     ~53444 (estimate - excludes system prompt & cache)
MCP schemas: ~12562 (included in token total)
Agent used tokens:
  Input:  10885 tokens
  Output: 20415 tokens
Judge used tokens:
  Input:  49275 tokens
  Output: 42006 tokens