SRE-skills-bench

Can Language Models Resolve SRE Tasks?

SRE-skills-bench evaluates LLMs on tasks commonly performed by Site Reliability Engineers, helping reliability practitioners choose the right model for the job, whether it's powering IDE assistants, automating operational workflows, or improving incident response. Think of SRE-skills-bench as the SWE-bench of Site Reliability Engineering.

At the Rootly AI Labs, we run SRE-skills-bench on frontier models the day they are released, and we share our findings on our social media platforms (LinkedIn, X). We also present our benchmarks at leading ML research conferences, including as workshop papers at NeurIPS, ICML, and ACL.

Benchmarks

SRE-skills-bench is organized as sub-benchmarks along a comprehend → write → act capability ladder. Each is self-contained under benchmarks/<name>/ with its own README, dependencies, and runner, and reports its own score (there is no single blended number).

Track	Capability	What it tests	Status
General Knowledge	Comprehend	Given a bug-fix issue and four candidate PRs from the same repo, identify the PR that closed it (GMCQ)	Available — runs via openbench eval rootly_gmcq
Code Reasoning	Comprehend → Write	Reason about real PR diffs across five tasks (match the closing diff, reconstruct the issue, complete a masked change, detect hallucination) — all environment-free	Available — data + judge prompts
Terraform	Write	Generate executable Terraform from a natural-language prompt; graded by running the full fmt → init → validate → plan → apply → destroy lifecycle against LocalStack	Available
Incident Response	Act	Replay real postmortems as live scenarios; grade an agent across detect → localize → diagnose → mitigate → verify	Planned

benchmarks/
├── general-knowledge/   # comprehend  (GMCQ — runs via openbench)
├── code-reasoning/      # comprehend→write  (PR-diff reasoning, environment-free)
├── terraform/           # write       (executable Terraform generation)
└── incident-response/   # act         (postmortem replay — planned)

📰 News

• [Dec. 2, 2025]: presenting our work at ER – NeurIPS in San Diego, USA.
• [Nov. 24, 2025]: released ~3,000 new tasks testing LLMs on compute, network, and storage actions across AWS, GCP, and Azure.
• [Jul. 27, 2025]: presented our work at KnowFM – ACL 2025 in Vienna, Austria.
• [Jul. 19, 2025]: presented our work at New In ML – ICML 2025 in Vancouver, Canada.

Getting Started

This project uses mise for tool version management and uv for Python packaging.

# Install mise (if not already installed)
curl https://mise.run | sh
mise trust && mise install

Each benchmark is run from its own directory — see the per-track README for setup and commands:

• Terraform → benchmarks/terraform/README.md
• General Knowledge → benchmarks/general-knowledge/README.md
• Code Reasoning → benchmarks/code-reasoning/README.md
• Incident Response → benchmarks/incident-response/README.md

🔗 About the Rootly AI Labs

SRE-skills-bench is built with ❤️ by the Rootly AI Labs for engineering teams everywhere. The Rootly AI Labs is a fellow-led community designed to redefine reliability engineering. We develop innovative prototypes, create open-source tools, and produce research that's shared to advance the standards of operational excellence. We want to thank Anthropic, Google Cloud, and Google DeepMind for their support.