Tempest

Tempest is a YAML-based HTTP API test runner built for readable, composable test suites.

Features

• Declarative HTTP tests in YAML
• CEL and Liquid Templates are used for test preparation and assertions
• Automatic retries with flaky test detection
• Built-in console and json reports - or build your own!
• Nested structure with cascading directory and descriptor options

Quick start

Create a directory for your tests and add my-test.spec.yml:

# my-test.spec.yml
name: "Fetch the octocat/Hello-World repo"
test:
  route: "https://api.github.com/repos/octocat/Hello-World"
  headers:
    Accept: "application/vnd.github.v3+json"
  assert:
    - status == 200
    - body.json().full_name == "octocat/Hello-World"
    - body.json().owner.login == "octocat"

Mount that directory at /etc/tests and run the test command:

# Bash
docker run --rm --volume "$PWD:/etc/tests" mattisthegreatest/tempest:latest test

The Docker image supports Linux AMD64 and ARM64.

More executable scenarios are available under examples/tests.

CLI

Tempest currently provides one command:

tempest test [OPTIONS]

Option	Default	Description
--path <PATH>	/etc/tests	Test-project root to discover.
-r, --run <PATH>		An array of files/subdirectories relative to --path that are to be executed.
-d, --debug	false	Render the resolved route and detailed response information. This may expose sensitive response headers or bodies.
--retries <N>	0	Default number of additional attempts after an assertion failure.
--workers <N>	unset	Maximum number of spec files in flight. Must be greater than zero and enables file concurrency when supplied.
-s, --strict	false	Return exit code 2 when the run contains flaky tests but no failures.
-w, --warn-as-err	false	Return exit code 1 when Tempest emits any warning.
-h, --help		Print command help.
-e, --env		An array of base environment variables. Eg; -e KEY1=value1 -e KEY2=value2

Project files and discovery

Tempest recursively scans the directory selected by --path and recognizes these files:

Pattern	Purpose
*.spec.yml, *.spec.yaml	Test descriptor trees.
*.config.yml, *.config.yaml	Cascading run options.
*.template.yml, *.template.yaml	Custom report templates.
*.env	Simple KEY=value inputs exposed through Liquid's env object.

The filename suffix is significant: users.spec.yml is discovered, while users.yml is ignored.

Environment files support simple KEY=value lines. Blank lines are ignored, and # begins a comment even when it appears after a value. These files are project inputs rather than a full shell parser, so shell expansion, export, and complex quoting should not be relied upon.

API_BASE_URI=https://api.example.com
API_TOKEN=secret

test:
  route: "{{ env.API_BASE_URI }}/users"
  headers:
    Authorization: "Bearer {{ env.API_TOKEN }}"

Configuration and environment values discovered in a directory are available to its descendants. Prefer one config and one environment file at each configuration level so precedence remains obvious.

--run filters which specs execute; Tempest still discovers the project from --path so relevant configuration, environment values, and report templates remain available.

Test specifications

A spec file contains a descriptor. A descriptor can contain a test, nested descriptors under describe, or both.

name: "Display name"
description: "Optional longer explanation"
tags:
  - smoke
  - api

options:
  base_uri: https://api.example.com
  retries: 1

describe:
  - name: "Nested section"
    test:
      route: /health
      assert:
        - status == 200

test:
  route: /status
  assert:
    - status == 200

Nested descriptors execute depth-first in source order. Options on a descriptor are inherited by its descendants; options on one sibling do not affect another sibling. tags is currently metadata only and cannot yet be used to select tests.

Loops and profiles

Use options.loop to repeat a descriptor and its subtree a positive number of times. loop is descriptor-local rather than inherited: the parent already repeats its complete subtree, so applying the same loop again to every child would multiply it twice.

name: "Health check {{ iteration.loop_index }}"
options:
  loop: 2
test:
  route: /health

Use a non-empty descriptor-level profiles list to run the descriptor once for each input mapping:

name: "Post {{ profile.post_id }}"
profiles:
  - post_id: 1
  - post_id: 2
test:
  route: "/posts/{{ profile.post_id }}"
  assert:
    - body.json().id == {{ profile.post_id }}

When both are present, profiles are the outer expansion and loop is the inner expansion. Nested profiles form a Cartesian product. A child profile is shallow-merged over the active parent profile; matching child keys temporarily override parent keys, and the parent profile is restored before the next sibling. Expanded cases execute sequentially in source order.

HTTP test fields

test:
  route: /posts?postId=1
  verb: POST
  body: '{"title":"Tempest","userId":1}'
  headers:
    Content-Type: application/json
    Accept: application/json
  let:
    post: body.json()
  assert:
    - status == 201u
    - let.post.title == "Tempest"
  vars:
    post_id: let.post.id

Field	Required	Description
route	Yes	Absolute HTTP(S) URL or path joined to base_uri. Query parameters should be written directly in this string.
verb	No	HTTP method. Defaults to GET; matching is case-insensitive.
body	No	String request body sent with POST, PUT, and PATCH.
headers	No	Mapping of request header names to string values.
let	No	Ordered mapping of CEL expressions available under the test-scoped let namespace.
assert	No	List of CEL expressions. Every expression must evaluate to a boolean.
vars	No	Mapping of names to CEL expressions saved for later tests in this spec file.

Supported methods are GET, POST, PUT, PATCH, DELETE, and HEAD. An unsupported method currently produces an empty result with status 0 and sends no request.

HTTP status alone does not determine whether a test passes. A test with no assertions is considered passed, even if the response is a 4xx or 5xx response. Write the expected status explicitly.

Transport failures produce an empty response with status 504; the underlying error is available in status_message.

Configuration

Run options can be placed directly in a *.config.yml file:

base_uri: https://api.example.com
debug: false
retries: 1
reports:
  - console
  - json
concurrent: true

The same fields can appear under options on any descriptor:

name: "Eventually consistent endpoint"
options:
  retries: 3
test:
  route: /jobs/123
  assert:
    - status == 200

Option	Description
base_uri	Prefix for routes that do not start with http:// or https://. Joining normalizes the slash between the two values.
debug	Enables detailed request and response output through reporters that define a debug template.
retries	Number of additional attempts allowed after assertion failure.
retry_delay_ms	Unsigned integer denoting the number of milliseconds of delay in between retries. Default: 1000
reports	Names of report templates to use. Defaults to console. A configured list replaces, rather than extends, the inherited list.
concurrent	Enables concurrent spec-file execution. Treat this as a root project setting.
skip	When true, matching tests are reported as skipped without sending a request or evaluating assertions/variables.
loop	Positive number of times to execute this descriptor and its subtree. Valid only under descriptor options and not inherited as a run option.
quiet_retry	When true, do not report on the given test(s) when they retry. This includes summary values
quiet_run	When true, do not report on the given test(s) when they run. This includes summary values
quiet_fail	When true, do not report on the given test(s) when they fail. This includes summary values

Note

Sections without tests are not counted as skipped

Options cascade from command defaults through parent-directory configs, child-directory configs, and nested descriptor options. The more local configured value wins.

If a config or descriptor defines reports, include console explicitly if console output should remain enabled:

reports:
  - console
  - json

Liquid input interpolation

Tempest renders spec inputs with Liquid before sending a request. This includes descriptor names and descriptions, base_uri, routes, verbs, bodies, header names and values, assertions, and variable expressions.

Available input globals are:

Global	Description
env	Values loaded from discovered *.env files.
vars	Values saved by earlier tests in the same spec file.
file_name	Current spec-file path as a string.
retry_attempts	Zero-based attempt number: 0 for the initial attempt, 1 for the first retry, and so on.
profile	Effective shallow-merged profile for the current expanded case.
profile_stack	Individual active profiles ordered from outermost to innermost.
iteration	Current case, profile, and loop indexes and counts.
iteration_stack	Active iteration metadata ordered from outermost to innermost.

iteration contains case_index, case_count, profile_index, profile_count, loop_index, and loop_count. All indexes are zero-based.

test:
  route: "/users/{{ vars.user_id }}"
  headers:
    Authorization: "Bearer {{ env.API_TOKEN }}"

CEL and Liquid have different jobs:

• Use CEL to inspect the current HTTP response in let, assert, and on the right-hand side of vars.
• Use Liquid to interpolate environment values, retry state, and values saved by earlier tests.

When a Liquid value is rendered into a CEL string comparison, keep the CEL string quotes:

assert:
  - 'body.json().name == "{{ vars.expected_name }}"'

Numbers and booleans can normally be rendered without quotes:

assert:
  - 'body.json().id == {{ vars.expected_id }}'

Input interpolation and custom report rendering use the same Liquid engine, but each receives a different set of globals.

CEL assertions

Every expression under assert is evaluated against the HTTP response and must return a boolean. false, parse errors, execution errors, and non-boolean results all fail the assertion; evaluation errors are included in reporter output.

Response variables

Variable	CEL type	Description
status	unsigned integer	HTTP status code.
status_message	string	Canonical HTTP status text, or the transport error message.
body	string	Response bytes decoded lossily as text.
bytes	bytes	Raw response body.
headers	map	Response headers, normally addressed with lowercase names.
duration	unsigned integer	Request duration in whole milliseconds.

assert:
  - status == 200
  - status_message == "OK"
  - headers["content-type"].contains("application/json")
  - duration < 1000u
  - body.contains("Tempest")

Normal CEL operators, methods, macros, and literals supported by the bundled interpreter are available. Existing examples use contains, startsWith, endsWith, matches, size, all, exists, exists_one, filter, and map.

JSON

Call .json() on a string to parse it into CEL-compatible JSON data:

assert:
  - body.json().id == 1
  - body.json()["display-name"] != ""
  - body.json().all(item, item.id > 0)

Dotted access works for identifier-like keys; bracket access works for keys containing punctuation. Invalid JSON produces an assertion evaluation error.

HTML and CSS selectors

Call .css(selector) on an HTML string. It returns a list of matches containing tag, normalized text, and an attrs map:

assert:
  - body.css("title").exists(element, element.text == "Example Domain")
  - body.css("a[href]").all(element, element.attrs.href.startsWith("https://"))
  - body.css("main article").size() > 0

An invalid selector produces an assertion evaluation error.

XML and XPath

Call .xpath(expression) on an XML string. Boolean, number, and string XPath results become the corresponding CEL scalar; node sets become a list of strings.

assert:
  - body.xpath("count(/slideshow/slide)") == 2.0
  - body.xpath("string(/slideshow/@title)") == "Sample Slide Show"
  - body.xpath("/slideshow/slide/title") == ["First", "Second"]

Bytes and files

fileBytes(path) reads a file and returns bytes. Relative paths resolve from the current spec file's directory. Paths beginning with / resolve from the suite root selected by --path; paths may not escape that root.

assert:
  - bytes == fileBytes("fixtures/avatar.png")
  - bytes == fileBytes("/shared/avatar.png")

Use .toBase64() to encode bytes and .fromBase64() to decode a Base64 string:

assert:
  - bytes.toBase64() == "aGVsbG8="
  - '"aGVsbG8=".fromBase64() == b"hello"'

Missing files, rejected paths, and invalid Base64 produce evaluation errors.

Test-scoped bindings

Use let to name response-derived CEL values for the current test attempt. Bindings are evaluated in declaration order after the response arrives and before assertions run. Each expression can reference bindings declared above it through the let namespace.

test:
  route: /albums
  let:
    json: body.json()
    album_ids: let.json.map(album, album.id)
  assert:
    - status == 200
    - let.json.all(album, album.title.size() > 0)
    - let.album_ids.size() == 100
  vars:
    first_album_id: let.album_ids[0]

Bindings retain their CEL types, are recomputed for every retry attempt, and do not persist into later tests. Saved-variable expressions in the same test may reference them. If a binding cannot be parsed or evaluated, the attempt fails, later bindings and assertions are not evaluated, and vars are not assigned for that attempt.

Saved variables

Use vars to evaluate response-derived values and make them available to later tests in the same spec file. Each value is a CEL expression evaluated with the same variables, functions, and test-scoped let bindings available to assertions.

name: "Load a user"
test:
  route: /users/1
  assert:
    - status == 200
  vars:
    user_id: body.json().id
    user_name: body.json().name
    response_type: headers["content-type"]

Later descriptors access those values through Liquid's vars object:

name: "Use the saved user"
test:
  route: "/users/{{ vars.user_id }}"
  assert:
    - status == 200
    - 'body.json().name == "{{ vars.user_name }}"'

Variable behavior:

• Producer and consumer tests must be in the same spec file.
• The producer must appear before the consumer in execution order.
• Variables do not cross spec-file boundaries.
• CEL values are converted to JSON-compatible Liquid values.
• If a variable expression fails or cannot be converted, Tempest stores null; assignment failure does not itself fail the test.
• Mutations from a failed retry attempt are rolled back before the next attempt.
• A variable exported under an active loop or profile becomes an ordered array and appends once per terminal test attempt.
• Collected variables bleed into later expanded cases and subsequent descriptors, allowing later criteria to depend on earlier results.
• Once a variable becomes collected, later assignments to that name continue appending so its type remains stable.
• A pre-existing scalar is preserved as the first element when an expanded assignment promotes it to a collected variable.
• Collection indexes follow actual assignment order. Skipped tests and paths that do not evaluate vars do not reserve entries.

profiles:
  - post_id: 1
  - post_id: 2
test:
  route: "/posts/{{ profile.post_id }}"
  vars:
    post_ids: body.json().id

# A later descriptor can use vars.post_ids[0] and vars.post_ids[1].

Retries and flaky tests

retries is the number of additional attempts after an assertion failure. With retries: 1, a test can run at most twice.

options:
  retries: 2
test:
  route: /eventually-consistent-resource
  assert:
    - status == 200

A test that fails and later passes is reported as flaky. Every attempt is rendered, but the final summary counts the descriptor once.

The zero-based retry_attempts Liquid value can vary request inputs by attempt:

options:
  retries: 1
test:
  route: "/status/{% if retry_attempts == 0 %}500{% else %}200{% endif %}"
  assert:
    - status == 200

Flaky tests normally return exit code 0. Use --strict to return exit code 2 when a run contains flaky tests and no failures.

Concurrency

Spec files execute serially by default. Tests within one spec file always remain sequential so saved variables and retries have deterministic ordering.

Enable concurrent spec files in the root project config:

concurrent: true

Without an explicit worker limit, Tempest uses the machine's available parallelism. Use --workers N to set a positive cap; providing --workers enables file concurrency even when concurrent is absent or false.

# At most four spec files at once.
tempest test --path ./tests --workers 4

# Explicitly force serial execution.
tempest test --path ./tests --workers 1

concurrent is a suite scheduler setting, so configure it at the project root rather than on individual descriptors.

Reporters

Tempest includes two built-in report templates:

Reporter	Description
console	Default human-readable terminal output. (default)
json	JSON output written under ./tempest-reports/report-<timestamp>.json.

The console reporter automatically prefixes expanded tests with their one-based location, for example [profile #1/2] [loop #2/3]. Nested expansions include each active profile and loop from outermost to innermost.

Select reporters with the reports option:

reports:
  - console
  - json

Custom report templates

Create a *.template.yml or *.template.yaml file anywhere in the discovered test tree. Its registered name is the lowercase filename without .template.yml; for example, JUnit.template.yml is selected as junit.

# concise.template.yml
title_template: |
  Running {{ test_count }} tests

section_template: |
  # {{ full_name }}

test_template: |
  {% if skipped %}
    {{ expansion_prefix }} {{ full_name }}: skipped
  {% else %}
    {{ expansion_prefix }} {{ full_name }}: {{ status }} {{ status_message }}
  {% endif %}

summary_template: |
  Passed: {{ passed }}, flaky: {{ flaky }}, failed: {{ failed }}, skipped: {{ skipped }}

error_template: |
  Template error: {{ liquid_error_message }}

debug_template: |
  {{ debug_message }}

file:
  dir: ./tempest-reports
  file_name: report-{{ start_timestamp }}.txt

If file is omitted, rendered content is printed to the console. File-backed reporters create their directory when needed and append output to the target file. start_timestamp is available when rendering file_name.

Template fields may contain inline Liquid or refer to a .liquid file beside the YAML template:

test_template: concise.test.liquid
summary_template: concise.summary.liquid

Unknown report names are ignored.

Report template globals

Event/template	Available globals
title_template	test_count
section_template	name, description, title_path, full_name, expansion_prefix, passed, test_count, retry_count, assertions
test_template	Section globals plus status, status_message, body, duration_ms, skipped, and headers
summary_template	passed, failed, flaky, skipped
error_template	liquid_error_message
debug_template	debug_message

Each item in assertions contains expr, passed, and error.

expansion_prefix is empty for an ordinary descriptor. For expanded descriptors it contains the same one-based profile and loop locations used by the console reporter.

The Liquid engine includes its standard library plus Tempest's json, color_status, and color_duration filters and ANSI color filters such as red, green, yellow, bright_red, and their supported on_* background variants. The built-in templates under tempest/src/builtin_reporters provide complete examples.

The Liquid json report filter remains valid and is unrelated to the obsolete CEL json response variable.

Exit codes

Code	Meaning
0	No failed tests. Flaky tests also return 0 unless --strict is enabled.
1	One or more tests failed, or a warning occurred with --warn-as-err.
2	One or more tests were flaky, none failed, and --strict was enabled.

Warning-as-error handling takes precedence over the flaky exit code.

Current limitations

• Tempest currently runs HTTP tests only.
• tags are metadata and cannot currently filter --run selections.
• File-backed reports append to existing output files.
• The repository examples call public services and therefore require network access.

Developing Tempest

Run the passing examples from the repository:

cd tempest
cargo run -- test --path ../examples/tests --run pass

Run the automated checks:

cd tempest
cargo test
cargo clippy --all-targets --all-features -- -D warnings

License

Tempest is available under either of the following licenses, at your option:

• Apache License 2.0
• MIT License