Skip to content

TEST-GUIDE.md

Latest commit

 

History

History
173 lines (139 loc) · 6.03 KB

TEST-GUIDE.md

File metadata and controls

173 lines (139 loc) · 6.03 KB
Raw

Writing Test Suites

The full suite grammar, with the reasoning behind it. examples/quickstart.yaml exercises everything on this page against httpbin.

Suite shape

name: Orders            # unique name — required for serial suites (see below)
description: Order API contract
serial: false           # optional; true = run tests one at a time, in file order
tests:
  - id: ORD-001         # shown in output and reports; filterable via --test
    name: list orders returns an array
    method: GET         # GET | POST | PUT | DELETE
    endpoint: /api/orders?limit=10
    tags: [read-only]   # optional; filterable via --tags
    expectedStatus: 200
    expectedFields:
      data.orders: type:array

Field names are camelCase in YAML (expectedStatus, alternateStatuses, expectedFields, expectedHeaders, preconditions, setup, teardown, ignoreFailure, storeAs, extractField, failMessage).

Status assertions

expectedStatus: 200
alternateStatuses: [403]   # optional — accept these too

Use alternateStatuses for endpoints whose status legitimately varies by deployment (e.g. admin gating). Body and header assertions run only when the primary status matched — a 403 or redirect usually has no JSON body to validate, and a spurious field failure there would be noise.

Non-200 contracts are first-class: expectedStatus: 404 asserts the 404.

Field assertions

expectedFields maps dot-separated JSON paths to operator strings:

Operator Meaning
* Field exists, any value
type:string type:number type:bool type:array type:object JSON kind check
exact:VALUE Case-insensitive equality
contains:SUBSTR Case-insensitive substring
gt:N gte:N lt:N lte:N Numeric comparison (value must be a JSON number)
arrayLength:N / arrayMinLength:N Exact / minimum element count
bare string Legacy exact match 
expectedFields:
  data.total: type:number
  data.mode: exact:strict
  url: contains:/api/
  data.score: gt:0.5

Gate only what you care about. Unlisted fields are not asserted. Leave volatile fields (timings, generated ids, scores) out of the suite and it will replay green for years.

Invariant gates — each: / all:

A path may contain one [*] array wildcard, combined with an each: (alias all:) prefix wrapping any single-value operator:

expectedFields:
  data.results: arrayMinLength:1
  data.results[*].artist: each:contains:Miles Davis
  data.results[*].id: all:type:string

Every element must satisfy the inner operator; an empty array passes vacuously (pair with arrayMinLength: when emptiness itself would be a failure).

Gates are how you freeze behavior rather than data: assert the property that must hold for any dataset ("every result matches the filter"), not the payload you happened to see today. Match the endpoint's real contract — an exact filter earns each:exact:; a fuzzy or ranked match only supports a weaker invariant like each:contains:. An overstated gate is a false-red waiting to happen.

Header assertions

expectedHeaders:
  Content-Type: contains:json
  Cache-Control: "*"

Same operators as fields. Content headers (Content-Type, Content-Length, Content-Range) are merged into the assertable set.

Setup, teardown, and variables

Steps are API calls that run before (setup) or after (teardown) a test. Teardown always runs — even when the test fails or throws.

- id: ORD-002
  name: created order is retrievable
  method: GET
  endpoint: /api/orders/${orderId}
  expectedStatus: 200
  expectedFields:
    data.id: exact:${orderId}        # variables substitute in assertion values too
  setup:
    - description: create an order to look up
      method: POST
      endpoint: /api/orders
      body: '{"widget": "sprocket"}'
      storeAs: orderId
      extractField: data.id
  teardown:
    - description: clean up
      method: DELETE
      endpoint: /api/orders/${orderId}
      ignoreFailure: true            # cleanup must not mask the test result
  • storeAs names the variable; extractField selects a dot-path from the step's response (a numeric segment indexes into an array: data.results.0.url). Without extractField, the whole response body is stored.
  • ${name} substitutes into endpoints, bodies, and assertion values. The stored value is percent-encoded, safe for URL paths.
  • ${name:json} is the JSON-string-escaped flavor — use it when embedding the value inside a JSON request body, where percent-encoding would corrupt it.
  • A failing setup step fails the test (unless ignoreFailure: true).

Preconditions

Skip — don't fail — tests the environment can't support:

preconditions:
  - type: endpoint-returns
    endpoint: /api/features
    expectedStatus: 200
    failMessage: feature service not deployed
  - type: field-equals
    endpoint: /api/features
    field: data.search
    value: "true"
    failMessage: search disabled

Types: endpoint-returns, field-equals, field-exists. Unknown types pass (forward compatibility). Skipped tests are reported but do not affect the exit code.

Serial suites

Tests run concurrently by default. When a suite's tests form a stateful transition cycle (start → assert running → stop → assert stopped), mark the suite serial: true so they run one at a time in file order. Serial suites need a unique top-level name: — unnamed suites merge into one display group and would serialize together.

Running

mbxhval validate --host 127.0.0.1 --port 8080 --suite-file smoke.yaml
mbxhval validate --suite-dir tests/captures            # every *.yaml, name-sorted
mbxhval validate --suite-file smoke.yaml --test ORD-002
mbxhval validate --suite-dir suites --tags read-only
mbxhval validate --suite-dir suites --report junit --output results.xml
mbxhval validate --suite-dir suites --fail-fast --concurrency 4

Exit 0 = all executed tests passed (skips allowed). Exit 1 = any failure, no suites found, or target unreachable. That contract is what makes a suite a CI gate.