Note

Building your own reporter? Check out features.md for a checklist.

Flakiness JSON Report Specification

Official specification for the Flakiness.io report format.

In a nutshell, Flakiness Report is a JSON file that follows this specification. Oftentimes, this JSON file is accompanied by a set of files - attachments. This format defines a standardized file system layout for storing these test artifacts.

This repository contains:

• TypeScript type definitions for type-safe usage of Flakiness Report data structures
• Zod schema validation for runtime validation of report JSON
• Directory layout to store report and its attachments on file system

Design Goals

• Portability: Generate and process reports on any platform combination without compatibility issues.
• Scalability: Efficiently store tens of thousands of test results using attachment references.
• Simplicity: Standard JSON format that's easy to parse and work with in any language or tool.
• Flexibility: Supports diverse testing framework features: tags, annotations, multiple errors, attachments, and execution environments.

Usage

The easiest way to generate a Flakiness JSON Report is to use one of the Flakiness.io integrations: https://flakiness.io/docs/integrations/overview/

Once you have a flakiness-report on the file system, you can view & upload it using the Flakiness CLI Tool:

# view report
flakiness show ./flakiness-report
# upload report to flakiness.io
flakiness upload ./flakiness-report

Learn more in the official documentation.

Specification

JSON format

• TypeScript definitions - report structure as TypeScript types with inline documentation
• Zod schema - runtime validator for report JSON

💡 Tip: The TypeScript type definitions include extensive inline comments that describe each entity and field in detail. Be sure to read through the comments in flakinessReport.ts for a comprehensive understanding of the report format structure.

Minimal Example

Here's a minimal Flakiness Report with one environment, one test and one attachment:

flakiness-report/
├── report.json
└── attachments/
    └── 5d41402abc4b2a76b9719d911017c592

{
  "title": "E2E tests on qa backend / run-tests-web-2 (01-03)",
  "category": "pytest",
  "commitId": "a1b2c3d4e5f6789012345678901234567890abcd",
  "environments": [
    {
      "name": "Python Tests"
    }
  ],
  "tests": [
    {
      "title": "should pass basic test",
      "location": {
        "file": "tests/test_example.py",
        "line": 3,
        "column": 1
      },
      "attempts": [
        {
          "environmentIdx": 0,
          "expectedStatus": "passed",
          "status": "passed",
          "startTimestamp": 1703001600000,
          "duration": 1500,
          "attachments": [
            {
              "name": "screenshot.png",
              "contentType": "image/png",
              "id": "5d41402abc4b2a76b9719d911017c592"
            }
          ]
        }
      ]
    }
  ],
  "startTimestamp": 1703001600000,
  "duration": 2000
}

Report Concepts

1. File Paths All file paths within the report are POSIX-formatted paths relative to the repository root, regardless of the platform on which tests are executed.

2. Test A test represents a specific location in the source code.

3. Suite A suite is a logical grouping of tests. Suites can be of various types and may have associated file locations.

4. Environment An environment is a set of key-value pairs that describe the execution context. Environments capture information about the operating system, browser, and other testing properties.

5. Run Attempts A run attempt represents a single execution of a test within a specific environment. When test runners automatically retry failed tests, each retry is recorded as a separate run attempt for the same test in the same environment.

6. Test Statuses Each run attempt has both an actual status and an expected status. The expected status is typically passed, but some test runners allow marking tests as expected to fail, in which case the expected status is failed.

   The Flakiness Report viewer supports filtering reports by status.

7. Test Tags Test tags are case-insensitive markers assigned to tests. Tags are static and cannot be dynamically attached during test execution; they are typically modified only when source code changes. Common examples include smoke, e2e, and regression.

   The Flakiness Report viewer supports filtering reports by tags.

8. Report Title The optional top-level title field gives the overall run a human-readable name. This is useful for CI matrix shards, manually triggered runs, or any workflow where viewers need more context than a job URL alone.

9. Annotations Annotations are metadata attached to individual run attempts, unlike tags which are attached to tests themselves. Annotations are dynamic and can vary across different test executions.

   Each annotation has a type and a description. Common use cases include:

   • skip annotations to mark tests as skipped (without additional description)
   • owner annotations to assign ownership of specific tests

   The Flakiness Report viewer supports filtering by annotations.

10. Attachments Each run attempt can have attachments: screenshots, videos, logs, or other debugging artifacts referenced by ID. Actual attachment contents are stored on the file system, following the directory layout explained in the "Attachments" section.

11. Producer & runtime metadata Optional top-level fields capture who produced the report and where it ran:

    • generatedBy — the reporter package that emitted the JSON (e.g., @flakiness/playwright, version 1.8.0).
    • testRunner — the underlying test runner the reporter integrated with (e.g., playwright, version 1.57.0).
    • runtime — the language runtime executing the test process (e.g., node 22.10, bun 1.1, cpython 3.13). Flakiness can be runtime-dependent, so capturing this helps diagnose reproducibility issues.

    Each is a { name, version } object. All three are optional; when present, both fields must be set.

Attachments

Attachments (screenshots, videos, logs, etc.) are referenced in the report by ID rather than embedded directly.

Each attachment in a RunAttempt contains:

• name - Descriptive name of the attachment (often the filename)
• contentType - MIME type of the attachment
• id - Unique identifier used to retrieve the actual attachment content. It is recommended to use the MD5 hash of the attachment content as the identifier.

The actual attachment files are stored in the attachments/ directory alongside the report.json, with their id as the filename (without extension).

The report JSON and its attachments should be organized as follows:

flakiness-report/
├── report.json
└── attachments/
    ├── 5d41402abc4b2a76b9719d911017c592
    ├── 7d865e959b2466918c9863afca942d0f
    └── 9bb58f26192e4ba00f01e2e7b136bbd8

Important: Report generators shall not compress attachments manually. The Flakiness.io CLI tool automatically applies optimal compression for different file types during upload.

NPM Package

The repository is published to NPM and is compatible with both Node.js and browser environments:

# Install this type definition
npm install @flakiness/flakiness-report

The package provides TypeScript types and a Zod schema for runtime validation:

import { FlakinessReport, Schema } from '@flakiness/flakiness-report';

// Type-safe report handling
const report: FlakinessReport.Report = { /* ... */ };

// Runtime validation using Zod schema
const result = Schema.Report.safeParse(report);
if (!result.success)
  console.error(`Validation failed:`, result.error);

Development

For information on building, watching, and contributing to this project, see CONTRIBUTING.md.