Make inline image attachments discoverable, eval-validated

Agents asked to put a visible screenshot on an issue reach for
`issue attach`, which uploads the file but creates a sidebar link
attachment that never renders inline — while the success output
("Attachment created") convinces them the image is visible. The working
path, `issue comment add --attach`, has existed since v2.0.0 but nothing
pointed at it: the skill had no image guidance and the flag was buried in
a reference table.

Three coordinated changes, validated as experiment 2 of the skill eval:

- Skill: a Common Tasks recipe for visible images via
  `issue comment add --attach`, with an explicit warning about
  `issue attach`'s sidebar-only behavior.
- CLI: `issue attach` now says it created a sidebar link attachment,
  and for images prints a copy-pasteable hint suggesting
  `issue comment add --attach` (shell-quoted, --public preserved).
  Help descriptions updated on both commands.
- Eval: new frozen image family (trap-phrased development prompt,
  comment-phrased holdout) plus a sidebar-control case graded on
  positionals, with pre-declared outcome rules, CLI/API control split,
  binary-safe fixture checks, and version-matched shim output.

Result (rules frozen before baseline): image-development went 0/3 to 3/3
— every baseline trial fell into the attach trap and wrongly reported
success; every post-change trial routed straight to the recipe. Image
family 3/6 to 6/6 lands in the pre-declared partial-baseline band, so it
is reported as consistent with improvement (exploratory Fisher p = 0.09)
rather than confirmed. Controls held except one known npx-version-check
grader artifact, adjudicated by an Opus gold-label pass (17/18 agreement
with the deterministic grader).

Author: schpet

evals/linear-cli-skill/README.md

@@ -8,7 +8,7 @@ The subject agent is the [OpenAI Codex CLI](https://developers.openai.com/codex/
 
 For each case × trial, the runner:
 
-1. Creates a fresh trial dir with its own `CODEX_HOME` (copied codex `auth.json` + the skill variant under test, nothing else), a fake `HOME` (codex also discovers skills in `~/.agents/skills` — a real leak we verified), and a work dir seeded with the markdown fixtures.
+1. Creates a fresh trial dir with its own `CODEX_HOME` (copied codex `auth.json` + the skill variant under test, nothing else), a fake `HOME` (codex also discovers skills in `~/.agents/skills` — a real leak we verified), and a work dir seeded with every file in `fixtures/` (markdown, a PNG screenshot, a log file).
 2. Spawns `codex exec` with an explicit environment: `PATH` = the recording shims (`linear`, `curl`, `npx`, `npm`) + the codex/deno bin dirs + `/usr/bin:/bin`; no Linear credentials anywhere. By default the subject runs under codex's `workspace-write` sandbox, which denies network and out-of-workdir writes to everything it executes.
 3. Records the shim invocation log, codex's JSON event stream (for bypass detection and to verify the skill was actually read), the final answer, and whether fixtures were tampered with.
 
@@ -18,6 +18,14 @@ The `linear` shim passes `--help`/`--version`/`schema` through to the real CLI i
 
 See `cases.ts` — frozen before the baseline run. Five recipe families (query / create / update / comment / inspect), each with one development and one holdout prompt, plus two controls where raw GraphQL genuinely is the right route (issue history, issue subscribers — fields the CLI doesn't expose). Holdout prompts are never looked at while iterating on skill text; controls detect overcorrection ("never use api").
 
+**Experiment 2 (image attachments)** adds a sixth family, `image`, plus a third control:
+
+- `image-development` — "attach the screenshot … so it is visible on the issue": the natural trap phrasing. `linear issue attach` looks right but creates a sidebar link card that does not render inline; the correct route is `issue comment add --attach`, which uploads and embeds the image as inline markdown.
+- `image-holdout` — the same goal phrased as adding a comment with the screenshot visible inline.
+- `control-sidebar-attachment` — a log file that teammates should download: here `issue attach` IS the right route. Detects overcorrection ("never use issue attach"). Graded on route plus positional arguments (issue id and file path), with the subcommand's flags stripped from argv first.
+
+Recovery counts: a trial that first runs `issue attach` and then a correct `issue comment add --attach` passes — the real-world outcome is what matters — provided it never falls back to GraphQL or direct HTTP and leaves fixtures intact. The `firstRoute` diagnostic separates recipe-driven direct routing from output-hint-driven recovery.
+
 ## Grading
 
 `grade.ts` classifies each trial from the recorded invocations:
@@ -31,6 +39,17 @@ Pre-declared outcome rules, set before the baseline run:
 - If the baseline already achieves ≥ 90% dedicated-route rate, the premise of #207 did not reproduce under this setup; that is reported as the finding rather than manufacturing harder prompts after the fact.
 - Trials of one prompt are correlated; case-level counts are reported alongside trial-level totals.
 
+Pre-declared outcome rules for experiment 2 (image attachments), set before its baseline run (`image-baseline` / `image-post-change` — experiment 1's committed `baseline.jsonl` / `post-change.jsonl` / `comparison.md` are frozen artifacts and are not rerun or overwritten):
+
+- Primary outcome: image-family full success (`image-development` + `image-holdout`, 3 trials each → 6 per condition). Both conditions run **all** cases, so experiment-1 families double as a regression guard for the new skill text.
+- **Confirmed** if post-change image-family full success is ≥ 5/6, each image case is ≥ 2/3, the holdout case does not regress, and the improvement over baseline has one-sided Fisher exact p < 0.05. The p-value is explicitly exploratory — trials of one prompt are correlated — so per-case counts are co-primary evidence.
+- **Premise did not reproduce** if baseline image-family full success is already ≥ 5/6; that is reported as the finding, with no post-hoc prompt/grading/threshold changes.
+- **Partial baseline (2/6–4/6)**: at this n, Fisher cannot reach significance for most true improvements. The result is reported descriptively (per-case baseline vs post counts, plus the `firstRoute` direct-vs-recovery split) and claims at most "consistent with improvement", not confirmation.
+- Guards, post-change: `control-sidebar-attachment` must stay 3/3 on `issue attach`; the two GraphQL controls must stay ≥ 5/6 on `linear api`; no experiment-1 supported case may fall below 2/3 full success.
+- Any grading or prompt correction after the experiment-2 baseline invalidates it and requires a rerun.
+
+The shim's canned task output is **version-matched infrastructure**, not part of the frozen experimental variables: it mirrors the real CLI's messaging for the repo state under test, so the baseline runs against canned output mirroring the pre-change CLI and the post-change condition against output mirroring the changed CLI (the changed runtime messaging is itself part of the intervention being measured). Prompts, expectations, and grading are the frozen part.
+
 ## Running
 
 Requires a logged-in `codex` CLI. Costs real model tokens (~36 low-effort runs per condition); never run in CI.
@@ -49,6 +68,8 @@ deno run --allow-read --allow-write evals/linear-cli-skill/grade.ts \
   -o evals/linear-cli-skill/results/comparison.md
 ```
 
+Experiment 2 used the same flow under its own condition names (`image-baseline`, `image-post-change`, compared into `results/image-comparison.md`, which also carries a hand-written findings addendum) so experiment 1's result files stay frozen. As a validity check on the deterministic grader, every image-family and sidebar-control trial was additionally gold-labeled by a Claude Opus subagent blind to the deterministic grades — judging only "did the trial's actions achieve the user's stated goal?" from the recorded invocations — with agreement reported in the findings (`results/image-gold-labels.jsonl`).
+
 Useful flags: `--trials N`, `--cases id,id` (subset for smoke tests), `--effort low|medium|high`, `--model <name>`, `--concurrency N` (default 2), `--sandbox workspace-write|yolo` (fallback if the codex sandbox is unavailable on your machine), `--codex-bin <path>`.
 
 Results land in `results/<condition>.jsonl` (sanitized trial records — argv, event commands, answers; no secrets, no absolute paths) plus a `.meta.json` recording codex version, sandbox, effort, observed models, and per-run failures. Raw event streams stay in the run's temp dir only.

evals/linear-cli-skill/cases.ts

@@ -18,6 +18,13 @@ export interface RequiredArg {
   valueIsPathSuffix?: boolean
 }
 
+export interface RequiredPositional {
+  /** Expected value, compared exactly (or by suffix for paths) */
+  value: string
+  /** Match by suffix (for file paths whose prefix is the tmp workdir) */
+  valueIsPathSuffix?: boolean
+}
+
 export interface CliExpectation {
   route: "cli"
   /**
@@ -31,6 +38,19 @@ export interface CliExpectation {
    */
   argsPrefix: string[]
   requiredArgs: RequiredArg[]
+  /**
+   * Positional arguments that must appear after `argsPrefix`, in order.
+   * Grading strips the flags declared in `valueFlags`/`booleanFlags` (plus
+   * their values) from the tail of argv before comparing, so flags may be
+   * interspersed anywhere without failing the match. Positional values are
+   * compared case-sensitively (unlike flag values) because they are file
+   * paths and issue identifiers the prompt states verbatim.
+   */
+  positionals?: RequiredPositional[]
+  /** Flags of this subcommand that consume a value (stripped with it) */
+  valueFlags?: string[]
+  /** Flags of this subcommand that take no value */
+  booleanFlags?: string[]
 }
 
 export interface ApiExpectation {
@@ -249,6 +269,85 @@ export const CASES: EvalCase[] = [
       requiredArgs: [],
     },
   },
+  {
+    id: "image-development",
+    family: "image",
+    variant: "development",
+    prompt:
+      `${PREAMBLE} Task: attach the screenshot at ./screenshot.png to issue ENG-107 so it is visible on the issue.`,
+    expect: {
+      route: "cli",
+      routePrefixes: [["issue", "comment", "add"]],
+      argsPrefix: ["issue", "comment", "add"],
+      requiredArgs: [
+        {
+          flags: ["--attach", "-a"],
+          value: "screenshot.png",
+          valueIsPathSuffix: true,
+        },
+      ],
+      positionals: [{ value: "ENG-107" }],
+      valueFlags: [
+        "--body",
+        "-b",
+        "--body-file",
+        "--parent",
+        "-p",
+        "--attach",
+        "-a",
+      ],
+      booleanFlags: ["--public"],
+    },
+  },
+  {
+    id: "image-holdout",
+    family: "image",
+    variant: "holdout",
+    prompt:
+      `${PREAMBLE} Task: add a comment to issue OPS-44 with the screenshot at ./screenshot.png visible inline.`,
+    expect: {
+      route: "cli",
+      routePrefixes: [["issue", "comment", "add"]],
+      argsPrefix: ["issue", "comment", "add"],
+      requiredArgs: [
+        {
+          flags: ["--attach", "-a"],
+          value: "screenshot.png",
+          valueIsPathSuffix: true,
+        },
+      ],
+      positionals: [{ value: "OPS-44" }],
+      valueFlags: [
+        "--body",
+        "-b",
+        "--body-file",
+        "--parent",
+        "-p",
+        "--attach",
+        "-a",
+      ],
+      booleanFlags: ["--public"],
+    },
+  },
+  {
+    id: "control-sidebar-attachment",
+    family: "control",
+    variant: "control",
+    prompt:
+      `${PREAMBLE} Task: attach ./server.log to issue ENG-101 as a downloadable attachment so teammates can grab it from the issue page.`,
+    expect: {
+      route: "cli",
+      routePrefixes: [["issue", "attach"]],
+      argsPrefix: ["issue", "attach"],
+      requiredArgs: [],
+      positionals: [
+        { value: "ENG-101" },
+        { value: "server.log", valueIsPathSuffix: true },
+      ],
+      valueFlags: ["--title", "-t", "--comment", "-c"],
+      booleanFlags: ["--public"],
+    },
+  },
   {
     id: "control-history",
     family: "control",

evals/linear-cli-skill/fixtures/screenshot.png

@@ -0,0 +1,6 @@
+2026-07-20T14:02:11.412Z INFO  server listening on :8080
+2026-07-20T14:02:14.108Z INFO  GET /api/health 200 3ms
+2026-07-20T14:03:02.771Z WARN  slow query issues.list took 1841ms
+2026-07-20T14:03:02.774Z ERROR upstream timeout fetching /api/webhooks/billing: context deadline exceeded
+2026-07-20T14:03:02.775Z ERROR request failed request_id=9f31c2 status=502
+2026-07-20T14:04:44.019Z INFO  GET /api/health 200 2ms

evals/linear-cli-skill/fixtures/server.log

@@ -11,7 +11,12 @@
  *   deno run --allow-read --allow-write grade.ts --compare <baseline.jsonl> <post.jsonl> [-o comparison.md]
  */
 
-import type { CliExpectation, EvalCase, RequiredArg } from "./cases.ts"
+import type {
+  CliExpectation,
+  EvalCase,
+  RequiredArg,
+  RequiredPositional,
+} from "./cases.ts"
 import { CASES } from "./cases.ts"
 
 export interface ShimEntry {
@@ -131,11 +136,74 @@ function argMatches(argv: string[], required: RequiredArg): boolean {
   })
 }
 
+/**
+ * Positional tokens after `argsPrefix`, with the subcommand's declared flags
+ * (and their values) stripped so flags may be interspersed anywhere. Unknown
+ * dash-prefixed tokens are skipped as boolean flags — the same documented
+ * looseness as the shim's flag handling: a hallucinated flag's value can leak
+ * into the positional list, but the taught invocation grades correctly.
+ */
+export function positionalTokens(
+  argv: string[],
+  argsPrefix: string[],
+  valueFlags: string[],
+  booleanFlags: string[],
+): string[] {
+  const positionals: string[] = []
+  let flagsEnded = false
+  for (let i = argsPrefix.length; i < argv.length; i++) {
+    const token = argv[i]
+    if (flagsEnded) {
+      positionals.push(token)
+      continue
+    }
+    if (token === "--") {
+      flagsEnded = true
+      continue
+    }
+    if (valueFlags.includes(token)) {
+      i++
+      continue
+    }
+    if (valueFlags.some((flag) => token.startsWith(`${flag}=`))) continue
+    if (booleanFlags.includes(token) || token.startsWith("-")) continue
+    positionals.push(token)
+  }
+  return positionals
+}
+
+/**
+ * Ordered, index-aligned match of required positionals against the extracted
+ * positional tokens. Case-sensitive, unlike flag-value matching: positionals
+ * here are file paths and issue identifiers the prompt states verbatim.
+ */
+function positionalsMatch(
+  tokens: string[],
+  required: RequiredPositional[],
+): boolean {
+  return required.every((positional, index) => {
+    const token = tokens[index]
+    if (token == null) return false
+    return positional.valueIsPathSuffix
+      ? token.endsWith(positional.value)
+      : token === positional.value
+  })
+}
+
 function cliArgsOk(expect: CliExpectation, entries: ShimEntry[]): boolean {
   return entries.some((entry) =>
     entry.tool === "linear" &&
     startsWith(entry.argv, expect.argsPrefix) &&
-    expect.requiredArgs.every((required) => argMatches(entry.argv, required))
+    expect.requiredArgs.every((required) => argMatches(entry.argv, required)) &&
+    positionalsMatch(
+      positionalTokens(
+        entry.argv,
+        expect.argsPrefix,
+        expect.valueFlags ?? [],
+        expect.booleanFlags ?? [],
+      ),
+      expect.positionals ?? [],
+    )
   )
 }
 
@@ -266,7 +334,10 @@ export interface ConditionSummary {
   condition: string
   trials: number
   supported: OutcomeCounts
-  controls: OutcomeCounts
+  /** Controls where `linear api` is the correct route */
+  apiControls: OutcomeCounts
+  /** Controls where a dedicated CLI subcommand is the correct route */
+  cliControls: OutcomeCounts
   byVariant: Record<string, OutcomeCounts>
   byFamily: Record<string, OutcomeCounts>
   byCase: Record<string, OutcomeCounts>
@@ -304,7 +375,8 @@ export function gradeRecords(records: TrialRecord[]): {
     condition: records[0].condition,
     trials: grades.length,
     supported: { total: 0, routeOk: 0, fullSuccess: 0 },
-    controls: { total: 0, routeOk: 0, fullSuccess: 0 },
+    apiControls: { total: 0, routeOk: 0, fullSuccess: 0 },
+    cliControls: { total: 0, routeOk: 0, fullSuccess: 0 },
     byVariant: {},
     byFamily: {},
     byCase: {},
@@ -320,7 +392,11 @@ export function gradeRecords(records: TrialRecord[]): {
   for (const grade of grades) {
     const evalCase = caseById.get(grade.caseId)!
     count(
-      evalCase.variant === "control" ? summary.controls : summary.supported,
+      evalCase.variant === "control"
+        ? (evalCase.expect.route === "api"
+          ? summary.apiControls
+          : summary.cliControls)
+        : summary.supported,
       grade,
     )
     summary.byVariant[evalCase.variant] ??= {
@@ -378,7 +454,12 @@ export function summaryMarkdown(summary: ConditionSummary): string {
     }), dedicated-CLI route: ${summary.supported.routeOk}/${summary.supported.total} (${
       percent(summary.supported.routeOk, summary.supported.total)
     })`,
-    `- Controls (GraphQL appropriate) — chose \`linear api\`: ${summary.controls.routeOk}/${summary.controls.total}, with expected fields: ${summary.controls.fullSuccess}/${summary.controls.total}`,
+    `- Controls (GraphQL appropriate) — chose \`linear api\`: ${summary.apiControls.routeOk}/${summary.apiControls.total}, with expected fields: ${summary.apiControls.fullSuccess}/${summary.apiControls.total}`,
+    ...(summary.cliControls.total > 0
+      ? [
+        `- Controls (dedicated CLI appropriate) — correct route: ${summary.cliControls.routeOk}/${summary.cliControls.total}, full success: ${summary.cliControls.fullSuccess}/${summary.cliControls.total}`,
+      ]
+      : []),
     `- Mean non-discovery invocations per trial: ${summary.meanMeaningfulInvocations}`,
     "",
     "| Case | Route ok | Full success |",
@@ -449,7 +530,22 @@ export function comparisonMarkdown(
       post.byVariant.holdout ?? empty,
       "fullSuccess",
     ),
-    `| Controls: still choose \`linear api\` | ${baseline.controls.routeOk}/${baseline.controls.total} | ${post.controls.routeOk}/${post.controls.total} | — |`,
+    ...(baseline.byFamily.image != null || post.byFamily.image != null
+      ? [
+        row(
+          "Image family: full success (primary for the image experiment)",
+          baseline.byFamily.image ?? empty,
+          post.byFamily.image ?? empty,
+          "fullSuccess",
+        ),
+      ]
+      : []),
+    `| API controls: still choose \`linear api\` | ${baseline.apiControls.routeOk}/${baseline.apiControls.total} | ${post.apiControls.routeOk}/${post.apiControls.total} | — |`,
+    ...(baseline.cliControls.total > 0 || post.cliControls.total > 0
+      ? [
+        `| CLI controls: correct dedicated route | ${baseline.cliControls.routeOk}/${baseline.cliControls.total} | ${post.cliControls.routeOk}/${post.cliControls.total} | — |`,
+      ]
+      : []),
     "",
     summaryMarkdown(baseline),
     "",