Skip to content

feat(tokens): exclude selected paths from token accounting - #83

Open
choplin wants to merge 1 commit into
agent-ecosystem:mainfrom
choplin:feature/exclude-token-paths
Open

feat(tokens): exclude selected paths from token accounting#83
choplin wants to merge 1 commit into
agent-ecosystem:mainfrom
choplin:feature/exclude-token-paths

Conversation

@choplin

@choplin choplin commented Jul 24, 2026

Copy link
Copy Markdown

What this PR does

Some skill packages commit generated output for portability or because the
generated reference site is part of the package contract. Those files are not
agent instructions or references, but they currently contribute to
non-standard token warnings and the holistic structure ratio.

This PR adds --exclude-token-paths to validate structure and check.
Explicitly selected, skill-relative subtrees are excluded from non-standard
per-file token output, the Other files aggregate, and the holistic
standard-to-non-standard ratio. Non-excluded content remains counted, and the
option does not suppress structure, orphan, frontmatter, Markdown, or link
validation.

Paths are normalized across operating systems, matched on path boundaries, and
rejected when absolute or when they escape the skill root. A visible info result
is emitted whenever an exclusion is applied.

The main implementation lives in:

  • structure/paths.go — portable relative-path validation and subtree matching
  • structure/tokens.go — filtered non-standard file traversal and accounting
  • cmd/{validate_structure,check}.go — matching CLI flags and option propagation
  • structure/*_test.go, cmd/exitcode_integration_test.go, and
    testdata/token-exclusion-skill/ — aggregation, ratio, boundary, CLI, and
    output-format coverage

How to test

go test -race ./... -count=1
golangci-lint run

The integration tests cover both commands, Windows separators, invalid
absolute/traversal paths, similar-prefix siblings, retained non-excluded
content, and text, JSON, and Markdown output.

Checklist

  • Tests pass locally (go test -race ./... -count=1)
  • Lint passes locally (golangci-lint run)
  • New functionality includes tests
  • Breaking changes are noted above (if any)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant