agentskill-evals-required

Require evals/evals.json for each skill (opt-in)

Severity	warning (disabled)
Autofix	-
Since	v0.1.0
Repo Types	agentskills, dot-claude, marketplace, single-plugin
Category	agentskills.io

Why

Skills without evals have no automated way to verify they still work after changes. This opt-in rule enforces that every skill directory includes an evals/evals.json file, ensuring eval coverage is a gating requirement.

Examples

Bad:

my-skill/
  SKILL.md

Good:

my-skill/
  SKILL.md
  evals/
    evals.json

How to fix

Create an evals/evals.json file inside the skill directory with at least one test case covering the skill's primary use case. This rule is disabled by default — enable it in your config when you want to enforce eval coverage:

rules:
  agentskill-evals-required:
    enabled: true

Configuration

rules:
  agentskill-evals-required:
    enabled: false  # true | false | auto
    severity: warning

Run skillsaw explain agentskill-evals-required to see this documentation and the rule's effective configuration in your terminal.