promptfoo-valid

Validate promptfoo eval YAML config structure and file references

Severity	error (auto)
Autofix	-
Since	v0.1.0
Repo Types	promptfoo
Category	Promptfoo Evals

Why

Promptfoo eval YAML configs define test suites for skills and plugins. Invalid YAML, missing required fields, or broken file references mean evals cannot run — regressions in skill behavior go undetected.

Examples

Bad:

prompts:
  - file://nonexistent.txt

Good:

prompts:
  - file://../../SKILL.md
tests:
  - vars:
      input: "deploy to staging"
    assert:
      - type: contains
        value: "deployed"

How to fix

Fix the YAML syntax or structural issue identified in the violation. Ensure prompts and tests arrays exist, file references point to real files, and the overall structure matches the promptfoo config schema.

Configuration

rules:
  promptfoo-valid:
    enabled: auto  # true | false | auto
    severity: error

Run skillsaw explain promptfoo-valid to see this documentation and the rule's effective configuration in your terminal.