promptfoo-metadata

Require specific metadata keys on all promptfoo eval tests

Severity	warning (disabled)
Autofix	-
Since	v0.1.0
Repo Types	promptfoo
Category	Promptfoo Evals

Why

Metadata keys on eval tests (like category, priority, or owner) enable filtering and reporting across a large test suite. This opt-in rule enforces that every test case includes the metadata keys you configure.

Examples

Bad:

tests:
  - vars:
      input: "deploy"
    assert:
      - type: contains
        value: "deployed"

Good:

tests:
  - vars:
      input: "deploy"
    metadata:
      category: deployment
      priority: high
    assert:
      - type: contains
        value: "deployed"

How to fix

Add the required metadata keys to each test case. Configure which keys are required:

rules:
  promptfoo-metadata:
    enabled: true
    required-keys:
      - category

Configuration

rules:
  promptfoo-metadata:
    enabled: false  # true | false | auto
    severity: warning

Parameter	Description	Default
required-keys	Metadata keys required on every test case	[]

Run skillsaw explain promptfoo-metadata to see this documentation and the rule's effective configuration in your terminal.