promptfoo-assertions

Require specific assertion types in all promptfoo eval tests

Severity	warning (disabled)
Autofix	-
Since	v0.1.0
Repo Types	promptfoo
Category	Promptfoo Evals

Why

Eval tests without assertions pass unconditionally — they verify that the skill runs without crashing but say nothing about whether the output is correct. This opt-in rule enforces that every test case includes specific assertion types you configure.

Examples

Bad:

tests:
  - vars:
      input: "deploy to staging"

Good:

tests:
  - vars:
      input: "deploy to staging"
    assert:
      - type: contains
        value: "staging"
      - type: cost
        threshold: 0.05

How to fix

Add assertion objects to each test case. Configure which assertion types are required:

rules:
  promptfoo-assertions:
    enabled: true
    required-types:
      - contains
      - cost

Configuration

rules:
  promptfoo-assertions:
    enabled: false  # true | false | auto
    severity: warning

Parameter	Description	Default
required-types	Assertion types that every test must include (via test-level or defaultTest assertions)	[]
threshold-constraints	Per-assertion-type threshold bounds, e.g. {cost: {max: 2.0}, latency: {max: 30000}}	{}

Run skillsaw explain promptfoo-assertions to see this documentation and the rule's effective configuration in your terminal.