Lint prompts in CI

prompt.check() is a pure advisory lint — it never renders, never mutates, and takes no vars. Wire it as a CI gate over every Prompt the application constructs.

What check() catches

The hard construction invariants (undeclared-variable references, template syntax errors, un-analyzable templates, reserved variant names) are enforced at construction — a constructed Prompt is already free of them. The only live finding check() can return is:

Finding kind	Meaning
`untrusted_without_guard`	A variable is declared `trusted: false` but the prompt carries no `guard` key in its `metadata` map.

This is an advisory — the library does not block rendering when this finding is present. It is the CI gate that chooses to fail on it.

Write it as a test, not a script

A CI gate is a test that fails the build. Place shipped prompts under version control, load each one, and assert check() returns no findings. When CI runs the normal test command (cargo test / pytest / node --test), a flagged prompt fails the run — no separate lint binary is required.

Each example below loads every *.yaml under a prompts/ directory and asserts a clean report, naming the offending prompt + finding on failure so the CI log is actionable.

//! Wiring `Prompt::check()` as a CI gate: a test that loads every `*.yaml` under a
//! `prompts/` directory, constructs each prompt, and asserts `check()` returns no
//! findings — failing the build (and naming the offender) otherwise.
//!
//! Standalone — `cargo run --example guides_lint-in-ci_prompts-lint-test`. To keep the
//! program self-contained it first materializes a `prompts/` directory of shipped
//! fixtures in a temp dir and `cd`s into it; a real repo checks its own `prompts/` in.

use prompting_press::Prompt;
use std::fs;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Materialize the `prompts/` directory a real repo would keep under version control.
    // A clean, shipped prompt: its untrusted-free variable needs no guard, so check() passes.
    let dir = std::env::temp_dir().join("pp_lint_in_ci_prompts");
    let prompts = dir.join("prompts");
    fs::create_dir_all(&prompts)?;
    fs::write(
        prompts.join("assistant.yaml"),
        r#"
name: assistant
role: system
body: "You are a support assistant for {{ company }}. Keep your replies under {{ max_words }} words."
variables:
  company:
    type: string
    trusted: true
  max_words:
    type: integer
    trusted: true
"#,
    )?;
    std::env::set_current_dir(&dir)?;

    // ── The CI gate itself: a `#[test]` in a real crate; here inlined into `main`. ──
    // tests/prompts_lint.rs  — runs under `cargo test`
    let mut failures = Vec::new();

    for entry in fs::read_dir("prompts").expect("prompts/ dir") {
        let path = entry.unwrap().path();
        if path.extension().and_then(|e| e.to_str()) != Some("yaml") {
            continue;
        }
        let text = fs::read_to_string(&path).unwrap();
        // Construction itself enforces the hard invariants — surface a load/agreement
        // failure as a test failure too, not a panic.
        let prompt = match Prompt::from_yaml(&text) {
            Ok(p) => p,
            Err(e) => {
                failures.push(format!("{}: construction failed: {e:?}", path.display()));
                continue;
            }
        };
        for f in &prompt.check().findings {
            failures.push(format!("{}: {} — {}", path.display(), f.prompt, f.detail));
        }
    }

    assert!(
        failures.is_empty(),
        "prompt lint findings:\n{}",
        failures.join("\n")
    );

    println!("prompt lint: clean — {} findings", failures.len());
    Ok(())
}

CheckReport::passed() returns true iff findings is empty (is_empty() is an alias); here the findings are collected directly so the assertion message lists every offender.

"""Wiring ``Prompt.check()`` as a CI gate under pytest.

A CI gate is a test that fails the build: load every ``*.yaml`` under a ``prompts/``
directory, construct each prompt, and assert ``check()`` returns no findings — naming
the offender otherwise. Standalone: this program first materializes a ``prompts/``
directory of shipped fixtures in a temp dir and ``chdir``s into it (a real repo keeps
its own ``prompts/`` under version control), then runs the documented parametrized test.

Run it directly (``python guides_lint-in-ci_prompts-lint-test.py``) or under pytest.
"""

from __future__ import annotations

import os
import tempfile
from pathlib import Path

import pytest
from prompting_press import Prompt, PromptingPressError

# ── Materialize the `prompts/` directory a real repo would keep under version control. ──
# A clean, shipped prompt: its untrusted-free variable needs no guard, so check() passes.
_TMP = Path(tempfile.mkdtemp(prefix="pp_lint_in_ci_"))
(_TMP / "prompts").mkdir()
(_TMP / "prompts" / "assistant.yaml").write_text(
    """
name: assistant
role: system
body: "You are a support assistant for {{ company }}. Keep your replies under {{ max_words }} words."
variables:
  company:
    type: string
    trusted: true
  max_words:
    type: integer
    trusted: true
"""
)
os.chdir(_TMP)

# ── The CI gate itself. ──
# tests/test_prompts_lint.py  — runs under pytest
PROMPT_FILES = sorted(Path("prompts").glob("*.yaml"))


@pytest.mark.parametrize("path", PROMPT_FILES, ids=lambda p: p.name)
def test_shipped_prompt_passes_check(path: Path) -> None:
    try:
        prompt = Prompt.from_yaml(
            path.read_text()
        )  # construction enforces hard invariants
    except PromptingPressError as e:
        pytest.fail(f"{path.name}: construction failed: {e}")

    report = prompt.check()
    findings = [f"{f.kind}: {f.detail}" for f in report.findings]
    assert report.passed(), f"{path.name} lint findings:\n" + "\n".join(findings)


if __name__ == "__main__":
    # Run the documented per-file check directly (no pytest runner needed).
    for _path in PROMPT_FILES:
        _prompt = Prompt.from_yaml(_path.read_text())
        _report = _prompt.check()
        assert _report.passed(), f"{_path.name} lint findings: {_report.findings}"
    print(f"prompt lint: clean — checked {len(PROMPT_FILES)} prompt(s)")

Parametrizing over the files gives one pass/fail per prompt in the pytest report. report.passed() → True when there are no findings (report.is_empty() is an alias).

// Wiring `prompt.check()` as a CI gate under `node --test`.
//
// A CI gate is a test that fails the build: load every `*.yaml` under a `prompts/`
// directory, construct each prompt, and assert `check()` returns no findings — naming
// the offender otherwise. Standalone: this program first materializes a `prompts/`
// directory of shipped fixtures in a temp dir and `chdir`s into it (a real repo keeps
// its own `prompts/` under version control), then registers the documented test cases.
//
// Run it: `node --test guides_lint-in-ci_prompts-lint-test.ts`.

import { test } from "node:test";
import assert from "node:assert/strict";
import { mkdirSync, mkdtempSync, readdirSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { Prompt } from "prompting-press";

// ── Materialize the `prompts/` directory a real repo would keep under version control. ──
// A clean, shipped prompt: its untrusted-free variable needs no guard, so check() passes.
const dir = mkdtempSync(join(tmpdir(), "pp-lint-in-ci-"));
mkdirSync(join(dir, "prompts"));
writeFileSync(
  join(dir, "prompts", "assistant.yaml"),
  `
name: assistant
role: system
body: "You are a support assistant for {{ company }}. Keep your replies under {{ max_words }} words."
variables:
  company:
    type: string
    trusted: true
  max_words:
    type: integer
    trusted: true
`,
);
process.chdir(dir);

// ── The CI gate itself. ──
// test/prompts-lint.test.mjs  — runs under `node --test`
for (const file of readdirSync("prompts").filter((f) => f.endsWith(".yaml"))) {
  test(`shipped prompt ${file} passes check`, () => {
    // Construction throws on the hard invariants; let that fail the test directly.
    const prompt = Prompt.fromYaml(readFileSync(`prompts/${file}`, "utf-8"));
    const report = prompt.check();
    const findings = report.findings.map((f) => `${f.kind}: ${f.detail}`);
    assert.ok(report.passed(), `${file} lint findings:\n${findings.join("\n")}`);
  });
}

One node:test case per prompt file; the assertion message lists the findings. These tests need no network or model — check() is pure analysis — so they run in the same fast unit-test lane as the rest of the test suite.

Suppressing a finding (the guard key)

To suppress untrusted_without_guard, add a guard key to the prompt’s metadata map. The library checks only for the presence of the key, not its shape — any value (including true) is sufficient:

name: ask
role: user
body: "Tell me about {{ topic }}."
variables:
  topic:
    type: string
    trusted: false
metadata:
  guard:
    enabled: true

With the guard key present, check() returns a passing report. At render time, pass guard=GuardConfig(enabled=True) (Python) / guard: { enabled: true } (TypeScript) / &GuardConfig { enabled: true, .. } (Rust) to receive the advisory guard text in RenderResult.guard. See The advisory guard.

docs current as of 0.2.0