04.02 — Cram tests (prysk)#

CLI tests in cram format: a .t file contains shell commands and their expected output. Tool: prysk (modern rewrite of cram).

.t file#

Valid args produce a deterministic parsed config

  $ touch driver
  $ "$PYTHON" "$TESTDIR/_demo_cli.py" --browser firefox --driver-path driver --workers 3 --wait-timeout 20 --logger terminal
  browser=firefox
  driver_path=driver
  headless=True
  workers=3
  wait_timeout=20
  logger=terminal
  only=()
  exclude=()

• First line: title (comment for the human).
• Blank line.
• Lines starting with $: shell command run.
• Following lines (indent 2): expected output.

Actual output differs → test fails.

_demo_cli.py#

# tests/cram/_demo_cli.py
"""Tiny CLI that pushes a SeleniumCliStoreSingleton and prints its content."""

from ocarina.opinionated.cli.selenium.cli_store_singleton import (
    SeleniumCliStoreSingleton as CliStoreSingleton,
)
from ocarina.opinionated.cli.selenium.create_cli_store import (
    create_selenium_auto_cli_store,
)

if __name__ == "__main__":
    CliStoreSingleton().push(create_selenium_auto_cli_store())
    for key in ("browser", "driver_path", "headless", "workers", "wait_timeout", "logger", "only", "exclude"):
        print(f"{key}={CliStoreSingleton().get(key)}")

→ Prints the store contents deterministically. Cram diffs the output line by line.

.t files#

cli_help.t#

CLI --help lists every declared flag

Pipe through grep to stay resilient to unrelated wording changes in argparse's
help output. Sorted unique output → alphabetical by first-differing character.

  $ "$PYTHON" "$TESTDIR/_demo_cli.py" --help 2>&1 | grep -o -- '--\(driver-path\|profile-path\|browser\|not-headless\|workers\|logger\|wait-timeout\|dont-force-delete-tmp-dirs\|only\|exclude\)' | sort -u
  --browser
  --dont-force-delete-tmp-dirs
  --driver-path
  --exclude
  --logger
  --not-headless
  --only
  --profile-path
  --wait-timeout
  --workers

We don’t test --help’s full output (it shifts with argparse versions), just that the listed flags appear at minimum. grep each flag, then sort -u. Missing flag → output differs, test fails.

We test what matters, nothing more.

The Playwright .t files#

The Playwright launcher has its own CLI (PlaywrightCliStoreSingleton), hence its own cram runner: _demo_pw_cli.py, which prints the Playwright store instead of the Selenium one.

# tests/cram/_demo_pw_cli.py
from ocarina.opinionated.cli.playwright.cli_store_singleton import (
    PlaywrightCliStoreSingleton as CliStoreSingleton,
)
from ocarina.opinionated.cli.playwright.create_cli_store import create_playwright_cli_store

CliStoreSingleton().push(create_playwright_cli_store())
store = CliStoreSingleton()
for key in ("browser", "profile_path", "headless", "workers", "wait_timeout",
            "logger", "video_dir", "trace_dir", "only", "exclude"):
    print(f"{key}={store.get(key)}")

Five dedicated .t files, mirroring the Selenium surface by dropping --driver-path (Playwright ships its own browsers directly) and adding --video-dir / --trace-dir:

Same philosophy as the Selenium side: grep the flags, sort -u, never test argparse’s full output. The CLI is a user surface, so cram is the natural tool, whatever the backend.

Makefile#

.PHONY: cram-test
cram-test:
ifeq ($(OS),Windows_NT)
	@echo "N A P O L E O N  D I S A P P R O V E S  W I N D O W S"
else
	@echo "Running cram tests..."
	PYTHON="$(CURDIR)/$(VENV_PYTHON)" "$(CURDIR)/$(VENV_BIN)/prysk" tests/cram/
endif

1. Skip on Windows. Prysk doesn’t have the same shell support on Windows; Napoleon told the author to skip cleanly instead of fighting it.
2. PYTHON injected: Prysk substitutes $PYTHON in .t files with the venv Python’s path.
3. prysk tests/cram/: runs every .t in the folder.

Why cram rather than pytest for the CLI#

The CLI is a user interface, not Python code. Cram is the natural tool — the most appropriate “system test” here.

make test#

.PHONY: test
test: cram-test
	@echo "Running tests..."
	-pytest --alluredir=$(ALLURE_RESULTS) -vv --hypothesis-show-statistics

→ make test chains:

1. cram-test (the .ts).
2. pytest with Allure output + --hypothesis-show-statistics.

If cram-test fails, pytest doesn’t run (Makefile target dependency). The - before pytest lets execution continue even if pytest fails (so we still get the Allure report).