Chapter 04 — Framework internal tests on Ocarina — manual

04.01 — From the outside like a user strategy

Wed, 20 May 2026 00:00:00 +0000

04.01 — "From the outside like a user" strategy#

The posture lives in the scenario tests’ conftest.py. No test cheats by peeking at internals.

`tests/scenarios/conftest.py`#

Component	Role
`FakeDriver` (dataclass)	Minimal driver: `title`, `disposed`. No Selenium method.
`make_built_driver()`	Factory `(FakeDriver, dispose)`, i.e. the signature `WebDriversPool` expects.
`make_pool(max_size=2)`	`WebDriversPool[FakeDriver]` ready to go.
`RecordingPOM(POMBase)`	POM that records its calls and can be configured to raise.
`acting(pom, step)`	Full `ActionSuccess[RecordingPOM]` (failure+success as no-ops).
`scenario_of("ok", "ok2")`	`TestScenario[FakeDriver]` with N steps.
`failing_scenario(step="boom", exc=...)`	1-step scenario that raises.
`make_test(name, scenario=..., test_id=..., skipped=False)`	`Test[FakeDriver]`
`make_suite(name, tests, pool=..., transient_errors=..., max_retries_per_test=...)`	`TestSuite[FakeDriver]`
`make_campaign(name, suites, max_workers=1)`	`TestCampaign[FakeDriver]`
`make_cycle(name="cycle", campaigns=..., smoke=..., mode=...)`	`TestCycle[FakeDriver]`
`run_chain(runner)`	Helper that executes a `ChainRunner` and returns the `ActionChain`.

`FakeDriver`#

@dataclass
class FakeDriver:
    title: str = "fake"
    disposed: bool = False

That’s all. Ocarina’s core calls no Selenium API.

04.02 — Cram tests (prysk)

Wed, 20 May 2026 00:00:00 +0000

04.02 — Cram tests (`prysk`)#

CLI tests in cram format: a .t file contains shell commands and their expected output. Tool: prysk (modern rewrite of cram).

`.t` file#

Valid args produce a deterministic parsed config

  $ touch driver
  $ "$PYTHON" "$TESTDIR/_demo_cli.py" --browser firefox --driver-path driver --workers 3 --wait-timeout 20 --logger terminal
  browser=firefox
  driver_path=driver
  headless=True
  workers=3
  wait_timeout=20
  logger=terminal
  only=()
  exclude=()

First line: title (comment for the human).
Blank line.
Lines starting with $: shell command run.
Following lines (indent 2): expected output.

Actual output differs → test fails.

04.03 — Scenario tests (pytest + allure)

Wed, 20 May 2026 00:00:00 +0000

04.03 — Scenario tests (pytest + allure)#

15 test_*.py files in tests/scenarios/ cover the DSL, orchestration and the Playwright actor. Dynamic tests.

Listing#

tests/scenarios/
├── conftest.py                              # FakeDriver, RecordingPOM, make_*
├── test_railway_and_action_chain.py         # create_act, drive_page, Result, ChainRunner
├── test_match_page.py                       # match_page / when (first/last branches, exceptions)
├── test_validate.py                         # validate, assert_that, otherwise, then, chain_validations
├── test_invariants_properties.py            # hypothesis (see 06-hypothesis-properties)
├── test_drivers_pool.py                     # WebDriversPool (acquire, warmup, shutdown, semaphore leaks)
├── test_screenshotter.py                    # Screenshotter (single shot, burst, healthcheck, threadsafety)
├── test_driver_builder.py                   # DriverBuilder (profile tmp dir, dispose)
├── test_watcher.py                          # Watcher (start, stop, callback errors swallowed, dedup cache)
├── test_playwright_driver_actor.py          # PlaywrightDriver actor (sync_playwright mocked, no browser)
├── test_playwright_adapter.py               # real-browser smoke (skipped if no Chromium installed)
├── test_test_suite.py                       # TestSuite (parallel, saturation, only/exclude, transient_errors)
├── test_test_cycle_and_bootstrap.py         # TestCycle, mode fail-fast vs wait-for-all, bootstrap
├── test_loggers_and_reports.py              # PrintLogger, FileLogger, pretty_print, JSON
├── test_docx_tests_proofs.py                # generate_docx_proof (parse logs, insert images)
└── test_env.py                              # smoke test: versions, Python 3.14+

Example internal test#

@allure.epic("Railway / action chain")
@allure.feature("Action chain")
@allure.tag("act", "handlers", "happy-path")
@allure.severity(allure.severity_level.CRITICAL)
@allure.label("layer", "unit")
@allure.title("A successful act reports success; the success handler fires once")
def test_success_act_fires_success_handler() -> None:
    pom = RecordingPOM()
    calls = {"success": 0, "failure": 0}

    chain = (
        create_act(pom, lambda p: p.step("click"))
        .failure(lambda _: calls.__setitem__("failure", calls["failure"] + 1))
        .success(lambda: calls.__setitem__("success", calls["success"] + 1))
        .execute()
    )

    assert chain.is_ok()
    assert pom.calls == ["click"]
    assert calls == {"success": 1, "failure": 0}

Assertion	Target
`chain.is_ok()`	Public API
`pom.calls == ["click"]`	Observable POM behavior (recording)
`calls == {"success": 1, "failure": 0}`	Right handler called

Allure annotations#

Category	Typical values
`epic`	“Railway / action chain”, “Invariants”, “Orchestration”, “Watcher”, “Drivers pool”, “Reports”, “Match page”
`feature`	“Action chain”, “Validate”, “TestSuite”, “TestCycle”, “Bootstrap”, “Pretty print”, “JSON”, “DOCX proofs”
`tag`	“act”, “handlers”, “happy-path”, “error-handling”, “retry”, “transient”, “smoke”, “skip”, “saturation”, “parallel”
`severity`	CRITICAL > NORMAL > MINOR > TRIVIAL
`label("layer", ...)`	“unit”, “integration”

`test_test_suite.py`#

Tests pass in sequential mode (max_workers=1).
Tests pass in parallel mode (max_workers=N).
--only / --exclude filtering.
--only + --exclude mutex.
Saturation: tests cloned with [COPY N].
Retries on transient errors.
Setup failure → SKIPPED when every attempt fails at setup.
A skipped=True test stops immediately.
Guardrails (unique names, unique IDs).
take_screenshot called on fail if autoscreen_on_fail.
Exact attempt counting on max_retries (one test covers the loop edge).

`test_match_page.py`#

Case	Assertion
First branch matches → executed	`chain.is_ok()`, later branches not evaluated
Second branch matches → executed	id.
No branch matches	`NoMatchingBranchError` in the `Fail`
Branch matches, but its `then` fails	`Fail` propagated
`condition` raises an ordinary `Exception`	Treated as `False`, later branches tried
`condition` raises an exception from the `raised_exceptions` tuple	Re-raise
Branch `then=[]` matches	`Ok(None)`
`match_page` in the middle of a `test_chain`	Short-circuits if failed

`test_drivers_pool.py`#

Uses a FakeDriverFactory that can be configured to raise / block / sleep, and counts its calls.

04.04 — Static typing tests (pytest-mypy-plugins)

Wed, 20 May 2026 00:00:00 +0000

04.04 — Static typing tests (`pytest-mypy-plugins`)#

Five *test_types.yml files that test the checker’s behavior. A testing dimension unique to Ocarina in its way: we make sure that what should be a mypy error actually is one.

Listing#

tests/
├── dsl/
│   ├── invariants/test_types.yml
│   └── testing_with_railway/test_types.yml
└── opinionated/
    ├── cli/test_types.yml
    ├── infra/test_types.yml
    └── plugins/reports/...   (snapshots — not YAML)

YAML#

- case: compatible_predicate_types
  description: Predicates should accept values of compatible types
  main: |
    from ocarina.dsl.invariants.validate import validate
    from ocarina.dsl.invariants.assertions import is_email, is_positive

    validate(1234).assert_that(is_positive)
    validate("a@a.com").assert_that(is_email)

Without out: → we expect mypy to pass without error on the given code.

04.05 — Snapshot testing (syrupy)

Wed, 20 May 2026 00:00:00 +0000

04.05 — Snapshot testing (`syrupy`)#

syrupy is a pytest plugin that serializes a test’s output into a .ambr file and compares to subsequent runs. Used for the output of pretty_print_results and results_to_json.

`.ambr`#

tests/opinionated/plugins/reports/__snapshots__/
├── test_pretty_print_results.ambr
└── test_results_to_json.ambr

Generated and managed by syrupy.

Example#

# tests/opinionated/plugins/reports/test_pretty_print_results.py
def test_pretty_print_results_renders_campaign_suite_test(snapshot, capsys):
    results = {
        "Dashboard": {
            "Login happy paths": {
                "Login - without OTP": (Ok(None), 5, "login_no_otp"),
                "Login - with OTP": (Fail(error=RuntimeError("OTP missed")), 8, "login_otp"),
                "Skipped one": (None, -1, "skipped_one"),
            },
        },
    }
    pretty_print_results(results, with_colors=False)
    out = capsys.readouterr().out
    assert out == snapshot

The snapshot (syrupy fixture):

04.06 — Property-based testing (hypothesis)

Wed, 20 May 2026 00:00:00 +0000

04.06 — Property-based testing (`hypothesis`)#

A single file (test_invariants_properties.py), but the pattern is interesting: we generate inputs and verify universal properties of the invariant predicates.

The concept#

hypothesis auto-generates test cases that satisfy given constraints, then checks a property holds for every generated case. On failure, hypothesis does shrinking to find the smallest counter-example.

from hypothesis import given, strategies as st

@given(st.integers())
def test_is_positive_passes_on_positive(value):
    if value >= 0:
        is_positive(value)             # must not raise
    else:
        with pytest.raises(InvariantViolationError):
            is_positive(value)         # must raise

→ hypothesis generates integers (typically 100 by default), checks the property on each. Unexpected exception → fail. The shrinker finds the smallest failing case (0? -1? MIN_INT?).

04.07 — Coverage policy: what's tested, what isn't, why

Wed, 20 May 2026 00:00:00 +0000

04.07 — Coverage policy: what’s tested, what isn’t, why#

Line coverage is explicitly scoped to the pure DSL and agnostic infra. The rest gets tested by other means (cram, types, snapshots, e2e) or stays out of scope (inert shapes).

`pyproject.toml#tool.coverage.run`#

[tool.coverage.run]
omit = [
    "*/__init__.py",
    "*/tests/*",
    "*/test_*.py",
    "src/ocarina/custom_errors/*",
    "src/ocarina/custom_types/*",
    "src/ocarina/ports/*",
    "src/ocarina/opinionated/loggers/custom_types/*",
    "src/**/*singleton.py",
    "src/**/consts/**",
    # Selenium adapter layer — exercised only against a real browser.
    "src/ocarina/infra/selenium/*",
    "src/ocarina/dsl/testing/selenium/*",
    "src/ocarina/opinionated/cli/selenium/*",
    "src/ocarina/pom/selenium/muted.py",
    # Playwright adapter layer — exercised only against a real browser.
    "src/ocarina/infra/playwright/*",
    "src/ocarina/dsl/testing/playwright/*",
    "src/ocarina/opinionated/cli/playwright/*",
    "src/ocarina/pom/playwright/*",
    "src/ocarina/opinionated/cli/phantoms.py",
    # Opinionated loggers — only file_logger.py stays in scope.
    "src/ocarina/opinionated/loggers/create_matching_logger.py",
    "src/ocarina/opinionated/loggers/muted_logger.py",
    "src/ocarina/opinionated/loggers/print_logger.py",
    "src/ocarina/opinionated/loggers/print_and_file_logger.py",
    "src/ocarina/opinionated/loggers/utils/*",
    # Traversed by cram tests
    "src/ocarina/opinionated/cli/store.py",
    "src/ocarina/opinionated/cli/builder.py",
]

Categories#

1. Obvious exclusions#

Pattern	Justification
`*/__init__.py`	Empty modules (Python 3 often doesn’t need them)
`/tests/`, `/test_.py`	Tests don’t test themselves

2. Types / errors / ports / shapes#

Path	Justification
`src/ocarina/custom_types/*`	Just `type X = ...` and frozen dataclasses. No runtime logic.
`src/ocarina/custom_errors/*`	Just Exception subclasses. No logic.
`src/ocarina/ports/*`	ABCs and Protocols. No behavior.
`src/ocarina/opinionated/loggers/custom_types/*`	id.
`src//consts/`	Constants (`LOGGERS_CHOICES = (...)`).
`src/*/singleton.py`	Trivial Singleton wrappers.

3. Selenium and Playwright layers (e2e only)#

Path	Justification
`src/ocarina/infra/selenium/*`	Real Selenium code (`Chrome()`, `Firefox()`). Only runs with a real browser.
`src/ocarina/dsl/testing/selenium/*`	`create_selenium_test`, `create_selenium_watcher` (trivial factories on `Test` / `Watcher`).
`src/ocarina/opinionated/cli/selenium/*`	`create_selenium_*_cli_store` (reads `platform.system`, instantiates).
`src/ocarina/pom/selenium/muted.py`	Utility `MutedPOM` (nearly empty).
`src/ocarina/infra/playwright/*`	Real Playwright code + the `PlaywrightDriver` actor. The end-to-end only proves out against a real browser.
`src/ocarina/dsl/testing/playwright/*`	`create_playwright_test`, `create_playwright_watcher` (same trivial factories).
`src/ocarina/opinionated/cli/playwright/*`	`create_playwright_cli_store` (Playwright CLI).
`src/ocarina/pom/playwright/*`	`MutedPlaywrightPOM` + `mixins` (Null Object, nearly empty).

These files are covered by:

04.08 — Allure + composite action allure-history + GH Pages deploy

Wed, 20 May 2026 00:00:00 +0000

04.08 — Allure + composite action `allure-history` + GH Pages deploy#

The Allure report gets generated on every CI, archived on a dedicated Git branch, then published on GitHub Pages. Live URL: https://mojo-molotov.github.io/ocarina/allure-report/.

Pipeline#

┌────────────────────────────────────────────────────────┐
│ main_ci.yml (push/PR main, or dispatch)                │
└──────────────────┬─────────────────────────────────────┘
                   ▼
   ┌───────────────────────────────────┐
   │ job : test                        │
   │  - matrix (ubuntu / windows × py) │
   │  - make test (cram + pytest)      │
   │  - upload-artifact allure-results │
   └───────────────┬───────────────────┘
                   ▼
   ┌───────────────────────────────────┐
   │ job : allure-history              │
   │  - download all allure-results    │
   │  - local composite action:        │
   │       allure-history/action.yml   │
   │  - upload-artifact allure-report  │
   └───────────────┬───────────────────┘
                   ▼
   ┌───────────────────────────────────┐
   │ job : deploy                      │
   │  - download allure-report         │
   │  - prepare pages-root/            │
   │  - actions/upload-pages-artifact  │
   │  - actions/deploy-pages           │
   └───────────────────────────────────┘

`allure-history`#

.github/actions/allure-history/action.yml: composite action local to the repo (uses: ./.github/actions/allure-history).

Chapter 04 — Framework internal tests on Ocarina — manual

04.01 — From the outside like a user strategy

04.01 — "From the outside like a user" strategy#

tests/scenarios/conftest.py#

FakeDriver#

04.02 — Cram tests (prysk)

04.02 — Cram tests (prysk)#

.t file#

04.03 — Scenario tests (pytest + allure)

04.03 — Scenario tests (pytest + allure)#

Listing#

Example internal test#

Allure annotations#

test_test_suite.py#

test_match_page.py#

test_drivers_pool.py#

04.04 — Static typing tests (pytest-mypy-plugins)

04.04 — Static typing tests (pytest-mypy-plugins)#

Listing#

YAML#

04.05 — Snapshot testing (syrupy)

04.05 — Snapshot testing (syrupy)#

.ambr#

Example#

04.06 — Property-based testing (hypothesis)

04.06 — Property-based testing (hypothesis)#

The concept#

04.07 — Coverage policy: what's tested, what isn't, why

04.07 — Coverage policy: what’s tested, what isn’t, why#

pyproject.toml#tool.coverage.run#

Categories#

1. Obvious exclusions#

2. Types / errors / ports / shapes#

3. Selenium and Playwright layers (e2e only)#

04.08 — Allure + composite action allure-history + GH Pages deploy

04.08 — Allure + composite action allure-history + GH Pages deploy#

Pipeline#

allure-history#

`tests/scenarios/conftest.py`#

`FakeDriver`#

04.02 — Cram tests (`prysk`)#

`.t` file#

`test_test_suite.py`#

`test_match_page.py`#

`test_drivers_pool.py`#

04.04 — Static typing tests (`pytest-mypy-plugins`)#

04.05 — Snapshot testing (`syrupy`)#

`.ambr`#

04.06 — Property-based testing (`hypothesis`)#

`pyproject.toml#tool.coverage.run`#

04.08 — Allure + composite action `allure-history` + GH Pages deploy#

`allure-history`#