Orchestration on Ocarina — manual

02.05.01 — Test[Driver]

Wed, 20 May 2026 00:00:00 +0000

02.05.01 — `Test[Driver]`#

Source file: src/ocarina/dsl/testing/oc_test.py

Signature#

@final
class Test[Driver]:
    def __init__(
        self,
        *,
        name: TestName,
        test_id: str | None = None,
        test_scenario: TestScenario[Driver],
        pre_test_scenarios_fragments: Sequence[TestScenarioFragment[Driver]] | None = None,
        post_test_scenarios_fragments: Sequence[TestScenarioFragment[Driver]] | None = None,
        skipped: bool = False,
    ) -> None:
        if test_id is None:
            test_id = name
        self.name = name
        self.test_id = test_id
        self._test_scenario = test_scenario
        self._pre_test_scenarios_fragments = pre_test_scenarios_fragments or []
        self._post_test_scenarios_fragments = post_test_scenarios_fragments or []
        self._skipped = skipped

Six parameters:

Parameter	Type	Role
`name`	`TestName` (`str`)	Human label; appears in the report, and becomes the log file name (hence subject to `is_valid_filename`).
`test_id`	`str \| None`	Stable identifier for `--only` / `--exclude`. If absent: takes `name`.
`test_scenario`	`TestScenario[Driver]` (alias = `Callable[[Driver, ILogger], Scenario[Driver]]`)	Factory that builds the `Scenario` at execution time.
`pre_test_scenarios_fragments`	`Sequence[TestScenarioFragment[Driver]] \| None`	`(driver, logger) -> TestChain` functions executed before the main scenario.
`post_test_scenarios_fragments`	`Sequence[TestScenarioFragment[Driver]] \| None`	`(driver, logger) -> TestChain` functions executed after the main scenario.
`skipped`	`bool`	If `True`, the test is registered but not executed.

1. `@final`#

No user inheritance. Want a “special” test? Compose via fragments or a scenario — don’t subclass.

02.05.02 — TestExecutor[Driver]

Wed, 20 May 2026 00:00:00 +0000

02.05.02 — `TestExecutor[Driver]`#

Source file: src/ocarina/dsl/testing/internals/test_executor.py

One job: run one attempt of a test with one driver. Knows nothing about retries, the driver pool, or suite-level aggregation.

`ExecutionOutcome`#

@final
@dataclass(frozen=True, slots=True)
class ExecutionOutcome:
    result: TestResult
    skipped: bool
    setup_failed: bool
    should_retry: bool
    steps_count: int

Field	Type	Meaning
`result`	`TestResult` = `Result[Any] \| None`	The chain’s result (Ok, Fail), or `None` if skip / setup_failed.
`skipped`	`bool`	True if `test_runner.skipped` (i.e. `Test(skipped=True)`).
`setup_failed`	`bool`	True if the scenario’s `setup()` raised.
`should_retry`	`bool`	True if the retry rule applies (transient_error detected).
`steps_count`	`int`	Number of `act` calls; `-1` if skip or setup_failed.

slots=True: blocks dynamic attribute addition, saves memory. This object rides the hot path — slots earns its keep.
frozen=True: immutable, safe to share across threads.
@final: no inheritance.

Execution order for one attempt#

┌──────────────────────────────────────────────────────────────────────┐
│  test_runner = test.spawn(driver, logger_with_taxonomy)              │
└──────────────────────────────┬───────────────────────────────────────┘
                               │
                               ▼
              ┌─────────────────────────────────┐
              │  test_runner.skipped ?          │── True ─► return Outcome(skipped=True, ...)
              └────────────────┬────────────────┘
                               │ False
                               ▼
              ┌─────────────────────────────────┐
              │  logger.test_name(test.name)    │   (annotation)
              └────────────────┬────────────────┘
                               │
                               ▼
              ┌─────────────────────────────────┐
              │  setup() (if not None)          │── raises ─► teardown() (if not None)
              └────────────────┬────────────────┘            ↓
                               │                  return Outcome(setup_failed=True, should_retry=True, ...)
                               ▼
              ┌─────────────────────────────────┐
              │  watchers.start(driver, logger, │
              │                 take_screenshot)│   (1 daemon thread per watcher)
              └────────────────┬────────────────┘
                               │
                               ▼
              ┌─────────────────────────────────┐
              │  _run_chain(chain_runners, ...) │── returns (result, should_retry)
              └────────────────┬────────────────┘
                               │
                               ▼
              ┌─────────────────────────────────┐
              │  watchers.stop()                │   (always)
              └────────────────┬────────────────┘
                               │
                               ▼
              ┌─────────────────────────────────┐
              │  teardown() (if not None)       │   (ALWAYS, even if chain failed)
              └────────────────┬────────────────┘     (exceptions are logged & swallowed)
                               │
                               ▼
              ┌─────────────────────────────────┐
              │  steps_count = act_counter.get()│
              │  return Outcome(...)            │
              └─────────────────────────────────┘

`execute`#

def execute(
    self, test: Test[Driver], *,
    driver: Driver,
    taxonomy: tuple[str, ...],
    logger_with_taxonomy: ILogger,
    logger_without_taxonomy: ILogger,
    attempt: int, max_attempts: int,
) -> ExecutionOutcome:
    test_runner = test.spawn(driver, logger_with_taxonomy)

    if test_runner.skipped:
        return ExecutionOutcome(result=None, skipped=True, setup_failed=False,
                                should_retry=False, steps_count=-1)

    logger_without_taxonomy.test_name(test.name)

    if test_runner.setup is not None:
        try:
            test_runner.setup()
        except Exception as exc:
            msg = f"{test.name} -- Setup failed (attempt {attempt}/{max_attempts}): {exc}"
            logger_with_taxonomy.warning(msg)
            if test_runner.teardown is not None:
                self._run_teardown(test_runner.teardown, test_name=test.name, logger=logger_with_taxonomy)
            return ExecutionOutcome(result=None, skipped=False, setup_failed=True,
                                    should_retry=True, steps_count=-1)

    watchers: Sequence[Watcher[Driver]] = test_runner.watchers or []
    self._start_watchers(watchers, driver=driver, test_name=test.name, taxonomy=taxonomy)

    result, should_retry = self._run_chain(
        test_runner.chain_runners, test_name=test.name, attempt=attempt,
        driver=driver, logger=logger_without_taxonomy,
        logger_with_taxonomy=logger_with_taxonomy, max_attempts=max_attempts,
    )

    self._stop_watchers(watchers)

    if test_runner.teardown is not None:
        self._run_teardown(test_runner.teardown, test_name=test.name, logger=logger_with_taxonomy)

    steps_count = self._act_counter.get()
    return ExecutionOutcome(result=result, skipped=False, setup_failed=False,
                            should_retry=should_retry, steps_count=steps_count)

Two loggers: why#

logger_with_taxonomy vs logger_without_taxonomy:

02.05.03 — TestFlow[Driver] — retry policy

Wed, 20 May 2026 00:00:00 +0000

02.05.03 — `TestFlow[Driver]` — retry policy#

Source file: src/ocarina/dsl/testing/internals/test_flow.py

One job: for a single Test, run the retry loop, acquire a fresh driver per attempt, apply the backoff policy.

Why an intermediate level between `TestSuite` and `TestExecutor`#

Level	Doesn’t know about
`TestExecutor`	Retries, the driver pool
`TestFlow`	Inter-test concurrency, suite-level aggregation
`TestSuite`	One attempt’s execution, retries

A genuinely strict separation. TestFlow is the only level that knows both the pool and the retry policy — combining them is its whole job.

02.05.04 — TestSuite[Driver] — parallelized engine

Wed, 20 May 2026 00:00:00 +0000

02.05.04 — `TestSuite[Driver]` — parallelized engine#

Source file: src/ocarina/dsl/testing/oc_test_suite.py

One job: run a sequence of Tests in parallel, handle saturation, ID filtering, and pre-execution invariant validation.

Constructor#

def __init__(
    self,
    *,
    name: str,
    tests: Sequence[Test[Driver]],
    create_logger: Thunk[ILogger],
    drivers_pool: WebDriversPool[Driver],
    take_screenshot: ITakeScreenshot[Driver],
    act_counter: ActCounter | None = None,
    transient_errors: tuple[type[Exception], ...] = (),
    copy_indicator: str = "COPY",
    put_space_after_copy_indicator: bool = True,
    max_retries_per_test: int | None = None,
    autoscreen_on_fail: bool = False,
    saturate_workers: bool | None = None,
    only_ids: Iterable[str] = (),
    exclude_ids: Iterable[str] = (),
) -> None:
    self._guards_on_invoke(tests)
    # ... assignments ...
    self._tests = filter_tests_by_ids(
        tests,
        only=only_ids,
        exclude=exclude_ids,
        logger=create_logger().set_prefix(lambda: f"{self.name}: filtering tests..."),
    )

Parameter	Type	Role	Default
`name`	`str`	Suite name (appears in logs & reports)	—
`tests`	`Sequence[Test[Driver]]`	List of tests to run	—
`create_logger`	`Thunk[ILogger]`	Logger factory	—
`drivers_pool`	`WebDriversPool[Driver]`	Shared pool	—
`take_screenshot`	`ITakeScreenshot[Driver]`	Callable `(driver, logger, category) -> None`	—
`act_counter`	`ActCounter \| None`	If `None` → `ThreadsBasedActCounter()`	`None`
`transient_errors`	`tuple[type[Exception], ...]`	Exceptions that trigger a retry	`()`
`copy_indicator`	`str`	Prefix for saturation-cloned tests	`"COPY"`
`put_space_after_copy_indicator`	`bool`	`[COPY 1]` (`True`) vs `[COPY1]` (`False`)	`True`
`max_retries_per_test`	`int \| None`	Max retries for a potentially flaky test	`None` → `_DEFAULT_MAX_RETRIES_PER_TEST` (8)
`autoscreen_on_fail`	`bool`	Automatically capture one or several screenshots (burst) on failure	`False`
`saturate_workers`	`bool \| None`	Clone tests to saturate every worker if the pool is large enough	`None` → cascade: suite > campaign > bootstrap
`only_ids`	`Iterable[str]`	`--only` filter	`()`
`exclude_ids`	`Iterable[str]`	`--exclude` filter	`()`

Guardrails#

At `init` time#

def _guards_on_invoke(self, tests: Sequence[Test[Driver]]) -> None:
    validate_test_runners_ids(tests=tests, name="tests").execute().raise_if_invalid()

→ Every test_id must be unique. Otherwise: AggregateInvariantViolationError. Raised at construction, before any execution attempt. Duplicate a test_id by accident and you find out instantly.

02.05.05 — Worker saturation

Wed, 20 May 2026 00:00:00 +0000

02.05.05 — Worker saturation#

Ocarina-specific trick: if a suite has fewer tests than workers, tests get randomly cloned until the worker count is filled. Copies are renamed [COPY 1] <name>, [COPY 2] <name>, etc.

Why#

The Holy Book formalizes the rationale in the “First real-world hurdles” chapter:

Its saturate_workers option lets you force random cloning of tests inside a suite.

As soon as there are more available workers in the DriversPool than there are tests to run in a suite, Ocarina will randomly clone tests, start every driver, and assign each one a test to perform.

02.05.06 — TestCampaign[Driver]

Wed, 20 May 2026 00:00:00 +0000

02.05.06 — `TestCampaign[Driver]`#

Source file: src/ocarina/dsl/testing/oc_test_campaign.py

One job: run a sequence of suites in order, share a workers config, and declare campaign_has_failed.

Code#

class TestCampaign[Driver]:
    def __init__(
        self,
        *,
        name: str,
        suites: Sequence[TestSuite[Driver]],
        max_workers: int,
        saturate_workers: bool | None = None,
    ) -> None:
        validate_test_suites_names(suites=suites, name="suites").execute().raise_if_invalid()

        self.name = name
        self._suites = suites
        self._results: TestCampaignResults = {}
        self._max_workers = max_workers
        self._saturate_workers = saturate_workers

        for suite in self._suites:
            suite._campaign_name = self.name

    def run_all(
        self, *, skip_all: bool = False, saturate_workers: bool = True
    ) -> TestCampaignResults:
        self._results.clear()

        resolved_saturate_workers = (
            self._saturate_workers
            if self._saturate_workers is not None
            else saturate_workers
        )

        if skip_all:
            for suite in self._suites:
                self._results[suite.name] = {
                    name: (None, -1, test_id)
                    for name, test_id in suite.test_names_and_ids
                }
            return self._results

        for suite in self._suites:
            self._results[suite.name] = suite.run(
                max_workers=self._max_workers,
                saturate_workers=resolved_saturate_workers,
            )

        return self._results


def campaign_has_failed(results: TestCampaignResults) -> bool:
    return any(
        is_test_result_fail(outcome)
        for campaign_results in results.values()
        for outcome, _, _ in campaign_results.values()
    )

Anatomy#

Constructor guard#

validate_test_suites_names(suites=suites, name="suites").execute().raise_if_invalid()

→ Suite names inside a campaign must be unique — case-insensitively since 1.1.10 (NFC + casefold). Raised at construction.

02.05.07 — TestCycle[Driver] + modes

Wed, 20 May 2026 00:00:00 +0000

02.05.07 — `TestCycle[Driver]` + modes#

Source file: src/ocarina/dsl/testing/oc_test_cycle.py

One job: orchestrate smoke then main campaigns, and apply a smoke-failure-handling mode.

Two modes#

type Mode = Literal[
    "fail-fast-on-first-smoke-campaigns-sequence-fail",
    "wait-for-all-smoke-tests",
]

Mode	Behavior
`fail-fast-on-first-smoke-campaigns-sequence-fail` (default)	First smoke campaign that fails → following ones get skipped.
`wait-for-all-smoke-tests`	Every smoke campaign runs. If any fails, the main campaigns get skipped.

Case	Mode
Dependency-style smoke (e.g. “login” then “dashboard accessible”)	`fail-fast` (login dies → dashboard pointless)
Parallel smoke (e.g. “homepage” + “API ping”)	`wait-for-all` (we want to see both)

fail-fast is the default — it’s the common case.

02.05.08 — filter_tests_by_ids

Wed, 20 May 2026 00:00:00 +0000

02.05.08 — `filter_tests_by_ids`#

Source file: src/ocarina/dsl/testing/filter_tests_by_ids.py

Called by TestSuite.__init__ to apply the CLI flags --only <ids> and --exclude <ids>. Mutually exclusive — can’t pass both.

Code#

def filter_tests_by_ids[Driver](
    tests: Sequence[Test[Driver]],
    *,
    only: Iterable[str] = (),
    exclude: Iterable[str] = (),
    logger: ILogger,
) -> Sequence[Test[Driver]]:
    only_set = set(only)
    exclude_set = set(exclude)

    if only_set and exclude_set:
        raise ValueError("--only and --exclude cannot be used together")

    known_ids = {test.test_id for test in tests}

    if only_set:
        matched = only_set & known_ids
        if matched:
            joined = ", ".join(sorted(matched))
            logger.info(f"--only: matched test IDs: {joined}")
        return [test for test in tests if test.test_id in only_set]

    if exclude_set:
        matched = exclude_set & known_ids
        if matched:
            joined = ", ".join(sorted(matched))
            logger.info(f"--exclude: matched test IDs: {joined}")
        return [test for test in tests if test.test_id not in exclude_set]

    return tests

Four guarantees#

1. Mutex `--only` / `--exclude`#

if only_set and exclude_set:
    raise ValueError("--only and --exclude cannot be used together")

Passing both is a runtime error (not a mypy one). Semantically nonsense: “keep only A, B” AND “exclude C, D”? At the CLI level, doesn’t make sense. The mutex kills ambiguity.

Orchestration on Ocarina — manual

02.05.01 — Test[Driver]

02.05.01 — Test[Driver]#

Signature#

1. @final#

02.05.02 — TestExecutor[Driver]

02.05.02 — TestExecutor[Driver]#

ExecutionOutcome#

Execution order for one attempt#

execute#

Two loggers: why#

02.05.03 — TestFlow[Driver] — retry policy

02.05.03 — TestFlow[Driver] — retry policy#

Why an intermediate level between TestSuite and TestExecutor#

02.05.04 — TestSuite[Driver] — parallelized engine

02.05.04 — TestSuite[Driver] — parallelized engine#

Constructor#

Guardrails#

At __init__ time#

02.05.05 — Worker saturation

02.05.05 — Worker saturation#

Why#

02.05.06 — TestCampaign[Driver]

02.05.06 — TestCampaign[Driver]#

Code#

Anatomy#

Constructor guard#

02.05.07 — TestCycle[Driver] + modes

02.05.07 — TestCycle[Driver] + modes#

Two modes#

02.05.08 — filter_tests_by_ids

02.05.08 — filter_tests_by_ids#

Code#

Four guarantees#

1. Mutex --only / --exclude#

02.05.01 — `Test[Driver]`#

1. `@final`#

02.05.02 — `TestExecutor[Driver]`#

`ExecutionOutcome`#

`execute`#

02.05.03 — `TestFlow[Driver]` — retry policy#

Why an intermediate level between `TestSuite` and `TestExecutor`#

02.05.04 — `TestSuite[Driver]` — parallelized engine#

At `init` time#

02.05.06 — `TestCampaign[Driver]`#

02.05.07 — `TestCycle[Driver]` + modes#

02.05.08 — `filter_tests_by_ids`#

1. Mutex `--only` / `--exclude`#