Parallelization on Ocarina — manual

02.05.04 — TestSuite[Driver] — parallelized engine

Wed, 20 May 2026 00:00:00 +0000

02.05.04 — `TestSuite[Driver]` — parallelized engine#

Source file: src/ocarina/dsl/testing/oc_test_suite.py

One job: run a sequence of Tests in parallel, handle saturation, ID filtering, and pre-execution invariant validation.

Constructor#

def __init__(
    self,
    *,
    name: str,
    tests: Sequence[Test[Driver]],
    create_logger: Thunk[ILogger],
    drivers_pool: WebDriversPool[Driver],
    take_screenshot: ITakeScreenshot[Driver],
    act_counter: ActCounter | None = None,
    transient_errors: tuple[type[Exception], ...] = (),
    copy_indicator: str = "COPY",
    put_space_after_copy_indicator: bool = True,
    max_retries_per_test: int | None = None,
    autoscreen_on_fail: bool = False,
    saturate_workers: bool | None = None,
    only_ids: Iterable[str] = (),
    exclude_ids: Iterable[str] = (),
) -> None:
    self._guards_on_invoke(tests)
    # ... assignments ...
    self._tests = filter_tests_by_ids(
        tests,
        only=only_ids,
        exclude=exclude_ids,
        logger=create_logger().set_prefix(lambda: f"{self.name}: filtering tests..."),
    )

Parameter	Type	Role	Default
`name`	`str`	Suite name (appears in logs & reports)	—
`tests`	`Sequence[Test[Driver]]`	List of tests to run	—
`create_logger`	`Thunk[ILogger]`	Logger factory	—
`drivers_pool`	`WebDriversPool[Driver]`	Shared pool	—
`take_screenshot`	`ITakeScreenshot[Driver]`	Callable `(driver, logger, category) -> None`	—
`act_counter`	`ActCounter \| None`	If `None` → `ThreadsBasedActCounter()`	`None`
`transient_errors`	`tuple[type[Exception], ...]`	Exceptions that trigger a retry	`()`
`copy_indicator`	`str`	Prefix for saturation-cloned tests	`"COPY"`
`put_space_after_copy_indicator`	`bool`	`[COPY 1]` (`True`) vs `[COPY1]` (`False`)	`True`
`max_retries_per_test`	`int \| None`	Max retries for a potentially flaky test	`None` → `_DEFAULT_MAX_RETRIES_PER_TEST` (8)
`autoscreen_on_fail`	`bool`	Automatically capture one or several screenshots (burst) on failure	`False`
`saturate_workers`	`bool \| None`	Clone tests to saturate every worker if the pool is large enough	`None` → cascade: suite > campaign > bootstrap
`only_ids`	`Iterable[str]`	`--only` filter	`()`
`exclude_ids`	`Iterable[str]`	`--exclude` filter	`()`

Guardrails#

At `init` time#

def _guards_on_invoke(self, tests: Sequence[Test[Driver]]) -> None:
    validate_test_runners_ids(tests=tests, name="tests").execute().raise_if_invalid()

→ Every test_id must be unique. Otherwise: AggregateInvariantViolationError. Raised at construction, before any execution attempt. Duplicate a test_id by accident and you find out instantly.

02.05.05 — Worker saturation

Wed, 20 May 2026 00:00:00 +0000

02.05.05 — Worker saturation#

Ocarina-specific trick: if a suite has fewer tests than workers, tests get randomly cloned until the worker count is filled. Copies are renamed [COPY 1] <name>, [COPY 2] <name>, etc.

Why#

The Holy Book formalizes the rationale in the “First real-world hurdles” chapter:

Its saturate_workers option lets you force random cloning of tests inside a suite.

As soon as there are more available workers in the DriversPool than there are tests to run in a suite, Ocarina will randomly clone tests, start every driver, and assign each one a test to perform.

Chapter 02.05 — Orchestration

Wed, 20 May 2026 00:00:00 +0000

Chapter 02.05 — Orchestration#

Test → TestSuite → TestCampaign → TestCycle: how each level slots into the next, who owns parallelization, who owns retries, who decides skipping, and where the pre-execution invariants live.

Plan#

#	File	Subject
01	`01-test.md`	The `Test[Driver]` class — `spawn`, pre/post fragments, skip.
02	`02-test-executor.md`	`TestExecutor` — execution of one attempt, order: setup → watchers.start → chain → watchers.stop → teardown.
03	`03-test-flow-retries.md`	`TestFlow` — retry loop (1 + max_retries), linear backoff, setup-failure handling.
04	`04-test-suite.md`	`TestSuite` — parallelization with `ThreadPoolExecutor`, ID filtering, guardrails.
05	`05-saturation.md`	Worker saturation: random cloning `[COPY N]`. Why and how.
06	`06-test-campaign.md`	`TestCampaign` — sequence of suites, `campaign_has_failed`.
07	`07-test-cycle-modes.md`	`TestCycle` — smoke + main, `fail-fast` vs `wait-for-all` modes, `has_test_cycle_failed`.
08	`08-filter-tests-by-ids.md`	`filter_tests_by_ids` — `--only`/`--exclude`, mutex, ignores unknown IDs.

Diagram#

            ┌────────────────────────────────────────────────┐
            │                  TestCycle                     │
            │                                                │
            │  ┌──────────────────────────────────────────┐  │
            │  │ smoke_tests_campaigns                    │  │ ◄── first, gate
            │  │  └─ TestCampaign                         │  │     mode: fail-fast |
            │  │      └─ TestSuite                        │  │           wait-for-all
            │  │          └─ Test                         │  │
            │  └──────────────────────────────────────────┘  │
            │                                                │
            │  ┌──────────────────────────────────────────┐  │
            │  │ campaigns (main)                         │  │ ◄── skipped if a
            │  │  └─ TestCampaign                         │  │     smoke fails
            │  │      └─ TestSuite                        │  │
            │  │          └─ Test                         │  │
            │  └──────────────────────────────────────────┘  │
            └────────────────────────────────────────────────┘

Who does what?#

Level	Single responsibility	Concurrency	Out of scope
`Test`	Metadata + `spawn(driver, logger)`	none	execution
`TestExecutor`	One attempt	none	retries, driver acquisition
`TestFlow`	Retry loop	none	parallelization, aggregation
`TestSuite`	Parallelization + saturation + ID filtering	N threads	inter-suite sequence
`TestCampaign`	Sequence of suites	suite-level	smoke vs main
`TestCycle`	Smoke + main + mode	campaign-level	bootstrap / post-exec plugins

The strict separation is deliberate. TestExecutor knows nothing about retries. TestFlow knows nothing about concurrency. TestSuite knows nothing about campaign-level aggregation. One axis of responsibility per class.

02.05.06 — TestCampaign[Driver]

Wed, 20 May 2026 00:00:00 +0000

02.05.06 — `TestCampaign[Driver]`#

Source file: src/ocarina/dsl/testing/oc_test_campaign.py

One job: run a sequence of suites in order, share a workers config, and declare campaign_has_failed.

Code#

class TestCampaign[Driver]:
    def __init__(
        self,
        *,
        name: str,
        suites: Sequence[TestSuite[Driver]],
        max_workers: int,
        saturate_workers: bool | None = None,
    ) -> None:
        validate_test_suites_names(suites=suites, name="suites").execute().raise_if_invalid()

        self.name = name
        self._suites = suites
        self._results: TestCampaignResults = {}
        self._max_workers = max_workers
        self._saturate_workers = saturate_workers

        for suite in self._suites:
            suite._campaign_name = self.name

    def run_all(
        self, *, skip_all: bool = False, saturate_workers: bool = True
    ) -> TestCampaignResults:
        self._results.clear()

        resolved_saturate_workers = (
            self._saturate_workers
            if self._saturate_workers is not None
            else saturate_workers
        )

        if skip_all:
            for suite in self._suites:
                self._results[suite.name] = {
                    name: (None, -1, test_id)
                    for name, test_id in suite.test_names_and_ids
                }
            return self._results

        for suite in self._suites:
            self._results[suite.name] = suite.run(
                max_workers=self._max_workers,
                saturate_workers=resolved_saturate_workers,
            )

        return self._results


def campaign_has_failed(results: TestCampaignResults) -> bool:
    return any(
        is_test_result_fail(outcome)
        for campaign_results in results.values()
        for outcome, _, _ in campaign_results.values()
    )

Anatomy#

Constructor guard#

validate_test_suites_names(suites=suites, name="suites").execute().raise_if_invalid()

→ Suite names inside a campaign must be unique — case-insensitively since 1.1.10 (NFC + casefold). Raised at construction.

Parallelization on Ocarina — manual

02.05.04 — TestSuite[Driver] — parallelized engine

02.05.04 — TestSuite[Driver] — parallelized engine#

Constructor#

Guardrails#

At __init__ time#

02.05.05 — Worker saturation

02.05.05 — Worker saturation#

Why#

Chapter 02.05 — Orchestration

Chapter 02.05 — Orchestration#

Plan#

Diagram#

Who does what?#

02.05.06 — TestCampaign[Driver]

02.05.06 — TestCampaign[Driver]#

Code#

Anatomy#

Constructor guard#

02.05.04 — `TestSuite[Driver]` — parallelized engine#

At `init` time#

02.05.06 — `TestCampaign[Driver]`#