02.10.04 — ActCounter + ThreadsBasedActCounter

02.10.04 — ActCounter + ThreadsBasedActCounter#

Thread-local counter of act calls per attempt. Lets the report say “this test made 17 steps before dying at step 18.”

Interface#

# src/ocarina/infra/act_counter.py
class ActCounter:
    def get(self) -> int: ...
    def reset(self) -> None: ...
    def incr_act_call_count(self) -> None: ...

Default implementation#

# src/ocarina/opinionated/infra/act_counter.py
from threading import local

_thread_local = local()
_COUNTER_KEY: Final[str] = "ocarina_counter"


class ActCounter(_ActCounter):
    def get(self) -> int:
        return getattr(_thread_local, _COUNTER_KEY, 0)

    def reset(self) -> None:
        setattr(_thread_local, _COUNTER_KEY, 0)

    def incr_act_call_count(self) -> None:
        if not hasattr(_thread_local, _COUNTER_KEY):
            self.reset()
        setattr(_thread_local, _COUNTER_KEY, getattr(_thread_local, _COUNTER_KEY) + 1)
  • threading.local(): namespace where attributes are per thread. Each thread has its own value.
  • _COUNTER_KEY = "ocarina_counter": attribute key.
  • No lock: each thread only writes its own value, so no race.

Why thread-local rather than a shared counter + lock?#

Ocarina’s architecture says one thread = one test. We’ve got no appetite for a whole state monad — might as well park this “impurity” (still a piece of state) right here. That’s it.

02.11.04 — Loggers: PrintLogger, FileLogger, PrintAndFileLogger, MutedLogger

02.11.04 — Loggers: PrintLogger, FileLogger, PrintAndFileLogger, MutedLogger#

Source folder: src/ocarina/opinionated/loggers/

Four canonical ILogger implementations + a create_matching_logger factory. All opt-in.

create_matching_logger#

LOGGERS_CHOICES = ("terminal", "file", "terminal+file", "muted")

def create_matching_logger(mode: SupportedLogger) -> ILogger:
    if mode == "terminal":
        return PrintLogger()
    if mode == "file":
        return FileLogger(base_dir=get_default_log_dir())
    if mode == "terminal+file":
        return PrintAndFileLogger(base_dir=get_default_log_dir())
    if mode == "muted":
        return MutedLogger()
    raise ValueError(f"Unsupported logger mode: {mode}")

And:

def get_default_log_dir() -> Path:
    return Path.cwd() / ".ocarina_logs"

→ By default, logs land in .ocarina_logs/ at the user project root. Gitignored by convention.

PrintLogger#

class PrintLogger(ILogger):
    def __init__(self) -> None:
        self._prefix_thunk: Thunk[str] = lambda: ""
        self._domain_taxonomy: tuple[str, ...] = ("",)

    def set_prefix(self, prefix_thunk: Thunk[str]) -> Self:
        self._prefix_thunk = prefix_thunk
        return self

    def set_domain_taxonomy(self, taxonomy: tuple[str, ...]) -> Self:
        self._domain_taxonomy = taxonomy
        return self

    def raw(self, *args, stream=None, **kwargs) -> None:
        if stream is None:
            stream = sys.stdout
        print(*args, file=stream, **kwargs)

    def info(self, msg, *args, exc=None, **kwargs):
        self._print_log("[INFO]", msg, *args, exc=exc, stream=sys.stdout)
    # ... id. for critical / error / warning / debug / success / test_name ...
  • State: _prefix_thunk (Thunk) + _domain_taxonomy (tuple).
  • Output: sys.stdout (info, debug, success, test_name) or sys.stderr (critical, error, warning, exception).
  • Format: typically <prefix> [LEVEL] <message>, with ANSI colors if terminal.
  • cleanup(): no-op.

FileLogger#

@final
class FileLogger(PrintLogger):
    def __init__(
        self, *,
        base_dir: Path,
        with_flush_effect: bool = True,
        with_fallback_on_print_logger_when_no_taxonomy_effect: bool = True,
    ) -> None:
        self._base_dir = base_dir
        self._with_flush_effect = with_flush_effect
        self._with_fallback_on_print_logger_when_no_taxonomy_effect = (
            with_fallback_on_print_logger_when_no_taxonomy_effect
        )
        super().__init__()

    @property
    def _log_file_path(self) -> Path:
        if len(self._domain_taxonomy) == 1:
            folder_path = self._base_dir
            file_name = f"{self._domain_taxonomy[0]}.log"
        else:
            folder_path = self._base_dir.joinpath(*self._domain_taxonomy[:-1])
            file_name = f"{self._domain_taxonomy[-1]}.log"
        folder_path.mkdir(parents=True, exist_ok=True)
        return folder_path / file_name
TaxonomyFolderFile
("") — → Fallback to PrintLogger if enabled
("annoncements",)base_dirannoncements.log
("e2e", "Login")base_dir/e2eLogin.log
("e2e", "Login", "happy paths", "Login without OTP")base_dir/e2e/Login/happy pathsLogin without OTP.log

The taxonomy is the folder tree. The DOCX reports (see 05-plugins-reports.md) walk it recursively and emit one .docx per test case.

Chapter 02.04 — Invariants

Chapter 02.04 — Invariants#

Ocarina’s sub-DSL for typed, composable invariants. Used everywhere: CLI, POMs, the custom_invariants/testing/ that check suite consistency before any test runs.

Plan#

#FileSubject
0101-validate-flow.mdThe full flow validate(value) → assert_that → execute → raise_if_invalid + the _ValidationChain class.
0202-assertions.mdCatalog of builtin assertions (is_str, is_email, is_positive, is_in, has_unique_elements, each, is_valid_filename, …).
0303-otherwise-any-of.md.otherwise(...) and the OR combination via _any_of.
0404-then-chain-of-validations.md.then(new_value) and chain_validations(...) for composition.
0505-business-vs-framework-validator.mdBusinessInvariantValidator vs FrameworkInvariantValidator
0606-invariant-errors.mdInvariantViolationError, DuplicatesError, AggregateInvariantViolationError.

Why the invariants DSL#

  1. Pre-execution guard. TestSuite.__init__ checks — before launching anything — that test names / IDs are unique, max_workers >= 1, etc.
  2. CLI validation. Every flag carries its validate=lambda chain: chain.assert_that(...) inside the CliStore. Errors aggregate into one message.
  3. POM-side validation. POM actions validate their parameters ("retries must be positive") via validate(retries, name="retries").assert_that(is_positive).execute().raise_if_invalid().

The Holy Book sums it up:

02.03.05 — create_act and its hooks

02.03.05 — create_act and its hooks#

Source file: src/ocarina/dsl/testing_with_railway/constructors/create_act.py

The low-level primitive that builds a test step constructor. You call it once to mint a project-specific act verb. In theory a complex project might want several distinct act verbs with different hooks, but no real-world need has shown up yet — recommendation: create one. Not a singleton, just a convention.

Signature#

def create_act(
    pom: TPOM,
    action: Callable[[TPOM], TPOM],
    *,
    on_failure: Callable[[TPOM, Exception], Fail] | None = None,
    on_run_effect: Effect | None = None,
    act_counter_effect: Effect | None = None,
) -> ActionStart[TPOM]:
ParameterPositionTypeRole
pompositionalTPOM (bound POMBase)The page (or any POMBase subclass) you’re acting on.
actionpositionalCallable[[TPOM], TPOM]The function that acts on the page and returns the page (fluent). Identity-typed.
on_failurekeyword-onlyCallable[[TPOM, Exception], Fail] | NoneHook for refining the Fail: if you reach this, you’ve already failed; the hook just lets you type the error more precisely (e.g. HttpErrorPageReachedError).
on_run_effectkeyword-onlyEffect | NoneSide effect called before the action (e.g.: log a step number).
act_counter_effectkeyword-onlyEffect | NoneAdditional side effect called before the action. Default: increments the act counter.

Code#

def run_action() -> Result[TPOM]:
    try:
        if act_counter_effect:
            act_counter_effect()
        else:
            ThreadsBasedActCounter().incr_act_call_count()

        if on_run_effect:
            on_run_effect()

        result_pom = action(pom)
        return Ok(result_pom)
    except Exception as exc:  # noqa: BLE001
        if on_failure:
            return on_failure(pom, exc)
        return Fail(error=exc)

return ActionStart(run_action)
  1. The try wraps EVERYTHING: act_counter_effect, on_run_effect, and the action. If any side effect raises (say on_run_effect hits a dead resource), it counts as a test-step failure. Everything that runs “during the test step” IS the test step.
  2. Order is deterministic: counter → run effect → action. The counter ticks even if the action ends up failing. Intentional — we want to know how many steps were attempted, not how many passed.
  3. on_failure can return an enriched Fail: that’s where a WebDriverException becomes an HttpErrorPageReachedError.

Default ActCounter: ThreadsBasedActCounter#

from ocarina.opinionated.infra.act_counter import ActCounter as ThreadsBasedActCounter
# ...
if act_counter_effect:
    act_counter_effect()
else:
    ThreadsBasedActCounter().incr_act_call_count()

Source: src/ocarina/opinionated/infra/act_counter.py

02.04.05 — BusinessInvariantValidator vs FrameworkInvariantValidator

02.04.05 — BusinessInvariantValidator vs FrameworkInvariantValidator#

Two factories, identical code, exist purely to signal intent.

Code#

class _BaseCustomInvariantValidator:
    @staticmethod
    def create[T](
        value: T,
        name: str,
        build_chain: Callable[[ValidationStartBlock[T], T], ValidationAssertBlock[T]],
    ) -> ValidationAssertBlock[T]:
        return _create_business_invariant_validator(
            value, name, lambda chain: build_chain(chain, value)
        )


@final
class BusinessInvariantValidator(_BaseCustomInvariantValidator):
    """Factory for creating business domain invariant validators.

    Use this for domain-specific validation logic (e.g., "user must be adult",
    "order total must match items").
    """


@final
class FrameworkInvariantValidator(_BaseCustomInvariantValidator):
    """Factory for creating framework-level invariant validators.

    Use this for technical/framework validation logic (e.g., "config must be valid",
    "test structure must be correct").
    """
def _create_business_invariant_validator[T](
    value: T,
    name: str,
    build_chain: ValidationChainBuilder[T],
) -> ValidationAssertBlock[T]:
    start_block = validate(value, name=name)
    return build_chain(start_block)

Why two classes?#

  1. BusinessInvariantValidator flags a business rule; FrameworkInvariantValidator, a technical framework rule. Intent jumps off the page.
  2. Grep. FrameworkInvariantValidator.create finds every framework invariant in one search. Same for business rules on the project side.
  3. Maintainability. Want to add behavior (different log, different stack trace) to one or the other? Do it without touching every call-site.

Usage#

InvariantCategoryWhere it lives
validate_workers_amountFrameworksrc/ocarina/custom_invariants/testing/workers.py
validate_test_runners_idsFrameworksrc/ocarina/custom_invariants/testing/oc_test_runners_ids.py
validate_test_runners_namesFrameworksrc/ocarina/custom_invariants/testing/oc_test_runners_names.py
validate_test_suites_namesFrameworksrc/ocarina/custom_invariants/testing/oc_test_suites_names.py
validate_test_campaigns_namesFrameworksrc/ocarina/custom_invariants/testing/oc_test_campaigns_names.py
validate_test_cycle_nameFrameworksrc/ocarina/custom_invariants/testing/oc_test_cycles_names.py
Business example: “the username must be on the whitelist”BusinessTo be written on the user project side
Business example: “the booking date must be future and < 1 year”BusinessTo be written on the user project side

Full pattern#

# src/ocarina/custom_invariants/testing/workers.py
from ocarina.dsl.invariants.assertions import is_not_zero, is_positive
from ocarina.dsl.invariants.validate import FrameworkInvariantValidator


def _workers_amount_chain(
    chain: ValidationStartBlock[int],
    value: int,
) -> ValidationAssertBlock[int]:
    msg = f"Value Error: Number of workers must be at least 1 (got: {value})."
    return chain.assert_that(is_positive, msg=msg).assert_that(is_not_zero, msg=msg)


def validate_workers_amount(
    *, workers_amount: int, name: str
) -> ValidationAssertBlock[int]:
    """Validate that workers amount is at least 1."""
    return FrameworkInvariantValidator.create(
        workers_amount, name, _workers_amount_chain
    )
  1. Private _workers_amount_chain(chain, value) -> ValidationAssertBlock defines the what.
  2. Public validate_workers_amount(*, workers_amount, name) is the API and delegates to FrameworkInvariantValidator.create.
  3. Caller in TestSuite.run: validate_workers_amount(...).execute().raise_if_invalid().

Which factory for which case#

Holy Book (excerpt from the “Extensibility” chapter):

02.05.05 — Worker saturation

02.05.05 — Worker saturation#

Ocarina-specific trick: if a suite has fewer tests than workers, tests get randomly cloned until the worker count is filled. Copies are renamed [COPY 1] <name>, [COPY 2] <name>, etc.

Why#

The Holy Book formalizes the rationale in the “First real-world hurdles” chapter:

Its saturate_workers option lets you force random cloning of tests inside a suite.

As soon as there are more available workers in the DriversPool than there are tests to run in a suite, Ocarina will randomly clone tests, start every driver, and assign each one a test to perform.

02.10.05 — Selenium adapters

02.10.05 — Selenium adapters#

Source folder: src/ocarina/infra/selenium/

Everything that materializes the abstractions (POMBase, WebDriversPool, Screenshotter) in Selenium flavor. Lives outside the pure DSL — and since 1.1.3 it has a twin: src/ocarina/infra/playwright/ ships the exact same contracts in Playwright flavor. Swapping backend is now picking a folder, not writing one.

Files in the folder#

FileRole
create_driver.py_build_firefox, _build_chrome, _build_edge, _build_safari + dispatch
create_drivers_pool.pycreate_selenium_drivers_pool (uses DriverBuilder + WebDriversPool)
create_screenshotter.pycreate_selenium_screenshotter + _selenium_save_full_page (Firefox-only)
driver_healthcheck.pydriver_healthcheck(driver) — pings driver.title
mixins.pySeleniumTitleMixin

create_driver.py#

def _build_firefox(*, profile_path, driver_path, headless, wait_timeout) -> WebDriver:
    service = FirefoxService(executable_path=driver_path)
    options = FirefoxOptions()
    if headless:
        options.add_argument("-headless")
    if profile_path:
        options.profile = FirefoxProfile(profile_path)
    driver = Firefox(service=service, options=options)
    driver.implicitly_wait(wait_timeout)
    return driver


def _build_chrome(*, profile_path, driver_path, headless, wait_timeout) -> WebDriver:
    service = ChromeService(executable_path=driver_path)
    options = ChromeOptions()
    if headless:
        options.add_argument("--headless=new")
    if profile_path:
        options.add_argument(f"--user-data-dir={profile_path}")
    driver = Chrome(service=service, options=options)
    driver.implicitly_wait(wait_timeout)
    return driver


def _build_edge(*, profile_path, driver_path, headless, wait_timeout) -> WebDriver:
    # … same as Chrome with EdgeService/EdgeOptions/Edge …


def _build_safari(*, profile_path, driver_path, headless, wait_timeout) -> WebDriver:
    # … Safari supports neither driver_path (uses macOS' native safaridriver),
    #   nor profile_path (no custom profile), nor headless …

1. implicitly_wait set only once#

Each builder calls driver.implicitly_wait(wait_timeout) once. Any find_element that doesn’t find the element immediately waits up to wait_timeout seconds before raising NoSuchElementException.

02.11.05 — Report plugins

02.11.05 — Report plugins#

Source folder: src/ocarina/opinionated/plugins/reports/

pretty_print_results, results_to_json, generate_docx_proof, timing. All opt-in. Run sequentially or in parallel via run_plugins.

pretty_print_results#

# src/ocarina/opinionated/plugins/reports/pretty_print_results.py
def pretty_print_results(results: TestCycleResults, *, with_colors: bool = False) -> None:
    """Display the full test results."""
Campaign
• Suite
  > Test case 1
    » PASSED
  > Test case 2
    » FAILED
      → Error message
        ⫸ At step 3
  > Test case 3
    » SKIPPED

Test results:
1 FAILED | 1 PASSED | 1 SKIPPED
AspectDetail
Hierarchy3 levels: campaign , suite >, case »
LevelsPASSED (green), FAILED (red), SKIPPED (gray)
ErrorMessage + step number where it failed ("At step 3")
SummaryAggregated count at the bottom

results_to_json#

def generate_json_results(*, results: TestCycleResults, output_dir: Path, logger: ILogger) -> None:
    """Write test results to a JSON file in results_dir."""
    output_dir.mkdir(parents=True, exist_ok=True)
    max_stem_length = 8
    max_attempts = 500
    for _ in range(max_attempts):
        filename = f"{uuid.uuid4().hex[:max_stem_length]}.json"
        file_path = output_dir / filename
        if not file_path.exists():
            break
    else:
        raise RuntimeError(f"Can't generate unique JSON filename in: {output_dir}.")
{
  "Campaign": {
    "Suite": {
      "test_case": [
        {"status": "success" | "fail", "error": "..."},
        counter,
        metadata
      ]
    }
  }
}
def _result_to_serializable(v: Any) -> dict[str, str]:
    if is_ok(v):
        return {"status": "success"}
    if is_fail(v):
        return {"status": "fail", "error": str(v.error)}
    raise TypeError(f"Expected Ok or Fail instances, but got: {type(v)}")

Filename is an 8-char hex UUID → 16⁸ ≈ 4 billion options. No collisions in practice. 500 safety retries.

Chapter 02.05 — Orchestration

Chapter 02.05 — Orchestration#

Test → TestSuite → TestCampaign → TestCycle: how each level slots into the next, who owns parallelization, who owns retries, who decides skipping, and where the pre-execution invariants live.

Plan#

#FileSubject
0101-test.mdThe Test[Driver] class — spawn, pre/post fragments, skip.
0202-test-executor.mdTestExecutor — execution of one attempt, order: setup → watchers.start → chain → watchers.stop → teardown.
0303-test-flow-retries.mdTestFlow — retry loop (1 + max_retries), linear backoff, setup-failure handling.
0404-test-suite.mdTestSuite — parallelization with ThreadPoolExecutor, ID filtering, guardrails.
0505-saturation.mdWorker saturation: random cloning [COPY N]. Why and how.
0606-test-campaign.mdTestCampaign — sequence of suites, campaign_has_failed.
0707-test-cycle-modes.mdTestCycle — smoke + main, fail-fast vs wait-for-all modes, has_test_cycle_failed.
0808-filter-tests-by-ids.mdfilter_tests_by_ids — --only/--exclude, mutex, ignores unknown IDs.

Diagram#

            ┌────────────────────────────────────────────────┐
            │                  TestCycle                     │
            │                                                │
            │  ┌──────────────────────────────────────────┐  │
            │  │ smoke_tests_campaigns                    │  │ ◄── first, gate
            │  │  └─ TestCampaign                         │  │     mode: fail-fast |
            │  │      └─ TestSuite                        │  │           wait-for-all
            │  │          └─ Test                         │  │
            │  └──────────────────────────────────────────┘  │
            │                                                │
            │  ┌──────────────────────────────────────────┐  │
            │  │ campaigns (main)                         │  │ ◄── skipped if a
            │  │  └─ TestCampaign                         │  │     smoke fails
            │  │      └─ TestSuite                        │  │
            │  │          └─ Test                         │  │
            │  └──────────────────────────────────────────┘  │
            └────────────────────────────────────────────────┘

Who does what?#

LevelSingle responsibilityConcurrencyOut of scope
TestMetadata + spawn(driver, logger)noneexecution
TestExecutorOne attemptnoneretries, driver acquisition
TestFlowRetry loopnoneparallelization, aggregation
TestSuiteParallelization + saturation + ID filteringN threadsinter-suite sequence
TestCampaignSequence of suitessuite-levelsmoke vs main
TestCycleSmoke + main + modecampaign-levelbootstrap / post-exec plugins

The strict separation is deliberate. TestExecutor knows nothing about retries. TestFlow knows nothing about concurrency. TestSuite knows nothing about campaign-level aggregation. One axis of responsibility per class.

02.10.06 — The Playwright actor: one owner thread

02.10.06 — The Playwright actor: one owner thread#

Source folder: src/ocarina/infra/playwright/

The Playwright adapter mirrors the Selenium one file for file (create_driver, create_drivers_pool, create_screenshotter, driver_healthcheck, mixins), with one extra file: driver.py. That file deserves a chapter of its own. It is what reconciles Playwright’s sync API — inherently bound to one thread — with Ocarina’s threaded model (pool, warmup, Watcher). The model has been stressed and stabilized; it holds.

The problem: Playwright’s sync API is thread-affine#

Playwright’s sync API binds every object it hands out — Playwright, Browser, BrowserContext, Page, Locator, … — to the thread that called sync_playwright().start(). Under the hood it is a greenlet pinned to that thread. Touching any of them from another thread raises: