Watcher on Ocarina — manual

08.01 — Code: 99% Claude, 1% Igor, intelligence: 50-50

Wed, 20 May 2026 00:00:00 +0000

08.01 — Code: 99% Claude, 1% Igor, intelligence: 50-50#

README#

A note to the reader#

This project is an experiment in AI-driven test engineering. In the interest of honesty about how it was built:

Claude Code Human

Code written 99% 1%

Intelligence 50% 50%

Almost every line was machine-written. The judgement behind it — what to test, what to distrust, when to dig and when to stop — was shared.

	Claude Code	Human
Code written	99%	1%
Intelligence	50%	50%

09.03.01 — Review skills

Wed, 20 May 2026 00:00:00 +0000

09.03.01 — Review skills#

Static reads, surface findings. A large family. Lets the AI review its project systematically.

Listing (potentially non-exhaustive)#

Skill	Target
`review-spec-gaps`	Clarification questions on the FRDs
`review-watcher-misuse`	Verifies the “negative-only” principle of `watcher.report(...)`
`review-compartmentalisation-leaks`	Detects URLs, selectors, magic numbers in wrong places
`review-dead-code`	Detects unused connectors / POMs / scenarios / suites / fragments / constants
`review-report`	Classifies each FAIL / SKIP of an execution
`review-type-ignore`	Audits `# type: ignore`s (are they justified?)
`review-match-candidates`	Spots places where a `match` (Python, pattern matching) could be used instead of `if` `elif` `elif` `if` `if` `elif`
`review-unverified-transitions`	Verifies that every page transition has a `verify`
`review-submit-dispatchers`	Audits input-confirmation methods (click vs enter key…)
`review-comment-drift`	Detects comments that have drifted from the code
`review-suite-stability`	Evaluates a suite’s stability (proportion of retries, transient_errors hits)
`review-intent-collisions`	Detects tests that overwrite each other (conflicting intents) and asks/proposes clarifications
`review-watcher-emissions`	Audits watcher emissions (volume, dedup, relevance)
`review-hierarchy-naming`	Audits the test-hierarchy naming (`TestCycle` / `TestCampaign` / `TestSuite` / `Test`) for the lazy-naming antipattern where a child carries the parent’s name

`review-dead-code`#

input  : test base
output : list of unused elements (connectors / POMs / scenarios / fragments / constants)
         + per-element recommendation:
            - delete
            - move to incubator (<root-source>/incubator/, dependency tree preserved)
            - keep (justify)

`review-report`#

input  : a recent execution (logs + reports)
output : per-test classification:
            - PASS                  (nothing to do)
            - SKIP                  (why?)
            - intentional gap FAIL  (G-DATA-*, G-SEC-*, ...)
            - cross-browser FAIL    (B-BROWSER-*)
            - transient FAIL        (A-ENV-*)
            - regression            ⚠️ ALERT

`review-watcher-misuse`#

input  : all watcher.report(...) calls
output : list of reports that look positive ("success", "completed", "ok", ...)
         → recommendation: remove or rephrase as negative

`review-comment-drift`#

input  : every code comment
output : list of comments that no longer match the adjacent code
         (typical: comment mentions foo, code mentions bar)

Helps remove stale comments.
“Teach the pattern, not the symptom.”

02.05.02 — TestExecutor[Driver]

Wed, 20 May 2026 00:00:00 +0000

02.05.02 — `TestExecutor[Driver]`#

Source file: src/ocarina/dsl/testing/internals/test_executor.py

One job: run one attempt of a test with one driver. Knows nothing about retries, the driver pool, or suite-level aggregation.

`ExecutionOutcome`#

@final
@dataclass(frozen=True, slots=True)
class ExecutionOutcome:
    result: TestResult
    skipped: bool
    setup_failed: bool
    should_retry: bool
    steps_count: int

Field	Type	Meaning
`result`	`TestResult` = `Result[Any] \| None`	The chain’s result (Ok, Fail), or `None` if skip / setup_failed.
`skipped`	`bool`	True if `test_runner.skipped` (i.e. `Test(skipped=True)`).
`setup_failed`	`bool`	True if the scenario’s `setup()` raised.
`should_retry`	`bool`	True if the retry rule applies (transient_error detected).
`steps_count`	`int`	Number of `act` calls; `-1` if skip or setup_failed.

slots=True: blocks dynamic attribute addition, saves memory. This object rides the hot path — slots earns its keep.
frozen=True: immutable, safe to share across threads.
@final: no inheritance.

Execution order for one attempt#

┌──────────────────────────────────────────────────────────────────────┐
│  test_runner = test.spawn(driver, logger_with_taxonomy)              │
└──────────────────────────────┬───────────────────────────────────────┘
                               │
                               ▼
              ┌─────────────────────────────────┐
              │  test_runner.skipped ?          │── True ─► return Outcome(skipped=True, ...)
              └────────────────┬────────────────┘
                               │ False
                               ▼
              ┌─────────────────────────────────┐
              │  logger.test_name(test.name)    │   (annotation)
              └────────────────┬────────────────┘
                               │
                               ▼
              ┌─────────────────────────────────┐
              │  setup() (if not None)          │── raises ─► teardown() (if not None)
              └────────────────┬────────────────┘            ↓
                               │                  return Outcome(setup_failed=True, should_retry=True, ...)
                               ▼
              ┌─────────────────────────────────┐
              │  watchers.start(driver, logger, │
              │                 take_screenshot)│   (1 daemon thread per watcher)
              └────────────────┬────────────────┘
                               │
                               ▼
              ┌─────────────────────────────────┐
              │  _run_chain(chain_runners, ...) │── returns (result, should_retry)
              └────────────────┬────────────────┘
                               │
                               ▼
              ┌─────────────────────────────────┐
              │  watchers.stop()                │   (always)
              └────────────────┬────────────────┘
                               │
                               ▼
              ┌─────────────────────────────────┐
              │  teardown() (if not None)       │   (ALWAYS, even if chain failed)
              └────────────────┬────────────────┘     (exceptions are logged & swallowed)
                               │
                               ▼
              ┌─────────────────────────────────┐
              │  steps_count = act_counter.get()│
              │  return Outcome(...)            │
              └─────────────────────────────────┘

`execute`#

def execute(
    self, test: Test[Driver], *,
    driver: Driver,
    taxonomy: tuple[str, ...],
    logger_with_taxonomy: ILogger,
    logger_without_taxonomy: ILogger,
    attempt: int, max_attempts: int,
) -> ExecutionOutcome:
    test_runner = test.spawn(driver, logger_with_taxonomy)

    if test_runner.skipped:
        return ExecutionOutcome(result=None, skipped=True, setup_failed=False,
                                should_retry=False, steps_count=-1)

    logger_without_taxonomy.test_name(test.name)

    if test_runner.setup is not None:
        try:
            test_runner.setup()
        except Exception as exc:
            msg = f"{test.name} -- Setup failed (attempt {attempt}/{max_attempts}): {exc}"
            logger_with_taxonomy.warning(msg)
            if test_runner.teardown is not None:
                self._run_teardown(test_runner.teardown, test_name=test.name, logger=logger_with_taxonomy)
            return ExecutionOutcome(result=None, skipped=False, setup_failed=True,
                                    should_retry=True, steps_count=-1)

    watchers: Sequence[Watcher[Driver]] = test_runner.watchers or []
    self._start_watchers(watchers, driver=driver, test_name=test.name, taxonomy=taxonomy)

    result, should_retry = self._run_chain(
        test_runner.chain_runners, test_name=test.name, attempt=attempt,
        driver=driver, logger=logger_without_taxonomy,
        logger_with_taxonomy=logger_with_taxonomy, max_attempts=max_attempts,
    )

    self._stop_watchers(watchers)

    if test_runner.teardown is not None:
        self._run_teardown(test_runner.teardown, test_name=test.name, logger=logger_with_taxonomy)

    steps_count = self._act_counter.get()
    return ExecutionOutcome(result=result, skipped=False, setup_failed=False,
                            should_retry=should_retry, steps_count=steps_count)

Two loggers: why#

logger_with_taxonomy vs logger_without_taxonomy:

05.02 — Igoristan routes

Wed, 20 May 2026 00:00:00 +0000

05.02 — Igoristan routes#

`src/config/routes.ts`#

const __ROOT = "/igoristan/";
const DASHBOARD = "dashboard";

const __ROUTES = {
  DONKEY_SAUSAGE_DETECTOR: "donkey-sausage-eater-detector",
  DASHBOARD_NESTED: `${DASHBOARD}/nested`,
  RANDOM_LOADERS: "random-loaders",
  SACRED_UPLOAD: "sacred-upload",
  RANDOM_ERROR: "random-error",
  CHAOTIC_FORM: "chaotic-form",
  CORSICAMON: "corsicamon",
  MADNESS: "madness",
  DASHBOARD,
  HOME: "",
} as const satisfies Routes;

const ROUTES = createRoutes(__ROUTES, __ROOT);

Route	URL	Role	Deliberate chaos
`HOME`	`/igoristan/`	Home with links to other pages, autoplay audio	—
`RANDOM_LOADERS`	`/igoristan/random-loaders`	Pseudo-random loaders	Loaders that stay visible for a variable time
`SACRED_UPLOAD`	`/igoristan/sacred-upload`	`react-dropzone` upload form	—
`DASHBOARD`	`/igoristan/dashboard`	Login (password `figatellu`), optional MFA OTP	`useAuth` may fail 10% of the time even with the right pwd
`DASHBOARD_NESTED`	`/igoristan/dashboard/nested`	Protected page, requires MFA	—
`CORSICAMON`	`/igoristan/corsicamon`	“Corsican Pokédex”, 3 random picks via API key	1/5 chance of artificially raising `Error('lol')`
`RANDOM_ERROR`	`/igoristan/random-error`	Fake error page	Title matched by `ERROR_PAGE_REGEX` on the test side
`CHAOTIC_FORM`	`/igoristan/chaotic-form`	Unstable form	`.catch-me-if-you-can` elements appear at random
`MADNESS`	`/igoristan/madness`	Alternating stories (Cors, ThisIsBastia)	Renders one OR the other at random
`DONKEY_SAUSAGE_DETECTOR`	`/igoristan/donkey-sausage-eater-detector`	Detector for filthy Sicilian shits	30% random fail (BSOD-style)

Page by page#

const links = [
  { href: ROUTES.RANDOM_LOADERS, label: "Random loaders" },
  { href: ROUTES.SACRED_UPLOAD, label: "Sacred upload" },
  { href: ROUTES.DASHBOARD, label: "Dashboard" },
  { href: ROUTES.CORSICAMON, label: "Corsicamon" },
  { href: ROUTES.RANDOM_ERROR, label: "Random Error" },
  { href: ROUTES.CHAOTIC_FORM, label: "Chaotic form" },
  { href: ROUTES.MADNESS, label: "Madness" },
  { href: ROUTES.DONKEY_SAUSAGE_DETECTOR, label: "Donkey Sausage Detector" },
];

home#

8 links to the other pages + autoplay audio Angelus of Jerusalem (amen 🙏).

03.03 — Lazy evaluation

Wed, 20 May 2026 00:00:00 +0000

03.03 — Lazy evaluation#

In Ocarina, nothing is executed until you explicitly trigger it. That’s what makes scenarios composable as values.

Zzz#

Location	Form	Trigger
`ChainRunner[T]`	`Thunk[ActionChain[T]]`	`runner.run()`
`validate(...)`	`ValidationStartBlock` / `ValidationAssertBlock`	`.execute()`
`match_page(...)`	returns a `ChainRunner[Any]`	`.run()` (via the surrounding chain)
`Watcher.callback`	`Callable[[Watcher], None]`	`_loop` when `start()` is called
`logger.set_prefix(thunk)`	`Thunk[str]`	recomputed at each log call
`Scenario.setup` / `teardown`	`Effect`	called by `TestExecutor`
`bootstrap(post_exec=...)`	`Callable[[TestCycleResults], None]`	called after `run_plugins`
`CliBuilder(effects_factory=lambda ns: (...))`	`Effects`	called after the argparse parse
`test_scenario: TestScenario[Driver]`	`Callable[[Driver, ILogger], Scenario[Driver]]`	called by `Test.spawn`
`dispatch[mode]()` in `TestCycle.run_all`	dict of `Thunk[bool]`	called on lookup

`ChainRunner`#

runner = drive_page(act1, act2, act3)        # ⚠️  nothing executed
# … later …
chain = runner.run()                         # ▶︎  execution

Capability	Without laziness	With laziness
Store a scenario in a variable	impossible (already executed)	trivial
Multiply `[runner] * 5`	executes once, you get 5 refs to the result	executes 5 times
Pass a `runner` to another `runner` (composition)	impossible	trivial
Reorder `act`s in a test refactor	hard	trivial

`validate(...).execute()`#

v = validate(value, name="x").assert_that(is_positive).assert_that(is_not_zero)
# … you can compose …
combined = chain_validations(v, other_validation)
# … nothing executed so far …
combined.execute().raise_if_invalid()        # ▶︎  execution + aggregation

That’s how _ValidationChain collects every error before raising a single one (AggregateInvariantViolationError).

07.05 — Randomness scenarios (4 levels)

Wed, 20 May 2026 00:00:00 +0000

07.05 — Randomness scenarios (4 levels)#

Four progressive levels of chaos. We start simple, ramp up complexity, stress Ocarina more and more.

Campaign#

# src/tests/campaigns/randomness.py
def create_igoristan_randomness_campaign(*, drivers_pool) -> TestCampaign:
    return TestCampaign(
        name="Randomness",
        suites=[create_randomness_test_suite(drivers_pool=drivers_pool)],
    )

Suite#

# src/tests/suites/randomness.py
def create_randomness_test_suite(*, drivers_pool) -> TestSuite:
    return TestSuite(
        name="Randomness",
        tests=[
            test_random_error_page,                     # level 1
            test_random_loaders_page,                   # level 1
            test_dsed,                                  # level 2
            test_madness,                               # level 2
            test_chaotic_form,                          # level 3
            test_walkthrough,                           # level 4
        ],
        drivers_pool=drivers_pool,
    )

Levels#

Level	Mechanic exercised	Difficulty
1	Base DSL (`drive_page`, `act`)	easy
2	`match_page` / `when`	medium
3	`HumanizedDriver` + `Watcher`	hard
4	Multi-page chaotic walkthrough	hard

Level 1 — Random Error Page#

# tests/scenarios/randomness/level_1/random_error_page.py
def scenario_random_error_page(driver, logger):
    page = RandomErrorPage(driver=driver)
    return [
        drive_page(
            act(page, open_random_error_page)...,
            act(page, verify_random_error_page)...,
        ),
    ]


test_random_error_page = create_selenium_test(
    name="Random Error Page - smoke",
    test_scenario=lambda driver, logger: Scenario(test_chain=scenario_random_error_page(driver, logger)),
)

Deliberately flaky — exercises the on_failure hook.

02.06 — Scenario[Driver]

Wed, 20 May 2026 00:00:00 +0000

02.06 — `Scenario[Driver]`#

Source file: src/ocarina/custom_types/scenario.py

Dataclass#

@final
@dataclass(frozen=True)
class Scenario[Driver]:
    test_chain: TestChain
    setup: TestSetup = field(default=None)
    teardown: TestTeardown = field(default=None)
    watchers: TestWatchers[Driver] | None = field(default=None)

# src/ocarina/custom_types/test_components.py
type TestChain = Sequence[ChainRunner[Any]]
type TestSetup = Effect | None
type TestTeardown = Effect | None
type TestWatchers[Driver] = Sequence[Watcher[Driver]] | None

Lifecycle (per attempt)#

1. setup()           — optional, free-form Effect (DB, API, …)
       → raises    :  test_chain skipped, jump to teardown,
                      return Outcome(setup_failed=True, should_retry=True)
       → ok        :  continue to test_chain

2. test_chain        — the actual chain (Sequence[ChainRunner])
       (watchers run during this time)

3. teardown()        — optional, always executed
       → raises    :  log warning + ignore (does not affect the verdict)

If EVERY attempt raises during setup:
    → test marked SKIPPED (not FAILED)
    → warning log « setup keeps failing »

`setup` and `teardown`#

setup and teardown are driver-free and injection-free by design.

02.07 — Watcher[Driver]

Wed, 20 May 2026 00:00:00 +0000

02.07 — `Watcher[Driver]`#

Source file: src/ocarina/dsl/testing/watcher.py

Parallelized observer running as a daemon thread alongside the test_chain. Built to catch unpredictable frictions with no direct impact on the scenario that pop up while the scenario runs: random error toasts, parasitic validations, unexpected popups.

The problem#

Holy Book quote (First real-world hurdles):

Even more surprisingly: apps displaying error toasts for no apparent reason, or forms flagging validation errors on inputs that are actually correct, without blocking the journey.

07.09 — catch_me_if_you_can

Wed, 20 May 2026 00:00:00 +0000

07.09 — `catch_me_if_you_can`#

Detects parasite elements that pop on the page while the test is running, and traces them.

Callback#

# src/lib/ext/selenium/watchers/catch_me_if_you_can_watcher.py
def catch_me_if_you_can_cb(watcher: SeleniumWatcher) -> None:
    elements = watcher.driver.execute_script(
        "return Array.from(document.querySelectorAll('.catch-me-if-you-can'));"
    )

    if not elements:
        return

    raw = watcher.driver.execute_script(
        """
        return arguments[0].map(el => ({
            tag:       el.tagName.toLowerCase(),
            text:      el.innerText.trim(),
            id:        el.id,
            cls:       el.className,
            name:      el.getAttribute('name') || '',
            testid:    el.getAttribute('data-testid') || '',
        }));
        """,
        elements,
    )

    for attrs in raw:
        fingerprint = ":".join(
            filter(
                None,
                [
                    attrs["tag"],
                    attrs["text"],
                    attrs["id"],
                    attrs["cls"],
                    attrs["name"],
                    attrs["testid"],
                ],
            )
        )

        if fingerprint in watcher.cache:
            continue

        watcher.cache.add(fingerprint)
        watcher.report(
            f"catch-me-if-you-can element detected: <{attrs['tag']}> {attrs['text']!r}",
            label="CATCH_ME_IF_YOU_CAN",
        )

Mechanics#

1. Javascript to bypass implicit wait#

elements = watcher.driver.execute_script(
    "return Array.from(document.querySelectorAll('.catch-me-if-you-can'));"
)

Not find_elements(By.CSS_SELECTOR, ...).

Watcher on Ocarina — manual

08.01 — Code: 99% Claude, 1% Igor, intelligence: 50-50

08.01 — Code: 99% Claude, 1% Igor, intelligence: 50-50#

README#

A note to the reader#

09.03.01 — Review skills

09.03.01 — Review skills#

Listing (potentially non-exhaustive)#

review-dead-code#

review-report#

review-watcher-misuse#

review-comment-drift#

02.05.02 — TestExecutor[Driver]

02.05.02 — TestExecutor[Driver]#

ExecutionOutcome#

Execution order for one attempt#

execute#

Two loggers: why#

05.02 — Igoristan routes

05.02 — Igoristan routes#

src/config/routes.ts#

Page by page#

home#

03.03 — Lazy evaluation

03.03 — Lazy evaluation#

Zzz#

ChainRunner#

validate(...).execute()#

07.05 — Randomness scenarios (4 levels)

07.05 — Randomness scenarios (4 levels)#

Campaign#

Suite#

Levels#

Level 1 — Random Error Page#

02.06 — Scenario[Driver]

02.06 — Scenario[Driver]#

Dataclass#

Lifecycle (per attempt)#

setup and teardown#

02.07 — Watcher[Driver]

02.07 — Watcher[Driver]#

The problem#

07.09 — catch_me_if_you_can

07.09 — catch_me_if_you_can#

Callback#

Mechanics#

1. Javascript to bypass implicit wait#

`review-dead-code`#

`review-report`#

`review-watcher-misuse`#

`review-comment-drift`#

02.05.02 — `TestExecutor[Driver]`#

`ExecutionOutcome`#

`execute`#

`src/config/routes.ts`#

`ChainRunner`#

`validate(...).execute()`#

02.06 — `Scenario[Driver]`#

`setup` and `teardown`#

02.07 — `Watcher[Driver]`#

07.09 — `catch_me_if_you_can`#