08.01 — Code: 99% Claude, 1% Igor, intelligence: 50-50

08.01 — Code: 99% Claude, 1% Igor, intelligence: 50-50#

README#

A note to the reader#

This project is an experiment in AI-driven test engineering. In the interest of honesty about how it was built:

Claude CodeHuman
Code written99%1%
Intelligence50%50%

Almost every line was machine-written. The judgement behind it — what to test, what to distrust, when to dig and when to stop — was shared.

09.03.01 — Review skills

09.03.01 — Review skills#

Static reads, surface findings. A large family. Lets the AI review its project systematically.

Listing (potentially non-exhaustive)#

SkillTarget
review-spec-gapsClarification questions on the FRDs
review-watcher-misuseVerifies the “negative-only” principle of watcher.report(...)
review-compartmentalisation-leaksDetects URLs, selectors, magic numbers in wrong places
review-dead-codeDetects unused connectors / POMs / scenarios / suites / fragments / constants
review-reportClassifies each FAIL / SKIP of an execution
review-type-ignoreAudits # type: ignores (are they justified?)
review-match-candidatesSpots places where a match (Python, pattern matching) could be used instead of if elif elif if if elif
review-unverified-transitionsVerifies that every page transition has a verify
review-submit-dispatchersAudits input-confirmation methods (click vs enter key…)
review-comment-driftDetects comments that have drifted from the code
review-suite-stabilityEvaluates a suite’s stability (proportion of retries, transient_errors hits)
review-intent-collisionsDetects tests that overwrite each other (conflicting intents) and asks/proposes clarifications
review-watcher-emissionsAudits watcher emissions (volume, dedup, relevance)
review-hierarchy-namingAudits the test-hierarchy naming (TestCycle / TestCampaign / TestSuite / Test) for the lazy-naming antipattern where a child carries the parent’s name

review-dead-code#

input  : test base
output : list of unused elements (connectors / POMs / scenarios / fragments / constants)
         + per-element recommendation:
            - delete
            - move to incubator (<root-source>/incubator/, dependency tree preserved)
            - keep (justify)

review-report#

input  : a recent execution (logs + reports)
output : per-test classification:
            - PASS                  (nothing to do)
            - SKIP                  (why?)
            - intentional gap FAIL  (G-DATA-*, G-SEC-*, ...)
            - cross-browser FAIL    (B-BROWSER-*)
            - transient FAIL        (A-ENV-*)
            - regression            ⚠️ ALERT

review-watcher-misuse#

input  : all watcher.report(...) calls
output : list of reports that look positive ("success", "completed", "ok", ...)
         → recommendation: remove or rephrase as negative

review-comment-drift#

input  : every code comment
output : list of comments that no longer match the adjacent code
         (typical: comment mentions foo, code mentions bar)

Helps remove stale comments.
Teach the pattern, not the symptom.”

02.05.02 — TestExecutor[Driver]

02.05.02 — TestExecutor[Driver]#

Source file: src/ocarina/dsl/testing/internals/test_executor.py

One job: run one attempt of a test with one driver. Knows nothing about retries, the driver pool, or suite-level aggregation.

ExecutionOutcome#

@final
@dataclass(frozen=True, slots=True)
class ExecutionOutcome:
    result: TestResult
    skipped: bool
    setup_failed: bool
    should_retry: bool
    steps_count: int
FieldTypeMeaning
resultTestResult = Result[Any] | NoneThe chain’s result (Ok, Fail), or None if skip / setup_failed.
skippedboolTrue if test_runner.skipped (i.e. Test(skipped=True)).
setup_failedboolTrue if the scenario’s setup() raised.
should_retryboolTrue if the retry rule applies (transient_error detected).
steps_countintNumber of act calls; -1 if skip or setup_failed.
  • slots=True: blocks dynamic attribute addition, saves memory. This object rides the hot path — slots earns its keep.
  • frozen=True: immutable, safe to share across threads.
  • @final: no inheritance.

Execution order for one attempt#

┌──────────────────────────────────────────────────────────────────────┐
│  test_runner = test.spawn(driver, logger_with_taxonomy)              │
└──────────────────────────────┬───────────────────────────────────────┘
                               │
                               ▼
              ┌─────────────────────────────────┐
              │  test_runner.skipped ?          │── True ─► return Outcome(skipped=True, ...)
              └────────────────┬────────────────┘
                               │ False
                               ▼
              ┌─────────────────────────────────┐
              │  logger.test_name(test.name)    │   (annotation)
              └────────────────┬────────────────┘
                               │
                               ▼
              ┌─────────────────────────────────┐
              │  setup() (if not None)          │── raises ─► teardown() (if not None)
              └────────────────┬────────────────┘            ↓
                               │                  return Outcome(setup_failed=True, should_retry=True, ...)
                               ▼
              ┌─────────────────────────────────┐
              │  watchers.start(driver, logger, │
              │                 take_screenshot)│   (1 daemon thread per watcher)
              └────────────────┬────────────────┘
                               │
                               ▼
              ┌─────────────────────────────────┐
              │  _run_chain(chain_runners, ...) │── returns (result, should_retry)
              └────────────────┬────────────────┘
                               │
                               ▼
              ┌─────────────────────────────────┐
              │  watchers.stop()                │   (always)
              └────────────────┬────────────────┘
                               │
                               ▼
              ┌─────────────────────────────────┐
              │  teardown() (if not None)       │   (ALWAYS, even if chain failed)
              └────────────────┬────────────────┘     (exceptions are logged & swallowed)
                               │
                               ▼
              ┌─────────────────────────────────┐
              │  steps_count = act_counter.get()│
              │  return Outcome(...)            │
              └─────────────────────────────────┘

execute#

def execute(
    self, test: Test[Driver], *,
    driver: Driver,
    taxonomy: tuple[str, ...],
    logger_with_taxonomy: ILogger,
    logger_without_taxonomy: ILogger,
    attempt: int, max_attempts: int,
) -> ExecutionOutcome:
    test_runner = test.spawn(driver, logger_with_taxonomy)

    if test_runner.skipped:
        return ExecutionOutcome(result=None, skipped=True, setup_failed=False,
                                should_retry=False, steps_count=-1)

    logger_without_taxonomy.test_name(test.name)

    if test_runner.setup is not None:
        try:
            test_runner.setup()
        except Exception as exc:
            msg = f"{test.name} -- Setup failed (attempt {attempt}/{max_attempts}): {exc}"
            logger_with_taxonomy.warning(msg)
            if test_runner.teardown is not None:
                self._run_teardown(test_runner.teardown, test_name=test.name, logger=logger_with_taxonomy)
            return ExecutionOutcome(result=None, skipped=False, setup_failed=True,
                                    should_retry=True, steps_count=-1)

    watchers: Sequence[Watcher[Driver]] = test_runner.watchers or []
    self._start_watchers(watchers, driver=driver, test_name=test.name, taxonomy=taxonomy)

    result, should_retry = self._run_chain(
        test_runner.chain_runners, test_name=test.name, attempt=attempt,
        driver=driver, logger=logger_without_taxonomy,
        logger_with_taxonomy=logger_with_taxonomy, max_attempts=max_attempts,
    )

    self._stop_watchers(watchers)

    if test_runner.teardown is not None:
        self._run_teardown(test_runner.teardown, test_name=test.name, logger=logger_with_taxonomy)

    steps_count = self._act_counter.get()
    return ExecutionOutcome(result=result, skipped=False, setup_failed=False,
                            should_retry=should_retry, steps_count=steps_count)

Two loggers: why#

logger_with_taxonomy vs logger_without_taxonomy:

05.02 — Igoristan routes

05.02 — Igoristan routes#

src/config/routes.ts#

const __ROOT = "/igoristan/";
const DASHBOARD = "dashboard";

const __ROUTES = {
  DONKEY_SAUSAGE_DETECTOR: "donkey-sausage-eater-detector",
  DASHBOARD_NESTED: `${DASHBOARD}/nested`,
  RANDOM_LOADERS: "random-loaders",
  SACRED_UPLOAD: "sacred-upload",
  RANDOM_ERROR: "random-error",
  CHAOTIC_FORM: "chaotic-form",
  CORSICAMON: "corsicamon",
  MADNESS: "madness",
  DASHBOARD,
  HOME: "",
} as const satisfies Routes;

const ROUTES = createRoutes(__ROUTES, __ROOT);
RouteURLRoleDeliberate chaos
HOME/igoristan/Home with links to other pages, autoplay audio — 
RANDOM_LOADERS/igoristan/random-loadersPseudo-random loadersLoaders that stay visible for a variable time
SACRED_UPLOAD/igoristan/sacred-uploadreact-dropzone upload form — 
DASHBOARD/igoristan/dashboardLogin (password figatellu), optional MFA OTPuseAuth may fail 10% of the time even with the right pwd
DASHBOARD_NESTED/igoristan/dashboard/nestedProtected page, requires MFA — 
CORSICAMON/igoristan/corsicamonCorsican Pokédex”, 3 random picks via API key1/5 chance of artificially raising Error('lol')
RANDOM_ERROR/igoristan/random-errorFake error pageTitle matched by ERROR_PAGE_REGEX on the test side
CHAOTIC_FORM/igoristan/chaotic-formUnstable form.catch-me-if-you-can elements appear at random
MADNESS/igoristan/madnessAlternating stories (Cors, ThisIsBastia)Renders one OR the other at random
DONKEY_SAUSAGE_DETECTOR/igoristan/donkey-sausage-eater-detectorDetector for filthy Sicilian shits30% random fail (BSOD-style)

Page by page#

const links = [
  { href: ROUTES.RANDOM_LOADERS, label: "Random loaders" },
  { href: ROUTES.SACRED_UPLOAD, label: "Sacred upload" },
  { href: ROUTES.DASHBOARD, label: "Dashboard" },
  { href: ROUTES.CORSICAMON, label: "Corsicamon" },
  { href: ROUTES.RANDOM_ERROR, label: "Random Error" },
  { href: ROUTES.CHAOTIC_FORM, label: "Chaotic form" },
  { href: ROUTES.MADNESS, label: "Madness" },
  { href: ROUTES.DONKEY_SAUSAGE_DETECTOR, label: "Donkey Sausage Detector" },
];

home#

8 links to the other pages + autoplay audio Angelus of Jerusalem (amen 🙏).

03.03 — Lazy evaluation

03.03 — Lazy evaluation#

In Ocarina, nothing is executed until you explicitly trigger it. That’s what makes scenarios composable as values.

Zzz#

LocationFormTrigger
ChainRunner[T]Thunk[ActionChain[T]]runner.run()
validate(...)ValidationStartBlock / ValidationAssertBlock.execute()
match_page(...)returns a ChainRunner[Any].run() (via the surrounding chain)
Watcher.callbackCallable[[Watcher], None]_loop when start() is called
logger.set_prefix(thunk)Thunk[str]recomputed at each log call
Scenario.setup / teardownEffectcalled by TestExecutor
bootstrap(post_exec=...)Callable[[TestCycleResults], None]called after run_plugins
CliBuilder(effects_factory=lambda ns: (...))Effectscalled after the argparse parse
test_scenario: TestScenario[Driver]Callable[[Driver, ILogger], Scenario[Driver]]called by Test.spawn
dispatch[mode]() in TestCycle.run_alldict of Thunk[bool]called on lookup

ChainRunner#

runner = drive_page(act1, act2, act3)        # ⚠️  nothing executed
# … later …
chain = runner.run()                         # ▶︎  execution
CapabilityWithout lazinessWith laziness
Store a scenario in a variableimpossible (already executed)trivial
Multiply [runner] * 5executes once, you get 5 refs to the resultexecutes 5 times
Pass a runner to another runner (composition)impossibletrivial
Reorder acts in a test refactorhardtrivial

validate(...).execute()#

v = validate(value, name="x").assert_that(is_positive).assert_that(is_not_zero)
# … you can compose …
combined = chain_validations(v, other_validation)
# … nothing executed so far …
combined.execute().raise_if_invalid()        # ▶︎  execution + aggregation

That’s how _ValidationChain collects every error before raising a single one (AggregateInvariantViolationError).

07.05 — Randomness scenarios (4 levels)

07.05 — Randomness scenarios (4 levels)#

Four progressive levels of chaos. We start simple, ramp up complexity, stress Ocarina more and more.

Campaign#

# src/tests/campaigns/randomness.py
def create_igoristan_randomness_campaign(*, drivers_pool) -> TestCampaign:
    return TestCampaign(
        name="Randomness",
        suites=[create_randomness_test_suite(drivers_pool=drivers_pool)],
    )

Suite#

# src/tests/suites/randomness.py
def create_randomness_test_suite(*, drivers_pool) -> TestSuite:
    return TestSuite(
        name="Randomness",
        tests=[
            test_random_error_page,                     # level 1
            test_random_loaders_page,                   # level 1
            test_dsed,                                  # level 2
            test_madness,                               # level 2
            test_chaotic_form,                          # level 3
            test_walkthrough,                           # level 4
        ],
        drivers_pool=drivers_pool,
    )

Levels#

LevelMechanic exercisedDifficulty
1Base DSL (drive_page, act)easy
2match_page / whenmedium
3HumanizedDriver + Watcherhard
4Multi-page chaotic walkthroughhard

Level 1 — Random Error Page#

# tests/scenarios/randomness/level_1/random_error_page.py
def scenario_random_error_page(driver, logger):
    page = RandomErrorPage(driver=driver)
    return [
        drive_page(
            act(page, open_random_error_page)...,
            act(page, verify_random_error_page)...,
        ),
    ]


test_random_error_page = create_selenium_test(
    name="Random Error Page - smoke",
    test_scenario=lambda driver, logger: Scenario(test_chain=scenario_random_error_page(driver, logger)),
)

Deliberately flaky — exercises the on_failure hook.

02.06 — Scenario[Driver]

02.06 — Scenario[Driver]#

Source file: src/ocarina/custom_types/scenario.py

Dataclass#

@final
@dataclass(frozen=True)
class Scenario[Driver]:
    test_chain: TestChain
    setup: TestSetup = field(default=None)
    teardown: TestTeardown = field(default=None)
    watchers: TestWatchers[Driver] | None = field(default=None)
# src/ocarina/custom_types/test_components.py
type TestChain = Sequence[ChainRunner[Any]]
type TestSetup = Effect | None
type TestTeardown = Effect | None
type TestWatchers[Driver] = Sequence[Watcher[Driver]] | None

Lifecycle (per attempt)#

1. setup()           — optional, free-form Effect (DB, API, …)
       → raises    :  test_chain skipped, jump to teardown,
                      return Outcome(setup_failed=True, should_retry=True)
       → ok        :  continue to test_chain

2. test_chain        — the actual chain (Sequence[ChainRunner])
       (watchers run during this time)

3. teardown()        — optional, always executed
       → raises    :  log warning + ignore (does not affect the verdict)

If EVERY attempt raises during setup:
    → test marked SKIPPED (not FAILED)
    → warning log « setup keeps failing »

setup and teardown#

setup and teardown are driver-free and injection-free by design.

02.07 — Watcher[Driver]

02.07 — Watcher[Driver]#

Source file: src/ocarina/dsl/testing/watcher.py

Parallelized observer running as a daemon thread alongside the test_chain. Built to catch unpredictable frictions with no direct impact on the scenario that pop up while the scenario runs: random error toasts, parasitic validations, unexpected popups.

The problem#

Holy Book quote (First real-world hurdles):

Even more surprisingly: apps displaying error toasts for no apparent reason, or forms flagging validation errors on inputs that are actually correct, without blocking the journey.

07.09 — catch_me_if_you_can

07.09 — catch_me_if_you_can#

Detects parasite elements that pop on the page while the test is running, and traces them.

Callback#

# src/lib/ext/selenium/watchers/catch_me_if_you_can_watcher.py
def catch_me_if_you_can_cb(watcher: SeleniumWatcher) -> None:
    elements = watcher.driver.execute_script(
        "return Array.from(document.querySelectorAll('.catch-me-if-you-can'));"
    )

    if not elements:
        return

    raw = watcher.driver.execute_script(
        """
        return arguments[0].map(el => ({
            tag:       el.tagName.toLowerCase(),
            text:      el.innerText.trim(),
            id:        el.id,
            cls:       el.className,
            name:      el.getAttribute('name') || '',
            testid:    el.getAttribute('data-testid') || '',
        }));
        """,
        elements,
    )

    for attrs in raw:
        fingerprint = ":".join(
            filter(
                None,
                [
                    attrs["tag"],
                    attrs["text"],
                    attrs["id"],
                    attrs["cls"],
                    attrs["name"],
                    attrs["testid"],
                ],
            )
        )

        if fingerprint in watcher.cache:
            continue

        watcher.cache.add(fingerprint)
        watcher.report(
            f"catch-me-if-you-can element detected: <{attrs['tag']}> {attrs['text']!r}",
            label="CATCH_ME_IF_YOU_CAN",
        )

Mechanics#

1. Javascript to bypass implicit wait#

elements = watcher.driver.execute_script(
    "return Array.from(document.querySelectorAll('.catch-me-if-you-can'));"
)

Not find_elements(By.CSS_SELECTOR, ...).