09.03.02 — Analyse skills#

Dynamic analyses: use the logs / reports of a recent execution to diagnose a failure. Two root-cause skills sit in front of the family — diagnose-root-cause for a deterministic red, diagnose-flake-root-cause for an intermittent one — and route to the controlled analyse-* experiments.

Listing (potentially non-exhaustive)#

diagnose-root-cause#

input  : a deterministic red (fails every run)
output : structured root-cause analysis:
            - re-derive the failure, don't inherit prior assumptions
            - climb the synthetic-to-real ladder (fake driver → real browser)
            - read the source, confirm with a probe
            - root cause sorted into five buckets
         hands off to diagnose-flake-root-cause if the failure proves intermittent

diagnose-flake-root-cause#

input  : an intermittent failure (a flake)
output : distribution-based root-cause analysis:
            - establish a failure rate (the distribution is the evidence)
            - pin the signature, correlate
            - route to the right analyse-* experiment, raise the rate to confirm
            - classify into five flake buckets
         the orchestrator of the analyse-* family

analyse-flakiness#

input  : logs of the last N runs
output : list of tests that retry often (by % of retries)
         + recommendation:
            - add an exception to the transient_errors tuple
            - investigate a particular test (the issue might come from the test itself)

analyse-fixture-flakiness#

input  : tests with setup/teardown
output : tests whose setup fails often
         + recommendation:
            - verify setup idempotency
            - check teardown is clean (otherwise cross-test contamination)
            - move the seed out of setup into a manual init if too fragile

analyse-watcher-flakiness#

input  : watcher.report(...) logs over N runs
output : watchers with abnormal reporting:
            - too many emissions (dedup issue)
            - no emissions (the watcher is useless)
            - false positives (expected element reported as intruder)

analyse-screenshot-flakiness#

input  : screenshots from N runs, grouped by (test, step, browser)
output : visual flakiness:
            - tests whose screenshots vary abnormally between runs
            - tests that pass but whose look changes
            - identify whether the variation is semantic (real diff) or cosmetic

Uses heuristics (hash, dimensions, dominant palette). Not a real image diff.

Role#

• Consume an execution (runtime).
• Detect patterns over time (N runs).
• Propose actions (widen transient_errors, refactor a watcher, etc.).

Reminder from the Holy Book:

Failing cycle: review-report → analyse-* → write-a-probe → findings propagated to IDENTIFIED_GAPS.md / the FRDs / a scenario comment → probe deleted.