08.01 — Code: 99% Claude, 1% Igor, intelligence: 50-50

08.01 — Code: 99% Claude, 1% Igor, intelligence: 50-50#

README#

A note to the reader#

This project is an experiment in AI-driven test engineering. In the interest of honesty about how it was built:

Claude CodeHuman
Code written99%1%
Intelligence50%50%

Almost every line was machine-written. The judgement behind it — what to test, what to distrust, when to dig and when to stop — was shared.

08.02 — CURA Healthcare (SUT)

08.02 — CURA Healthcare (SUT)#

Ocarina’s first victim#

Perimeter#

PageFunction
/Marketing homepage
/profile.php#loginLogin
/appointment.phpBooking form (facility, programs, hospital_readmission, visit_date, comment)
/confirmation.phpBooking confirmation
/history.phpThe user’s booking history
/profile.php#profileProfile page (empty placeholder — gap §G-SPEC-2)

Why target CURA#

  1. Open source — we can read the PHP to understand why a given behavior exists.
  2. Plenty of gaps — rich territory for findings.
  3. Public demo — no authorization needed. Stable (maintained by Katalon).

There’s also a tacit reason: Katalon is a competing test-framework vendor (Katalon Studio). It’s poetic to test their demo app with Ocarina.

08.03 — Documentation

08.03 — Documentation#

The AI project is the most documented in the Ocarina ecosystem.

Listing#

DocumentStatusAudience
CLAUDE.mdClaude ↔ project working contractClaude + any contributor
CURA_FRD.mdReconstructed specMaintainers, stakeholders
CURA_TEST_STRATEGY.mdTest strategyMaintainers
IDENTIFIED_GAPS.mdTechnical inventory of gapsMaintainers

CLAUDE.md#

  1. Key documents: points to README.md, CURA_FRD.md, CURA_TEST_STRATEGY.md, IDENTIFIED_GAPS.md, CLAUDE.local.md.
  2. Project: 2 sentences on the project (CURA, Python 3.14+, demo creds).
  3. Project philosophy: “No tricks or hacks. Teach the pattern, not the symptom.
  4. CLAUDE.local.md template: what the gitignored file should contain.
  5. Layout: full src/ tree, audited (shell command that verifies CLAUDE.md and the real tree stay in sync).
  6. Ocarina hierarchy: reminder Test → Suite → Campaign → Cycle.
  7. Test strategy: cycle structure, smoke gate, taxonomy.
  8. Running tests: CLI commands.
  9. Reports and screenshots: “one screenshot per drive_page” rule.
  10. Scenario fragments: when to extract a fragment.
  11. Data-driven tests: when to use the pattern, naming conventions.
  12. Shared mixins: SeleniumBackAndForwardNavigationMixin.
  13. Conventions: 12 concrete rules (TYPE_CHECKING guard, log factories, etc.).
  14. Hard-won rules: rules drawn from experience, with past examples.

Hard-won rules#

Verify SUT behaviour — don’t theorise#

CURA is open source. Before building on a server-side claim, read the PHP:

08.04 — Test strategy

08.04 — Test strategy#

Six test types, six distinct roles. Documented in CURA_TEST_STRATEGY.md §3.

Test types#

TypeExpected statusAuthoritative source
Happy pathPASSNominal flows described from the FRDs
Unhappy pathPASS (the test passes when the app correctly refuses)Existing validations work
Edge case / boundaryPASS or FAILDepending on the test, often accompanied by a note added to the FRDs
Business logic vulnerability / gap test (functional tests trying to “break” the product and identify gaps)Intentional FAILBehaviors a system should prevent but CURA doesn’t
Exploratory / observed-behaviourPASSDocuments current behavior when the spec doesn’t cover it
Permanent security regression fixturePASS (= security holds)Verifies a security fix holds (fixes that stay permanently in the test heritage)

Happy path#

Exercises the nominal flow with valid inputs and an authenticated user. Verifies that the system produces the correct output (confirmation, history entry, etc.).

08.05 — Security gaps (G-SEC-1 to G-SEC-3)

08.05 — Security gaps (G-SEC-1 to G-SEC-3)#

Three documented security gaps. Not actively exploited (cf. the rule “security testing is functional and static, never active”), but observed via functional tests.

G-SEC-1 — No CSRF token on the front-end forms#

Symptom#

The <form> rendered on Heroku does not contain a CSRF token.

Source#

views/page_appointment.php in the GitHub repo calls $antiCSRF->insertHiddenToken() before the submit button. But on the deployed version, this token does not appear in the HTML.

08.06 — Data gaps (G-DATA-1, G-DATA-2)

08.06 — Data gaps (G-DATA-1, G-DATA-2)#

Two major gaps on data integrity.

G-DATA-1 — No server-side validation of visit_date#

Source#

// appointment.php
array_diff_key($_POST, array(
    "facility" => "",
    "hospital_readmission" => "",
    "programs" => "",
    "visit_date" => "",
    "comment" => ""
))

This checks for key presence only. It never validates the value of visit_date.

Symptom#

  1. Empty visit_date: if you strip the HTML5 required attribute via JS and submit with an empty date → booking accepted.
  2. Past visit_date: if you send a backdated booking → booking accepted.
  3. The booking is added to history as-is.
TestMethod
Appointment - Server accepts empty date when client bypass appliedStrip the required attribute, send with empty booking date → expects rejection by the system (FAIL)
Appointment - Past date booking acceptedSend a booking request for yesterday → expects rejection by the system (FAIL)

Both tests fail intentionally — they document the gap.

08.07 — Spec gaps (G-SPEC-1 to G-SPEC-3)

08.07 — Spec gaps (G-SPEC-1 to G-SPEC-3)#

Three gaps on business behavior vs the expected spec of a healthcare system.

G-SPEC-1 — History sorted by submission order, not by date#

Source#

// views/page_history.php
foreach ($_SESSION['history'] as $appointment) {
    // ... render ...
}

No usort. No sort. No sort step exists anywhere.

appointment.php pushes with array_push($_SESSION['history'], $_POST) — append to the end (it’s a pushback).

Unmet spec#

REQ-HIST-3 from the FRD:

08.08 — Chrome BFCache (B-BROWSER-1) + environment artifacts

08.08 — Chrome BFCache (B-BROWSER-1) + environment artifacts#

Chrome restores a no-store page after logout, from the BFCache, even though it’s a page protected by authentication.

B-BROWSER-1 — Chrome’s BFCache is a pain#

Symptom#

After logout, clicking the back button to be redirected to a page that was protected by authentication (e.g. history.php):

  • Chrome: the page stays displayed with its contents as shown to a still-authenticated user. The URL remains history.php, and the server isn’t re-contacted to validate or invalidate access.
  • Firefox: a new request goes to the server, which sees the session is dead and redirects.

Very annoying gap#

history.php correctly sends:

08.09 — CI: ai_proof_ci.yml + ai_proof_e2e.yml

08.09 — CI: ai_proof_ci.yml + ai_proof_e2e.yml#

Two workflows. Fast PR (lint+typecheck). Manual e2e with a Firefox/Chrome matrix + Heroku warm-up + ChromeDriver stacktrace filtering.

ai_proof_ci.yml#

name: ai_proof CI

on:
  push:
    branches: [main]
  pull_request:
  workflow_dispatch:

defaults:
  run:
    shell: bash

jobs:
  lint-and-typecheck:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.14"
          allow-prereleases: true
      - uses: actions/cache@v4
        with:
          path: .venv
          key: venv-${{ runner.os }}-py${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('pyproject.toml') }}
      - name: Install dependencies
        if: steps.cache-venv.outputs.cache-hit != 'true'
        run: |
          python -m venv .venv
          .venv/bin/pip install . ruff mypy mypy-extensions typing-extensions pre-commit

      - name: Ruff format check
        run: .venv/bin/ruff format src/ --check

      - name: Ruff lint
        run: .venv/bin/ruff check src/

      - name: Mypy
        run: .venv/bin/mypy src/
StepCmd
Ruff format checkruff format src/ --check
Ruff lintruff check src/
Mypymypy src/

ai_proof_e2e.yml#

name: ai_proof e2e

on:
  workflow_dispatch:

defaults:
  run:
    shell: bash

jobs:
  e2e:
    name: e2e (${{ matrix.browser }})
    runs-on: ubuntu-latest
    env:
      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
      WAIT_TIMEOUT: 15
    strategy:
      fail-fast: false
      matrix:
        include:
          - browser: firefox
            driver: geckodriver
          - browser: chrome
            driver: chromedriver

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.14"
          allow-prereleases: true
      - uses: actions/cache@v4
        with:
          path: .venv
          key: venv-${{ runner.os }}-py${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('pyproject.toml') }}
      - name: Install dependencies
        if: steps.cache-venv.outputs.cache-hit != 'true'
        run: |
          python -m venv .venv
          .venv/bin/pip install . ruff mypy mypy-extensions typing-extensions pre-commit

      # --- Firefox ---

      - name: Install geckodriver
        if: matrix.browser == 'firefox'
        run: |
          GECKO_VERSION=0.36.0
          wget -q https://github.com/mozilla/geckodriver/releases/download/v$GECKO_VERSION/geckodriver-v$GECKO_VERSION-linux64.tar.gz
          tar -xzf geckodriver-v$GECKO_VERSION-linux64.tar.gz
          chmod +x geckodriver

      - name: Setup Firefox
        if: matrix.browser == 'firefox'
        uses: browser-actions/setup-firefox@fcf821c621167805dd63a29662bd7cb5676c81a8

      # --- Chrome ---

      - name: Install chromedriver
        if: matrix.browser == 'chrome'
        run: |
          CHROME_VER=$(google-chrome-stable --version | grep -oP '\d+\.\d+\.\d+\.\d+')
          wget -q -O chromedriver.zip \
            "https://storage.googleapis.com/chrome-for-testing-public/${CHROME_VER}/linux64/chromedriver-linux64.zip"
          unzip -q chromedriver.zip
          mv chromedriver-linux64/chromedriver ./chromedriver
          chmod +x ./chromedriver

      # --- Warm-up ---

      - name: Warm up Heroku dyno
        run: |
          # CURA runs on a Heroku dyno that sleeps after inactivity.
          # Tests hitting a cold dyno produce timeouts (false fails) and slow
          # confirmations (false passes on gap tests). Wake it explicitly
          # before any browser opens.
          curl -sf --retry 6 --retry-delay 5 --retry-all-errors \
            --max-time 30 https://katalon-demo-cura.herokuapp.com/ > /dev/null
          echo "Dyno is warm"

      # --- Run ---

      - name: Run e2e cycle
        run: |
          # ChromeDriver release builds ship without debug symbols. On errors it
          # dumps raw C++ stack frames ("#N 0x<addr> <unknown>") into the WebDriver
          # response, which Selenium surfaces as the exception message — polluting
          # both the live log and the pretty-print summary with unreadable hex noise.
          # No Python-side fix exists. Strip the frames with sed before they land.
          # pipefail preserves Python's exit code through the pipe.
          set -o pipefail
          .venv/bin/python -u src/main.py \
            --driver-path ./${{ matrix.driver }} \
            --browser ${{ matrix.browser }} \
            --workers 3 \
            --wait-timeout ${{ env.WAIT_TIMEOUT }} \
            --logger terminal+file \
            2>&1 | sed -u \
              -e '/Stacktrace:/{N;/\n#[0-9][0-9]* 0x[0-9a-f][0-9a-f]* <unknown>/{s/Stacktrace:\n[^\n]*/HIDDEN/;b};P;D}' \
              -e '/^#[0-9][0-9]* 0x[0-9a-f][0-9a-f]* <unknown>/d'

      - name: Enumerate traces
        if: always()
        run: |
          sync
          find $GITHUB_WORKSPACE/.screenshots -type f 2>/dev/null || true
          find $GITHUB_WORKSPACE/.ocarina_logs -type f 2>/dev/null || true
          find $GITHUB_WORKSPACE/.reports -type f 2>/dev/null || true

      - name: Upload screenshots
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: screenshots-${{ matrix.browser }}
          path: .screenshots/
          retention-days: 7

      # … upload logs, reports similarly ...

Analysis#

fail-fast: false#

strategy:
  fail-fast: false
  matrix:
    include:
      - browser: firefox
      - browser: chrome

fail-fast: false — if Firefox fails, Chrome continues. Essential: the point of the run is seeing both browsers, even when one fails.