Ai-Example on Ocarina — manual

08.01 — Code: 99% Claude, 1% Igor, intelligence: 50-50

Wed, 20 May 2026 00:00:00 +0000

08.01 — Code: 99% Claude, 1% Igor, intelligence: 50-50#

README#

A note to the reader#

This project is an experiment in AI-driven test engineering. In the interest of honesty about how it was built:

Claude Code Human

Code written 99% 1%

Intelligence 50% 50%

Almost every line was machine-written. The judgement behind it — what to test, what to distrust, when to dig and when to stop — was shared.

	Claude Code	Human
Code written	99%	1%
Intelligence	50%	50%

08.02 — CURA Healthcare (SUT)

Wed, 20 May 2026 00:00:00 +0000

08.02 — CURA Healthcare (SUT)#

Ocarina’s first victim#

Hosting: https://katalon-demo-cura.herokuapp.com/
Source: https://github.com/katalon-studio/katalon-demo-cura (PHP, open source, MIT)
Host: Heroku eco-dyno (sleeps after inactivity)
Credentials: John Doe / ThisIsNotAPassword (public, hardcoded on the login page)

Perimeter#

Page	Function
`/`	Marketing homepage
`/profile.php#login`	Login
`/appointment.php`	Booking form (facility, programs, hospital_readmission, visit_date, comment)
`/confirmation.php`	Booking confirmation
`/history.php`	The user’s booking history
`/profile.php#profile`	Profile page (empty placeholder — gap §G-SPEC-2)

Why target CURA#

Open source — we can read the PHP to understand why a given behavior exists.
Plenty of gaps — rich territory for findings.
Public demo — no authorization needed. Stable (maintained by Katalon).

There’s also a tacit reason: Katalon is a competing test-framework vendor (Katalon Studio). It’s poetic to test their demo app with Ocarina.

08.03 — Documentation

Wed, 20 May 2026 00:00:00 +0000

08.03 — Documentation#

The AI project is the most documented in the Ocarina ecosystem.

Listing#

Document	Status	Audience
`CLAUDE.md`	Claude ↔ project working contract	Claude + any contributor
`CURA_FRD.md`	Reconstructed spec	Maintainers, stakeholders
`CURA_TEST_STRATEGY.md`	Test strategy	Maintainers
`IDENTIFIED_GAPS.md`	Technical inventory of gaps	Maintainers

`CLAUDE.md`#

Key documents: points to README.md, CURA_FRD.md, CURA_TEST_STRATEGY.md, IDENTIFIED_GAPS.md, CLAUDE.local.md.
Project: 2 sentences on the project (CURA, Python 3.14+, demo creds).
Project philosophy: “No tricks or hacks. Teach the pattern, not the symptom.”
CLAUDE.local.md template: what the gitignored file should contain.
Layout: full src/ tree, audited (shell command that verifies CLAUDE.md and the real tree stay in sync).
Ocarina hierarchy: reminder Test → Suite → Campaign → Cycle.
Test strategy: cycle structure, smoke gate, taxonomy.
Running tests: CLI commands.
Reports and screenshots: “one screenshot per drive_page” rule.
Scenario fragments: when to extract a fragment.
Data-driven tests: when to use the pattern, naming conventions.
Shared mixins: SeleniumBackAndForwardNavigationMixin.
Conventions: 12 concrete rules (TYPE_CHECKING guard, log factories, etc.).
Hard-won rules: rules drawn from experience, with past examples.

Hard-won rules#

“Verify SUT behaviour — don’t theorise”#

CURA is open source. Before building on a server-side claim, read the PHP:

08.04 — Test strategy

Wed, 20 May 2026 00:00:00 +0000

08.04 — Test strategy#

Six test types, six distinct roles. Documented in CURA_TEST_STRATEGY.md §3.

Test types#

Type	Expected status	Authoritative source
Happy path	PASS	Nominal flows described from the FRDs
Unhappy path	PASS (the test passes when the app correctly refuses)	Existing validations work
Edge case / boundary	PASS or FAIL	Depending on the test, often accompanied by a note added to the FRDs
Business logic vulnerability / gap test (functional tests trying to “break” the product and identify gaps)	Intentional FAIL	Behaviors a system should prevent but CURA doesn’t
Exploratory / observed-behaviour	PASS	Documents current behavior when the spec doesn’t cover it
Permanent security regression fixture	PASS (= security holds)	Verifies a security fix holds (fixes that stay permanently in the test heritage)

Happy path#

Exercises the nominal flow with valid inputs and an authenticated user. Verifies that the system produces the correct output (confirmation, history entry, etc.).

08.05 — Security gaps (G-SEC-1 to G-SEC-3)

Wed, 20 May 2026 00:00:00 +0000

08.05 — Security gaps (G-SEC-1 to G-SEC-3)#

Three documented security gaps. Not actively exploited (cf. the rule “security testing is functional and static, never active”), but observed via functional tests.

G-SEC-1 — No CSRF token on the front-end forms#

Symptom#

The <form> rendered on Heroku does not contain a CSRF token.

Source#

views/page_appointment.php in the GitHub repo calls $antiCSRF->insertHiddenToken() before the submit button. But on the deployed version, this token does not appear in the HTML.

08.06 — Data gaps (G-DATA-1, G-DATA-2)

Wed, 20 May 2026 00:00:00 +0000

08.06 — Data gaps (G-DATA-1, G-DATA-2)#

Two major gaps on data integrity.

G-DATA-1 — No server-side validation of `visit_date`#

Source#

// appointment.php
array_diff_key($_POST, array(
    "facility" => "",
    "hospital_readmission" => "",
    "programs" => "",
    "visit_date" => "",
    "comment" => ""
))

This checks for key presence only. It never validates the value of visit_date.

Symptom#

Empty visit_date: if you strip the HTML5 required attribute via JS and submit with an empty date → booking accepted.
Past visit_date: if you send a backdated booking → booking accepted.
The booking is added to history as-is.

Test	Method
`Appointment - Server accepts empty date when client bypass applied`	Strip the `required` attribute, send with empty booking date → expects rejection by the system (FAIL)
`Appointment - Past date booking accepted`	Send a booking request for yesterday → expects rejection by the system (FAIL)

Both tests fail intentionally — they document the gap.

08.07 — Spec gaps (G-SPEC-1 to G-SPEC-3)

Wed, 20 May 2026 00:00:00 +0000

08.07 — Spec gaps (G-SPEC-1 to G-SPEC-3)#

Three gaps on business behavior vs the expected spec of a healthcare system.

G-SPEC-1 — History sorted by submission order, not by date#

Source#

// views/page_history.php
foreach ($_SESSION['history'] as $appointment) {
    // ... render ...
}

No usort. No sort. No sort step exists anywhere.

appointment.php pushes with array_push($_SESSION['history'], $_POST) — append to the end (it’s a pushback).

Unmet spec#

REQ-HIST-3 from the FRD:

08.08 — Chrome BFCache (B-BROWSER-1) + environment artifacts

Wed, 20 May 2026 00:00:00 +0000

08.08 — Chrome BFCache (B-BROWSER-1) + environment artifacts#

Chrome restores a no-store page after logout, from the BFCache, even though it’s a page protected by authentication.

B-BROWSER-1 — Chrome’s BFCache is a pain#

Symptom#

After logout, clicking the back button to be redirected to a page that was protected by authentication (e.g. history.php):

Chrome: the page stays displayed with its contents as shown to a still-authenticated user. The URL remains history.php, and the server isn’t re-contacted to validate or invalidate access.
Firefox: a new request goes to the server, which sees the session is dead and redirects.

Very annoying gap#

history.php correctly sends:

08.09 — CI: ai_proof_ci.yml + ai_proof_e2e.yml

Wed, 20 May 2026 00:00:00 +0000

08.09 — CI: `ai_proof_ci.yml` + `ai_proof_e2e.yml`#

Two workflows. Fast PR (lint+typecheck). Manual e2e with a Firefox/Chrome matrix + Heroku warm-up + ChromeDriver stacktrace filtering.

`ai_proof_ci.yml`#

name: ai_proof CI

on:
  push:
    branches: [main]
  pull_request:
  workflow_dispatch:

defaults:
  run:
    shell: bash

jobs:
  lint-and-typecheck:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.14"
          allow-prereleases: true
      - uses: actions/cache@v4
        with:
          path: .venv
          key: venv-${{ runner.os }}-py${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('pyproject.toml') }}
      - name: Install dependencies
        if: steps.cache-venv.outputs.cache-hit != 'true'
        run: |
          python -m venv .venv
          .venv/bin/pip install . ruff mypy mypy-extensions typing-extensions pre-commit

      - name: Ruff format check
        run: .venv/bin/ruff format src/ --check

      - name: Ruff lint
        run: .venv/bin/ruff check src/

      - name: Mypy
        run: .venv/bin/mypy src/

Step	Cmd
Ruff format check	`ruff format src/ --check`
Ruff lint	`ruff check src/`
Mypy	`mypy src/`

`ai_proof_e2e.yml`#

name: ai_proof e2e

on:
  workflow_dispatch:

defaults:
  run:
    shell: bash

jobs:
  e2e:
    name: e2e (${{ matrix.browser }})
    runs-on: ubuntu-latest
    env:
      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
      WAIT_TIMEOUT: 15
    strategy:
      fail-fast: false
      matrix:
        include:
          - browser: firefox
            driver: geckodriver
          - browser: chrome
            driver: chromedriver

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.14"
          allow-prereleases: true
      - uses: actions/cache@v4
        with:
          path: .venv
          key: venv-${{ runner.os }}-py${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('pyproject.toml') }}
      - name: Install dependencies
        if: steps.cache-venv.outputs.cache-hit != 'true'
        run: |
          python -m venv .venv
          .venv/bin/pip install . ruff mypy mypy-extensions typing-extensions pre-commit

      # --- Firefox ---

      - name: Install geckodriver
        if: matrix.browser == 'firefox'
        run: |
          GECKO_VERSION=0.36.0
          wget -q https://github.com/mozilla/geckodriver/releases/download/v$GECKO_VERSION/geckodriver-v$GECKO_VERSION-linux64.tar.gz
          tar -xzf geckodriver-v$GECKO_VERSION-linux64.tar.gz
          chmod +x geckodriver

      - name: Setup Firefox
        if: matrix.browser == 'firefox'
        uses: browser-actions/setup-firefox@fcf821c621167805dd63a29662bd7cb5676c81a8

      # --- Chrome ---

      - name: Install chromedriver
        if: matrix.browser == 'chrome'
        run: |
          CHROME_VER=$(google-chrome-stable --version | grep -oP '\d+\.\d+\.\d+\.\d+')
          wget -q -O chromedriver.zip \
            "https://storage.googleapis.com/chrome-for-testing-public/${CHROME_VER}/linux64/chromedriver-linux64.zip"
          unzip -q chromedriver.zip
          mv chromedriver-linux64/chromedriver ./chromedriver
          chmod +x ./chromedriver

      # --- Warm-up ---

      - name: Warm up Heroku dyno
        run: |
          # CURA runs on a Heroku dyno that sleeps after inactivity.
          # Tests hitting a cold dyno produce timeouts (false fails) and slow
          # confirmations (false passes on gap tests). Wake it explicitly
          # before any browser opens.
          curl -sf --retry 6 --retry-delay 5 --retry-all-errors \
            --max-time 30 https://katalon-demo-cura.herokuapp.com/ > /dev/null
          echo "Dyno is warm"

      # --- Run ---

      - name: Run e2e cycle
        run: |
          # ChromeDriver release builds ship without debug symbols. On errors it
          # dumps raw C++ stack frames ("#N 0x<addr> <unknown>") into the WebDriver
          # response, which Selenium surfaces as the exception message — polluting
          # both the live log and the pretty-print summary with unreadable hex noise.
          # No Python-side fix exists. Strip the frames with sed before they land.
          # pipefail preserves Python's exit code through the pipe.
          set -o pipefail
          .venv/bin/python -u src/main.py \
            --driver-path ./${{ matrix.driver }} \
            --browser ${{ matrix.browser }} \
            --workers 3 \
            --wait-timeout ${{ env.WAIT_TIMEOUT }} \
            --logger terminal+file \
            2>&1 | sed -u \
              -e '/Stacktrace:/{N;/\n#[0-9][0-9]* 0x[0-9a-f][0-9a-f]* <unknown>/{s/Stacktrace:\n[^\n]*/HIDDEN/;b};P;D}' \
              -e '/^#[0-9][0-9]* 0x[0-9a-f][0-9a-f]* <unknown>/d'

      - name: Enumerate traces
        if: always()
        run: |
          sync
          find $GITHUB_WORKSPACE/.screenshots -type f 2>/dev/null || true
          find $GITHUB_WORKSPACE/.ocarina_logs -type f 2>/dev/null || true
          find $GITHUB_WORKSPACE/.reports -type f 2>/dev/null || true

      - name: Upload screenshots
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: screenshots-${{ matrix.browser }}
          path: .screenshots/
          retention-days: 7

      # … upload logs, reports similarly ...

Analysis#

`fail-fast: false`#

strategy:
  fail-fast: false
  matrix:
    include:
      - browser: firefox
      - browser: chrome

fail-fast: false — if Firefox fails, Chrome continues. Essential: the point of the run is seeing both browsers, even when one fails.

Ai-Example on Ocarina — manual

08.01 — Code: 99% Claude, 1% Igor, intelligence: 50-50

08.01 — Code: 99% Claude, 1% Igor, intelligence: 50-50#

README#

A note to the reader#

08.02 — CURA Healthcare (SUT)

08.02 — CURA Healthcare (SUT)#

Ocarina’s first victim#

Perimeter#

Why target CURA#

08.03 — Documentation

08.03 — Documentation#

Listing#

CLAUDE.md#

Hard-won rules#

“Verify SUT behaviour — don’t theorise”#

08.04 — Test strategy

08.04 — Test strategy#

Test types#

Happy path#

08.05 — Security gaps (G-SEC-1 to G-SEC-3)

08.05 — Security gaps (G-SEC-1 to G-SEC-3)#

G-SEC-1 — No CSRF token on the front-end forms#

Symptom#

Source#

08.06 — Data gaps (G-DATA-1, G-DATA-2)

08.06 — Data gaps (G-DATA-1, G-DATA-2)#

G-DATA-1 — No server-side validation of visit_date#

Source#

Symptom#

Related tests#

08.07 — Spec gaps (G-SPEC-1 to G-SPEC-3)

08.07 — Spec gaps (G-SPEC-1 to G-SPEC-3)#

G-SPEC-1 — History sorted by submission order, not by date#

Source#

Unmet spec#

08.08 — Chrome BFCache (B-BROWSER-1) + environment artifacts

08.08 — Chrome BFCache (B-BROWSER-1) + environment artifacts#

B-BROWSER-1 — Chrome’s BFCache is a pain#

Symptom#

Very annoying gap#

08.09 — CI: ai_proof_ci.yml + ai_proof_e2e.yml

08.09 — CI: ai_proof_ci.yml + ai_proof_e2e.yml#

ai_proof_ci.yml#

ai_proof_e2e.yml#

Analysis#

fail-fast: false#

`CLAUDE.md`#

G-DATA-1 — No server-side validation of `visit_date`#

08.09 — CI: `ai_proof_ci.yml` + `ai_proof_e2e.yml`#

`ai_proof_ci.yml`#

`ai_proof_e2e.yml`#

`fail-fast: false`#