00.01 — Ecosystem cartography#

The Ocarina ecosystem is six public repos under the mojo-molotov GitHub account. They cover five distinct roles:

The big picture#

┌─────────────────────────────────────────────────────────────────────────────────┐
│                              OCARINA (BIG PICTURE)                              │
└─────────────────────────────────────────────────────────────────────────────────┘

   ┌──────────────────────────────┐         ┌──────────────────────────────────┐
   │  [PACKAGE]  ocarina          │ ──────► │  [PACKAGE]  ocarina-example      │
   │  Python framework 3.14+      │   pip   │  e2e suite against the Igoristan │
   │  ROP / DSL / orchestration   │ install │  (Selenium + Firefox + Redis)    │
   └──────────────────────────────┘         └──────────────────────────────────┘
            │  │                                       │  ▲   │
            │  │                                       │  │  uses
            │  │ pip install ocarina                   │  │  IGOR_API_KEY
            │  ▼                                       ▼  │   │
            │  ┌────────────────────────────────────┐  drives │
            │  │ [PACKAGE]  ocarina-with-ai-example │  ┌──────┴──────────────────┐
            │  │ e2e suite against CURA Healthcare  │  │ [WEBSITE]  igoristan    │
            │  │ Claude Code, 99% machine-written   │  │ React 19 + Vike         │
            │  └────────────────────────────────────┘  │ GitHub Pages            │
            │              │                           └──────┬──────────────────┘
            │              │ drives                           │ fetches OTP
            │              ▼                                  ▼
            │   ┌──────────────────────────────┐    ┌──────────────────────────────┐
            │   │ [WEBSITE]  CURA Healthcare   │    │ [BACKEND]  tests-workers     │
            │   │ Heroku, open-source PHP      │    │ Vercel Edge Functions        │
            │   └──────────────────────────────┘    │ + Upstash Redis              │
            │                                       │ /api/otp, /otp-history,      │
            │                                       │ /api/corsicadex              │
            │                                       └──────────────────────────────┘
            │                                                 ▲
            │                                                 │ x-api-key
            │                                                 │ (IGOR_API_KEY ≡ API_SECRET)
            │                                                 │
            ▼
   ┌──────────────────────────────┐
   │  [DOCS]  ocarina-holy-book   │
   │  VitePress EN + FR + RU      │
   │  AI skills, PDFs, llms.txt   │
   │  GitHub Pages                │
   └──────────────────────────────┘

1. ocarina → example suites: plain pip dependency (pip install ocarina).
2. Example suites → SUT (Igoristan / CURA): browser automation through Selenium.
3. Igoristan suite ↔ tests-workers: HTTP calls (OTP, Corsicadex). Shared secret: IGOR_API_KEY client-side ≡ API_SECRET server-side.

Responsibilities#

Three licenses#

Deliberate friction between repos#

The design is intentionally asymmetric along certain axes. The friction is the testing tool:

06.01 — Vercel Edge stack#

package.json#

{
  "name": "tests-workers",
  "version": "1.0.0",
  "scripts": {
    "vercel-build": "echo 'Build skipped'"
  },
  "packageManager": "pnpm@11.2.2",
  "dependencies": {
    "@upstash/redis": "^1.36.2",
    "otplib": "^13.2.1"
  },
  "devDependencies": {
    "@types/node": "^25.2.0",
    "next": "^16.1.6",
    "@vercel/node": "^5.5.28",
    "typescript": "^5.9.3"
  },
  "license": "ISC"
}

No runtime dependency beyond @upstash/redis + otplib. Bundle is microscopic → cold start < 100ms.

06.02 — GET /api/otp#

Source file: api/otp.ts

Code#

import { generate } from "otplib";
import type { NextRequest } from "next/server";
import redis from "../lib/redis";
import isAuthorized from "../lib/isAuthorized";

export const config = { runtime: "edge" };

const SECRET_QUERY_PARAM_KEY = "secret";
const OTP_SECRET_MIN_LENGTH = 32;
const OTP_SECRET_MAX_LENGTH = 64;
const BASE32_ALPHABET = "ABCDEFGHIJKLMNOPQRSTUVWXYZ234567";
const OTP_TTL = 60 * 5; // 5 minutes
const OTP_HISTORY_TTL = OTP_TTL + 60; // 6 minutes
const MAX_FREE_PAYLOAD_SIZE = 4096;

const corsHeaders = {
  "Access-Control-Allow-Origin": "*",
  "Access-Control-Allow-Methods": "GET, OPTIONS",
  "Access-Control-Allow-Headers": "x-api-key, Content-Type",
} as const;

function generateRandomBase32Secret(length: number = 32): string {
  let secret = "";
  const randomValues = new Uint8Array(length);
  crypto.getRandomValues(randomValues);
  for (let i = 0; i < length; i++) {
    secret += BASE32_ALPHABET[randomValues[i] % 32];
  }
  return secret;
}

function ensureSecretLength(secret: string): string {
  const cleaned = secret.toUpperCase().replace(/\s/g, "");
  if (cleaned.length > OTP_SECRET_MAX_LENGTH) {
    const validLength = Math.floor(OTP_SECRET_MAX_LENGTH / 8) * 8;
    return cleaned.substring(0, validLength);
  }
  if (cleaned.length >= OTP_SECRET_MIN_LENGTH) {
    const validLength = Math.floor(cleaned.length / 8) * 8;
    return cleaned.substring(0, validLength);
  }
  const repetitions = Math.ceil(OTP_SECRET_MIN_LENGTH / cleaned.length);
  const repeated = cleaned.repeat(repetitions);
  const validLength = Math.floor(repeated.length / 8) * 8;
  return repeated.substring(0, Math.max(validLength, OTP_SECRET_MIN_LENGTH));
}

export default async function handler(request: NextRequest) {
  if (request.method === "OPTIONS") {
    return new Response(null, { status: 200, headers: corsHeaders });
  }

  if (!isAuthorized(request)) {
    return new Response(JSON.stringify({ error: "Unauthorized" }), {
      status: 401,
      headers: { "Content-Type": "application/json", ...corsHeaders },
    });
  }

  const { searchParams } = new URL(request.url);
  const freePayload: Record<string, string> = {};
  let payloadSize = 0;

  searchParams.forEach((value, key) => {
    if (key === SECRET_QUERY_PARAM_KEY) return;
    const entrySize = new TextEncoder().encode(key + value).length;
    if (payloadSize + entrySize > MAX_FREE_PAYLOAD_SIZE) {
      payloadSize += entrySize;
      return;
    }
    freePayload[key] = value;
    payloadSize += entrySize;
  });

  if (payloadSize > MAX_FREE_PAYLOAD_SIZE) {
    return new Response(
      JSON.stringify({
        error: "Bad Request",
        message: `Free payload exceeds maximum size of ${MAX_FREE_PAYLOAD_SIZE} bytes (got ${payloadSize} bytes)`,
      }),
      {
        status: 400,
        headers: { "Content-Type": "application/json", ...corsHeaders },
      },
    );
  }

  const rawSecret =
    searchParams.get(SECRET_QUERY_PARAM_KEY) ?? generateRandomBase32Secret();
  const secret = ensureSecretLength(rawSecret);
  const otpCode = await generate({ secret });
  const now = Date.now();
  const expiresAtMs = now + OTP_TTL * 1000;
  const createdAtTimestampLackingMsPrecision = new Date(
    now - (now % 1000),
  ).toISOString();

  const event = {
    ...freePayload,
    otpCode,
    secret,
    createdAtTimestampLackingMsPrecision,
    expiresAt: new Date(expiresAtMs).toISOString(),
  };

  const key = `otp:${now}:${crypto.randomUUID()}`;
  await redis.set(key, JSON.stringify(event), { ex: OTP_HISTORY_TTL });

  return new Response(JSON.stringify(event), {
    headers: { "Content-Type": "application/json", ...corsHeaders },
  });
}

Flow#

1. Request arrives
   ↓
2. If OPTIONS → 200 + CORS headers (preflight)
   ↓
3. isAuthorized(request)? No → 401
   ↓
4. Parse searchParams; everything except "secret" goes into freePayload
   ↓
5. payloadSize > 4 KB? → 400
   ↓
6. secret = ?secret=<X> (if given) else random Base32 32-chars
   ↓
7. ensureSecretLength: enforces 32..64 chars, multiple of 8
   ↓
8. otpCode = generate({ secret })   ← otplib TOTP RFC 6238
   ↓
9. createdAt = floor(now / 1000) * 1000  ← ms stripped
   ↓
10. event = { ...freePayload, otpCode, secret, createdAt, expiresAt }
    ↓
11. key = `otp:<now>:<uuid>`
    ↓
12. redis.set(key, JSON.stringify(event), { ex: 360 })  ← TTL 6 min
    ↓
13. Response = JSON.stringify(event), 200

CORS headers#

const corsHeaders = {
  "Access-Control-Allow-Origin": "*",
  "Access-Control-Allow-Methods": "GET, OPTIONS",
  "Access-Control-Allow-Headers": "x-api-key, Content-Type",
} as const;

• * origin: lets Igoristan (https://mojo-molotov.github.io) call the service.
• GET, OPTIONS: the only supported verbs.
• x-api-key listed explicitly — otherwise the browser blocks it as a non-simple header.

as const makes the object readonly on the TypeScript side.

05.03 — The fake useAuth + MFA OTP#

Igoristan’s useAuth hook is deliberately fake. Two mechanics to exercise Ocarina’s replay and worker coordination: a 10% random failure and an optional MFA OTP.

Code#

// src/hooks/useAuth.ts
const VALID_PASSWORD = "figatellu";
const NOT_AUTHENTICATED = -1;
const AUTHENTICATED_WITHOUT_MFA = 1;
const AUTHENTICATED_WITH_MFA = 2;

// * ... This is FAKED
export const useAuth = () => {
  const [_isAuthenticated, setIsAuthenticated] = useLocalStorage(
    "isAuthenticated",
    NOT_AUTHENTICATED,
  );
  const [isLoading, setIsLoading] = useState(true);

  const isAuthenticated = useCallback(
    () => _isAuthenticated !== NOT_AUTHENTICATED,
    [_isAuthenticated],
  );
  const isAuthenticatedWithMFA = useCallback(
    () => _isAuthenticated === AUTHENTICATED_WITH_MFA,
    [_isAuthenticated],
  );
  const logout = useCallback(
    () => setIsAuthenticated(NOT_AUTHENTICATED),
    [setIsAuthenticated],
  );

  useEffect(() => {
    const delay = 500 * randint(1, 5); // ◄── random delay 0.5–2.5s
    const timer = setTimeout(() => setIsLoading(false), delay);
    return () => clearTimeout(timer);
  }, []);

  const login: LoginFn = ({ pre = false, password, withMFA }) => {
    if (password === VALID_PASSWORD && Math.random() < 0.9) {
      // ◄── 10% fail despite correct password
      if (!pre) {
        setIsAuthenticated(
          withMFA ? AUTHENTICATED_WITH_MFA : AUTHENTICATED_WITHOUT_MFA,
        );
      }
      return true;
    }
    return false;
  };

  return { isAuthenticatedWithMFA, isAuthenticated, isLoading, logout, login };
};

Mechanics#

1. LocalStorage#

const [_isAuthenticated, setIsAuthenticated] = useLocalStorage(
  "isAuthenticated",
  NOT_AUTHENTICATED,
);

Three values: -1 (not authenticated), 1 (no MFA), 2 (with MFA).
Persisted in localStorage, easy to inspect for testing / manual debugging.

06.03 — GET /api/otp-history#

Source file: api/otp-history.ts

Code#

import type { NextRequest } from "next/server";
import redis from "../lib/redis";
import isAuthorized from "../lib/isAuthorized";

export const config = { runtime: "edge" };

const corsHeaders = {
  "Access-Control-Allow-Origin": "*",
  "Access-Control-Allow-Methods": "GET, OPTIONS",
  "Access-Control-Allow-Headers": "x-api-key, Content-Type",
} as const;

export default async function handler(request: NextRequest) {
  if (request.method === "OPTIONS") {
    return new Response(null, { status: 200, headers: corsHeaders });
  }

  if (!isAuthorized(request)) {
    return new Response(JSON.stringify({ error: "Unauthorized" }), {
      status: 401,
      headers: { "Content-Type": "application/json", ...corsHeaders },
    });
  }

  let cursor = "0";
  const allEvents = [];
  const BATCH_SIZE = 100;

  do {
    const [newCursor, keys] = await redis.scan(cursor, {
      match: "otp:*",
      count: BATCH_SIZE,
    });
    cursor = newCursor;

    if (keys.length > 0) {
      const values = await redis.mget(...keys);
      allEvents.push(...values.filter(Boolean));
    }
  } while (cursor !== "0");

  return new Response(JSON.stringify(allEvents), {
    headers: { "Content-Type": "application/json", ...corsHeaders },
  });
}

Redis SCAN mechanics#

let cursor = "0";
const allEvents = [];
const BATCH_SIZE = 100;

do {
  const [newCursor, keys] = await redis.scan(cursor, {
    match: "otp:*",
    count: BATCH_SIZE,
  });
  cursor = newCursor;

  if (keys.length > 0) {
    const values = await redis.mget(...keys);
    allEvents.push(...values.filter(Boolean));
  }
} while (cursor !== "0");

1. cursor = "0" (initialization).
2. Loop:
   • SCAN <cursor> MATCH otp:* COUNT 100 returns [newCursor, keys].
   • If keys non-empty: MGET k1 k2 ... fetches the values.
   • cursor = newCursor.
3. We stop when cursor === "0" (cycle complete).

Why SCAN rather than KEYS?

07.03 — Dashboard login scenarios#

Three families: happy paths, unhappy paths, data-driven multi-login. All exercise the 10%-fail useAuth and OTP coordination.

dashboard_login#

# src/tests/campaigns/dashboard_login.py
def create_igoristan_login_campaign(*, drivers_pool: SeleniumWebDriversPool) -> TestCampaign:
    return TestCampaign(
        name="Dashboard login",
        suites=[
            create_igoristan_login_happy_paths_test_suite(drivers_pool=drivers_pool),
            create_igoristan_login_unhappy_paths_test_suite(drivers_pool=drivers_pool),
            create_igoristan_login_data_driven_test_suite(drivers_pool=drivers_pool),
        ],
    )

Happy paths#

# src/tests/suites/dashboard/access/happy_paths.py
def create_igoristan_login_happy_paths_test_suite(*, drivers_pool) -> TestSuite:
    return TestSuite(
        name="Login happy paths",
        tests=[
            test_dashboard_login_without_otp_happy_path,
            test_dashboard_login_with_otp_happy_path,
            test_dashboard_login_page_back_to_igoristan_button,
        ],
        drivers_pool=drivers_pool,
    )

Test 1: login without OTP#

def dashboard_login_without_otp_happy_path(driver, logger):
    dashboard_creds = create_env_getters().get_credentials("dashboard")
    on_dashboard_login_page = DashboardLoginPage(driver=driver)
    on_dashboard_welcome_page = DashboardWelcomePage(driver=driver)

    just_log_error = create_just_log_error(logger=logger)
    log_error_with_current_url = create_log_error_with_current_url(logger=logger, driver=driver)
    just_log_success = create_just_log_success(logger=logger)
    log_success_with_current_url_and_take_screenshot = create_log_success_with_current_url_and_take_screenshot(...)

    retries_amount = max(get_max_workers(), 10)

    return [
        drive_page(
            act(on_dashboard_login_page, open_dashboard_login_page)
                .failure(just_log_error("Failed to open the dashboard login page..."))
                .success(just_log_success("Opened the dashboard login page!")),
            act(on_dashboard_login_page, verify_dashboard_login_page)
                .failure(log_error_with_current_url("Failed to verify..."))
                .success(log_success_with_current_url_and_take_screenshot("Verified!")),
            act(on_dashboard_login_page,
                login_without_otp_and_with_retries(dashboard_creds, retries_amount, logger=logger))
                .failure(just_log_error("Failed to connect to the dashboard without OTP..."))
                .success(just_log_success("Connected to the dashboard!")),
        ),
        drive_page(
            act(on_dashboard_welcome_page, verify_dashboard_welcome_page)
                .failure(log_error_with_current_url("Failed to verify..."))
                .success(log_success_with_current_url_and_take_screenshot("Verified!")),
        ),
    ]

1. retries_amount = max(get_max_workers(), 10): at least 10 internal login retries to beat the 10% random failure.
2. login_without_otp_and_with_retries(creds, retries_amount, logger=logger): closure capturing creds, retry count, and logger. Returns Callable[[DashboardLoginPage], DashboardLoginPage].
3. 2 drive_pages: login page then welcome page. Each opens, verifies, screenshots.
4. Log factories everywhere: no inline lambdas.
5. SeleniumTitleMixin: get_current_title() feeds act’s on_failure hook.

Test 2: login with OTP#

def dashboard_login_with_otp_happy_path(driver, logger):
    # ... same setup ...

    cache = in_memory_cache_with_30m_ttl
    fresh_cache_key_for_username = reserve_free_cache_key(cache)
    fresh_cache_key_for_otp_send_button_click_date = reserve_free_cache_key(cache)

    return [
        drive_page(
            act(on_dashboard_login_page, open_dashboard_login_page)...,
            act(on_dashboard_login_page, verify_dashboard_login_page)...,
            act(on_dashboard_login_page,
                start_to_login_with_otp_and_with_retries(
                    dashboard_creds, retries_amount,
                    cache=cache, logger=logger,
                    username_cache_key=fresh_cache_key_for_username,
                    otp_send_button_click_date_cache_key=fresh_cache_key_for_otp_send_button_click_date,
                ))...,
            act(on_dashboard_login_page, verify_otp_screen)...,
            act(on_dashboard_login_page,
                type_otp_with_retries(
                    retries_amount,
                    cache=cache, logger=logger,
                    username_cache_key=fresh_cache_key_for_username,
                    otp_send_button_click_date_cache_key=fresh_cache_key_for_otp_send_button_click_date,
                ))...,
        ),
        drive_page(
            act(on_dashboard_welcome_page, verify_dashboard_welcome_page)...,
            act(on_dashboard_welcome_page, click_on_go_to_nested_page_btn)...,
        ),
        drive_page(
            act(on_dashboard_protected_page, verify_dashboard_protected_page)...,
        ),
    ]

Three drive_pages, five acts in the first. L1 cache + reserved keys share values between acts (see 08-caches-locks.md).

00.04 — Relations between the six repositories#

Every edge of the graph is a contract — a secret, a URL, a payload type, or a version. One pair per section below.

ocarina ↔ ocarina-example#

The example writes its own adapters on top of the framework:

• lib/ext/ocarina/adapters/agnostic/act.py → adds an on_failure hook that turns an HTTP error page (title matched by ERROR_PAGE_REGEX) into HttpErrorPageReachedError.
• lib/ext/ocarina/adapters/agnostic/match_page.py → create_match_page(raised_exceptions=transient_errors).
• lib/ext/ocarina/adapters/agnostic/env_getters.py → typed EnvGetters[_CredsKeys, _ValuesKeys].
• lib/ext/ocarina/adapters/selenium/test_suite.py → pins max_retries_per_test=8, transient_errors=..., autoscreen_on_fail=True, propagates --only/--exclude.
• lib/ext/ocarina/adapters/selenium/test_campaign.py → pins max_workers=get_max_workers().

ocarina ↔ ocarina-with-ai-example#

• pip install . ruff mypy mypy-extensions typing-extensions pre-commit — the AI example pins its dep in pyproject.toml: ocarina>=1.0.3.
• act adapter is minimal here. No on_failure (CURA doesn’t throw deliberately random HTTP error pages). See ../08-ai-example/
• create_drivers_pool adapter is overridden to build a clean Chrome (password manager off, leak detection off). See ../08-ai-example/06-data-gaps.md

ocarina-example ↔ igoristan#

src/constants/pages/:

06.07 — OTP coordination flow + deliberate imprecision#

The backend’s reason to exist: let N parallel Ocarina workers retrieve the right OTP for their user, even when several OTPs are generated.

Flow#

   ┌───────────────────────────────────────────────────────────────────┐
   │               Worker (among --workers 3 of Ocarina)               │
   └───────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │ Selenium opens the Dashboard login page                           │
   └─────────────────────────────────┬─────────────────────────────────┘
                                     ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │ Acquire the distributed Redis lock (OTP_SEND_LOCK_KEY)            │
   │   → only this worker can click "Send OTP" during ACQ              │
   └─────────────────────────────────┬─────────────────────────────────┘
                                     ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │ min_utc_date = datetime.now(UTC)                                  │
   └─────────────────────────────────┬─────────────────────────────────┘
                                     ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │ L1 cache: record min_utc_date + username in the cache             │
   │   (keys reserved by reserve_free_cache_key)                       │
   └─────────────────────────────────┬─────────────────────────────────┘
                                     ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │ Selenium types username + password + ticks OTP + click "REQ OTP"  │
   └─────────────────────────────────┬─────────────────────────────────┘
                                     ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │ IGORISTAN UI: fetch /api/otp?_user=<username>                     │
   │   (x-api-key typed by Selenium into the UI)                       │
   └─────────────────────────────────┬─────────────────────────────────┘
                                     ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │ TESTS-WORKERS /api/otp:                                           │
   │   generate(secret) → otpCode                                      │
   │   createdAt = floor(now/1000)*1000  ← ms stripped                 │
   │   event = { _user, otpCode, createdAt, expiresAt, ... }           │
   │   redis.set("otp:<now>:<uuid>", JSON, EX 360)                     │
   │   return event                                                    │
   └──────────────────────────────────┬────────────────────────────────┘
                                      ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │ IGORISTAN UI: displays OTP screen                                 │
   └──────────────────────────────────┬────────────────────────────────┘
                                      ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │ Release the OTP_SEND_LOCK_KEY Redis lock                          │
   └──────────────────────────────────┬────────────────────────────────┘
                                      ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │ Selenium: retrieve_dashboard_otp_code(min_utc_date, _user)        │
   │   ↓                                                               │
   │   GET /api/otp-history  (x-api-key = IGOR_API_KEY)                │
   │   ↓                                                               │
   │   TESTS-WORKERS: SCAN otp:* + MGET → all events                   │
   │   ↓                                                               │
   │   client-side filter by _user                                     │
   │   filter createdAt >= min_utc_date - 1s                           │
   │   sort ASC by createdAt                                           │
   │   return first.otpCode                                            │
   └──────────────────────────────────┬────────────────────────────────┘
                                      ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │ Selenium types the OTP into the Igoristan UI                      │
   │   → AUTHENTICATED_WITH_MFA                                        │
   └───────────────────────────────────────────────────────────────────┘

Race conditions#

1. Worker A: min_utc_date_A = 13:27:53.123, generates OTP_A at 13:27:53.250.
2. Worker B: min_utc_date_B = 13:27:53.130, generates OTP_B at 13:27:53.470.

Without coordination, A could pick up OTP_B instead of OTP_A — the timestamps are close and truncated to the second.

Chapter 06 — tests-workers, Vercel Edge backend#

Three HTTP endpoints, two libs (@upstash/redis, otplib), zero Next.js app code. The bare minimum to coordinate parallel tests against Igoristan.

Outline#

Goals#

• 3 .ts endpoints.
• 2 lib/ files.
• 1 consts/ file.
• Total: ~7 TypeScript code files, plus package.json and vercel.json.

Goal: deployable in 30 seconds by a human.