How Cradle was rebuilt and measured

The 2022 code is preserved in legacy/ exactly as it stood when work stopped (one database password removed). Every rule the site enforces is ported from those files, and parity tests read them directly.

The seed (web/data/seed.db) is a deterministic SQLite file: 19 users (the system inviter, the two real founders, two shared demo accounts and 14 fictional members), 21 invite codes and 5 demo mails. Its timings are synthetic. Details, and what it must not be used for, are in the data card.

Live data on the production site comes from visitors using the demo. Nothing is emailed, visitors are asked not to enter real details, and admins see every table with hashes and tokens hidden.

Cradle is invite-only: you join with a code from a member, and each member can hold two unused codes at a time. The checks, their order and every status message are the 2022 ones (DR-002). Two rules are enforced inside single SQL statements so they hold when requests race: the quota check is part of the insert, and a code is consumed by a conditional update in the sign-up transaction.

The stack moved from Express + MongoDB to one Next.js app on SQLite / Turso (DR-001), sessions became server-side, hashed and rotating (DR-003), passwords moved to argon2id (DR-004), and security events go to an append-only log (DR-006).

Before release, two independent reviews tried to break the upgrade on a production build. Everything they found that let the AI features, the SQL sandbox or the estimates fail open was fixed at its cause, with a test that reproduces the original problem (DR-008).

Unit and parity tests

The ported rules, code generation, validation limits and status tables, checked against the preserved 2022 files.

Integration tests

The service layer on a real SQLite file with the committed migrations: sign-up order, racing sign-ups, racing code requests, mail ownership, hashed sessions, rotation, triggers, rate limits.

Property-based tests (fast-check)

Seed 20220107. 1,000 random operation sequences against the pure rules; 100 runs of up to 8 rounds of 4 concurrent operations against SQLite; 1,000 random member graphs (cycles included); 300 tampered audit chains; 500 rate-limit streams.

What “no counterexample” is worth

Under the generators, the exact one-sided 95% upper bound on the chance a single scenario breaks an invariant is 0.30% for the model runs and 3.0% for the database runs. Evidence, not proof. A deliberately broken quota rule is caught, so the properties have teeth. Admins can check the same invariants against the live data on /admin/invariants.

Text-to-SQL evaluation

24 questions (cradle-sql-eval/v2) with hand-written reference SQL run against a frozen copy of the seed. A model's query scores when its result matches the reference (same rows, column names and order ignored, row order only when asked, numbers to four decimals). Accuracy carries a Wilson interval; two configurations are compared as paired data. Rate limits, network errors and answers from a fallback model are excluded and counted separately rather than scored as wrong, and each question gets one generation, so the intervals describe which questions were asked, not run-to-run variation. With 24 questions, 17 correct is 70.8% (95% CI 50.8–85.1%): only large differences can be told apart. No results are published: there is no budget for API calls, so numbers exist only when a visitor runs the harness with their own key. Admins can run it at /admin/ask/eval.

Password hashing benchmark

40 paired rounds on Apple M4 (Node v26.11.0): bcryptjs cost 10 61.5 ms (95% CI 58.7–63.1) per hash, argon2id 11.0 ms (95% CI 10.2–11.7); ratio of medians 0.18 (95% CI 0.17–0.19). The comparison mixes scheme and implementation (pure JavaScript against native Rust) and comes from a laptop, so it says this app's sign-in got cheaper in CPU time, nothing more. See DR-004.

• Counts are a census, not estimates. The member count on the landing page is every row in the database, so it has no interval. Figures that generalise (how likely a code is to be used, how accurate a model is) are estimates and always come with a 95% interval and their n.
• Proportions use the Wilson score interval, which behaves at small n and near 0 or 1 where the textbook Wald interval doesn't.
• Medians, means and ratios use bootstrap percentile intervals with 2,000 resamples and a fixed seed (20220107), so every interval on the site can be reproduced. Percentile intervals under-cover a little at small n.
• Open cases are censored, not failures. An invite code that hasn't been used yet has only been waiting so far, so /tree estimates time to redeem with a Kaplan–Meier median (checked against R's survival package) and counts redemption within 7 days only for codes at least 7 days old.
• No variation, no interval. When every observation is the same (the seed's synthetic 26-hour timings), a bootstrap interval has zero width; the site says so instead of printing a 95% interval that looks like certainty.
• Paired comparisons (two models on the same questions, two hash schemes in the same rounds) resample pairs together, and binary outcomes use the exact McNemar test. Effects are reported as differences or ratios with intervals, not as p-values alone.
• Verified against reference software. The helpers in web/src/lib/stats/ are unit-tested against values from SciPy 1.18.1 and NumPy 2.5.3 (generated by scripts/stats_reference.py with uv), and spot-checked against R.

• Server-side sessions: a signed httpOnly cookie points at a row that stores only the SHA-256 of the session id; 24-hour absolute limit; new id every 15 minutes of use, on sign-in and after a password change; “sign out everywhere”.
• CSRF: Next.js's Origin check, our own Origin and Fetch Metadata check (also on the JSON routes), and a per-session token in every signed-in form.
• Passwords: argon2id (19 MiB, 2 passes); 2022-era bcrypt hashes are upgraded on the next sign-in.
• Rate limits shared by every server instance, per IP and per account (DR-007).
• An append-only, hash-chained audit log, verified each time an admin opens it. A failed sign-in with a username that doesn't exist is recorded as a keyed hash, never as the typed text.
• The SQL sandbox runs a query only if a worst-case bound on the rows it can visit, read from SQLite's query plan, is at most 2,000,000; recursion and table-valued functions are refused from the plan (DR-008).
• In production, a Content-Security-Policy stops page scripts from opening fetch, XHR, beacon or WebSocket connections to hosts other than this site and the two AI providers. It doesn't cover image loads or navigations, so on its own it doesn't stop a stolen key from leaving the page.

Optional and paid by you. Everything works without AI. If you add your own Anthropic key (default model Claude Haiku 4.5) or OpenAI key, calls go from your browser straight to the provider. The key stays in your browser (session storage, cleared when the tab closes or you sign out, unless you choose “remember on this device”) and is never sent to Cradle's server or logged. Each provider keeps its own key, and a key is only ever sent to its own provider.

Data sent to the provider. For a note: your username, the tone and an optional hint. For SQL: your question and the sandbox schema (names of tables and columns). Never rows, never the invite code, never your friend's address.

Labelled and logged. Every output is marked “AI-generated” with the model that wrote it (both models, if a fallback answered). After each call, including failures, your browser reports it to the AI log with what was sent and received, latency, tokens and the human decision; rows can't be edited or deleted, and the log exports to CSV or JSON. The log fails closed: the browser checks first that the call can be logged, and an answer whose entry can't be written is never shown or used. Because the call itself goes from your browser to the provider, the server can't confirm that every call was reported; the log is what the browser reports.

Who can read the log. The demo admin account is open to every visitor and can read every member's AI log, which can't be deleted. On the shared demo accounts, text you type into the AI features (and model prose that may repeat it) is stored only as a fingerprint: its length and a SHA-256 prefix. With your own account, the text is stored as written, so don't type real names or details.

Frameworks. The approach is informed by the Australian Government's Policy for the responsible use of AI in government (DTA), the transparency principles of the EU AI Act and the NIST AI Risk Management Framework. This is not a claim of compliance with any of them. More in the model card and DR-005.

• The 2022 quota meant “two codes held at a time”, because used codes were deleted. The revival keeps used codes and counts only unused ones.
• Treating codes and members so far as a sample of the invite process (on /tree) is a modelling choice; with a fictional cohort and synthetic timings it demonstrates the method rather than describing people. Codes still open are assumed to be censored at random (whether a code has been used yet says nothing extra about how long it would take).
• Visitors who bring a key accept their provider's terms for API traffic.
• The reference SQL for each evaluation question is the right answer. Each was checked by hand against the seed, and the answers are pinned by a test.

• The seeded cohort is 18 fictional members with synthetic timings.
• 24 evaluation questions give wide intervals, and execution match fails reasonable answers with an extra column while passing answers that are right by coincidence.
• No evaluation run has been published yet.
• The database property suite has 100 runs, so its bound is weak (3.0%).
• The audit chain proves internal consistency, not authenticity: someone with write access could rebuild it. It has no external anchor yet.
• The SQL sandbox has no statement timeout. Its cost is bounded before a query runs (a worst-case row bound from the query plan, plus row caps and rate limits); the bound is conservative, so a legitimate query on a much larger table could be refused.
• Evaluation runs make one generation per question, so they say nothing about how much a model's answers vary between runs.
• The AI log is reported by the browser; the server can't prove that every call was reported.
• Benchmarks ran on a laptop, not on the production platform.

1. Publish one funded evaluation run (two models, all 24 questions) with its export.
2. Grow the question set to 100 or more, stratified by tag.
3. Anchor the audit chain's head outside the database.
4. Measure page latency and hashing on the production platform, with intervals.
5. Give the sandbox a real statement timeout (a worker thread that can be stopped).
6. Run each evaluation question several times to measure run-to-run variation, and have the server issue a log id before each AI call so unreported calls show up.

Each record states the decision first, then the context, the options, why, what happened (weak numbers included) and what I'd change. Records are never edited after acceptance; a new record supersedes an old one.

1. DR-001From Express + MongoDB to one Next.js app on TursoRebuild Project Cradle as a single Next.js 16 (App Router) application on SQLite through libSQL and Drizzle ORM, with Turso as the production database. The 2022 code stays untouched in legacy/, and the 2022 rules are ported as pure functions that parity tests check against the preserved files.
2. DR-002The invite quota counts held codes, enforced in one statementA member may hold at most two unused codes at a time. Used codes are kept (marked with used_by_id and used_at) instead of deleted, and only unused, non-reusable codes count against the quota. The quota check and the insert are a single INSERT ... SELECT ... WHERE (count of held codes) < 2, and a sign-up consumes its code with a conditional UPDATE ... WHERE used_by_id IS NULL inside the sign-up transaction. Codes issued by the system inviter "Cradle" stay reusable, as in 2022.
3. DR-003Server-side sessions with hashed, rotating ids and CSRF tokensKeep sessions on the server. The browser holds an httpOnly, SameSite=Lax cookie (Secure in production) containing an HS256-signed JWT that points at a session row; the table stores only the SHA-256 of the session id. A session ends 24 hours after sign-in however it is used. Its id is replaced every 15 minutes of activity (the old row lives for a 60-second grace period), on every sign-in (so a planted id is worthless) and after a password change (which also signs out every other browser). Members can "sign out everywhere". State-changing requests carry three CSRF defences: Next.js's own Origin check for server actions, our own Origin and Fetch Metadata check (which also covers the JSON routes), and a per-session synchronizer token that comes back in a hidden form field.
4. DR-004Hash passwords with argon2id and upgrade bcrypt hashes on sign-inNew password hashes use argon2id with 19 MiB of memory, 2 passes and 1 lane (the first configuration the OWASP Password Storage Cheat Sheet recommends), through @node-rs/argon2. Existing bcrypt hashes still verify, and each is replaced with an argon2id hash the next time its owner signs in successfully. The seed's demo accounts are hashed with argon2id using deterministic salts, so data/seed.db stays byte-for-byte reproducible.
5. DR-005Optional AI with the visitor's own key, a SQL sandbox and an evaluation harnessAI features are optional and use only the visitor's own API key (Anthropic by default, or OpenAI). The key is stored in the visitor's browser (session storage by default, local storage only if they ask), and calls go from the browser straight to the provider through the official SDKs; the key never reaches Cradle's server. Every call is then recorded in an append-only ai_audit_log table by a server action that never receives the key. Every AI output is labelled "AI-generated" and needs a person to accept, edit or reject it, and the decision is logged. AI-written SQL runs only in a read-only, in-memory sandbox holding analytic columns. An evaluation harness measures text-to-SQL accuracy against hand-written reference queries, with intervals.
6. DR-006An append-only, hash-chained audit log in the databaseSecurity-relevant events go to an audit_log table that database triggers make append-only (every UPDATE and DELETE aborts). Each entry stores the SHA-256 of the previous entry's hash plus its own fields, starting from a fixed genesis value, and prev_hash is unique so concurrent writers can't fork the chain. Admins see the log in /admin/records with a chain check run on every view, and can export it as CSV. Client IPs are stored only as an HMAC with the signing key.
7. DR-007Rate limits shared by every server instanceRate limits are fixed-window counters in a rate_limits table, updated with a single atomic upsert, so every Vercel function instance shares them. Keys are HMACs of the bucket and the subject (an IP or a username), never the raw values.
8. DR-008Fail closed: fixes from the pre-release reviewEvery way the review found for the AI features, the SQL sandbox or the estimates to fail open is closed in code, each with a test that reproduces the original problem, before the upgrade ships.