# GaiaLab Forward-Skill Scorecard — Methodology

**Status:** live, computed on every page load · **Source of truth:** `GET /api/predictions/monitoring-count` (`src/server.js`) · **Displayed at:** [/new-trials](https://www.gailabai.com/new-trials#forward-skill-scorecard)

This document formalizes a definition that already runs, unchanged, as a SQL query behind the
Forward-Skill Scorecard panel on `/new-trials`. It does not introduce any new criterion — it
exists so the definition is citable on its own, independent of reading the query.

The goal, same as `PROSPECTIVE_VALIDATION_PROTOCOL.md`: make the claim **falsifiable and
un-gameable**. A scorecard whose denominator can be quietly widened or narrowed after the fact
is not evidence of anything. Every quantity below is defined once, computed the same way every
time, and reported **whether or not it flatters GaiaLab** — the live page currently reports
0 confirmed forward hits, and says so.

---

## 1. What this scorecard measures (and what it does not)

The Forward-Skill Scorecard answers one question: **of the predictions GaiaLab could possibly
have gotten right prospectively, how many actually were?**

It does **not** measure: overall prediction volume, retrospective concordance with trials that
already existed, semantic/AI-assisted similarity matches, or drug efficacy. Those are real,
separately reported numbers elsewhere on the site (`/predictions`, `/validation`,
`/case-study-*`) — this scorecard exists specifically to isolate the one number that can't be
inflated by counting things that were never a fair test of foresight.

## 2. The fair denominator: `prospectiveEligible`

A drug–disease pair is **prospective-eligible** only if, at the moment GaiaLab made the
prediction, no ClinicalTrials.gov trial for that exact pair existed yet. Concretely:

```
prospectiveEligible = unmatchedCount + prospectiveHits
```

- **`unmatchedCount`** — distinct drug–disease pairs (last 365 days) where no trial has *ever*
  matched, as of now (`bool_or(has_trial) = FALSE` across every row recorded for that pair).
  These are still open: a trial could register tomorrow and turn one into a hit.
- **`prospectiveHits`** — distinct pairs where a matching trial *did* later register, strictly
  **after** the prediction's own `recorded_at` timestamp (`is_prospective = TRUE`, exact-name
  drug+disease match — not a semantic/fuzzy match).

Both counts are computed as `COUNT(DISTINCT drug_canonical || '||' || disease_context)` over
`gaialab_predictions`, scoped to the trailing 365 days, so the denominator is always a live,
rolling window — not a cherry-picked historical slice.

## 3. What is explicitly excluded, and why

- **`retrospectiveIneligible`** — distinct pairs with a trial match where `is_prospective` is
  `NULL` or `FALSE`, i.e. the trial already existed at (or before) prediction time. **A trial
  that pre-existed the prediction cannot test foresight**, so these pairs are excluded from the
  denominator entirely rather than counted as either a hit or a miss. They're still reported
  (visibly, on the same panel) so the exclusion is auditable, not hidden.
- **Semantic-similarity matches** (`verified_semantic_matches`, `match_type = 'prospective'`) —
  a looser, AI-assisted similarity layer (pgvector-based, not expert peer review) that finds
  candidate matches text-approximately rather than by exact drug+disease name. As of this
  writing it has surfaced 15 such matches, 13 of them on-label (an approved drug getting a new
  trial for its own indication is expected, not evidence of foresight) and 2 off-label. These
  are reported separately on `/new-trials` and **do not count toward this scorecard** — only an
  exact-name match qualifies as a confirmed forward hit here.

## 4. The forward hit-rate

```
forward hit-rate = prospectiveHits / prospectiveEligible
```

Reported as a percentage with both raw counts shown alongside it, never as a bare percentage —
a rate with a denominator of 3 and a rate with a denominator of 3,000 are not comparable
claims, and the live panel always shows both numbers together for exactly this reason.

## 5. Coverage: the honest ceiling on how many hits are even possible

Forward skill can only be demonstrated in the open (`prospectiveEligible`) fraction of all
matched-or-eligible pairs — a pair that's already `retrospectiveIneligible` can never become a
forward hit, no matter what GaiaLab does next. The site reports this ratio (`% open-space` vs.
`% trial-saturated`) as the honest bound on how large the numerator in §4 could ever get, so a
low current hit-rate is legible against how much of the space was even open to begin with,
rather than read as a flat failure rate.

## 6. What would change this document

This methodology changes only if the underlying SQL definition changes (`src/server.js`'s
`/api/predictions/monitoring-count` handler) — at which point this file should be updated in
the same commit, and the change itself becomes part of the public record, the same way the
lockbox protocol treats its own frozen parameters. It is not re-tuned to make a number look
better after the fact.

## 7. Related documents

- [`PROSPECTIVE_VALIDATION_PROTOCOL.md`](./PROSPECTIVE_VALIDATION_PROTOCOL.md) — the sealed
  lockbox protocol (357 predictions vs. 148 shuffled controls, frozen 2026-07-17, reads out
  2028-01-17), a stricter, pre-registered sibling test to this always-live scorecard.
- [`/new-trials`](https://www.gailabai.com/new-trials) — the live scorecard itself.
- [`/verify`](https://www.gailabai.com/verify) — independently recompute the Merkle/Bitcoin
  timestamp proof behind any individual prediction's `recorded_at` claim.