← Forward-Skill Scorecard  ·  Verify a timestamp  ·  Calibration data

GaiaLab Forward-Skill Scorecard — Methodology

Status: live, computed on every page load · Source of truth: GET /api/predictions/monitoring-count (src/server.js) · Displayed at: [/new-trials](https://www.gailabai.com/new-trials#forward-skill-scorecard)

This document formalizes a definition that already runs, unchanged, as a SQL query behind the Forward-Skill Scorecard panel on /new-trials. It does not introduce any new criterion — it exists so the definition is citable on its own, independent of reading the query.

The goal, same as PROSPECTIVE_VALIDATION_PROTOCOL.md: make the claim **falsifiable and un-gameable**. A scorecard whose denominator can be quietly widened or narrowed after the fact is not evidence of anything. Every quantity below is defined once, computed the same way every time, and reported whether or not it flatters GaiaLab — the live page currently reports 0 confirmed forward hits, and says so.

1. What this scorecard measures (and what it does not)

The Forward-Skill Scorecard answers one question: **of the predictions GaiaLab could possibly have gotten right prospectively, how many actually were?**

It does not measure: overall prediction volume, retrospective concordance with trials that already existed, semantic/AI-assisted similarity matches, or drug efficacy. Those are real, separately reported numbers elsewhere on the site (/predictions, /validation, /case-study-*) — this scorecard exists specifically to isolate the one number that can't be inflated by counting things that were never a fair test of foresight.

2. The fair denominator: prospectiveEligible

A drug–disease pair is prospective-eligible only if, at the moment GaiaLab made the prediction, no ClinicalTrials.gov trial for that exact pair existed yet. Concretely:

`` prospectiveEligible = unmatchedCount + prospectiveHits `

- unmatchedCount — distinct drug–disease pairs (last 365 days) where no trial has *ever* matched, as of now (bool_or(has_trial) = FALSE across every row recorded for that pair). These are still open: a trial could register tomorrow and turn one into a hit.

Both counts are computed as COUNT(DISTINCT drug_canonical || '||' || disease_context) over gaialab_predictions, scoped to the trailing 365 days, so the denominator is always a live, rolling window — not a cherry-picked historical slice.

3. What is explicitly excluded, and why

- retrospectiveIneligible — distinct pairs with a trial match where is_prospective is NULL or FALSE, i.e. the trial already existed at (or before) prediction time. **A trial that pre-existed the prediction cannot test foresight**, so these pairs are excluded from the denominator entirely rather than counted as either a hit or a miss. They're still reported (visibly, on the same panel) so the exclusion is auditable, not hidden.

` forward hit-rate = prospectiveHits / prospectiveEligible `

Reported as a percentage with both raw counts shown alongside it, never as a bare percentage — a rate with a denominator of 3 and a rate with a denominator of 3,000 are not comparable claims, and the live panel always shows both numbers together for exactly this reason.

5. Coverage: the honest ceiling on how many hits are even possible

Forward skill can only be demonstrated in the open (prospectiveEligible) fraction of all matched-or-eligible pairs — a pair that's already retrospectiveIneligible can never become a forward hit, no matter what GaiaLab does next. The site reports this ratio (% open-space vs. % trial-saturated) as the honest bound on how large the numerator in §4 could ever get, so a low current hit-rate is legible against how much of the space was even open to begin with, rather than read as a flat failure rate.

6. What would change this document

This methodology changes only if the underlying SQL definition changes (src/server.js's /api/predictions/monitoring-count handler) — at which point this file should be updated in the same commit, and the change itself becomes part of the public record, the same way the lockbox protocol treats its own frozen parameters. It is not re-tuned to make a number look better after the fact.

7. Related documents

- [PROSPECTIVE_VALIDATION_PROTOCOL.md](./PROSPECTIVE_VALIDATION_PROTOCOL.md) — the sealed lockbox protocol (357 predictions vs. 148 shuffled controls, frozen 2026-07-17, reads out 2028-01-17), a stricter, pre-registered sibling test to this always-live scorecard.