TrialGPT Evaluation Report

Executive Summary

Both the original TrialGPT approach described in the TrialGPT paper (original) and the Health Universe agentic approach (agentic) achieved identical discriminative ability for trial eligibility (ROC AUC = 0.736 for both arms). When the two approaches disagree, ground truth adjudication shows the Agentic approach is correct 69% of the time (20/29 divergent cases with ground truth).

Our agentic pipeline matches TrialGPT's output at 89.9% concordance while adding structured clinical interpretability. Independent LLM-as-judge evaluation corroborates this: agentic reasoning was preferred in 60% of divergent cases (9 of 15).

The ground truth evaluation was underpowered, with only 176 trial pairs, representing 2.5% of the paired dataset. In this setting, Agentic achieves PR AUC = 0.38 vs. Original 0.30 (27% improvement), but additional evaluation could be done if larger ground truth data sets were available.

Key Findings

Introduction

TrialGPT (Jin et al., Nature Communications 2024; DOI: 10.1038/s41467-024-53081-z) is an end-to-end framework for patient-to-trial matching using large language models, comprising three modules: TrialGPT-Retrieval, TrialGPT-Matching, and TrialGPT-Ranking. It was originally evaluated on 183 synthetic patients across three cohorts (SIGIR 2016, TREC CT 2021, TREC CT 2022) with over 75,000 trial annotations.

We evaluate our agentic oncology pipeline on oncology-specific patients against this baseline. The agentic approach adds a structured patient summarization step upstream of the matching algorithm, producing a Patient Fact Sheet that organizes clinical information into standardized categories before trial matching. This evaluation tests whether this additional processing improves or maintains matching quality on a subset of cancer patients.

Methods

Study Design

We anchored our dataset to the original TrialGPT study data sets. Our study focused on cancer patients and cancer-related clinical trials. Using the original data sources for TrialGPT, we identified 17 oncology patients and 417 cancer-related clinical trials. These formed the basis of the data for our trial.

We created a two-arm evaluation using 17 cancer patients and 417 cancer-related clinical trials extracted from the original TrialGPT datasets:

Models & Infrastructure

Both the Original and Agentic pipelines use OpenAI GPT-4o as the underlying large language model for all matching, scoring, and reasoning steps. The original TrialGPT paper reports results using GPT-4; our reproduction of the Original pipeline uses GPT-4o to ensure a controlled comparison between arms. The LLM-as-Judge evaluation uses GPT-5 as an independent adjudicator.

Results

With 17 patients and 417 trials, the total number of possible matches is 7,089. We categorized each match as not a match, a weak match, a moderate match, or a strong match. While the agentic approach was more conservative with fewer stronger matches (100 vs. 158), it had nearly identical moderate matches (705 vs. 726) — more conservative top-tier scoring with fewer false-positive strong matches.

Agreement Metrics

LLM-as-Judge Results

GPT-5 evaluated 15 patient-trial pairs with the largest score divergence between approaches, assessing clinical reasoning quality on a 1–5 scale.

The Agentic approach was preferred in 9 of 15 cases (60%). The following vignettes illustrate the most clinically significant divergences.

Vignette 1: Pediatric Age Restriction Missed

Patient trec-202111 is a 75-year-old male with metastatic papillary thyroid cancer. Original scored 2.6 (strongly eligible) for NCT02375451, a pediatric radioiodine study. Agentic correctly scored −0.5, recognizing the pediatric design makes this elderly patient clearly ineligible.

The trial targets patients who received radioiodine therapy in childhood/pediatric populations, whereas the patient is 75 years old. System A overlooks this key age-related criterion and declares eligibility, while System B correctly flags the pediatric focus and infers ineligibility based on age.

Vignette 2: Logical Inversion of Eligibility Criteria

Patient trec-202167 is a 54-year-old woman with cervical dysplasia. Original scored −0.0 by misinterpreting a “women ≥ 40” inclusion criterion as an exclusion criterion. Agentic correctly scored 2.9, recognizing clear eligibility.

System A makes a critical error by treating age ≥ 40 as an exclusion criterion, contradicting the trial design. System B correctly interprets the criteria. Overall, B's reasoning is more clinically sound.

Vignette 3: Missing VTE/LMWH Requirements

Patient trec-20214 was evaluated for NCT01803022, a trial requiring VTE diagnosis and LMWH treatment. Original scored 2.90 (strongly eligible) despite the patient lacking the required VTE diagnosis. Agentic scored −0.33, correctly identifying the required conditions are absent.

The trial requires a recent VTE and treatment with LMWH. System A overinterprets eligibility by equating DVT history with recent VTE without verifying LMWH use. System B appropriately flags the missing key criteria.

Vignette 4: Complex Comorbidity Handling

Patient trec-202113 is a 62-year-old male with non-Hodgkin lymphoma, SLE, diabetes, and hepatitis C — the most divergent patient (ρ = 0.41). For NCT01737567, Original scored 2.6 while Agentic scored −1.0. The approaches diverge on missing information: Original tends to assume missing data implies eligibility; Agentic treats missing critical information as grounds for caution.

System B correctly withholds inclusion due to lack of evidence the patient is undergoing a screening colonoscopy, whereas System A incorrectly assumes this criterion is met. Although B overstates ineligibility, its assessment is more cautious and clinically sound.

Ground Truth Analysis

Ground-truth relevance judgments (qrels) from the TREC and SIGIR shared tasks provide an external benchmark. Labels: 0 = irrelevant, 1 = excluded (right condition but failing specific criteria), 2 = eligible.

Coverage Caveat

Ranking Metrics vs. Ground Truth

Classification Metrics vs. Ground Truth

The Agentic approach misses more eligible trials but when it says “match” it is more likely to be correct. At the ≥ 1.0 threshold, Original generates 33 false positives per 176 trials (19%) vs. Agentic's 18 (10%). In a workflow with 400+ trials per patient, that translates to roughly 15 fewer irrelevant trial reviews per patient with the Agentic approach.

The NDCG vs. PR AUC Discrepancy

There is an apparent contradiction between the two arms' metrics: Original has higher NDCG@10 (0.627 vs. 0.604) but Agentic has higher PR AUC for eligible-vs-not (0.38 vs. 0.30, a 27% improvement). The NDCG uses graded relevance (0/1/2) and rewards ranking excluded (label = 1) trials higher — something Original does more aggressively (mean score for excluded trials: +0.105 vs. Agentic's −0.379). For the binary clinical question “is this patient eligible?” the Agentic approach is actually better.

Score Distributions by Ground-Truth Label

Key insight: Original scores excluded (label = 1) trials at mean = +0.105, nearly identical to its irrelevant trial mean (−0.009). It effectively cannot distinguish excluded from irrelevant. Agentic pushes excluded trials much lower (−0.379). This mechanism — not superior clinical accuracy — explains Original's higher NDCG.

Divergent Case Resolution vs. Ground Truth

Discussion

At the clinically meaningful threshold of 1.0, concordance was 89.9% with Cohen's Kappa of 0.519 (“moderate agreement”). The Agentic approach produced scores that were on average 0.031 points lower than the Original (p < 0.001), reflecting its more conservative approach to eligibility scoring.

Benchmarking Implications

On the oncology subset, both approaches achieve identical discriminative ability (ROC AUC = 0.736) for eligible-vs-not classification. The Agentic approach shows a 27% improvement in PR AUC (0.38 vs. 0.30; n = 176 trial pairs), indicating a better precision-recall tradeoff for the binary eligibility question. When the two approaches diverge, Agentic matches ground truth 69% of the time.

The NDCG advantage for Original (0.627 vs. 0.604) is explained by its more aggressive scoring of excluded (label = 1) trials — behavior that inflates graded relevance metrics but is not clinically desirable.

The structured fact sheet layer adds clinical interpretability without degrading — and in the key PR AUC metric, improving — matching accuracy. On the oncology subset, our pipeline demonstrates equivalent or superior performance against expert ground truth, with a more conservative scoring profile that produces higher precision and fewer false-positive strong matches. These characteristics are desirable for clinical deployment, where clinician time is limited and false positives carry real workflow costs.

Limitations

References

Jin, Q., et al. (2024). Matching patients to clinical trials with large language models. Nature Communications, 15, 9074. DOI: 10.1038/s41467-024-53081-z

©2026 Health Universe, Inc. Confidential. All rights reserved.