Workshop Goal
The goal of the NCI AI Benchmarks Workshop is to identify actionable, near-, mid-, and long-term pathways for improving cancer AI benchmarking. Through partnerships, shared infrastructure, and pilot efforts we aim to assess rigor, generalizability, and success of real-world adoption across academia, industry, healthcare systems, and government.
Background and Objectives
To fully harness AI’s potential in cancer research and care, it is essential for models to demonstrate accuracy, reproducibility, and generalizability. Benchmarks play a key role in advancing AI by enabling the assessment and comparison of model performance. They are critical for ensuring the accuracy and trustworthiness of AI models, thereby building confidence that AI can be safely and effectively integrated into real-world scenarios. Despite their importance, there is a lack of robust, high-quality benchmarks in cancer research and care.Addressing this gap will be critical to enabling rigorous evaluation, fostering trust, and translating AI innovation into patient impact.
Workshop Definition of an "AI Benchmark"
An AI benchmark is a meaningful relative comparison by which to evaluate an AI system on a well-defined task using representative data and agreed-upon metrics. By establishing an AI benchmark framework, we will enable rigorous validation protocols to assess not only accuracy, but also generalizability, clinical relevance, and real-world performance and impact. An AI system includes the model, data inputs, workflow integration, user interface, and deployment context, reinforcing the idea that we need to think beyond technical performance/accuracy. The intended focus for this workshop is on:
- supporting developers and early to mid-stage benchmarks while acknowledging downstream clinical impact as the ultimate goal
- enabling more systematic benchmarking in earlier stages, with an eye to how those benchmarks could connect to later clinical endpoints, our “north star”
NCI AI Benchmarks Early Draft Meeting Summary

