NIH | National Cancer Institute | NCI Wiki  

WIKI MAINTENANCE NOTICE

NCI Wiki will be undergoing scheduled maintenance on Monday, November 16th between 5:00 and 6:00 PM ET.
Wiki will remain available, but users may experience screen refreshes or HTTP 502 errors during the maintenance period. If you encounter these errors, wait 1-2 minutes, then refresh your page.
You will be required to re-authenticate into Wiki upon refresh if you were logged in prior.
As a precautionary measure, editors should ensure all work is saved prior to the maintenance period.

If you have any questions or concerns, contact the CBIIT Atlassian Management Team.


Workshop Goal

The goal of the NCI AI Benchmarks Workshop is to identify actionable, near-, mid-, and long-term pathways for improving cancer AI benchmarking. Through partnerships, shared infrastructure, and pilot efforts we aim to assess rigor, generalizability, and success of real-world adoption across academia, industry, healthcare systems, and government.

Background and Objectives

To fully harness AI’s potential in cancer research and care, it is essential for models to demonstrate accuracy, reproducibility, and generalizability. Benchmarks play a key role in advancing AI by enabling the assessment and comparison of model performance. They are critical for ensuring the accuracy and trustworthiness of AI models, thereby building confidence that AI can be safely and effectively integrated into real-world scenarios. Despite their importance, there is a lack of robust, high-quality benchmarks in cancer research and care.Addressing this gap will be critical to enabling rigorous evaluation, fostering trust, and translating AI innovation into patient impact.

Workshop Definition of an "AI Benchmark"

An AI benchmark is a meaningful relative comparison by which to evaluate an AI system on a well-defined task using representative data and agreed-upon metrics. By establishing an AI benchmark framework, we will enable rigorous validation protocols to assess not only accuracy, but also generalizability, clinical relevance, and real-world performance and impact. An AI system includes the model, data inputs, workflow integration, user interface, and deployment context, reinforcing the idea that we need to think beyond technical performance/accuracy. The intended focus for this workshop is on:

  • supporting developers and early to mid-stage benchmarks while acknowledging downstream clinical impact as the ultimate goal
  • enabling more systematic benchmarking in earlier stages, with an eye to how those benchmarks could connect to later clinical endpoints, our “north star”

AI Benchmarks Event Website

Breakout Room Selection

Shared Workshop Photos

NCI AI Benchmarks Early Draft Meeting Summary


  • No labels