NIH | National Cancer Institute | NCI Wiki  

Versions Compared

Old Version 30

changes.mady.by.user Unknown User (kimbroughmg)

Saved on

compared with

New Version 31

changes.mady.by.user Unknown User (kimbroughmg)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Welcome to the CBIIT HPC Wiki!  This page is actively under construction and will soon serve as a central hub to provide you with the latest developments and resources related to the CBIIT HPC Program.  We look forward to serving you!

Reliable, innovative solutions that make it easy for NCI investigators to use high performance computing.  

 

 

Table of Contents

Button Hyperlinkiconsearch


 

Horizontal Navigation Bar
titleSite Map

 

 

Horizontal Navigation Bar Page
title
Related Links
Background
type
Expand
standard
title
url
Context

CBIIT HPC

Related Links

Strategy

With a focus on providing robust and reliable solutions enabling NCI investigators to utilize HPC in their efforts, the CBIIT HPC strategy is focused on these interconnected areas:

  • Working closely with investigators across NCI to enable broader utilization of HPC through HPC training, education and reliable system access
  • Guided by investigator challenges and opportunities, provide support and consulting for HPC needs as well as development, optimization and/or validation of HPC applications useful to cancer research and clinical applications
  • Effective data management and information delivery solutions in support of HPC applications used in cancer research
  • Exploration and evaluation of emerging HPC technologies for use in cancer research, information delivery and data center operations
  • Developing essential partnerships within NCI, NIH, HHS, government, academically, commercially, nationally and internationally fostering the expanded use of HPC in cancer research
  • Develop and deliver services supporting current and future HPC needs of NCI investigators
  • Continual incorporation of investigator input to improve and evolve HPC services, capabilities and opportunities

 

Long Range Guiding Objectives for HPC in Cancer Research

With guidance and insight provided by the cancer research and clinical community within NCI, deliver robust, reliable HPC capabilities and support that:

  • Enable broader understanding of cancer, cancer system dynamics and cancer characterizations
  • Enable rapid identification of potential cancer risks and presence of cancer in individuals
  • Enable rapid determination of optimal treatment options for patients
  • Expand treatment options through improved discovery and rapid, reliable validation
  • Foster computational integration and cooperation across the global cancer research community
  • Enable transfer and flow of HPC technologies between NCI and other stakeholders
  • Enable NCI to take full advantage of computational advances to accelerate cancer research

 

Foundations for Successful HPC in Cancer Research and Clinical Development

  • Useful
    • Performing needed functions and delivering key capabilities
    • Enabling technologies can be eventually used in clinical application
  • Reliable
    • Assuring new computing technologies and applications are functionally reliable
    • Assuring technologies and applications are validated and verified
    • Assuring appropriate reproducibility of delivered solutions over time
  • Adaptive
    • Exploring new and emerging technologies and applications for use in cancer research and translation
    • Utilizing multiple sources of input (internal and external) to improve overall HPC capabilities
  • Portable
    • Enabling intellectual investments to transition across emerging and evolving technology platforms
  • Efficient
    • Providing solutions in as rapid as possible yet in a cost-effective manner
Expand
titleOverview

 

Biomedical research computation enables a growing number of scientists and clinicians to analyze, use, manage, and share their data—and their discoveries.

At NCI, wehave developed innovative High Performance (HPC) and Scientific Computing services that help investigators accelerate and advance their cancer research.

 

Our services include HPC support and consultation, education, and reliable access to various state-of-the-art HPC systems including the NIH Biowulf system, the NCI MOAB system, and HPC programs at the Department of Energy.  

 

We also work closely with investigators across NCI to develop, optimize and/or validate HPC applications—and computing technologies—that enable rapid data transfer and provide effective data management. With ongoing input from the cancer researchers we serve, we continue to improve and evolve our HPC services, capabilities and opportunities. As partners to cancer researchers worldwide, we are guided by investigators’ challenges and opportunities.

 

Above all, we seek to expand scientists’ use of high performance computing to accelerate discoveries in predictive oncology research and advance effective methods of prevention, treatment, and cure. 

 

 

Horizontal Navigation Bar Page
titleServices
Tabs Container
titleService Descriptions
directionhorizontal
Tabs Page
titleHPC Consultation


Name of Service: HPC Consultation


Type of Service: Scientific Computing


Short Description: Consultation, training, and support for high performance computing (HPC) systems used by NCI researchers 


Intro

Our innovative High Performance (HPC) and Scientific Computing services accelerate and advance cancer research.

Our services include HPC support and consultation, education, and reliable access to various state-of-the-art HPC systems including the NIH Biowulf system, the NCI MOAB system, and HPC programs at the Department of Energy.  

We also work closely with investigators across NCI to develop, optimize and/or validate HPC applications that expedite numerous data management functions.


What We Do

We provide consultation, training, and support for high-performance computing (HPC) systems used by NCI researchers.


Our Service Includes

Consultation and Evaluation of Your Scientific Computing Needs. We help you determine which HPC resources will help accelerate your research. These resources include:

  • NIH’s HPC systems Helix and Biowulf
  • NCI’s HPC system, MOAB
  • Temporary storage space to share or move data files
  • HPC application development or optimization
  • Application testing
  • Extended application profiling

Training and Education on High Performance Computing (HPC) Systems. We offer the guidance you need, including:

  • How to use and program the NIH HPC systems, Helix and Biowulf and the NCI HPC system, MOAB
  • Individualized training—we meet with you one-on-one to explain how to use specific computing tools and processes for your research study
  • Recommendations for HPC-related presentations on the NCI HCP wiki and NIH training materials

Access and Login Assistance to the HPC systems. We provide assistance to access and log into the HPC systems you need to use, including:

Assistance on the Optimal Use of HPC Resources. We help you optimize the HPC resources you can use for your research. For example:

  • How to make HPC applications run fast using graphical processing units (GPU)
  • Which combination of HPC applications to use
  • How to transfer your data efficiently using GLOBUS, the electronic service


FAQ

How Much Does It Cost?

There is no cost to you for HPC consultation.

There is a nominal monthly charge for an NIH HPC account. This gives you access to all of the NIH HPC systems and associated services (i.e. Helix, Biowulf).

There are no additional charges for CPU or storage on the NIH HPC systems.

How Does It Work? What Do I Need to Do?

If you’re not sure what you need or if you need any guidance on using high performance computing, we’re here to help. Use the Request Service button to request a consultation.

For additional information about our HPC program visit the NCI High Performance Computing (HPC) Program wiki.

When is the Service Available?

HPC Consultation is available during normal business hours: Monday through Friday, 8:00 a.m. to 5:30 p.m., excluding holidays.

How Long Does It Take to Receive the Service?

We generally respond to requests for consultation within 1 business day.


Additional Resources

NIH HPC Systems

Department of Energy HPC Systems

Argonne National Lab Leadership Computing Facility

Oak Ridge National Lab Leadership Computing Facility 

 

Related Services

High Performance and Scientific Computing Data Transfer Resources

High Performance and Scientific Computing Application Development

Research Computing and Data Management and Analysis

 

 

 

Tabs Page
titleHPC Application Development


Name of Service: HPC and Scientific Computing Application Development


Type of Service: Scientific Computing

 

Short Description: High performance (HPC) and scientific computing application development for NCI researchers

 

Intro

Cancer researchers generate a massive amount—and high volume—of data. This presents continual challenges for scientists to access, analyze, manage, and share their data with other researchers across the globe. Scientific computational tools must be robust, reliable, effective, and affordable.

At NCI, wehave developed innovative High Performance (HPC) and Scientific Computing services that offer reliable, high-speed data management and information delivery solutions to help investigators accelerate and advance their cancer research.

We work closely with investigators across NCI to develop, optimize and/or validate HPC applications—and computing technologies—that enable rapid data transfer and provide effective data management.

 

What We Do

We optimize and develop HPC applications to improve performance and increase speed on (HPC) systems.

 

Our Service Includes

HPC Application Optimization

  • We help your scientific applications run faster on HPC systems

HPC Application Development

  • Creation of new applications and algorithms for high-performance computing (HPC) systems

Script and Workflow Developmentfor applications that use HPC resources. This includes:

  • Job submission
  • Data transfer
  • Integration of multiple applications

 

FAQ

How Much Does It Cost?

There is no cost to you for this service.

How Does It Work? What Do I Need to Do?

If you’re not sure what you need or if you need guidance on using high performance computing, we’re here to help. Use the Request Service button to request a consultation or to request HPC application development.

See the NCI High Performance Computing (HPC) Program wiki for details on our high performance computing services.

When is the Service Available?

The service is available during normal business hours: Monday through Friday, 8:00 a.m. to 5:30 p.m., excluding holidays.

How Long Does It Take to Receive the Service?

We generally respond to service requests within 1 business day.

 

Additional Resources

For information about the NIH HPC Systems see the NIH HPC Systems  website.


Related Services

High Performance Computing (HPC) Consultation

High Performance and Scientific Computing Data Transfer Resources

Research Computing and Data Management and Analysis

 

 

 

 

Tabs Page
titleHPC Data Transfer Resources

 

 

 

Tabs Page
titleHPC Research Computing and Data Analysis

 

 

 

 

 

Horizontal Navigation Bar Page
titleCommunications

 

 

 

Horizontal Navigation Bar Page
titleTraining and Education

 

 

 

 

 

CBIIT

 

FOR APPROVAL 

Name of Service:     High Performance and Scientific Computing Data Transfer Resources

 [proposed]

                                    High Performance Computing (HPC) General Support [current] 

 

Type of Service:       Scientific Computing [proposed]

 

***********************************************************************************

Short Description: [for possible cover page listing this category]

High performance and scientific computing data transfer resources for NCI cancer research studies/projects[BL([1]  

**************************************************************************************

Intro Text

Managing and sharing data effectively, reliably, and affordably has become key to cancer research. Working collaboratively across NCI, NIH, and the broader cancer research community, we [help to bring forward] facilitate the use of[BL([2]  new services and technologies to meet these ever-changing needs.

 

What We Do

We provide high performance and scientific computing data transfer resources to help NCI researchers manage and share data effectively, reliably, and affordably.

 

We also offer information on long-term resources to archive and store your data. 

 

Our Service Includes

Consultation on Research Data Resources[BL([3] . We discuss your needs with you and offer guidance on the specific electronic resources available to move and share data internally and externally. [based[BL([4]  on your business/scientific needs]  

 

Information and Guidance on the Use of Globus, NIH’s Electronic Data Transfer Service. Globus allows you to transfer and share data files with NCI intramural and extramural researchers.

 

Immediate Short-Term Storage. We provide storage for shared data that is offloaded from high-performance computing systems[BL([5]  

 

Long-Term Storage and Data Archive[BL([6] . We provide [BL([7] long-term data storage and archiving solutions for your long-term data retention needs. 

 

How Much Does It Cost?

There is no cost to you for this service.

 

 

How Does It Work? What Do I Need to Do?

If you’re not sure what you need or if you need any guidance on using high performance computing, we’re here to help. Use the Request Service button to request a consultation or to request HPC Data Transfer Resources. See the NCI HPC Wiki for additional information.

 

When is the Service Available?

High performance computing systems and applications are available 24/7.

Consultation and support is available during normal business hours: Monday through Friday, 8:00 a.m. to 5:30 p.m., excluding holidays.

 

How Long Does It Take to Receive the Service?

We generally respond to requests for consultation within 1 business day.

 

Additional Resources[BL([8]  –if any  

NCIP HPC page?

 

Related Services

High Performance Computing Consultation

High Performance and Scientific Computing Application Development

Research Computing and Data Management and Analysis

 

 

 

 [BL([1]Which word – studies or projects?

 [BL([2]OK to delete?

Are my words in red OK?

 [BL([3]I moved this from the Research Computing Service. 

 [BL([4]Omit the last part of this phrase? Should this be scientific instead of business needs?

 [BL([5]Explain a little? Do the HPC systems offload the data automatically? Does the researcher do this? For what/why?

 [BL([6]Is this the right form of the word?

 [BL([7]Original said find. Assume we provide?

 [BL([8]Do we want to link to the NIH HPC page?

To the NCIP page? – probably too confusing about where to go

Button Hyperlink
iconsearch
titleHelpful Links
typestandard
urlHPC Related Links

Round Rectangle
bgcolororange
titleBlog Posts
Expand
titleClick Here for Blog Posts

Blog Posts
reversetrue
contenttitles

 

 

The High-Performance Computing Program Development effort aims to foster the expanded use of a high-performance computing ecosystem to accelerate advances in predictive oncology research and clinical applications. Both driven and enabled by the rapid growth rates of information collected and generated about cancer, the opportunity for ever increasing computational capability grows as the data is analyzed, explored and utilized to provide critical insight into cancer. The program aims to develop the computational and data science ecosystem by addressing critical needs in compute, data transfer, data management, exploration and education in these areas required to advance the mission of the NCI.

*For support inquiries, please contact us at nci-cbiit-hpc@list.nih.gov

 

CLICK THE BELOW IMAGE for a larger view of CBIIT's HPC Overview

Image Removed

 

Blog Central

Blog Posts
reversetrue
contenttitles

Recent Updates

Round Rectangle
titleMonthly Updates
Expand
titleClick Here for Recent Updates

Table of Contents

Table of Contents
maxLevel4
minLevel3

3/6/17

  • High Performance Computing User Group & Office Hours – The HPC Program is developing a user group with the objective of serving those requiring HPC assistance, and expanding the HPC community within the NCI intramural community.  The program will be implementing office hours in Frederick on a recurring basis, to serve as points of interaction among the developing HPC community and to aid those with HPC-specific needs or questions.
  • CANDLE Workshop – Serving as one of the key computational frameworks to support the NCI DOE Collaboration, the CANcer Distributed Learning Environment is designed to use machine-learning algorithms to find patterns in large datasets with the goal of offering insights that may ultimately result in improved cancer treatment.  Using this computational architecture, participating DOE labs are focused on accelerating methods to identify promising new treatments; deepening understanding of cancer biology; and understanding the impact of new diagnostics, treatments and patient factors in cancer outcomes.  
    • There will be a workshop on April 18-19 at the NIH which will provide an opportunity to learn more about deep learning, share insights into research, and explore the potential through open-ended discussions and hands-on activities.
    • Stay tuned for more information and registration details – please contact Miles Kimbrough (miles.kimbrough@nih.gov) or Eric Stahlberg (eric.stahlberg@nih.gov) with questions or to discuss further.
  • Globus user-focused Webinar – Globus, a cloud-authenticated data management and transfer platform, will be hosting a user-focused webinar in mid-April to benefit those interested in exchanging datasets across a variety of sources.  The webinar will provide a high-level overview of Globus, steps to start using the service, and common use cases along the following topics:
    • When, where, and why to use Globus?
    • NIH account specs - distinction from Globus Plus
    • What do system administrators need to set up managed endpoints?
    • Which endpoints are already set up?
    • How to set up Globus on your own desktop
    • How to transfer and share data
    • If sharing with collaborator, what info does collaborator need?  What do you need to give to collaborator?
    • New Globus command line interface, allowing users to script their transfers
    • Encryption, verification, and expected data transfer speeds as compared to other resources (e.g. FTP)
  • More details are forthcoming – in the meantime please contact Miles Kimbrough (miles.kimbrough@nih.gov) to discuss further.


2/6/17

  • HPC Data Management Environment 1.0.0 Release
–
  •  – The HPC data management environment (DME) provides a number of application programming interfaces (APIs) to support common scientific data/metadata management functions across the NCI.  Currently in initial release,  HPC DME archive storage can be a permanent storage solution for user data, and can be used as a platform to search, manage and transfer data onto other storage systems.  Common use scenarios may be characterized as follows:
    • Register a collection (PI Lab, Project, Run, Sample or dataset)
    • Register a single data file/object into storage archive synchronously/asynchronously
    • Perform an update on a metadata attribute
    • Subscribe to a known event
    • Generate a report
    • Update/assign permission
    • Perform simple search functions
    • Download a data file/object to Globus share or local directory
  • To obtain an HPC DME account, please contact the DME team
at
at
  • 2017 Globus Hackathon
–
  • – Globus, a cloud-based data management and transfer platform, hosted a Hackathon in January at the NIH main campus to benefit those interested in exchanging datasets across a variety of sources.  The event provided hands-on training, sample code writing, and breakout sessions to the more than 40 attendees.  As the event was very well-received, and due to increasing demand, scheduling efforts are now underway for an NIH-wide user-focused webinar to provide a high-level overview of Globus and steps to start using the service.  More details are forthcoming – in the meantime please contact Miles Kimbrough (miles.kimbrough@nih.gov) to discuss further.


1/10/17

  • 2017 Globus Hackathon – Globus, a cloud-based data management and transfer platform, will be hosting a Hackathon at the NIH on January 24-25.  This event will benefit those interested in exchanging datasets across a variety of sources and will be separated into two training sessions based on prior technical experience – a high-level overview for researchers and system administrators, and a developer-focused session.   The Hackathon is free to NIH employees and will include hands-on training, sample code writing, and breakout sessions to promote cross-collaboration.  Register here and review the agenda here.
  • CBIIT TechScouts
–
  •  – CBIIT TechScouts currently serves the CBIIT community as an idea-sharing platform which allows members to share new tools, technologies, and methodologies relevant to added workplace efficiency.  In order to promote broader community engagement and increase awareness of this platform, the TechScouts will be undergoing some exciting new developments throughout 2017.  More updates will follow soon – in the meantime, please contact Miles Kimbrough (miles.kimbrough@nih.gov) to discuss further.


12/16/16

  • SuperComputing 2016 - As a follow-up to the 2016 International Conference for High Performance Computing, a blog has been made available on the NCIP Hub which summarizes NCI's involvement in the conference and sheds light on the challenges and opportunities that lay ahead.  The blog can be found at https://ncip.nci.nih.gov/blog/precision-medicine-inspires-hpc/.

12/8/16

  • Frontiers of Predictive Oncology and Computing Meeting (FPOC)– With over 100 attendees from across the Department of Energy, the National Cancer Institute, academia, industry and other government agencies, the FPOC meeting (hosted by Intel July 12-14, 2016) provided an opportunity to gain insight into challenges and opportunities for the future. The white paper summarizing the meeting is now available through Intel’s website and can be found
at
–
  •  – The 2016 International Conference for High Performance Computing welcomed NCI CIO, Dr. Warren Kibbe and other representatives from CBIIT in Salt Lake City, UT, November 12-18, for a series of presentations and workshop sessions.  These included a Computational Approaches for Cancer workshop, a Plenary Session with Dr. Kibbe serving as a panelist, and a Birds-of-a-Feather session.  Electronic copies of these presentations will be circulated over the next month – more information on the conference can be found
at
Hackathon
  • Hackathon – Globus, a cloud-based data management and transfer platform, will be hosting a Hackathon at the NIH on January 24-25.  The Hackathon is free to NIH employees and will include hands-on training, sample code writing, and breakout sessions to promote cross-collaboration. 
Register
and
  •  and review the
agenda
  • agenda here.
  • Communications about HPC in NCI
–
  •  – To reflect current and emerging efforts of the NCI’s collaboration with the DOE, a landing page has been created
at
which
  •  which will provide an anchor point for new developments and resources related to the collaboration.
  • HPC Education and Training
–
  •  – As a part of ongoing outreach efforts, The HPC Program will be coordinating a PI-level presentation with the CCR in early January.  Primary presentation will include application domains, NCI DOE collaborative updates, and Big Data management, among others.  For more information, please contact Eric Stahlberg (eric.stahlberg@nih.gov) or George Zaki (george.zaki@nih.gov). 

10/3/16

  • High Performance Computing (HPC) Education & Training
–
  •  – The HPC Program will be sponsoring a Graphics Processing Unit (GPU) Training Workshop on Tuesday, October 25th, in collaboration with CIT.  The workshop will cover the fundamentals of GPU architecture and programming through hands-on exercises.  Please contact either George Zaki (George.zaki@nih.gov) or Miles Kimbrough (miles.kimbrough@nih.gov) for more information.
  • 2017 Globus Hackathon
–
  •  – Globus, a cloud-based data management and exchange platform, will be hosting a Hackathon at the NCI in late January.  The Hackathon will include hands-on training, sample code writing, and breakout sessions to promote cross-collaboration.  Stay tuned for further information and registration details.
  • HPC and Data Management Support
–
  •  – For assistance with Globus, GPUs, and other high-performance computing or data management needs, visit NCI at Your Service (service.cancer.gov) and simply submit a request for support using the ‘High Performance Computing / Data Management’ request button. Or, simply email George Zaki (george.zaki@nih.gov) or Miles Kimbrough (miles.kimbrough@nih.gov) to get started. 

 

9/6/16

  • Archive Data
Management
  • Management – A new archive data service for large data is now available for evaluation. The new service provides application interfaces to readily access backend storage technologies such as the new Cleversafe storage moved into production last month. Individuals and groups interested in learning more about how this resource may benefit current initiatives may reach out and contact Eric Stahlberg, Miles Kimbrough or George Zaki. 
  • Education and Training -  Learn more about how high-performance computing (HPC) can be used to accelerate cancer research and clinical applications. Individuals and groups interested in learning more about HPC, either in general or with specific technologies and scientific challenges in mind may reach out and contact Eric Stahlberg, Miles Kimbrough or George Zaki.
  • The
upcoming
  • upcoming Computational Approaches for Cancer
workshop
  •  workshop scheduled for November 13, 2016 as part of the International Conference for High Performance Computing, Networking, Storage and Analysis has extended a call for extended abstracts until September 15, 2016. More information can be obtained at the
link

8/8/16

  • Frontiers of Predictive Oncology and Computing Meeting - With over 100 attendees from across the Department of Energy, NCI, academia, industry and other government agencies, the meeting (hosted by Intel July 12-14, 2016) provided an opportunity to gain insight into challenges and opportunities for the future. A white paper summarizing the meeting is to be developed.
  • New Data Services with Cleversafe
–
  •  – The Cleversafe storage system officially was moved into a production operational status at the beginning of August. Led by the IT Operations Group at Frederick National Laboratory and working with many stakeholders including CCR, CBIIT and NIH CIT, the new system is used within industry and in key efforts such as the Genomic Data Commons to provide a high level of data assurance for archive and stable data. Stay tuned for further information on opportunities to learn more how this new resource may benefit your scientific and operational needs.
  • Education and Training - Plans are underway to develop educational opportunities to learn more about how high-performance computing (HPC) can be used to accelerate cancer research and clinical applications. Individuals and groups interested in learning more about HPC, either in general or with specific technologies and scientific challenges in mind may reach out and
contact
or
  •  or George Zaki.
  • Computational Approaches for Cancer workshop - Scheduled for November 13, 2016 as part of the International Conference for High Performance Computing, Networking, Storage and Analysis. A call for papers has been issued. More information can be obtained at the
link

 

 

Team Calendar

Team Calendarsid599d4e59-fe32-4915-9da0-3fbe5494d737