Graphics Processing Units (GPUs) can enable researchers to accelerate scientific applications. However, properly leveraging this hardware and exposing parallelism in an application can be challenging. Further, changes in GPU hardware and software on 6 month to 2 year timescales makes it difficult for domain scientists to keep up with this rapidly changing environment. Hackathons provide short bursts of intense programming, where developers work alongside GPU programming experts to port and tune their software applications for the latest GPU hardware.
The 2018 Boulder GPU Hackathon was made possible through sponsorships from Nvidia, OpenACC, and Google Cloud, as well as organizational support from Fluid Numerics, CIRES/CU Boulder, and Oak Ridge Leadership Class Facility (OLCF). Developer teams came from NASA, Los Alamos National Laboratory, Pacific Northwest National Laboratory, Lawrence Berkeley National Laboratory, National Renewable Energy Laboratory, National Center for Atmospheric Research, NOAA National Severe Storms Laboratory, University of Chicago, University of Maryland, University of Colorado (Boulder), University of Washington, SilcsBio, and General Atomics. Mentors came from Nvidia, PGI, Google Cloud, ARM, LBNL and University of Tennessee.
Daily, teams worked with their mentors to accelerate their own software applications by porting to and optimizing for GPU hardware. In the early afternoon, scrum sessions were held for all teams to report on their progress and their struggles. This catalyzed collaborative conversations that enabled teams to overcome their struggles quickly. Developers often found that another group had already encountered similar problems they were facing and were able to help them along. In an anonymous survey of the hackathon attendees, one noted that “[it was] awesome to discuss problems with other attendees and hear similar situations and get ideas I would not have thought of.”
In addition to gaining experience in GPU programming, attendees were also given the opportunity to work on Google Cloud Platform. This was a unique experience given that the Boulder GPU Hackathon was the first to make use of cloud computing resources. With sponsorship from Google Cloud and help from Wyatt Gorman (HPC Specialist at Google), a HPC cluster was set up consisting of 20 virtual machine compute nodes outfitted with Nvidia Tesla V100 GPU’s.
This platform allowed us to customize and take control of the software and hardware environment to specifically meet the attendee needs for the hackathon. For this hackathon, the software environment included the CentOS operating system outfitted with SchedMD’s Slurm job scheduler and the latest Nvidia drivers ( v 3.96 ). Additionally, we were able to provide the latest CUDA toolkit ( v 9.2 ), along with the most recently released community edition of the PGI compilers ( v 18.5 ). With this as the base install, the spack package management tool allowed for rapid installation of many typical HPC packages, like OpenMPI, MPICH, NetCDF, Lapack, and BLAS. The ability to configure this environment through a startup script allowed the setup of an HPC cluster to happen within a day. Federico Halpern, from the GA Fusion team noted that “GCP allowed for easy installation of packages. The hardware was top-notch.”
By the end of the week, each team had made significant progress towards accelerating their applications. The chart to the left shows the speedup in four team’s applications relative to the run time measured at the beginning of the week. These four developer teams, AMReX, EF5, GA Fusion, and NASA-LAVA were able to achieve between 3x and 5x speedup in just five days. For the other six teams not shown in this chart, most had speedups on individual kernels within their code but not in the overall run time. This is the result of a typical struggle in GPU programming : minimizing data movement between the CPU and GPU. Nonetheless, every team left the hackathon with a path forward and a plan to continue working on their application.
At the end of the event, we conducted an anonymous survey to understand how attendees think about the hackathons. Of the 66 total attendees and mentors, we were able to obtain 29 responses.
The 2018 Boulder GPU Hackathon was the first hackathon for the majority ( 75% ) of our attendees and the majority accomplished what they thought they would for the one week event. Everyone surveyed indicated that they would recommend that other developers attend a hackathon.
When asked what they liked most about the hackathon, attendees responded with comments like
“The community and the structure.”
“The support and mentoring. Mentors dig deep into the details of each specific application with the groups.”
“There were many GPU/compiler experts in the same room, so we could ask questions and solve problems quickly.”
“It was a great event. Interacting with other developers helped a lot.”
“Getting lots of talented people with different backgrounds in the same room. It was the inter-project collaboration that was the biggest benefit.”
Overall, the success of the hackathon is due to the motivated people that came together to be a part of this growing community. Everyone came with the idea that they were going to learn something new and challenging while sharing their expertise with others.
What’s Next ?
As this community continues to grow and the number of hackathons increases, we feel it’s important to share this experience with others. As a part of this, we have developed guidelines for mentors, gleaned from conversations throughout the week with mentors and attendees. In the coming months, we’ll be coming out with documentation on “Preparing for a hackathon as an attendee” and “What it takes to organize a successful hackathon”.
Organization of hackathons in new locations to share and grow this development methodology is another part of sustaining this experience. Currently, applications are open for our next hackathon in Santa Fe, NM taking place at the end of October this year.
Sign up for the gpuhackathon e-mail list to hear more about GPU computing and our upcoming hackathons !
If you are interested in bringing a hackathon to your organization, reach out and we can organize a hackathon !
The Boulder GPU Hackathon could not have happened without the support of our sponsors at Nvidia and Google Cloud.
Sponsors are critical to help cover mentor travel costs and other workshop expenses. Sponsor a hackathon today !