Access to the OLCF's recently deployed Titan supercomputer, a GPU-accelerated Cray XK7 system with a peak performance of 27 petaflops, gave Jeroen Tromp's team the quickest route to making its computing-intensive vision a reality.
"We had this vision that we could use this GPU-accelerated architecture for a global imaging project", Jeroen Tromp stated. "I always felt like this was the frontier of seismic imaging, and I firmly believed that our techniques had wonderful insight to offer."
The team's method, called adjoint tomography, leverages forward waves that travel from an earthquake's point of origin to seismic receivers scattered across the globe. The method then derives artificial waves traveling from the receiver to the quake, called adjoint waves, to help fill gaps in the seismic data. Since their initial OLCF visit, Jeroen Tromp and his team have applied their iterative full waveform inversion technique on Titan at unprecedented scales, using nearly 500 million core hours over a period of 6 years.
In 2019, Jeroen Tromp's team completed its second-generation 3D model of the Earth's interior, called GLAD-M25. The new model draws from the seismic data generated by nearly 1500 earthquakes - a sixfold increase from the team's first-generation model - to capture smaller-scale features within the Earth's subduction zones, plumes, and massive hot spots.
In addition to incorporating more data, the team developed better techniques to address the spotty record of recorded seismic events and integrated more resilient workflow management software to help manage thousands of computing jobs simultaneously.
"These improvements, combined with the increase in the number of earthquake events incorporated into the model, has really helped bring things into focus", Jeroen Tromp stated. "GLAD-M25 is doing a fabulous job at bringing smaller-scale features to light."
To help evaluate GLAD-M25, the team juxtaposed it with regional models of Europe, Asia, and North America in a recent paper submitted to Geophysical Journal International. Though more ambitious in its size and scale, GLAD-M25 compared favorably to its regional brethren, which typically constrain imaging to the crust and upper mantle.
Improved pre- and postprocessing of simulation data and new software management tools played a leading role in advancing GLAD-M25 to this point, helping to sort, sift, and smooth the data to fit model parameters. In the past, manual supervision and finetuning of this workflow limited the team's output to about one model update a month. Automated software produced in cooperation with OLCF staffhelped to drastically reduce this cycle during creation of the second-generation model and paved the way to expand the number of seismic events incorporated into the team's model.
The path forward has not been without challenges, however. Early in the project's history at the OLCF, Jeroen Tromp's team worked with OLCF staff to overcome bottlenecks in the adjoint tomography workflow, implementing an ORNL-developed parallel I/O library called ADIOS to improve performance.
After several years of running on Titan, the team began to encounter job failures on the aging system that threatened to slow the pace of progress. Collaborating with fellow OLCF user Shantenu Jha of Rutgers University, the team incorporated a new tool called EnTK to automatically detect and quickly recover from these failures.
"If ten jobs out of 1480 jobs failed, EnTK could tell you exactly what ten they were and allow you to resubmit with the push of a button. I can't envision doing this ever again without these kinds of tools", Jeroen Tromp stated.
With Titan scheduled to be decommissioned in mid-2019, Jeroen Tromp's team is transitioning its project to Summit, the OLCFs newest supercomputer, currently the most powerful in the world. The team is eyeing a three- to fourfold increase in seismic data on the IBM AC922 Summit and an 8 second reduction in the shortest seismic wavelength the team can process - from 17 seconds to about 9 seconds.
Though Titan will be taken offline, the Cray XK7's contributions to science and global seismic imaging will reverberate well into the future, Jeroen Tromp said.
Titan changed everything for me, he said. Thanks to Titan, weve been able to demonstrate the feasibility of global full waveform inversion. It would have never happened without Titan. That has been an amazing opportunity for me and something that I think will be lasting.