According to Ahmed Taha, the NCSA senior computational resource coordinator who led this extreme benchmarking project, using such a massive number of cores requires a powerful, robust MPI protocol, which was provided by Cray. CD-adapco also fine-tuned their code to improve parallel I/O.
The result of this collaborative effort over the past eight months was a steady increase in scaling, from a 105 million cell model running on 13,000 cores to a 1 billion cell model hitting the 102,000-core peak.
In addition to demonstrating the robustness of STAR-CCM+, the project also showed off the code's impressive scaling efficiency. The 1 billion cell model was 93% efficient at 60,000 cores, 84% at 71,000 cores, 79% at 79,000 cores, and 75% at 102,000 cores.
"This shows even users who run on lower core counts that they can benefit as well by reducing runtime", Ahmed Taha stated.
Steve Feldman, Senior Vice President for IT at CD-adapco, commented: "Running STAR-CCM+ on the NCSA Blue Waters machine not only showcased our unrivaled technical competence in scaling large runs, but highlighted our industry-unique licensing model as well. CD-adapco is the only major CAE vendor to provide licenses that are 100% uncoupled from the number of compute cores used in parallel to solve a problem. The financial benefits are huge, whether you run on a hundred or a hundred thousand cores."
These results were presented at the 2015 NCSA Private Sector Programme Annual Meeting and the 2015 Blue Waters Symposium.
CD-adapco is now creating a massive 2 billion cell simulation model which will allow the NCSA team to continue exploring the code's scaling potential.