220.000 cores and counting: MIT math professor breaks record for largest ever Compute Engine job on Google Cloud

20 Apr 2017 Mountain View - On the Google Cloud blog it was announced that a MIT math professor recently broke the record for the largest ever Compute Engine cluster, with 220.000 cores on Preemptible VMs, the largest known high-performance computing cluster to ever run in the Google public cloud.

Alex Barrett, GCP Blog Editor & Michael Basilyan, writes in the blog about Andrew V. Sutherland who is a computational number theorist and Principal Research Scientist at MIT, and is using Compute Engine to explore generalizations of the Sato-Tate Conjecture and the conjecture of Birch and Swinnerton-Dyer to curves of higher genus. In his latest run, he explored 1017 hyperelliptic curves of genus 3 in an effort to find curves whose L-functions can be easily computed, and which have potentially interesting Sato-Tate distributions. This yielded about 70.000 curves of interest, each of which will eventually have its own entry in the L-functions and Modular Forms Database (LMFDB).

His computations are pleasantly parallel - they can be easily separated into multiple, independent tasks - and he grabs available instances across any and all Google Cloud Regions. An average of about 2-3% of his instances are typically preempted in any given hour, but a simple script automatically restarts them as needed until the whole job is complete.

Ad Emmen