Nick Johnson explains how to speed up Python on the ARCHER supercomputer
26 May 2017 Edinburgh - On the EPCC blog Nick Johnson explains some tricks to speed up Python on the ARCHER supercomputer. He is concentrating on a Marine tech project by the London Firedrake team. Running on ARCHER, it scales well in the computation phase, but not well at start-up. "Why?", Nick Johnson asks. At start-up, the Python interpreter parses its paths and loads a default set of modules, each accessing the same files. On 1000 nodes, this takes about 45 minutes for the 24,000 Python interpreters to go through this routine.
Nick Johnson solved the problem by creating a Python module which provides a script that packages an existing environment suitable for deployment and resolves the specific pathing issues.