Many scientists throughout the Netherlands are using these data for their specific research questions. But how to share and use 750+ files with over 100 TB of data in an efficient and scalable manner from multiple locations? This is where SURFsara and SURFnet play an important role.
In this specific case, the group from Paul de Bakker at the UMC Utrecht wants to analyze (subsets) of the GoNL data at the local High-performance Computing (HPC) cluster. After initial consultation with people from SURFsara and SURFnet, the following approach was undertaken. First, data need to be synchronized between UMC Groningen and the Grid storage at SURFsara. This synchronization was done using the existing 10 Gbit/s lightpath connection between LOFAR in Groningen and SURFsara in Amsterdam, which ensured a fast and reliable transfer of approx. 60 TB of data. The data then needed to be split up in smaller chunks, so that subsets can more easily be downloaded and used at the local HPC cluster. This last step is currently in progress.
For this project, the support from SURFsara and SURFnet has been crucial, both in terms of providing resources - storage, high-speed connections and compute power, as well as providing expert knowledge about the design and implementation of the solution.