The archive currently consists of more than 40 petabytes of data and is projected to grow by roughly 75 petabytes annually over the next five years. For comparison, a petabyte of data is equal to roughly 210,000 DVDs. As climate simulations are carried out on increasingly powerful supercomputers, massive amounts of data are produced that must be effectively stored and analyzed.
To efficiently manage and provide quick access to this amount of data for analysis and research through its Hierarchical Storage Management system, the Center has selected the High Performance Storage System software solution developed by IBM and the U.S. Department of Energy. Supported by IBM services, this solution is capable of handling more than 500 petabytes of data.
Organisations including the Intergovernmental Panel on Climate Change, the Max-Planck Institute for Meteorology, the University of Hamburg, the Helmholtz-Zentrum Geesthacht Center for Materials and Coastal Research, the Alfred Wegener Institute Helmholtz Center for Polar and Marine Research, the West African Science Service Center on Climate Change and Adapted Land Use, The Coordinated Energy and Water Cycle Observation Project, the Programme For Earth System Modelling, and the Center itself use data in the archive for a range of research and reporting activities, including:
"Addressing the challenges of climate change and its impact on the environment and daily life for everyone on the planet requires gathering, storing and analyzing massive amounts of data", stated Professor Thomas Ludwig, director of the German Climate Computing Center and professor of computer science at the University of Hamburg. "We rely on the technology and the expertise of IBM to help us provide this essential service to the world's foremost climate and environmental research institutions."
The upgrade of the Center's Hierarchical Storage Management system includes a solution consisting of the latest x86 servers and Red Hat Linux operating system running the HPSS software. This solution manages all simulation data and serves as the input and output interface between the Center's high performance computing systems and the tape storage library where the climate data is stored.
"At IBM, one of the values we try to live up to daily is providing innovation that matters for our company and the world", stated Markus Koerner, vice president, IBM Global Technology Services. "I can think nothing that matters more than helping to effectively predict and mitigate the impact of climate change, and we believe the technology and services we provide for this Big Data challenge will significantly help advance the science of climate change."
The solution uses a disk cache of five petabytes of storage capacity and is capable of providing data access speeds of up to 12 gigabytes per second, which will be upgraded to 18 gigabytes per second later this year. IBM Global Technology Services supports this solution with data centre managed services on site, project management services, and hardware vendor management services, and the HPSS solution incorporates IBM DB2 scalable relational database software.
HPSS technology is the result of a 20-year collaborative effort between IBM and the Lawrence Livermore, Los Alamos, Sandia, Lawrence Berkeley and Oak Ridge National Laboratories of the U.S. Department of Energy. It is used for very large scientific data archives at research facilities and universities, and for weather forecast systems.