Back to Table of contents

Primeur weekly 2018-03-26

Quantum computing

Physicists reveal material for high-speed quantum internet ...

Focus on Europe

TUM junior scientist wins prestigious award for Computational Studies of Surface Catalytic Processes ...

Cosmologists create largest simulation of galaxy formation and break their own record ...

GigaSpaces triples annual revenue in France, building on strong global 2017 results ...

Atos and GENCI launch the Atos Joseph Fourier Award 2018 to accelerate research and innovation in numerical simulation, artificial intelligence and quantum computing ...

Middleware

Cavium Collaborates with Microsoft to demonstrate ThunderX2 platform compliant with Microsoft's Project Olympus specifications ...

Hardware

Pioneers of computer architecture receive ACM Turing Award ...

Mellanox's advanced Ethernet solutions selected to connect the Tel-Aviv Stock Exchange network infrastructure ...

GIGABYTE announces ThunderXStation: Industry's first Armv8 workstation based on Cavium's ThunderX2 processor ...

Cavium announces Packet Trakker: A programmable network telemetry suite ...

Applications

Los Alamos releases file index product to software community ...

NERSC supercomputers help researchers create reference catalogue for rumen microbiome ...

Major AI breakthrough by Optalysys demonstrating how its patented Optical Processing Technology performs deep learning at the speed of light ...

NVIDIA RTX technology realizes dream of real-time cinematic rendering ...

Batteries out of thin air ...

New method manages and stores data from millions of nerve cells - in real time ...

Groundbreaking research results in artificial intelligence ...

Taming chaos: Calculating probability in complex systems ...

Physicists made crystal lattice from polaritons ...

Golden touch: Next-gen optical disk to solve data storage challenge ...

COSMIC impact: Next-gen X-ray microscopy platform now operational ...

The Cloud

Mellanox simplifies hybrid Cloud connectivity between enterprises and Microsoft Azure ...

Cavium Technologies power end-to-end NVMe over Fabrics solutions ...

Los Alamos releases file index product to software community


The Trinity supercomputer.
22 Mar 2018 Los Alamos - Resolving the supercomputer challenge of searching and retrieving files could now be far simpler, with a tool developed by Los Alamos National Laboratory and released to the GitHub open-source software site. The Grand Unified File Index (GUFI) is designed using a new, hierarchical approach to storing file metada, allowing rapid parallel searches across many internal databases. Queries that would previously have taken hours or days can now be run in seconds.

"We anticipate that the Grand Unified File Index will have a big impact on the ability for many levels of users to search data and get a fast response", stated Gary Grider, division leader for High Performance Computing at Los Alamos. "Compared with other methods, the Grand Unified File Index has the advantages of not requiring the system administrator to do the query, and it honors the user access controls allowing users and admins to use the same indexing system", he stated.

Why develop a new search-and-retrieval tool? At Los Alamos and other supercomputing facilities around the world, databases for file metadata may potentially hold hundreds of millions of records, yet they are typically inefficient for the kinds of searches that are actually needed.

In recent decades, a major issue has been providing storage that could handle huge flows of data going in and out of state-of-the-art supercomputers. Handling these huge volumes quickly and economically allows the machines to make progress on scientific calculations that support national security, as well as for basic scientific research in fields such as engineered materials, biological processes, and earth systems modelling.

One important solution to the storage bottleneck has been the Parallel File System (PFS). A PFS allows many related streams of data to be moved at the same time, without losing track of how they are related. Unfortunately, searching through the lists of files that are stored in such systems remains difficult.

"Simple queries, such as 'where is the simulation data that was done with that new computational method?' could bring the PFS to its knees", stated Jeff Inman, one of the developers of the new search tool. System administrators frequently need to query to find which files should be archived, who is using the most storage, whether a dataset has been moved from PFS to tape, and so forth, he noted, and any of these queries may seriously compromise the performance of the file-system to send and receive data.

The trick used by GUFI is to store file-metadata in a hierarchy of databases, matching the hierarchy of folders. This allows rapid parallel searches across many databases, and allows access-permissions to be managed in the same way they are managed in a normal hierarchy of folders. GUFI can hold file-metadata from tape-archives, PFS, and other kinds of file-systems, unifying information from all the places where a file might reside.

The Laboratory is planning to initially present the work at a Microsoft gathering in March and a subsequent HPE session, then rolling it out at the IEEE Massive Storage Systems and Technologies Conference (MSST) May 14.

GUFI is now available at GUFI GitHub as open-source software for interested users to download and explore.

The GUFI project was funded by the U.S. Department of Energy's National Nuclear Security Administration Advanced Simulation and Computing Programme.
Source: Los Alamos National Laboratory

Back to Table of contents

Primeur weekly 2018-03-26

Quantum computing

Physicists reveal material for high-speed quantum internet ...

Focus on Europe

TUM junior scientist wins prestigious award for Computational Studies of Surface Catalytic Processes ...

Cosmologists create largest simulation of galaxy formation and break their own record ...

GigaSpaces triples annual revenue in France, building on strong global 2017 results ...

Atos and GENCI launch the Atos Joseph Fourier Award 2018 to accelerate research and innovation in numerical simulation, artificial intelligence and quantum computing ...

Middleware

Cavium Collaborates with Microsoft to demonstrate ThunderX2 platform compliant with Microsoft's Project Olympus specifications ...

Hardware

Pioneers of computer architecture receive ACM Turing Award ...

Mellanox's advanced Ethernet solutions selected to connect the Tel-Aviv Stock Exchange network infrastructure ...

GIGABYTE announces ThunderXStation: Industry's first Armv8 workstation based on Cavium's ThunderX2 processor ...

Cavium announces Packet Trakker: A programmable network telemetry suite ...

Applications

Los Alamos releases file index product to software community ...

NERSC supercomputers help researchers create reference catalogue for rumen microbiome ...

Major AI breakthrough by Optalysys demonstrating how its patented Optical Processing Technology performs deep learning at the speed of light ...

NVIDIA RTX technology realizes dream of real-time cinematic rendering ...

Batteries out of thin air ...

New method manages and stores data from millions of nerve cells - in real time ...

Groundbreaking research results in artificial intelligence ...

Taming chaos: Calculating probability in complex systems ...

Physicists made crystal lattice from polaritons ...

Golden touch: Next-gen optical disk to solve data storage challenge ...

COSMIC impact: Next-gen X-ray microscopy platform now operational ...

The Cloud

Mellanox simplifies hybrid Cloud connectivity between enterprises and Microsoft Azure ...

Cavium Technologies power end-to-end NVMe over Fabrics solutions ...