Back to Table of contents

Primeur weekly 2020-01-06

Focus

The LUMI supercomputer is not just a very fast supercomputer, it is first of all a competence development platform - Interview with Kimmo Koski, CSC, Finland ...

Quantum computing

ORNL researchers advance performance benchmark for quantum computers ...

In leap for quantum computing, silicon quantum bits establish a long-distance relationship ...

The Quantum Information Edge launches to accelerate quantum computing R&D ...

Focus on Europe

The coolest LEGO in the universe ...

Middleware

BP looks to ORNL and ADIOS to help rein in data ...

Hardware

New year brings new directory structure for OLCF's high-performance storage system ...

GIGABYTE brings AI, Cloud solutions and smart applications to CES 2020 to enable future today ...

During its final hours of operation, the Titan supercomputer simulated the birth of supernovae ...

Big iron afterlife: How ORNL's Titan supercomputer was recycled ...

Applications

Stanford researchers build a particle accelerator that fits on a chip ...

Brain-like functions emerging in a metallic nanowire network ...

Award-winning engineer helps keep US nuclear deterrent safe from radiation ...

New algorithm could mean more efficient, accurate equipment for Army ...

Paul Ginsparg named winner of the 2020 AIP Karl Compton Medal ...

'Super' simulations offer fresh insight into serotonin receptors ...

Researchers accelerate plasma turbulence simulations on Oak Ridge supercomputers to improve fusion design models ...

New year brings new directory structure for OLCF's high-performance storage system

2 Jan 2020 Oak Ridge - The High-Performance Storage System (HPSS) at the Oak Ridge Leadership Computing Facility (OLCF) will soon transition to a project-based directory structure. This may sound like a small change, but for research teams storing their project data on HPSS, the update promises to simplify file management and streamline searching for (or sharing) the right data.

Under the current system, users can save data to both personal and project directories - but that habit has often created confusion over where specific project data can be found by other team members. For example, if a user is assigned to multiple projects and saves all of this different data into a personal directory, other team members may have difficulty identifying which data belongs to what project. So, starting in January 2020, all data storage will be restricted to project areas.

"By only allowing writing to project directories, we eliminate confusion as to what file belongs to which project", stated Mitchell Griffith, an archival storage software developer for HPSS at the OLCF, a US Department of Energy (DOE) Office of Science User Facility at DOE's Oak Ridge National Laboratory. "If a user writes a file to project C's directory, then that project owns the file."

The new hierarchy is based on the project directory layout used by the OLCF's Spider storage system, making the two systems more compatible and eventually allowing project teams with sensitive, proprietary, or export-controlled data to be able to use HPSS to archive their data. Today, those project teams do not have access to HPSS, partly due to the current layout structure.

Each project will have three writable directories:

  • /hpss/prod/PROJECTID/proj-shared: for sharing files between project members
  • /hpss/prod/PROJECTID/users/$USER: for data needed mainly by that one user
  • /hpss/prod/PROJECTID/world-shared: for sharing open data between projects

This project-based structure also helps avoid potential disorganisation when users leave a project as they move to other institutions, graduate, or start their own research projects.

"This is to prevent ambiguity in file ownership. Projects last longer than people, and we want to ensure some basic metadata about what is written to HPSS", Mitchell Griffith stated. "By doing this, we can associate a project's description with the file and have a better understanding of what is written to HPSS."

Users with data currently stored in their personal directories (at /home/$USER) will be encouraged to start transferring it to the correct project areas; Mitchell Griffith said the HPSS team expects the data migration to be fully completed in 12 to 18 months.

To easily transfer their data, Mitchell Griffith advises users to employ the "mv" (move) command instead of the "cp" (copy) command. The mv command is a quick metadata operation, whereas cp will copy data in HPSS, which can be a lengthy process.

Once a user's home directory is empty, it will become an unwritable managed directory and links will be automatically created to projects accessible by the user.

Meanwhile, the HPSS team will rename the current /proj/PROJECTID directories to /hpss/prod/PROJECTID/proj-shared, then add a link to that area.

"There will be an HPSS downtime when this is done, but this means technically the old project data is in a different location", Mitchell Griffith stated. "However, the link should allow the users to access the data like they have always accessed the data. We are doing metadata operations for the restructure instead of copying data."

Source: Oak Ridge Leadership Computing Facility - OLCF

Back to Table of contents

Primeur weekly 2020-01-06

Focus

The LUMI supercomputer is not just a very fast supercomputer, it is first of all a competence development platform - Interview with Kimmo Koski, CSC, Finland ...

Quantum computing

ORNL researchers advance performance benchmark for quantum computers ...

In leap for quantum computing, silicon quantum bits establish a long-distance relationship ...

The Quantum Information Edge launches to accelerate quantum computing R&D ...

Focus on Europe

The coolest LEGO in the universe ...

Middleware

BP looks to ORNL and ADIOS to help rein in data ...

Hardware

New year brings new directory structure for OLCF's high-performance storage system ...

GIGABYTE brings AI, Cloud solutions and smart applications to CES 2020 to enable future today ...

During its final hours of operation, the Titan supercomputer simulated the birth of supernovae ...

Big iron afterlife: How ORNL's Titan supercomputer was recycled ...

Applications

Stanford researchers build a particle accelerator that fits on a chip ...

Brain-like functions emerging in a metallic nanowire network ...

Award-winning engineer helps keep US nuclear deterrent safe from radiation ...

New algorithm could mean more efficient, accurate equipment for Army ...

Paul Ginsparg named winner of the 2020 AIP Karl Compton Medal ...

'Super' simulations offer fresh insight into serotonin receptors ...

Researchers accelerate plasma turbulence simulations on Oak Ridge supercomputers to improve fusion design models ...