IBM Builds Super Fast File System

Found on Betanews on Thursday, 09 March 2006
Browse Computer

IBM on Thursday announced that it had scored a breakthrough in file system technology that increases the speed of data access by seven times. Researchers were able to attain a 102-gigabyte per second transfer rate on the ASC Purple supercomputer at Lawrence Livermore National Laboratory in a recent test.

The file system was an astonishing 1.6 petabytes in size, the largest ever in the world, and performance was maintained even as 1,000 clients pushed workloads into the file. The project used 104 Power-based eServer p575 nodes and 416 storage controllers, IBM said in a statement.

Called the General Parallel File System (GPFS), the technology allows for high-speed access to files across multiple nodes of a Linux or AIX cluster. The file system could be used in a variety of fields, including engineering design, digital media and entertainment, data mining, financial analysis, seismic data processing and scientific research.

IBM will push GPFS on several fronts, including an effort to even promote its use on non-IBM hardware. The source code behind the file system will be released to eligible clients who can develop upon the technology and share their work with others.

There already is new filesystem around which is perhaps not as fast as GPFS, but still better. IBM aimed at speed, but the creators of ZFS had also safety in mind: transactional file writing and checksums for data blocks to detect errors. The limit of ZFS are 256 quadrillion zettabytes (IBM's 1.6 petabytes are 1,638 terabytes, but 1 zettabyte is equal to 1,073,741,824 terabytes).