GPFS Parallel File System Explained

Barbara. October 31, 2020

What is GPFS?

IBM Spectrum Scale, formerly the General Parallel File System (GPFS) is high-performance clustered file system software developed by IBM in 1998. It was originally designed to support high-streaming applications in the media and entertainment space. It can be deployed in shared-disk or shared-nothing distributed parallel modes, or a combination of these two. The predominant deployment is as a shared-disk solution utilizing SAN block storage for persistent storage.

Advantages of a Modern Parallel File System over GPFS

Although IBM Spectrum Scale (aka IBM GPFS) is a parallel file system, it was developed over 20 years ago to support the high throughput needs of multimedia applications. And for the subsequent 15+ years it was optimized to deliver the highest large file bandwidth performance from hard disk drives. But the world has changed dramatically in the last ten years, flash technology has revolutionized the data center and workloads have changed from large sequential to metadata intensive small File I/O intensive big data workloads. The technology designed and fine-tuned to solve streaming applications is not suitable for the age of big data and analytics. IBM continues to develop its Spectrum Scale product, yet the file system still can not deliver performance for small file, low latency, I/O intensive workloads.

A modern parallel file system leverages new technologies such as NVMe, high speed low latency networking to service small file I/O in a massively parallel manner. In addition to managing small files efficiently, the metadata demands for AI and GPU workloads is orders of magnitude higher. A modern parallel file system utilizes a distributed metadata model where metadata is spread evenly across all the storage nodes and each node contains equal amounts of data and metadata. By contrast, the traditional GPFS model maintains metadata in separate metadata servers. This creates a performance bottleneck when the system contains millions of files.

Advantages of modern file systems

Best choice for GPU workloads which contain millions of files, large and small
Full featured, Easy to use, easy to manage, no need for specialized administrators to configure, manage and tune the storage system
High throughput, high IOPS, with ultra-low latency, fully saturating the network and delivering applications linear performance at exabyte scale, especially for mixed workloads

GPFS vs. Modern Parallel File System (WEKA)

Hardware	Spectrum Scale (on ESS 3000)	WEKA (COTS Hardware)
Server Nodes per Chassis	1 in 2 RU, Custom	1 in 1RU or 4 in 2RU, COTS
# of Server Nodes	1 to Thousands	6 to Thousands
Server Generation	Intel Skylake	Intel Cascade Lake Refresh, AMD EPYC
Memory per Node	768GB	192GB, 6 channels
SSD Interface	NVMe (still has SAS/SATA baggage)	Natively NVMe
Front-End Network	Up to 100G	Up to 200Gb
NVMe over Fabric	Requires 3rd-Party Add-On	Native
Protocol Support
POSIX	Yes	Yes
GPUDirect	No	Yes
NFS	Yes	Yes
SMB	Yes	Yes, SMB 2/3
HDFS	Yes	Not Required: Mount from POSIX
S3	Yes	Yes
Performance¹
Read Throughput^1,2	40GB/s	56GB/s
Write Throughput^1,2	32GB/s	20GB/s
Read IOPS¹	No Data from Vendor	5.8M
Write IOPS¹	No Data from Vendor	1.6M
Single Mount Point, Full Coherency	No Data from Vendor	82GB/s
#1 on IO500 and SPEC	No	Yes

¹ At smallest cluster configuration
² Spectrum Scale on ESS 3000, performance will vary on other hardware

Feature Comparison: GPFS vs. Parallel File System (WEKA)

Feature	Spectrum Scale (on ESS 3000)	WEKA
Ease of Use	Hard	Easy
Snapshots	Yes, with Performance Impact	Yes – Instantaneous
Snapshot-to-Object Store/S3	No	Yes
Tiering/Hybrid Disk Storage	No, Tiering Impacts Performance	Yes
Independent Cap/Perf Scaling	No	Yes
Thousands of Nodes	Yes	Yes
Dynamic Performance Scaling	No	Yes
Encryption	In-Flight and At-Rest	In-Flight and At-Rest
Replication	Yes	Yes (via Snapshot)
Data Protection	N+3, Reed Solomon EC only on ESS Appliance	N+4 Distributed Data Protection with High Performance EC
Compression/Deduplication	Limited/No	Future Release
S/W Only, H/W Independent	Yes	Yes
End-to-End Checksum	No	Yes
Active Directory and LDAPP Authentication	Yes	Yes
Compression/Deduplication	Limited/No	Future Release
S3	via Swfi	Yes
SMB	Limited/No	Yes, SMB 2/3
User Quotas	Yes	Yes
Authenticated Mounts	Yes	Yes
High Availability Bonding	Yes	Yes
IB Support (Ideal for GPUs)	Yes	Yes