# NVIDIA DGX SuperPOD with WEKApod Data Platform Appliance NVIDIA DGX SuperPOD is fully integrated AI supercomputing for training deep learning models at scale. This reference architecture evaluates the WEKApod appliance as validated storage for those workloads. **Type:** Reference Architecture ## Abstract The [NVIDIA DGX SuperPOD](https://www.nvidia.com/en-us/data-center/dgx-superpod) with NVIDIA DGX systems is an artificial intelligence (AI) supercomputing infrastructure, providing the computational power necessary to train today's state-of-the-art deep learning (DL) models and to fuel future innovation. The DGX SuperPOD delivers groundbreaking performance, deploys as a fully integrated system, and is designed to solve the world's most challenging computational problems. This DGX SuperPOD reference architecture (RA) for DGX systems with x86 architecture CPUs is the result of collaboration between DL scientists, application performance engineers, and system architects to build a system capable of supporting the widest range of DL workloads. The groundbreaking performance delivered by the DGX SuperPOD with DGX systems enables the rapid training of DL models at great scale. The integrated approach of provisioning, management, compute, networking, and fast storage enables a diverse, multi-tenant system that can span data analytics, model development, and AI inference. The WEKA Data Platform on the WEKApod appliance was evaluated for suitability for supporting DL workloads when connected to the DGX SuperPOD. This fully integrated storage appliance is validated with the DGX SuperPOD, and WEKApod is fully supported by WEKA support services. DGX SuperPOD with the WEKApod Data Platform is designed to make large-scale AI simpler, faster, and easier to manage for every organization and their IT team. The WEKApod Data Platform has key capabilities that benefit organizations as they elevate their AI initiatives to DGX SuperPOD scale: - The WEKA file system provides all the performance required for the most demanding HPC and AI data management. - Easy to deploy, WEKApod Data Platform appliances are pre-configured, and provide a highly capable scale-out platform for capacity and performance. - Fully optimized for all types of I/O patterns and data layouts, WEKApod Data Platform appliances deliver data to applications, ensuring excellent GPU resource utilization even with distributed applications running on multiple computing servers. The WEKA shared parallel protocol engages the full capabilities of the data infrastructure, and AI applications achieve excellent results, reliably, and consistently. Learn more about the NVIDIA and WEKA collaboration at: [https://www.weka.io/resources/video/introducing-wekapod/](https://www.weka.io/resources/video/introducing-wekapod/) ## Storage Overview Training performance may be limited by the rate at which data can be read and reread from storage. The key to performance is the ability to write data multiple times, ideally from local storage. The closer the data is cached to the GPU, the faster it can be read. Both persistent and nonpersistent storage needs to be designed to balance the needs of performance, capacity, and cost. ### Storage Caching Hierarchy The storage caching hierarchy for DGX systems is shown in Table 1. Depending on data size and performance needs, each tier of the hierarchy can be leveraged to maximize application performance. ```json { "_key": "rp00073", "_type": "richTableBlock", "headerRows": 1, "markDefs": null, "rows": [ { "_key": "rp00046", "_type": "row", "cells": [ { "_key": "rp00034", "_type": "richTableCell", "content": [ { "_key": "rp00036", "_type": "block", "children": [ { "_key": "rp00035", "_type": "span", "marks": [], "text": "Storage Hierarchy Level" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00037", "_type": "richTableCell", "content": [ { "_key": "rp00039", "_type": "block", "children": [ { "_key": "rp00038", "_type": "span", "marks": [], "text": "Technology" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00040", "_type": "richTableCell", "content": [ { "_key": "rp00042", "_type": "block", "children": [ { "_key": "rp00041", "_type": "span", "marks": [], "text": "Total Capacity¹" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00043", "_type": "richTableCell", "content": [ { "_key": "rp00045", "_type": "block", "children": [ { "_key": "rp00044", "_type": "span", "marks": [], "text": "Performance¹" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00059", "_type": "row", "cells": [ { "_key": "rp00047", "_type": "richTableCell", "content": [ { "_key": "rp00049", "_type": "block", "children": [ { "_key": "rp00048", "_type": "span", "marks": [], "text": "RAM" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00050", "_type": "richTableCell", "content": [ { "_key": "rp00052", "_type": "block", "children": [ { "_key": "rp00051", "_type": "span", "marks": [], "text": "DRAM" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00053", "_type": "richTableCell", "content": [ { "_key": "rp00055", "_type": "block", "children": [ { "_key": "rp00054", "_type": "span", "marks": [], "text": "2 TB²" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00056", "_type": "richTableCell", "content": [ { "_key": "rp00058", "_type": "block", "children": [ { "_key": "rp00057", "_type": "span", "marks": [], "text": "> 200 GB/s" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00072", "_type": "row", "cells": [ { "_key": "rp00060", "_type": "richTableCell", "content": [ { "_key": "rp00062", "_type": "block", "children": [ { "_key": "rp00061", "_type": "span", "marks": [], "text": "Internal storage" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00063", "_type": "richTableCell", "content": [ { "_key": "rp00065", "_type": "block", "children": [ { "_key": "rp00064", "_type": "span", "marks": [], "text": "Flash storage" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00066", "_type": "richTableCell", "content": [ { "_key": "rp00068", "_type": "block", "children": [ { "_key": "rp00067", "_type": "span", "marks": [], "text": "30 TB³" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00069", "_type": "richTableCell", "content": [ { "_key": "rp00071", "_type": "block", "children": [ { "_key": "rp00070", "_type": "span", "marks": [], "text": "> 50 GB/s" } ], "markDefs": [], "style": "normal" } ] } ] } ] } ``` Table 1. DGX system storage and caching hierarchy 1. Total capacity and performance values are per system. 2. Shared between the operating system, application, and other system processes 3. PCIe Gen 4 NVMe SSD storage Caching data in local RAM provides the best performance for reads. This caching is transparent once the data is read from the filesystem. While local storage is fast, it is not practical to manage a dynamic environment with local disk alone in a multi-node environment. Functionally, centralized storage may be as quick as local storage on many workloads. ### Storage Performance Requirements Performance requirements for high-speed storage greatly depend on the types of AI models and data formats used. The DGX SuperPOD has been designed as a fully capable system that can manage any workload both today and in the future. However, if systems are going to focus on a specific workload, such as natural language processing (NLP), it may be possible to better estimate performance needs of the storage system. To allow customers to characterize their own performance requirements, some general guidance on common workloads and datasets is shown Table 2. ```json { "_key": "rp00130", "_type": "richTableBlock", "headerRows": 1, "markDefs": null, "rows": [ { "_key": "rp00099", "_type": "row", "cells": [ { "_key": "rp00090", "_type": "richTableCell", "content": [ { "_key": "rp00092", "_type": "block", "children": [ { "_key": "rp00091", "_type": "span", "marks": [], "text": "Storage Performance Level Required" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00093", "_type": "richTableCell", "content": [ { "_key": "rp00095", "_type": "block", "children": [ { "_key": "rp00094", "_type": "span", "marks": [], "text": "Example Workloads" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00096", "_type": "richTableCell", "content": [ { "_key": "rp00098", "_type": "block", "children": [ { "_key": "rp00097", "_type": "span", "marks": [], "text": "Dataset Size" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00109", "_type": "row", "cells": [ { "_key": "rp00100", "_type": "richTableCell", "content": [ { "_key": "rp00102", "_type": "block", "children": [ { "_key": "rp00101", "_type": "span", "marks": [], "text": "Good" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00103", "_type": "richTableCell", "content": [ { "_key": "rp00105", "_type": "block", "children": [ { "_key": "rp00104", "_type": "span", "marks": [], "text": "NLP" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00106", "_type": "richTableCell", "content": [ { "_key": "rp00108", "_type": "block", "children": [ { "_key": "rp00107", "_type": "span", "marks": [], "text": "Most to all datasets fit in cache" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00119", "_type": "row", "cells": [ { "_key": "rp00110", "_type": "richTableCell", "content": [ { "_key": "rp00112", "_type": "block", "children": [ { "_key": "rp00111", "_type": "span", "marks": [], "text": "Better" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00113", "_type": "richTableCell", "content": [ { "_key": "rp00115", "_type": "block", "children": [ { "_key": "rp00114", "_type": "span", "marks": [], "text": "Image processing with compressed images, ImageNet/ResNet-50" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00116", "_type": "richTableCell", "content": [ { "_key": "rp00118", "_type": "block", "children": [ { "_key": "rp00117", "_type": "span", "marks": [], "text": "Many to most datasets can fit within the local system’s cache." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00129", "_type": "row", "cells": [ { "_key": "rp00120", "_type": "richTableCell", "content": [ { "_key": "rp00122", "_type": "block", "children": [ { "_key": "rp00121", "_type": "span", "marks": [], "text": "Best" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00123", "_type": "richTableCell", "content": [ { "_key": "rp00125", "_type": "block", "children": [ { "_key": "rp00124", "_type": "span", "marks": [], "text": "Training with 1080p, 4K, or uncompressed images, offline inference, ETL" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00126", "_type": "richTableCell", "content": [ { "_key": "rp00128", "_type": "block", "children": [ { "_key": "rp00127", "_type": "span", "marks": [], "text": "Datasets are too large to fit into cache, massive first epoch I/O requirements, workflows that only read the dataset once" } ], "markDefs": [], "style": "normal" } ] } ] } ] } ``` Table 2: Characterizing different I/O workloads Achieving these performance characteristics may require the use of optimized file formats such as TFRecord, RecordIO, or HDF5. The high-speed storage provides a shared view of an organization’s data to all systems. It needs to be optimized for small, random I/O patterns, and provide high peak system performance and high aggregate filesystem performance to meet the variety of training workloads an organization may encounter. ## About the WEKApod Data Platform The WEKApod Data Platform meets performance characteristics with an architecture that delivers not just speed and scale but also operational efficiency and ease of use to quickly deploy and support large-scale AI initiatives with DGX SuperPOD. ![WEKApod Data Platform appliance: eight stacked 1U storage nodes with purple mesh bezels and WEKA wordmarks](https://cdn.sanity.io/images/ult5g8gw/production/56dcb7ec70b9cb07885fe402e77b181364a6d245-186x131.png) Figure 1. WEKApod Data Platform appliance Some WEKApod Data Platform appliance features include: - Appliance deployment as a scalable building block that can be easily clustered into a single namespace that scales seamlessly in capacity, performance, and capability. It is fully integrated which streamlines deployment and simplifies management operations. - Configuration for 500 TB or 1 PB usable capacity with TLC NVMe drives. The base appliance can be expanded seamlessly in a single namespace that provides up to several hundreds of petabytes of useable capacity, with linear performance scaling. - Native support of the NVIDIA Quantum InfiniBand Platform is a key feature that maximizes performance and minimizes CPU overhead. Since the DGX SuperPOD compute fabric is InfiniBand, designing the storage fabric as InfiniBand means only one high-speed fabric type needs to be managed, simplifying operations. - Appliance communications with DGX SuperPOD nodes using multiple NVIDIA Quantum InfiniBand connections for performance, load balancing, and resiliency. - The WEKA shared parallel protocol enables a single DGX node to access data at over 90 GB/s and 3M IOPS from the WEKApod appliance.. Multiple nodes can access data on the appliance simultaneously, with performance available to all systems and distributed dynamically. Nodes can be optimized for ultra-low latency workloads. - The all-NVMe architecture of the WEKApod Data Platform appliance provides excellent random read performance, often as fast as sequential read patterns. - The WEKApod Data Platform appliance is space and power efficient with dense performance and capacity to optimize data center utilization and improve operational economics. ## Validation Methodology Three classes of validation tests are used to evaluate a particular storage technology and its configuration for use with the DGX SuperPOD: microbenchmark performance, real application performance, and functional testing. The microbenchmarks measure key I/O patterns for DL training and can be run on CPU-only nodes. Real DL training applications are then run on a DGX SuperPOD to confirm that the applications meet expected performance. Beyond performance, storage solutions are evaluated for robustness and resiliency as part of functional testing. The NVIDIA DGX SuperPOD storage validation process leverages a “Pass or Fail” methodology. Specific targets are set for the microbenchmark test. Each benchmark result is graded as good, fair, or poor. A passing grade is one where at least 80% of the tests are good, and none are poor. In addition, there must be no catastrophic issues created during testing. For application testing, a passing grade is one where all cases are completed within 5% of the performance set by running the same tests with data staged on the DGX local RAID. For functional testing, a passing grade is one where all functional tests meet their expected outcomes. ### Microbenchmarks In the Storage Performance Requirements section, there are several high-level performance metrics that storage systems must meet to qualify as a DGX SuperPOD solution. Current testing recommends that the solutions meet the “Best” criteria discussed in Table 2. In addition to these high-level metrics, several groups of tests are run to validate the overall capabilities of the proposed solutions. These include single node tests where the number of threads is varied and multi-node tests where a single thread count is used and as the number of nodes vary. In addition, each test runs in both Buffered and DirectIO modes and when I/O is performed to separate files. Five different read patterns are run. The first is sequential write to the shared file system. The second read operation is sequential where no data is in the cache. The second read operation is executed immediately thereafter to evaluate the ability for the filesystem to cache data. The cache is purged and then the data is read again, this time randomly. Lastly, the data is reread again randomly, to evaluate data caching. ### Hero Benchmark Performance The hero benchmark helps establish the peak performance capability of the entire solution. Storage parameters, such as filesystem settings, I/O size, and controlling CPU affinity, were tuned to achieve the best read and write performance. Storage devices were expected to demonstrate that quoted performance was close to measured performance. Other tests are crafted to demonstrate performance of real workloads. The delivered solution for a single Scalable Unit (SU) had to demonstrate over 20 GiB/s for writes and 65 GiB/s for reads, all when using 8 threads per node. Ideally, the write performance should be at least 50% of the read performance. However, some storage architectures have a different balance between read and write performance, so this is only a guideline. ### Single-Node, Multi-File Performance For single-node performance, I/O read and write performance is measured by varying the number of threads in incremental steps. Each thread writes (and reads) to (and from) its own file in the same directory. For single-node performance tests, the number of threads varies from 1 to the ideal number of threads to maximize performance (typically more than half the cores but no more than the total physical cores). The I/O size varies between 128 KiB and 1 MiB and the tests are run with Buffered I/O and Direct I/O. The target performance for a single thread for these tests is shown in Table 3. ```json { "_key": "rp00309", "_type": "richTableBlock", "headerRows": 1, "markDefs": null, "rows": [ { "_key": "rp00208", "_type": "row", "cells": [ { "_key": "rp00184", "_type": "richTableCell", "content": [ { "_key": "rp00186", "_type": "block", "children": [ { "_key": "rp00185", "_type": "span", "marks": [], "text": "Thread Count" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00187", "_type": "richTableCell", "content": [ { "_key": "rp00189", "_type": "block", "children": [ { "_key": "rp00188", "_type": "span", "marks": [], "text": "Buffered or DirectIO" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00190", "_type": "richTableCell", "content": [ { "_key": "8f4e89500734", "_type": "block", "children": [ { "_key": "17cc61ca2eb0", "_type": "span", "marks": [], "text": "Performance (MiB/s)" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00193", "_type": "richTableCell", "content": [ { "_key": "f3f8a2615f61", "_type": "block", "children": [ { "_key": "de3d0a4ab794", "_type": "span", "marks": [], "text": "" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00196", "_type": "richTableCell", "content": [ { "_key": "7315fed0e9e8", "_type": "block", "children": [ { "_key": "a8bb394d3e4e", "_type": "span", "marks": [], "text": "" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00199", "_type": "richTableCell", "content": [ { "_key": "8d5b4831c9b1", "_type": "block", "children": [ { "_key": "125aed832054", "_type": "span", "marks": [], "text": "" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00202", "_type": "richTableCell", "content": [ { "_key": "c0d5c454ba89", "_type": "block", "children": [ { "_key": "d26719ecb971", "_type": "span", "marks": [], "text": "" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00205", "_type": "richTableCell", "content": [ { "_key": "a690815ae6f4", "_type": "block", "children": [ { "_key": "96107c31481d", "_type": "span", "marks": [], "text": "" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "833ba2757844", "_type": "row", "cells": [ { "_key": "ada8a688af29", "_type": "richTableCell", "content": [ { "_key": "f0571d61091c", "_type": "block", "children": [ { "_key": "2bbbab4f5550", "_type": "span", "marks": [], "text": "" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "59d3f5e5a823", "_type": "richTableCell", "content": [ { "_key": "6930715674cb", "_type": "block", "children": [ { "_key": "68f207fc3ec7", "_type": "span", "marks": [], "text": "" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "f94687d1db0b", "_type": "richTableCell", "content": [ { "_key": "2ccf7fb04457", "_type": "block", "children": [ { "_key": "rp00191", "_type": "span", "marks": [ "strong" ], "text": "I/O size (KiB)" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "64d07d345bf9", "_type": "richTableCell", "content": [ { "_key": "rp00195", "_type": "block", "children": [ { "_key": "rp00194", "_type": "span", "marks": [ "strong" ], "text": "Write (MiB/s)" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "e0da22b2f569", "_type": "richTableCell", "content": [ { "_key": "rp00198", "_type": "block", "children": [ { "_key": "rp00197", "_type": "span", "marks": [ "strong" ], "text": "Read (MiB/s)" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "f8aef8d6320b", "_type": "richTableCell", "content": [ { "_key": "rp00201", "_type": "block", "children": [ { "_key": "rp00200", "_type": "span", "marks": [ "strong" ], "text": "Reread (MiB/s)" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "fdb4da6ca9c1", "_type": "richTableCell", "content": [ { "_key": "rp00204", "_type": "block", "children": [ { "_key": "rp00203", "_type": "span", "marks": [ "strong" ], "text": "Random Read (MiB/s)" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "692cdf764da0", "_type": "richTableCell", "content": [ { "_key": "rp00207", "_type": "block", "children": [ { "_key": "rp00206", "_type": "span", "marks": [ "strong" ], "text": "Random Reread (MiB/s)" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00233", "_type": "row", "cells": [ { "_key": "rp00209", "_type": "richTableCell", "content": [ { "_key": "rp00211", "_type": "block", "children": [ { "_key": "rp00210", "_type": "span", "marks": [], "text": "1" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00212", "_type": "richTableCell", "content": [ { "_key": "rp00214", "_type": "block", "children": [ { "_key": "rp00213", "_type": "span", "marks": [], "text": "Buffered" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00215", "_type": "richTableCell", "content": [ { "_key": "rp00217", "_type": "block", "children": [ { "_key": "rp00216", "_type": "span", "marks": [], "text": "128" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00218", "_type": "richTableCell", "content": [ { "_key": "rp00220", "_type": "block", "children": [ { "_key": "rp00219", "_type": "span", "marks": [], "text": "512" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00221", "_type": "richTableCell", "content": [ { "_key": "rp00223", "_type": "block", "children": [ { "_key": "rp00222", "_type": "span", "marks": [], "text": "1,024" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00224", "_type": "richTableCell", "content": [ { "_key": "rp00226", "_type": "block", "children": [ { "_key": "rp00225", "_type": "span", "marks": [], "text": "1,536" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00227", "_type": "richTableCell", "content": [ { "_key": "rp00229", "_type": "block", "children": [ { "_key": "rp00228", "_type": "span", "marks": [], "text": "256" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00230", "_type": "richTableCell", "content": [ { "_key": "rp00232", "_type": "block", "children": [ { "_key": "rp00231", "_type": "span", "marks": [], "text": "1,536" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00258", "_type": "row", "cells": [ { "_key": "rp00234", "_type": "richTableCell", "content": [ { "_key": "rp00236", "_type": "block", "children": [ { "_key": "rp00235", "_type": "span", "marks": [], "text": "1" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00237", "_type": "richTableCell", "content": [ { "_key": "rp00239", "_type": "block", "children": [ { "_key": "rp00238", "_type": "span", "marks": [], "text": "Buffered" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00240", "_type": "richTableCell", "content": [ { "_key": "rp00242", "_type": "block", "children": [ { "_key": "rp00241", "_type": "span", "marks": [], "text": "1024" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00243", "_type": "richTableCell", "content": [ { "_key": "rp00245", "_type": "block", "children": [ { "_key": "rp00244", "_type": "span", "marks": [], "text": "800" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00246", "_type": "richTableCell", "content": [ { "_key": "rp00248", "_type": "block", "children": [ { "_key": "rp00247", "_type": "span", "marks": [], "text": "3,072" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00249", "_type": "richTableCell", "content": [ { "_key": "rp00251", "_type": "block", "children": [ { "_key": "rp00250", "_type": "span", "marks": [], "text": "4,608" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00252", "_type": "richTableCell", "content": [ { "_key": "rp00254", "_type": "block", "children": [ { "_key": "rp00253", "_type": "span", "marks": [], "text": "768" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00255", "_type": "richTableCell", "content": [ { "_key": "rp00257", "_type": "block", "children": [ { "_key": "rp00256", "_type": "span", "marks": [], "text": "1,024" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00283", "_type": "row", "cells": [ { "_key": "rp00259", "_type": "richTableCell", "content": [ { "_key": "rp00261", "_type": "block", "children": [ { "_key": "rp00260", "_type": "span", "marks": [], "text": "1" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00262", "_type": "richTableCell", "content": [ { "_key": "rp00264", "_type": "block", "children": [ { "_key": "rp00263", "_type": "span", "marks": [], "text": "Direct" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00265", "_type": "richTableCell", "content": [ { "_key": "rp00267", "_type": "block", "children": [ { "_key": "rp00266", "_type": "span", "marks": [], "text": "128" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00268", "_type": "richTableCell", "content": [ { "_key": "rp00270", "_type": "block", "children": [ { "_key": "rp00269", "_type": "span", "marks": [], "text": "768" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00271", "_type": "richTableCell", "content": [ { "_key": "rp00273", "_type": "block", "children": [ { "_key": "rp00272", "_type": "span", "marks": [], "text": "1,024" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00274", "_type": "richTableCell", "content": [ { "_key": "rp00276", "_type": "block", "children": [ { "_key": "rp00275", "_type": "span", "marks": [], "text": "1,024" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00277", "_type": "richTableCell", "content": [ { "_key": "rp00279", "_type": "block", "children": [ { "_key": "rp00278", "_type": "span", "marks": [], "text": "768" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00280", "_type": "richTableCell", "content": [ { "_key": "rp00282", "_type": "block", "children": [ { "_key": "rp00281", "_type": "span", "marks": [], "text": "768" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00308", "_type": "row", "cells": [ { "_key": "rp00284", "_type": "richTableCell", "content": [ { "_key": "rp00286", "_type": "block", "children": [ { "_key": "rp00285", "_type": "span", "marks": [], "text": "1" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00287", "_type": "richTableCell", "content": [ { "_key": "rp00289", "_type": "block", "children": [ { "_key": "rp00288", "_type": "span", "marks": [], "text": "Direct" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00290", "_type": "richTableCell", "content": [ { "_key": "rp00292", "_type": "block", "children": [ { "_key": "rp00291", "_type": "span", "marks": [], "text": "1024" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00293", "_type": "richTableCell", "content": [ { "_key": "rp00295", "_type": "block", "children": [ { "_key": "rp00294", "_type": "span", "marks": [], "text": "1,024" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00296", "_type": "richTableCell", "content": [ { "_key": "rp00298", "_type": "block", "children": [ { "_key": "rp00297", "_type": "span", "marks": [], "text": "1,024" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00299", "_type": "richTableCell", "content": [ { "_key": "rp00301", "_type": "block", "children": [ { "_key": "rp00300", "_type": "span", "marks": [], "text": "1,024" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00302", "_type": "richTableCell", "content": [ { "_key": "rp00304", "_type": "block", "children": [ { "_key": "rp00303", "_type": "span", "marks": [], "text": "1,024" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00305", "_type": "richTableCell", "content": [ { "_key": "rp00307", "_type": "block", "children": [ { "_key": "rp00306", "_type": "span", "marks": [], "text": "1,024" } ], "markDefs": [], "style": "normal" } ] } ] } ] } ``` Table 3. Single-node, multi-file performance targets When maximizing single-node performance, the thread count may vary; however, it is expected that performance does not drop significantly when additional threads are used beyond the optimal thread count. Target performance for single-node performance with multiple threads is in Table 4. The optimal number of threads may vary for any storage configuration. ```json { "_key": "rp00441", "_type": "richTableBlock", "headerRows": 1, "markDefs": null, "rows": [ { "_key": "rp00340", "_type": "row", "cells": [ { "_key": "rp00316", "_type": "richTableCell", "content": [ { "_key": "rp00318", "_type": "block", "children": [ { "_key": "rp00317", "_type": "span", "marks": [], "text": "Thread Count" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00319", "_type": "richTableCell", "content": [ { "_key": "rp00321", "_type": "block", "children": [ { "_key": "rp00320", "_type": "span", "marks": [], "text": "Buffered or DirectIO" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00322", "_type": "richTableCell", "content": [ { "_key": "rp00324", "_type": "block", "children": [ { "_key": "rp00323", "_type": "span", "marks": [], "text": "I/O size (KiB)" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00325", "_type": "richTableCell", "content": [ { "_key": "95fe21ace6f4", "_type": "block", "children": [ { "_key": "3c759e802120", "_type": "span", "marks": [], "text": "Performance (MiB/s)" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00328", "_type": "richTableCell", "content": [ { "_key": "a3e550bf0e84", "_type": "block", "children": [ { "_key": "ec8ad214eaf3", "_type": "span", "marks": [], "text": "" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00331", "_type": "richTableCell", "content": [ { "_key": "3f73761183a9", "_type": "block", "children": [ { "_key": "f5c5e453dc76", "_type": "span", "marks": [], "text": "" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00334", "_type": "richTableCell", "content": [ { "_key": "178c268500ce", "_type": "block", "children": [ { "_key": "249358ee7896", "_type": "span", "marks": [], "text": "" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00337", "_type": "richTableCell", "content": [ { "_key": "888608d849f7", "_type": "block", "children": [ { "_key": "778695df0b26", "_type": "span", "marks": [], "text": "" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "e8f1709a7643", "_type": "row", "cells": [ { "_key": "062f7cb2f9a4", "_type": "richTableCell", "content": [ { "_key": "a3c59ce1f694", "_type": "block", "children": [ { "_key": "88b2036232b2", "_type": "span", "marks": [], "text": "" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "a231a4576c5a", "_type": "richTableCell", "content": [ { "_key": "49e90ed8455b", "_type": "block", "children": [ { "_key": "e85c51e734ea", "_type": "span", "marks": [], "text": "" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "f283279941fb", "_type": "richTableCell", "content": [ { "_key": "c82afc76fabd", "_type": "block", "children": [ { "_key": "6fab00cd8c2e", "_type": "span", "marks": [], "text": "" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "fc30fd96c3fb", "_type": "richTableCell", "content": [ { "_key": "rp00327", "_type": "block", "children": [ { "_key": "rp00326", "_type": "span", "marks": [ "strong" ], "text": "Write (MiB/s)" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "eeed4d83a2a2", "_type": "richTableCell", "content": [ { "_key": "rp00330", "_type": "block", "children": [ { "_key": "rp00329", "_type": "span", "marks": [ "strong" ], "text": "Read (MiB/s)" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "8d7d79f837d2", "_type": "richTableCell", "content": [ { "_key": "rp00333", "_type": "block", "children": [ { "_key": "rp00332", "_type": "span", "marks": [ "strong" ], "text": "Reread (MiB/s)" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "417dc3d973e4", "_type": "richTableCell", "content": [ { "_key": "rp00336", "_type": "block", "children": [ { "_key": "rp00335", "_type": "span", "marks": [ "strong" ], "text": "Random Read (MiB/s)" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "06a6d13cec11", "_type": "richTableCell", "content": [ { "_key": "rp00339", "_type": "block", "children": [ { "_key": "rp00338", "_type": "span", "marks": [ "strong" ], "text": "Random Reread (MiB/s)" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00365", "_type": "row", "cells": [ { "_key": "rp00341", "_type": "richTableCell", "content": [ { "_key": "rp00343", "_type": "block", "children": [ { "_key": "rp00342", "_type": "span", "marks": [], "text": "Varies" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00344", "_type": "richTableCell", "content": [ { "_key": "rp00346", "_type": "block", "children": [ { "_key": "rp00345", "_type": "span", "marks": [], "text": "Buffered" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00347", "_type": "richTableCell", "content": [ { "_key": "rp00349", "_type": "block", "children": [ { "_key": "rp00348", "_type": "span", "marks": [], "text": "128" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00350", "_type": "richTableCell", "content": [ { "_key": "rp00352", "_type": "block", "children": [ { "_key": "rp00351", "_type": "span", "marks": [], "text": "8,000" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00353", "_type": "richTableCell", "content": [ { "_key": "rp00355", "_type": "block", "children": [ { "_key": "rp00354", "_type": "span", "marks": [], "text": "12,000" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00356", "_type": "richTableCell", "content": [ { "_key": "rp00358", "_type": "block", "children": [ { "_key": "rp00357", "_type": "span", "marks": [], "text": "18,000" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00359", "_type": "richTableCell", "content": [ { "_key": "rp00361", "_type": "block", "children": [ { "_key": "rp00360", "_type": "span", "marks": [], "text": "12,000" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00362", "_type": "richTableCell", "content": [ { "_key": "rp00364", "_type": "block", "children": [ { "_key": "rp00363", "_type": "span", "marks": [], "text": "18,000" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00390", "_type": "row", "cells": [ { "_key": "rp00366", "_type": "richTableCell", "content": [ { "_key": "rp00368", "_type": "block", "children": [ { "_key": "rp00367", "_type": "span", "marks": [], "text": "Varies" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00369", "_type": "richTableCell", "content": [ { "_key": "rp00371", "_type": "block", "children": [ { "_key": "rp00370", "_type": "span", "marks": [], "text": "Direct" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00372", "_type": "richTableCell", "content": [ { "_key": "rp00374", "_type": "block", "children": [ { "_key": "rp00373", "_type": "span", "marks": [], "text": "128" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00375", "_type": "richTableCell", "content": [ { "_key": "rp00377", "_type": "block", "children": [ { "_key": "rp00376", "_type": "span", "marks": [], "text": "8,000" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00378", "_type": "richTableCell", "content": [ { "_key": "rp00380", "_type": "block", "children": [ { "_key": "rp00379", "_type": "span", "marks": [], "text": "15,000" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00381", "_type": "richTableCell", "content": [ { "_key": "rp00383", "_type": "block", "children": [ { "_key": "rp00382", "_type": "span", "marks": [], "text": "15,000" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00384", "_type": "richTableCell", "content": [ { "_key": "rp00386", "_type": "block", "children": [ { "_key": "rp00385", "_type": "span", "marks": [], "text": "15,000" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00387", "_type": "richTableCell", "content": [ { "_key": "rp00389", "_type": "block", "children": [ { "_key": "rp00388", "_type": "span", "marks": [], "text": "15,000" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00415", "_type": "row", "cells": [ { "_key": "rp00391", "_type": "richTableCell", "content": [ { "_key": "rp00393", "_type": "block", "children": [ { "_key": "rp00392", "_type": "span", "marks": [], "text": "Varies" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00394", "_type": "richTableCell", "content": [ { "_key": "rp00396", "_type": "block", "children": [ { "_key": "rp00395", "_type": "span", "marks": [], "text": "Buffered" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00397", "_type": "richTableCell", "content": [ { "_key": "rp00399", "_type": "block", "children": [ { "_key": "rp00398", "_type": "span", "marks": [], "text": "1024" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00400", "_type": "richTableCell", "content": [ { "_key": "rp00402", "_type": "block", "children": [ { "_key": "rp00401", "_type": "span", "marks": [], "text": "10,000" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00403", "_type": "richTableCell", "content": [ { "_key": "rp00405", "_type": "block", "children": [ { "_key": "rp00404", "_type": "span", "marks": [], "text": "20,000" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00406", "_type": "richTableCell", "content": [ { "_key": "rp00408", "_type": "block", "children": [ { "_key": "rp00407", "_type": "span", "marks": [], "text": "30,000" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00409", "_type": "richTableCell", "content": [ { "_key": "rp00411", "_type": "block", "children": [ { "_key": "rp00410", "_type": "span", "marks": [], "text": "20,000" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00412", "_type": "richTableCell", "content": [ { "_key": "rp00414", "_type": "block", "children": [ { "_key": "rp00413", "_type": "span", "marks": [], "text": "30,000" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00440", "_type": "row", "cells": [ { "_key": "rp00416", "_type": "richTableCell", "content": [ { "_key": "rp00418", "_type": "block", "children": [ { "_key": "rp00417", "_type": "span", "marks": [], "text": "Varies" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00419", "_type": "richTableCell", "content": [ { "_key": "rp00421", "_type": "block", "children": [ { "_key": "rp00420", "_type": "span", "marks": [], "text": "Direct" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00422", "_type": "richTableCell", "content": [ { "_key": "rp00424", "_type": "block", "children": [ { "_key": "rp00423", "_type": "span", "marks": [], "text": "1024" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00425", "_type": "richTableCell", "content": [ { "_key": "rp00427", "_type": "block", "children": [ { "_key": "rp00426", "_type": "span", "marks": [], "text": "10,000" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00428", "_type": "richTableCell", "content": [ { "_key": "rp00430", "_type": "block", "children": [ { "_key": "rp00429", "_type": "span", "marks": [], "text": "20,000" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00431", "_type": "richTableCell", "content": [ { "_key": "rp00433", "_type": "block", "children": [ { "_key": "rp00432", "_type": "span", "marks": [], "text": "20,000" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00434", "_type": "richTableCell", "content": [ { "_key": "rp00436", "_type": "block", "children": [ { "_key": "rp00435", "_type": "span", "marks": [], "text": "20,000" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00437", "_type": "richTableCell", "content": [ { "_key": "rp00439", "_type": "block", "children": [ { "_key": "rp00438", "_type": "span", "marks": [], "text": "20,000" } ], "markDefs": [], "style": "normal" } ] } ] } ] } ``` Table 4. Single-node, multi-threaded performance targets For Buffered I/O tests, the reread performance relative to read performance can vary substantially between different storage solutions. The reread performance should be at least 50% of the read performance for both sequential and random reads. ### Multi-Node, Multi-File Performance The next test performed is a multi-node I/O read and write test to make sure that the storage appliance can provide the minimum required buffered read and write per system for the DGX SuperPOD. This benchmark determines the capacity of a filesystem to scale performance of different I/O patterns. Performance should scale linearly from one to a few nodes, reach a maximum performance, and not drop off significantly as more nodes are added to the job. The target performance for a single test cluster (1 scalable unit (SU)) is 65 GiB/s for reads with an I/O size of 128 KiB or 1,024 KiB, and if the I/O is Direct or Buffered. The write performance should be at least 20 GiB/s, but ideally it would be 50% of the read performance. Results from these tests must be interpreted carefully as it is possible to add more hardware to achieve these levels. Overall performance is the goal, but it is desirable that the performance comes from an efficient architecture that is not over designed for its use. ### Application Testing Microbenchmarks provide indications of the peak performance of key metrics. However, it is application performance that is most important. A subset of the MLPerf Training benchmarks is used to validate storage performance and function. Here, both single-node and multi-node configurations are evaluated to ensure that the filesystem can support different I/O patterns and workloads. Training performance when data is staged on the DGX RAID was used as the baseline for performance. The performance goal is for the total time to train when data is staged on the shared filesystem to be within 5% of those measured when data is staged on the local RAID. This is not just for individual runs, but also when multiple cases are run across the DGX SuperPOD at the same time. ### ResNet-50 ResNet-50 is the canonical image classification benchmark. The size of the training dataset is over 100 GiB and it has a requirement for fast data ingestion. On a DGX system, a single node training requires approximately 3 GiB per second and the dataset is small enough that it can fit into GPU memory. Preprocessing can vary, but the typical image size is approximately 128 KiB. One challenge of this benchmark is that at NVIDIA the processed images are stored in the RecordIO format (i.e. one large file for the entire dataset) since this provides the best performance for MLPerf. Since it is a single file, this can stress shared filesystem architectures that do not distribute the data across multiple targets or controllers. ## Summary NVIDIA evaluations show that the WEKApod Data Platform appliance meets the DGX SuperPOD performance and functionality requirements and is a certified choice to pair with a DGX SuperPOD to meet current and future storage needs. As storage requirements grow, WEKApod Data Platform appliances can be added to seamlessly scale capacity, performance, and capability. The combination of NVME hardware and WekaFS file system architecture provides excellent random read performance, often just as fast as sequential read patterns. With the 1U form factor of the storage appliance, storage pools can be configured to meet or exceed the performance requirements of a DGX SuperPOD. DGX SuperPOD customers should be assured that the WEKApod Data Platform appliance will meet their most demanding I/O needs for their most challenging AI workloads at any scale. **Notice** This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs. No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA. Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices. THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product. **Trademarks** NVIDIA, the NVIDIA logo, NVIDIA DGX POD, NVIDIA DGX SuperPOD are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. Copyright © 2024 NVIDIA Corporation. All rights reserved. [Download PDF](/api/resource-pdf?slug=nvidia-dgx-superpod-with-wekapod-data-platform-appliance)