Today, we’re excited to introduce WEKA support for AWS ParallelCluster, enabling enterprises to accelerate time to value for their High-Performance Compute (HPC) applications. AWS ParallelCluster is an open-source cluster management tool that makes it easy to deploy and manage HPC clusters on AWS. By integrating WEKA with ParallelCluster, organizations can deploy a high-performance data platform that shrinks epoch time from months to days without additional infrastructure investment. With the infrastructure performance and efficiency gains made possible with WEKA for AWS ParallelCluster, organizations can accelerate their own pace of innovation, maximize their utilization of GPU-accelerated infrastructure, and control costs. Also in the works in the coming weeks is WEKA support for AWS Parallel Computing Service for customers looking for a more managed experience for their HPC deployments on AWS.

Do More Science, Manage Less Infrastructure

Cloud computing platforms like AWS ParallelCluster enable a new generation of HPC workloads like molecular dynamics, computational fluid dynamics, electronic design automation, and seismic imaging. Historically, these workloads have required a lot of complex and time-consuming resource orchestration. Infrastructure resources need to come together across thousands of compute cores. This high-performance shared storage system is tuned to support high IO, high bandwidth, and low latency, a fast network to connect everything, a comprehensive set of libraries, and a job scheduler to keep everything running. With AWS ParallelCluster, organizations can bring all their favorite HPC toolsets and run them with cloud scalability and agility.

Accelerate Time to Discoveries

WEKA software combined with AWS ParallelCluster provides the fastest, most scalable data platform for HPC workloads across hybrid and cloud-native deployments on AWS. The WEKA zero-tuning, zero-copy architecture delivers high IO, high throughput, and low latency to support every stage of an organization’s HPC data pipelines. This unique approach enables storage performance increases of 10x or more. It is optimized for processing large-scale datasets consisting of millions of small files with sub-millisecond latency and high IO. Customers like Atomwise and Genomics England rely on this approach to collapse HPC epoch times from months to days. The step-function increase in data performance also drives increases in GPU utilization that enable these customers to do more science while decreasing their infrastructure investments. Finally, WEKA’s advanced hybrid cloud capabilities enable organizations to easily burst HPC workflows into the AWS or migrate data from a lab or office environment to AWS.

Figure 1: Sample Architecture: WEKA Support for AWS ParallelCluster

Seamless Cloud Integration

It’s easy to get started with WEKA for AWS ParallelCluster. WEKA software deploys directly into the customer VPC and integrates with existing security groups, IAM roles, as well as infrastructure as code workflows through a set of comprehensive Terraform templates. Installation is simplified through our recently introduced Cloud Deployment Manager. Click here to learn more about WEKA for AWS ParallelCluster.