Hybrid Cloud Data Pipelines with WekaFS™ on AWS Outposts®

Shailesh Manjrekar. September 15, 2020

WekaIO™ (Weka) has achieved the Amazon Web Services (AWS) Outposts Ready designation, part of the AWS Service Ready Program. This designation recognizes that the Weka File System (WekaFS™) has demonstrated successful integration with AWS Outposts deployments.

AWS Outposts is a fully managed service that extends AWS infrastructure, AWS services, APIs, and tools to virtually any datacenter, co-location space, or on-premises facility for a truly consistent hybrid cloud experience. With this, WekaFS extends hybrid cloud storage deployment capabilities for enterprise clients to accelerate and simplify data management operations. Achieving the AWS Outposts Ready designation differentiates Weka as an AWS Partner Network (APN) member with a product fully tested on AWS Outposts. AWS Outposts Ready products are generally available and supported for AWS customers, with clear deployment documentation for AWS Outposts. AWS Service Ready Partners have demonstrated success building products integrated with AWS services, helping AWS customers evaluate and use their technology productively, at scale and varying levels of complexity.

WekaFS on AWS

WekaFS is a high-performance, distributed, parallel file system that runs natively on AWS EC2 i3en instances with NVMe Flash, or any standard x86 bare-metal server. WekaFS enables hybrid cloud deployments between on-premises data centers and AWS Outposts, while extending the global namespace over S3 storage as the capacity tier. WekaFS holds the highest performance numbers for SpecSFS, IO500 and STAC benchmarks and has rich data management capabilities including containers and Kubernetes support for stateful applications, data protection and Disaster Recovery (DR) across AWS availability zones (AZ’s).

The State of Cloud Computing

Although Cloud computing has grown by leaps and bounds, several enterprise customers struggle to reap the benefits for these reasons:

There is no standard way to build a private cloud, and moving everything to the public cloud is not always possible due to high performance, latency and governance requirements.
Enterprise customers do not want to spend the time building IT infrastructure and integrating legacy systems, but want to focus on innovation, productivity and efficiencies by providing cloud elasticity.
There is no easy way to move to an OPEX model when it comes to accelerating innovation.
IT is increasingly becoming an innovation center from a cost center. How do we monetize data and launch new business models?
How do we modernize IT infrastructure for IoT, 5G Edge and AI/ML technologies, which are giving rise to “local clouds” and “lights-out management” and closer to manufacturing hubs, FSI exchanges and Healthcare customers?

Gone are the days of the notion that 100% of a company’s data would be processed and stored in the public cloud. The modern reality is that the needs of the lowest-latency applications, mission-critical workloads, and data gravity have driven the need for the hybrid cloud. The hybrid cloud computing market is predicted to be at $96B by 2023, with 71% of enterprises using hybrid deployments and 75% enterprise applications still on-premises, as per IDC and Forrester.

The WekaFS and AWS Outposts Solution

WekaFS and AWS Outposts have very similar value propositions and goals: delivering the lowest latency and highest IOPS for enterprise applications, thereby enabling customers’ success in their hybrid cloud journey. With this solution, customers can focus on their time-to-market advantages and solving business challenges, all with the lowest total cost of ownership.

A Hybrid Cloud AWS Outposts solution with WekaFS provides the following unique benefits:

Meets the emerging need for low latency and local data processing
Delivers the same experience on-premises and in the cloud
Delivers the same fully managed infrastructure services and API’s as in the cloud
Enables simplified and efficient IT, improved developer productivity, and pace of innovation

These prominent AWS services are enabled to run locally with AWS Outposts:

Compute and Storage–EC2, S3 services
Networking–Amazon Virtual Private Cloud ( VPC) service
Database–Amazon Relational Database Service ( RDS)
Containers–Amazon Elastic Container Service (ECS) and Elastic Kubernetes Service ( EKS)
Data Processing–Amazon Elastic Mapreduce ( EMR)
AI/ML–AWS Sagemaker

WekaFS and AWS Outposts Use Cases

AWS Outposts is offered in pre-validated configurations that are a mix of Amazon EC2 compute and storage capacity. It is the same infrastructure AWS uses in their datacenters. It offers AWS Nitro-based, general purpose, compute-optimized, memory-optimized, or GPU-optimized Amazon EC2 instance families (C5, M5, R5, I3en, and G4dn) as Opex model – 3 year term with partial upfront, all upfront and no upfront pricing. Price includes delivery, installation, servicing, and removal at the end of term.

The WekaFS on AWS Outposts solution is ideal for workloads that require low-latency access to on-premises systems, local data processing, or local data storage. WekaFS supports all leading industry use cases with AWS Outposts, such as the following:

Financial Services–kdb+ solution, SAS analytics, equities trading platforms
Healthcare–medical imaging and clinical diagnostics, healthcare records management, Genomics sequencing
Telecommunications–Edge Analytics
Artificial Intelligence/Machine Learning (AI/ML)–Valohai Deep Learning Management solution, Parabricks Genomics solution, AdvancedDriver-Assistance Systems (ADAS)
Retail–Deep Learning
Industry 4.0–Edge Analytics
Media and Entertainment–Studio as a Service, post-production

Deploying WekaFS with AWS Outposts

WekaFS provides a seamless experience to deploy either on AWS Outposts or in an AWS availability zone (AZ).

WekaFS has an easy to use Cloud formation front-end, which can quickly deploy a Weka cluster based on the storage requirements, based on Outposts i3en EC2 NVMe flash instances, and in the same VPC as your nearest AWS AZ. To get started go to https://start.weka.io/.
Any typeof EC2 client can be configured for running enterprise applications that require the lowest latency and the highest IOPS and throughput.
The WekaFS CSI plug-in can be used to provide stateful application support, and snap2object capabilities can be used to protect data in AWS S3 and can replicate to other AZs.

Weka for AWS

Refer to the webpage for WekaFS on Outposts and the AWS data sheet

Refer to Weka’s deployment and configuration instructions to get started with WekaFS on Outposts.

WekaIO, an AWS Advanced Technology partner, is very excited to be designated as AWS Outposts Ready and a part of the AWS Service Ready Program. As such, we look forward to bringing more hybrid solutions to this platform for enterprise customers.

Additional resources:

– Kx Qualifies Weka to Run With Its kdb+ Database in AWS EC2

– 5 Reasons Why the WekaIO File System is Ideal for AWS

– How to Install a Weka Cluster on AWS

– Weka and Destiny SAS Analytics solution

– WekaIO and Valohai Deep Learning Pipeline Management Solution