Enabling Native S3 Applications to Enjoy the World’s Fastest File System Performance

With WekaFS Version 3.12, WekaIO adds front-end support for Amazon’s Simple Storage Service (S3) protocol. Weka’s new S3 front end presents fast S3 object access to a global namespace leveraging NVMe SSDs for performance, putting an end to the debate of bringing compute to the data or bringing the data to the compute instances by having to copy the data. Internally, WekaFS leverages the industry-standard object stores of our partners to extend the namespace and offer great economics.  

Just How Fast Is Weka’s S3 Front-end Protocol?

Our preliminary per-server performance measurements exceed the latest published performance results from a leading competitor who was the acknowledged leader in dedicated fast object storage.

That’s not bad for our first results. Of course, Weka’s S3 performance scales linearly as more servers are added. 

Weka Multiprotocol Support Empowers a Unified Data Platform for the Entire Life Cycle

Organizations use Weka as a unified data platform to accelerate outcomes and gain a competitive advantage. With Weka’s S3 front end, the same data can now be accessed from POSIX, S3, NFS, SMB, and GPUDirect® Storage, as shown in the figure below, creating a huge strategic advantage for Weka customers. As cloud-native applications migrate to native S3 services, organizations can enjoy the ability to share a single data set between traditional applications and S3-enabled applications without the need to copy or move data.

Weka’s Multi-protocol Unified Data Platform

 

With Weka S3, customers can gradually move applications to S3 and access the same data via multiple protocols (POSIX, S3, SMB, NFS, GPUDirect Storage). You can do all this while enjoying Weka’s scale, performance, security, and resiliency:

  • Ingest data with S3 and then access the data with either S3 or Weka’s other protocols.
  • Expose existing data to S3 and migrate your application within the same data platform.
  • Burst to the cloud and use new applications without the need to migrate your data.

Uses Standard S3 Clients with No Weka Software Needed on the Compute Clients

Clients do not need to load any special software to gain the advantage of Weka’s file system performance. Customers simply use standard S3 clients like Boto3. We also support standard client-side load balancers like Sidekick. Weka’s S3 front end has comprehensive S3 API support including Bucket Lifecycle, Policy, Object Tagging, and Object Multipart support. We support S3 audit logs and non-disruptive upgrades to the S3 service. 

Built For Our Biggest Customers

Our S3 front end was built on the request of our large IoT and Advanced Driver Assistance Systems (ADAS) customers who must rapidly ingest and output massive data sets into the file system for further analysis and manipulation by multiple applications. 

Weka S3 High-Performance Object Store Use Cases

With S3, Weka supports fast object use cases and high-performance file system implementations in our three primary markets: Financial Services, Life Sciences, and AI/ML. Weka’s S3 support extends the benefits of parallel, distributed file systems to cloud-native applications, which need strong consistency, multiprotocol access, and high performance/low latency, particularly for small objects.

  • New cloud-native workloads and applications – Strong consistency and the ability to offer high bandwidth to very tiny files enable fast object use cases like NoSQL databases, IoT, cloud-native applications, and cloud-native data lake houses like Snowflake, etc. 
  • Applications needing multiprotocol access – Weka’s ability to ingest data with S3 and expose it via POSIX, NFS, SMB, and GPUDirect Storage make it ideal for deep learning, analytics, and IoT use cases.
  • Scale and ease of deployment – Weka’s performance scales linearly as S3 clients and WekaF enabled storage nodes are added. Weka can be installed on any x86 based storage server in 15 minutes, as well as being available in AWS EC2 instances.
  • Best economics – Weka’s data platform unifies both NVMe SSD and HDD media in a single namespace, providing fast object-over-NVMe flash and capacity object with HDD-enabled object storage from multiple partners.
  • Data consolidation with global namespace – Weka provides a single global namespace, consolidating storage and eliminating the need to copy data to multiple storage silos.
  • Hybrid cloud – Weka extends native S3 access, both on-premises as well for AWS deployments, uniquely extending AWS S3 for fast object use cases and frameworks like AWS Sagemaker.

WekaFS: The Ultimate High-Performance Object Protocol Access
The Weka Limitless Data Platform now has a fully compliant native S3 protocol access in line with its multi-protocol capabilities, making it the ultimate high-performance solution for S3 native appliances! S3 access is in addition to POSIX, NFS, SMB, NVIDIA GPUDirect Storage, and CSI plug-in. Weka has been able to demonstrate line-rate performance with S3 access for small object and file benchmarks with linear scalability.

Additional Helpful Resources

Hybrid Cloud Strategy
FSx for Lustre
BeeGFS Parallel File System Explained
Learn About HPC Storage, HPC Storage Architecture and Use Cases
Isilon vs. Flashblade vs. Weka
NAS vs. SAN vs. DAS
Network File System (NFS) and AI Workloads
Block Storage vs. Object Storage
Modern Data Architecture