S3 has transformed cloud storage, but its design, rooted in traditional on-premises object storage, isn’t optimized for the high-performance, low-latency I/O patterns required by modern AI workloads. WEKA’s S3 interface addresses this by ensuring high throughput and ultra-low latency, optimizing performance for millions of small objects and files to minimize IO wait and accelerate data ingestion and processing. With its scalable architecture and multi-protocol support, WEKA enables seamless integration of S3 with AI pipelines, allowing faster training, inference, and real-time data analysis. To fully understand the impact of S3 and how WEKA is optimizing it for modern workloads, it’s essential to look at the history of object storage, its original use cases, and how it has evolved to power today’s most data-intensive applications.

A Brief History of Object Storage

Object storage emerged as a solution to the challenges traditional storage systems faced when dealing with large, unstructured datasets and you needed a where you would guarantee you don’t lose data. Originally popularized for backup and archival purposes, these systems were designed to store data for services that required 24/7 availability, offering an interface that remained responsive even when issues arose. Object storage systems stored data as individual “objects,” each containing data, metadata, and a unique identifier. The eventual consistency allowed for lots of scaled out activity, which enabled lots of pools of hardware and led to making object storage ideal for handling large objects like videos, images, and massive datasets.

In its early days, object storage was implemented in large storage appliances optimized for big, unstructured data, rather than for performance-sensitive, smaller objects. A key feature of object storage was its connectionless nature, meaning data could be accessed or stored without needing a persistent connection between the client and the storage system. This architecture proved incredibly useful for large-scale backups and archives, where large volumes of data needed to be stored but were rarely accessed.

Enter S3: Object Storage in the Cloud

In 2006, Amazon Web Services (AWS) introduced Amazon S3 (Simple Storage Service), bringing object storage into the cloud. This was a revolutionary step because it made object storage easy and affordable for businesses of all sizes. S3 allowed users to store and retrieve vast amounts of data without the need for complex on-premises infrastructure, driving the mass adoption of cloud storage.

What made S3 stand out was its simplicity. It was designed with an easy-to-use interface, which allowed users to upload and access data from anywhere. The platform could automatically scale to meet storage needs, whether users were storing gigabytes or petabytes of data. As a result, S3 became the foundation for cloud storage, powering everything from small startups to massive enterprises across the globe with over 20 exabytes of storage today, and growing  by multiple petabytes a week.

S3 Everywhere: From Backup to AI Workloads

Over time, S3’s simplicity of the operations democratized IO and made it the standard way to interact with object storage and vendors adopted it as the go-to protocol for a wide range of products and applications, expanding beyond its origins in backup and archival use cases. S3 now powers the storage needs for cloud-native applications, media content distribution, data lakes, and AI/ML workloads.

One of object storage’s newest and most important roles is within AI data pipelines where it provides scalable, high-performance data storage for large datasets, enabling efficient data access and retrieval during model training, inference, and analytics. However, there’s a challenge when using S3 for AI workloads. Much like backups, AI workflows consist of massive amounts of data. But unlike backups, where the objects are often large, AI workloads are composed of millions of small objects —from individual training data samples to tiny sensor outputs. And even if you combine the data into large objects, the parts you need are small. And the data you need might be determined by the state of the neural network (as an example) which means that all the traditional methods of prefetching and other kinds of predictive data loading won’t work.

This is where the performance that an application can get out of traditional S3 can be limited. S3 is known for being a “chatty” protocol, meaning each object generates its own API call, creating significant overhead. This isn’t a problem when you are dealing with backup workloads with a limited lumber of large objects. But, this overhead can lead to performance bottlenecks when dealing with the tremendous number of small objects typically required for AI or machine learning tasks. Without optimization, this results in slower data processing times and reduced overall efficiency, which can be a significant drawback for AI workloads. While Amazon S3 Express One Zone offers performance improvements, delivering millisecond latency for initial bytes and high IO or throughput at the same time does deliver the same accelerated time to last byte, requires implementation in a single availability zone and at significant cost uplift. As a result, most enterprises and advanced model developers find they still need to implement caching or prefetching strategies to meet their needs which don’t work well with AI.

WEKA’s Optimized S3 for Small Objects and AI Workloads

This is where WEKA steps in, taking S3 performance to the next level for AI and other demanding workloads. The WEKA® Data Platform’s S3 front end is designed specifically to handle millions of small objects and files with the high throughput and low latency that AI applications require.

Our S3 interface was specifically developed to meet the needs of our large IoT and Advanced Driver Assistance Systems (ADAS) customers, who require the ability to rapidly ingest and output massive datasets into their cluster for their AI training and inferencing needs. These customers deal with immense volumes of small objects that need to be processed in real-time, and our solution ensures high throughput and low latency, allowing them to efficiently manage and analyze their data without performance bottlenecks. With an S3 protocol optimized for these complex workloads, we enable seamless data flow across their systems, supporting critical AI and machine learning processes.

Here’s how WEKA optimizes S3 data transfers for modern workloads:

  • Fast S3 Object Access: WEKA’s S3 interface offers fast object access to the WEKA® Data Platform’s global namespace which leverages NVMe SSDs for performance. This ensures that the system can handle high-intensity workloads like AI pipelines with ease.
  • High-Performance Data Access: WEKA uses DPDK (Data Plane Development Kit) to bypass the kernel, which allows for incredibly high throughput and extremely low latency, making it ideal for handling massive amounts of small objects.
  • Linear Scalability: One of WEKA’s standout features is its ability to scale performance linearly as more servers or cloud VMs are added. This means that as your AI or data workloads grow, WEKA’s performance scales right along with it, ensuring consistent, high-speed access to your data.
  • Multi-Protocol Access: WEKA’s S3 interface is part of its multi-protocol support that lets the same dataset can be accessed from POSIX, S3, NFS, SMB, and GPUDirect® Storage. This allows organizations to share a single dataset between traditional and S3-enabled applications without the need to copy or move data. This unified data platform eliminates the silos typically seen in hybrid storage environments.
  • The same WEKA software runs seamlessly on the cloud and on-premises, offers a unified S3-compatible object storage implementation that enables AI pipelines to run efficiently, regardless of the deployment location.
  • The WEKA® Data Platform’s patented data layout and virtual metadata servers distribute and parallelize all metadata and data across the cluster for incredibly low latency and high performance no matter the object’s size or number.

WEKA’s S3 vs. Traditional Competitors

WEKA’s S3 interface is designed to deliver high performance, with per-server capabilities that compare extremely favorably to published results from leading dedicated object storage solutions. This performance focus makes WEKA a strong option for workloads requiring fast object access and high throughput.

This performance advantage, combined with WEKA’s ability to handle large-scale AI workloads, makes it a compelling choice for organizations dealing with millions of small objects or other demanding applications like machine learning, data analytics, and high-performance computing.

Why WEKA for Your S3 AI Pipelines?

By switching to WEKA, whether on-prem and especially in the cloud, customers can experience radical improvements in performance for their AI data pipelines. WEKA’s optimized S3 platform offers the performance, scalability, and flexibility needed to power modern data-driven organizations.

For businesses working with AI, ML, or data analytics, this means faster data ingestion, quicker training times, and the ability to scale without the performance bottlenecks typically associated with traditional S3 implementations.

While S3 revolutionized object storage by making it accessible, scalable, and essential for modern workloads, its traditional architecture struggles to meet the demands of AI and other high-performance applications. WEKA takes S3 to the next level by delivering unmatched performance, low latency, and scalability for millions of small objects, transforming it into a powerhouse for AI pipelines. By optimizing S3 for the complexities of modern workloads, WEKA empowers organizations to harness the full potential of their data, accelerating innovation and enabling breakthroughs in AI, machine learning, and beyond.

See How WEKA Accelerates AI