• How to Manage Data in the AI Era – Accelerated DataOps, Part 1
  • Accelerated DataOps Paves the Pathway to Explainable AI, Part 3

Shailesh Manjrekar, Head of AI and Strategic Alliances at WekaIO, shares his perspective on Accelerated DataOps with Weka AI in Part 2 of a three-part series titled “Accelerated DataOps with Weka AI for Edge to Core to Cloud Pipelines.”

Introduction

Weka is excited to be launching Weka AI, a transformative solution framework for Accelerated DataOps. In this second of my three-part series leading up to our launch of Weka AI, I will explain how new workloads are driving the need for modern underlying architectures and also explain how Weka AI enables Accelerated DataOps.

New Architectures – Edge AI-to-Core-to-Cloud

With the advent of IoT, 5G, and low-powered devices, Edge AI is expected to be bigger than the cloud. Depending on the use case, an Edge endpoint could be an autonomous vehicle, a drone, a Point of Sale device, an IP camera, or a precision medicine device. There is often an Edge aggregation point that is increasingly seen in parking lots, post offices, retail stores, and Edge data centers. This Edge endpoint aggregates the Edge datasets and runs Edge-to-Core-to-Cloud data pipelines. As a result, the infrastructure increasingly needs to cater to Edge-to-Core-to-Cloud data pipelines.

Also, in order to cater to complex DNNs and the convergence of HPC, HPDA, and AI, architectures such as GPUDirect storage are becoming paramount to feeding the GPU memory directly, providing the highest bandwidth and lowest latencies. Sharing datasets for distributed training across 64 DGX-2s (5,000 cores x 16 Teslas x 64 DGX-2s) has become the norm. Imagine the parallelism that is involved at the compute layer. Transports such as NVMeOF (over InfiniBand or RoCE Fabrics) are making data locality a non-issue, especially with support for 100 Gb/sec and 200 Gb/sec networking.

Equally important is the ability to cater to performance at scale, with use cases such as ADAS. A single survey car being used to achieve a goal of SAE 2+ autonomy, with 8*2 MP cameras and 10 DNNs, easily generates 2 PB of datasets per year.

Storage architectures using traditional direct-attached NVMe storage (DAS), NAS, and object storage limit performance and data mobility. In addition, NVMe block solutions lack the shareability and parallelism to deliver timely insights at scale for these new workloads and architectures.

Digital Transformation with Accelerated DataOps – Business and IT Convergence

When it comes to new workloads and architecture, these projects typically originate with Line of Business personas – Chief Data Officers, Chief Analytics Officers, CIOs, Data Scientists, Data Engineers – and then are augmented by IT personas. It is important to have an IT infrastructure, and particularly a storage solution, that can cater to both Line of Business and IT requirements. If not, the result will be data and IT silos. Accelerated DataOps break these silos by enabling shared goals, vision, and infrastructure. Weka AI enables Accelerated DataOps as follows:

  • Accelerated DataOps for Analytics – derive actionable intelligence
    • Run Business Intelligence with GPU-accelerated libraries such as RAPIDS and query engines such as BlazingSQL while running Artificial Intelligence with DALI on the same storage substrate
    • Run Descriptive, Predictive (what-if) and Prescriptive (what-if-then) and Cognitive Analytics with same storage substrate
  • Accelerated DataOps for Operational Agility – improve productivity, reduce TCO
    • Data is new source code – Weka AI provides data versioning, test/dev for pipelines, data protection with backup and recovery and DR
    • Data Anywhere – Weka AI manages Edge-to-Core-to-Cloud pipelines
    • CloudStore – Weka AI manages performance (NVMe Flash) and capacity tiers (S3/HDD) as a single namespace. Weka AI supports a broad ecosystem of S3 on-prem and public object stores
  • Accelerated DataOps for Governance
    • Weka AI provides line-rate in-flight and at-rest encryption supported by popular Key Management Systems such as HashiCorp Vault and supports virtual filesystems.

Summary

Weka, an NVIDIA Partner Network Solution Advisor, is uniquely positioned to anticipate the needs from these market transitions and provide transformative solutions. Weka, through delivering these solutions, makes it easy for our customers to monetize their data, achieve faster time-to-market, and gain competitive differentiation with the best TCO. Stay tuned for Part 3 of the series “Accelerated DataOps with Weka AI for Edge to Core to Cloud Pipelines,” where I explain Weka AI, a transformative solution framework, and how it can cater to these market transitions and deliver compelling benefits to Line of Business personas.

To learn more about WekaIO’s solutions for Artificial Intelligence and Data Analytics, click here.