Fast Data, Now Simpler With the WEKA Cloud Deployment Manager
We recently released the first version of our new WEKA Cloud Deployment Manager (CDM), a new UI-driven interface for managing WEKA deployments to the cloud. The WEKA Cloud Deployment Manager provides a step-by-step guided approach to deploying WEKA in the cloud. In this blog, we walk through the major elements of the CDM experience using Azure as our example, though the tool also supports WEKA deployments in AWS and Google Cloud.
Introduction to the WEKA Cloud Deployment Manager
Traditional high-performance storage systems – usually based on Lustre – are famously complex and hard to manage. These systems typically include multiple systems for storage servers, management servers, and metadata servers, then even more components for storage protocol gateways to support the most common applications. As we recently discussed, scale-out NAS and disaggregated share-everything (DASE) approaches fare no better as performance limits result in similarly complex network configurations to support the needs of next-generation workloads for AI and HPC. When deployed to cloud infrastructure, managed versions of these systems offer marginal improvements in ease of use and deployment, largely due to the limitations in the underlying architecture.
From the start, we built WEKA with the view that world-class performance should not sacrifice simplicity. With this in mind, we built WEKA to run natively in the cloud, and it now runs on every major cloud platform—AWS, Azure, Google, and OCI. We showed how customers can complete their WEKA deployment in about the same time it takes to order a pizza. We added support for automated deployments using Terraform to provide a familiar experience for WEKA customers deploying to their cloud platform of choice, and for deployment consistency across increasingly multiple cloud deployments.
Now, we’re pleased to release the WEKA Cloud Deployment Manager (CDM) to give customers a standard way to deploy WEKA across AWS, Azure, and Google Cloud. The WEKA CDM provides a GUI-driven step-by-step experience for configuring WEKA software and the cloud infrastructure that supports the WEKA environment. Using WEKA Cloud Deployment Manager enables you to deploy and manage high-performance data in the cloud (or across multiple clouds) in a consistent way using a standard approach. The WEKA CDM gives you a completed Terraform template that you import into your cloud provider of choice (AWS, Azure, or Google Cloud), or integrate into your existing infrastructure as code (IAC) workflows. Whether you’re a veteran of WEKA or just getting to know us, this is a fantastic new tool to have in your arsenal that will make deploying WEKA in the cloud faster and easier than ever before.
If you’re an existing WEKA customer, you can log in to check out the new WEKA CDM at cloud.weka.io. If you’re new to WEKA, we can help you take your first steps with WEKA in the cloud by getting in touch with us here. Two prerequisites for getting started with the WEKA CDM tool are a properly licensed version of WEKA software for the cloud and a valid account with your cloud provider of choice. Let’s have a look at the new WEKA Cloud Deployment Manager!
WEKA Cloud Deployment Manager Walkthrough
Once you’ve logged in to the main cloud.weka.io page, the first step is to select your cloud provider of choice. For this example, let’s choose Azure.
Cloud Deployment Manager
Automate and simplify WEKA deployment and scaling in multiple clouds in a consistent way
Please choose a cloud provider for WEKA deployment:
Within the main Cloud Deployment Manager interface, there are three main sections – Workflow Navigation, Configuration Input, and the Dynamic Content Bar. The Workflow Navigation contains the major WEKA and cloud configuration elements required for a successful WEKA deployment, the input screen is where I’ll make all the configuration selections and provide inputs, and the dynamic content bar will show specific guidance on our configuration as well as the completed Terraform template we’ll download once we’re done.
Main Elements of CDM Interface
In the Basic Configuration screen, you can define all the fundamentals of the WEKA cluster. First select your version of Terraform and WEKA software. The get.weka.io token authenticates access to WEKA install binaries for download during the deployment process. To input a valid get.weka.io token, you’ll have to switch over the get.weka.io, where you can find your valid WEKA software license.
The Resource Name and Cluster Name will propagate to resource names for the Cloud VPCs, VMs, security, and other cloud resources needed, so you should select names that will be meaningful and conform with your enterprise resource naming policies.
Next, you can specify how many VMs will support this WEKA environment, the VM types, and the Azure region where you want to deploy. The minimum cluster size is 6, in this example, let’s keep the default Std_L8s_v3 VMs in the US East Azure Region.
Pro Tip! The number of VMs needed to support customer environments varies based on their workload requirements. So, it’s a good idea to consult your WEKA team at this point to validate that your configuration is suitable for your workload, and you have sufficient quotas for vCPUs, vNICs, and IP addresses to support the cluster.
To complete the basic configuration, just provide your valid Azure Subscription ID, Deployment Resource Groups and Vnet Resource Group Names aligned with your current Azure subscriptions and naming conventions.
Finally, select your placement group type. In almost all cases, you should select “Yes” here to ensure all VMs in the cluster deploy to the same placement group for optimal performance.
WEKA Cloud Deployment Manager Basic Configuration screen
The Network Configuration Workflow allows you to define the Azure Vnet and Subnet where your WEKA cluster will deploy. By selecting “Yes”, the CDM will define a completely new Vnet with all necessary settings completed. At this point, if you have any custom network settings, you can input them now. For this example, le’ts create a net new Vnet using the defaults.
WEKA Cloud Deployment Manager Network Configuration screen
The Security Group Workflow allows me to define the security group my WEKA environment will join. In this case, we want the CDM workflow to create a new security group for me. So select “Yes” – Create Network Security Group. If you already have defined security groups, you can use this screen to provide the Security Group Name here and the public key string for SSH access.
WEKA Cloud Deployment Manager Security Configuration screen
You can use the OBS Configuration Workflow to enable tiering from the WEKA cluster to object storage – in this case, Azure Blob. Simply select “Yes” to turn on storage tiering and provide your Azure Blob bucket name. You’ll need to create a new Azure Blob bucket, set to private access, in your Azure Subscription. Select the storage tiering percentage you expect (defined as the % of capacity to store in the WEKA cluster); for this case, let’s select 20%.
WEKA Cloud Deployment Manager OBS Configuration screen
Finally, select the clients your new WEKA environment will connect to once deployed. In this case, the application is running on a Standard_D8_v5 VM.
WEKA Cloud Deployment Manager Clients Configuration screen
The Advanced Configuration and Protocol Nodes allow more customization of the WEKA environment and add more protocols to support complex data pipelines and scenarios where multiple clients will access the data set across NFS and SMB protocols, in addition to POSIX, which is the WEKA default. Since these aren’t needed for my current example, I’ll skip them for now—but stay tuned!
Now that I’ve completed all the required fields, I can preview the Terraform file in the Dynamic Content Bar. I can now download the file and import it directly into my Azure account to deploy my new WEKA environment!
WEKA Cloud Deployment Manager – Terraform File Preview
As you’ve seen, the WEKA Cloud Deployment Manager is an easy way to deploy and manage WEKA in the cloud using an intuitive, feature-rich GUI-driven interface. The WEKA CDM takes all the work out of building a robust, secure, and highly-performant WEKA deployment in the cloud. To get started with your WEKA deployment in the cloud, log in to cloud.weka.io.
If you’re already a WEKA customer or new to WEKA, get in touch with us to Arrange a Demo