Cross-cloud Hybrid Data Plane for Kubernetes persistent workloads using Dell PowerFlex 4.5 Software Defined Storage

This article is part of the EKS Anywhere series EKS Anywhere, extending the Hybrid cloud momentum | by Ambar Hassani

Cross-cloud data mobility is one of the hottest requirements in a multi/hybrid cloud landscape as the surge for Data on Kubernetes and a data-first strategy gains significant momentum.

Technology outcomes such as On-demand Cloud DR, Local on-premises copies of cloud hosted workloads for compliance processes, cross-cloud migrations, repatriations, workload recalibrations, secondary copies of databases for analytical processing., all of these signify the nuances our customers are expecting to be delivered when bringing in an end-to-end solution. This is where the underlying technology support from Dell PowerFlex presents a unique opportunity with it’s ubiquitous support for physical, virtual and containerized workloads. In addition, the native features within PowerFlex to address performance, redundancy, scalability and the desired level of agility when it comes to cross-cloud presence provides a perfect fit for such scenarios.

To weigh it in, the absolute beast nature of PowerFlex lies in its ability to deal with performance at scale. Talk about:

• 10 Million Transactions per second for an Oracle RAC cluster
• 7 Million Transactions per second for a SQL server in a 8U rack space
• 1 Billion Indexing events in 3 hours for Elastic stack
• 2.2 Million operations per second for MongoDB on Kubernetes
• 325K operations per second for Cassandra on Kubernetes
• 16 Petabytes on a single storage cluster

In this technical blog, we will uncover the prowess of Dell’s PowerFlex Software Defined Storage by building a Hybrid data plane and showcasing a real-time data mobility solution for MySQL workloads across EKS and EKS Anywhere clusters.

In the advent, we will also be experiencing the implementation nuances of deploying CSI drivers for Dell PowerFlex for EKS Anywhere and EKS service in the cloud.

Before we start, let’s layout the context in which the solution is being deployed. We have two Kubernetes clusters, an EKS Anywhere cluster hosted along with a Dell PowerFlex system in Equinix and the second is a EKS cluster in AWS Ireland region which also has a software defined PowerFlex system as the storage provider (akin to APEX Block over AWS). Both entities are interconnected via a Direct connect and the AWS VPC has private-only networking, typical of an enterprise system.

The above diagram visually articulates two scenarios that will be demonstrated.

Scenario-1: Implementation of CSI drivers for Dell PowerFlex for EKS Anywhere and showcasing MySQL persistence along with volume snapshots and restore.

Scenario-2 Implementation of CSI drivers for Dell PowerFlex for EKS in AWS region and replicating the MySQL persistent volume from the EKS Anywhere cluster to the AWS hosted system.

While we delve into the implementation routine, you could always listen in to a great technical overview of Dell PowerFlex with my colleague Simon Stevens, Senior Principal Engineering Technologist in Dell’s GEOS team (Global Engineering Outreach Specialists)

The entire implementation is split into three separate videos and since these are live recorded., so expect a bit of troubleshooting too along with the lengthy details. Hopefully, all put together it provides the right insights into the technology as opposed to generic visualization.

Video-1: Deploying EKS Anywhere along with CSI drivers for Dell PowerFlex. Once the CSI drivers are installed, we deploy a MySQL workload and demonstrate various levels of persistence including volume snapshots and restoral of MySQL data.

Video-2: Herein, we deploy a Replication system across the two PowerFlex software defined storage systems, one in Equinix and the other in AWS Ireland region. This replication system is based on the differentiated techniques of Journalling which is much more effective than a typical snapshot shipping method.

Video-3: The final part of this sequence involves creating an EKS cluster in AWS along with CSI integration with the software defined version of Dell PowerFlex in the same region. Once the CSI integration is complete, we simulate a disaster on the on-premises hosted EKS Anywhere cluster and resurrecting the MySQL database with the EKS-AWS cluster via the replicated persistent volume.

A big thanks to the collaborative effort from my colleagues at Dell

• Simon Stevens (Global Engineering Outreach Organization)
• Alexander Hoppe (Container Storage Engineering)
• Madhavi G (Customer Centric Cloud and Containers)

Happy PowerFlexing with Kubernetes,

cheers,

Ambar@thecloudgarage

#iwork4dell

Important note: Dell PowerFlex requires a SDC (Software defined client) to be implemented on each the Kubernetes nodes that participate in the CSI integration. Normally on OpenShift, the CSI driver installation automatically creates and configures a SDC client for PowerFlex. However, in the case of EKS and EKS Anywhere, we use ubuntu as the Operating system for the Kubernetes nodes. Hence, a pre-requisite has to be completed of bundling and bootstrapping the ubuntu Software defined Client on the participating nodes. This is fully automated as a part of the custom ubuntu image creation process documented here EKS Anywhere, customizing ubuntu templates for specific requirements | by Ambar Hassani | Medium. For the EKS-AWS cluster, we use a canonical custom ubuntu AMI and use custom bootstrap scripts to install the PowerFlex software defined client as a part of the user data process.