Kubernetes Storage Solutions for vSphere

As I'm sure you've heard, VMware has been working hard to do what our customers want as they are making the move towards Cloud Native Applications. (If not, you can find articles HERE, HERE, and HERE.) VMware's vision is to develop solutions that allow our customers to run whatever applications they would like, where they want to run them, and provide connectivity, availability and resiliency solutions so that they can be confident that they have made a great choice in their hybrid cloud solution.

If you're reading this then you probably know I work in the Storage & Availability BU at VMware so naturally I'm most interested in how Kubernetes (K8s) is consumed by our customers running K8s on top of vSphere with vSAN or one of our core storage partners. As I started learning more about the K8s environments that our customers are running they have told me how complicated setting up and maintaining a K8s environment can be. Not only Day 0 installation, architecture and initial deployment but also the Day 2 operations of updating everything when something new is released or a security patch needs to be applied. Also during my research, a good article I found was a study done by Datadog in which they found that 7 of 10 cloud native technologies requires persistent storage. This means that customers require enterprise class data management capabilities and not just some JBOD storage capacity they found on a server under someones desk.

"What does it mean to persist data in a Kubernetes environment?" you may ask yourself. It means a few things:

  1. Workload data needs to persist and survive through the restart/re-scheduling of a container
  2. Underlying infrastructure should handle the complexity of unmounting and mounting without user intervention
  3. When a container is rescheduled it's disk needs to be re-associated with the correct ID
  4. Data must also be shared and concurrently accessed by containers across all pods

As VMware started developing solutions for cloud native technologies we created what we called the vSphere Cloud Provider (VCP) for Kubernetes (previously known as Project Hatchway). VCP has been available for a while and is in use by tons of customers. VCP is natively built into K8s and supports all storage primitives exposed by K8s:

  1. Volumes
  2. Persistent Volumes
  3. Persistent Volume Claims
  4. Stateful Sets
  5. Storage Classes

Because we built this into K8s we are able to support the entire HCL of vSphere storage products and protocols including vSAN, VMFS, NFS and vVOLs. This allows our customers to utilize the same tools that they have been using for years to provide storage to their K8s environments. By using Storage Policy Based Management (SPBM) they can utilize all of the storage services that their chosen brand of storage provides by simply creating a basic Storage Class within K8s to point to their SPBM policy.

VCP for K8s

Link to screenshot: clicky

Now since the K8s community is never one to sit still and are always making changes that benefit their end users they decided to create a new volume plugin framework called Container Storage Interface (CSI). Because the old framework was "in-tree" it made releases much less flexible which caused a ton of pain for K8s developers they decided to create CSI as an "out-of-tree" plugin. This gives back the flexibility for releases of non-CSI functions, makes troubleshooting bugs easier, etc. This also allows for collaboration with other orchestrators like MesoSphere if the end user chooses to do so.

Because VMware is participating in the K8s community (yes we have lots of commits out there) we decided to utilize CSI in our latest solution we are calling the Cloud Native Storage Control Plane in vSphere (CNS). CNS allows for end to end provisioning workflows through APIs and UI extensions in vSphere including performance and health monitoring of K8s volumes within vCenter. This also allows us to offer volume granular data services like volume snapshots, backup, migration, encryption, etc.

Link to screenshot: clicky

Again, this is all done through the use of SPBM and K8s Storage Classes. The biggest difference in CNS is that we've implemented the CNS control plane inside of vSphere management (vCenter+ESXi) and are simply calling it from the CSP plugins installed on the K8s nodes. And as you can see in these screenshots, monitoring and troubleshooting of container volumes are built directly into vCenter.

Link to screenshot: clicky

Capacity Management per Container Volume

Link to screenshot: clicky

Health Reporting for Container Volumes

Link to screenshot: clicky

As of this writing, the beta is currently running for CNS and we are gathering a ton of customer feedback. I have run through the Hands-On Labs we have available and I have to say CNS is a seamless implementation that "just works". There's no special configuration the virtualization admin has to do to vSphere to get this working. It's simply plug and play.

As I mentioned earlier in this post all forms of vSphere storage on the HCL are supported for CNS. vSAN is uniquely suited to run CNS and your containers IMHO and I'll share more about why in another post. For now, if you're looking for more great (and in depth) reading on CNS and K8s on vSphere you can check out Myles Gray's blog over at blah.cloud.  

Shameless plug: I'll be at the Seattle Usercon on February 19th and the Toronto Usercon on March 19th so if you're in town get registered and come day "Hi!".