IT training platform as code ebook khotailieu

As Kubernetes is getting widely adopted,
enterprise challenges have progressed from
deployment of Kubernetes cluster to building
and managing desired platform stacks on it.
Learn how Platform-as-Code approach offers
a common language to create Kubernetes
Native platform stacks in hybrid multi-cloud
environments.

Introduction to
Platform-as-Code

CloudARK Inc.

Last Updated on 31 July 2019
Published on 19 March 2019


www.cloudark.io

Table of Contents
Kubernetes Native platforms ....................................................................................................................................................... 2
Kubernetes API extensibility with Operators ......................................................................................................................... 2
Evolution of as-Code systems ...................................................................................................................................................... 3
Platform-as-Code approach .......................................................................................................................................................... 4
Platform-as-Code Challenges ....................................................................................................................................................... 5
Discovery of Custom Resource functionality ..................................................................................................................... 5
Interoperability between Custom Resources .................................................................................................................... 6
KubePlus approach to Platform-as-Code ................................................................................................................................. 6
Consistency across CRDs/Operators .................................................................................................................................... 6
KubePlus API add-on ................................................................................................................................................................. 6

Platform-as-Code tenets ................................................................................................................................................................ 7
Kubernetes Native ...................................................................................................................................................................... 7
Composable................................................................................................................................................................................... 7
Declarative..................................................................................................................................................................................... 7
Platform-as-Code benefits............................................................................................................................................................. 7
Hybrid multi-cloud application portability ........................................................................................................................ 7
Common language between Devs and Ops ......................................................................................................................... 8
Conclusion .......................................................................................................................................................................................... 8

1 | Platform-as-Code eBook


www.cloudark.io

Kubernetes Native platforms
Cloud Native technologies act as building blocks towards developing modern, scalable containerized
microservices based applications. The landscape of Cloud Native technologies is quite large and it includes
various types of softwares like databases, API gateways, load balancers, key management systems,
storage/networking systems, etc. In this Cloud Native landscape, Kubernetes has emerged as one of the fastest
growing projects and has become the de-facto standard for running containerized application workloads. While
the broader Cloud Native trends are essential, for enterprises that are adopting Kubernetes, it is more
important to understand what subset of these Cloud Native technologies are best suited to work with
Kubernetes so that they can build their Kubernetes platform layers using them. In this regard, it is useful to
split the Cloud Native technology landscape broadly into three categories:

Figure 1: Kubernetes Native as subset of Cloud Native technologies

●
●
●

Kubernetes Native technologies - These are tools/systems that are primarily built or distributed only
for Kubernetes. They don’t support any other container orchestration system.
Kubernetes accommodative technologies - These are tools/systems that embrace multiple container
orchestration systems, Kubernetes being one of them.
Non-Kubernetes technologies are Cloud Native but don’t support Kubernetes.

An interesting aspect of Kubernetes is the ability to extend its control plane by adding new custom APIs. This
is done by writing ‘Kubernetes API Extensions’. Popular terms used in the community to refer to Kubernetes
API extensions are ‘Custom Resource Definitions (CRDs)’ and ‘Operators’. CRDs/Operators are Kubernetes
Native by design. They work with Kubernetes CLI (‘kubectl’); they can be installed on a cluster with
Kubernetes’s popular package manager Helm; and they seamlessly integrate with Kubernetes features such as
RBAC, Service accounts, Audit logs, etc. CRD/Operator has become the preferred mechanism to bundle and
distribute third party platform softwares to run on Kubernetes. Today, more than 400+ GitHub repositories of
Kubernetes CRDs/Operators exist for variety of platform softwares, such as API gateways, relational databases
(MySQL, Postgres), non-relational databases (MongoDB, Cassandra, Elastic), Volume backup/restore functions,
SSL certificate management, application workloads (Spark, TensorFlow), etc.
Building a Kubernetes Native platform layer involves assembling and installing Kubernetes CRDs/Operators of
required platform elements in a cluster. Enterprises are finding this approach of building Kubernetes Native
platforms attractive as it not only offers them control over the underlying Kubernetes layer, but also provides
flexibility in choosing the desired platform elements in their platform layer.

Kubernetes API extensibility with Operators
The control plane of a Kubernetes cluster is made of built-in controllers for built-in resources/APIs. The builtin resources include, among others, Pod, Deployment, Service, Secret, Ingress, etc. The built-in controllers
continuously reconcile the state of a cluster by reacting to CRUD (create, read, update, delete) operations on
these resources/APIs. The resources are represented as YAML specifications and provide a declarative way for
running and managing containerized workloads on a cluster.

2 | Platform-as-Code eBook

www.cloudark.io
An API extension (CRD/Operator) consists of a Custom Controller that monitors CRUD operations on Custom
Resources/APIs and reconciles the state of the cluster. For instance, one can write a CRD/Operator to run
stateful databases like Cassandra on Kubernetes in which a new Custom Resource/API named ‘Cassandra’ and
its Custom Controller is added to a cluster’s control plane. The Custom Controller will handle CRUD operations
on ‘Cassandra’ Custom Resource instances to manage Cassandra clusters on that Kubernetes cluster. Following
figure shows a glimpse of this rich CRD/Operator ecosystem of Custom Resources + Custom Controllers and
how it enables extending Kubernetes API set.

Figure 2: Categories of Kubernetes APIs

As shown in the figure, Kubernetes APIs can be broadly sub-divided into three categories:
●

●
●

Built-in APIs - Built-in APIs consist of Kubernetes’s built-in resources and controllers. As mentioned
earlier, the built-in resources include Deployment, Pod, Service, Secret, ConfigMap, StatefulSet,
PersistentVolume, ResourceQuota, etc. As of 1.14 release of Kubernetes there are 57 such resources
natively supported by Kubernetes.
Standardized APIs - Standardized APIs consist of Custom Resources and Custom Controllers around
which some standardization has started to happen, such as monitoring (Prometheus) and Service Mesh
Interface (SMI).
Bespoke APIs - Bespoke API category consists of Custom Resources and Custom Controllers for which
there are no standards. These include most of the community Operators. The Custom Resources and
Custom Controllers shown in above figure are representative, picked from the 400+ Operators that exist
today on GitHub.

Typically, DevOps team holds the responsibility of installing required Operators in a Kubernetes cluster to
create a customized platform layer. Once CRDs/Operators are installed, the task of application developers is to
build required platform workflows leveraging this platform layer. Application developers define their platform
workflows as Kubernetes YAML definitions consisting of Custom and built-in resources.

Evolution of as-Code systems
Let’s look at how Kubernetes API extensions affect the traditional notion of ‘as-Code’ systems (Refer to figure
3).
Any ‘as-Code’ system is designed to offer a common language to provision a technology stack leveraging
underlying APIs and resources of a cloud. First generation systems like AWS CloudFormation focused on
delivering this for a single cloud (AWS). Second generation systems like Terraform focused on abstracting
underlying cloud APIs across multiple clouds, offering a common language to provision technology stack in
multi-cloud environments.

3 | Platform-as-Code eBook


www.cloudark.io

Figure 3: Evolution of as-Code systems

An important aspect of the first- and second-generation systems was that the set of underlying APIs that these
systems wrapped was static and known apriori. In the Kubernetes world, this is no longer the case as the set of
control plane APIs in a Kubernetes cluster can be extended by installing CRDs/Operators in a cluster anytime.
Hence, for an as-Code system in Kubernetes world, the goal is no more to abstract the underlying cloud
infrastructure APIs, but it is to enhance the customized platform layer made from CRDs/Operators by
simplifying creation of platform workflows consisting of Kubernetes YAMLs of built-in and Custom Resources.
Such a ‘Platform as Code’ system thus enables creating technology stacks declaratively for hybrid multi-cloud
environments.

Platform-as-Code approach
Application developers define their platform workflows as Kubernetes YAML definitions of Custom and builtin resources. Such an ensemble of Custom and built-in resources essentially represents ‘codified’ definition of
platform workflows and hence can be referred to as Platform-as-Code. Following figure presents this Platformas-Code (PaC) approach as a flow diagram.

Figure 4: Platform-as-Code

To understand the PaC approach further, let us take a real-life example of a Kubernetes Native platform layer
that supports creating multiple Moodle stacks. Moodle is an open source e-learning software. Each Moodle
stack consists of - Moodle software, MySQL DB instance, Domain Name for that stack, and SSL certificate.
The high-level requirements for this custom platform layer are as follows:
4 | Platform-as-Code eBook


www.cloudark.io
-

Create multiple Moodle stacks on a Kubernetes cluster (one stack per tenant).
Support ability to install ‘plugins’ on each Moodle stack.
Backup & Restore a Moodle instance. For storage, Moodle uses both file system and a relational database.
Backup both (file system and database).
- Provision SSL certificate from Let’s Encrypt for each Moodle instance.
In order to support above workflow actions, a Kubernetes native Moodle platform layer can be created using
following CRDs/Operators.
1. Moodle Operator (for Moodle instance management)
2. MySQL Operator (for MySQL database management)
3. Stash Operator (for Volume backup/restore)
4. Cert-Manager Operator (for SSL certificate management)
5. NGINX Ingress Controller (for API gateway)
DevOps Engineer installs these CRDs/Operators in a Kubernetes cluster and application developer leverages
Kubernetes’s built-in and Custom Resources to realize the required platform workflows.

The Custom Resources made available by this platform layer are - Moodle, MysqlCluster, MysqlBackup, Restic,
Recovery, Certificate. The built-in resources that complement these Custom Resources include - Secret,
PersistentVolumeClaim, PersistentVolume, Ingress.
The custom workflow actions that are supported by the Moodle platform layer include - creating multiple
Moodle stacks, installing/deleting Moodle plugins on each stack, taking backups of MySQL database for a stack,
restoring a database from backup, backup/restore of Moodle stack Volume, obtaining and installing SSL
certificate per stack.
These actions are realized through Kubernetes YAML definitions leveraging above mentioned Custom and
built-in resources - moodle.yaml, mysqlcluster.yaml, restic.yaml, mysqlbackup.yaml, recovery.yaml,
secret.yaml, persistentvolumeclaim.yaml. These YAML definitions are nothing but Platform-as-Code for a
Moodle platform stack. As an example, in the following figure two Kubernetes Custom Resources are shown –
moodle.yaml and mysqlcluster.yaml. The Moodle Custom Resource provides a declarative approach for
managing plugins on a Moodle instance through its spec.plugins attribute. For its database needs it uses
MysqlCluster Custom Resource instance by referring to its Service resource in the spec definition
(spec.mySQLServiceName property).

Figure 5: Kubernetes Custom Resources – Moodle and MysqlCluster

Platform-as-Code Challenges
When application developers are leveraging Kubernetes built-in and Custom Resources to orchestrate required
platform workflows, they face following key challenges:

Discovery of Custom Resource functionality
Today it is not easy for application developers to understand new functionality introduced by every new
Custom Resource made available through CRDs/Operators installed in their clusters. Kubernetes natively
provides a way to discover information about Custom Resource Spec properties through ‘kubectl explain’
command. However, for effectively using Custom Resources to realize workflow actions, broader usage

5 | Platform-as-Code eBook

www.cloudark.io
information is needed, such as what higher-level actions are supported by a Custom Resource, how to perform
such actions, what are the assumptions made by Operator developer in handling Custom Resource actions, etc.

Interoperability between Custom Resources
Platform-as-Code process enables creating platform stacks and orchestrating platform workflows leveraging
different Kubernetes Resources (Custom and built-in). For this, various Custom and built-in resources need to
be bound/tied together in specific ways. This may involve passing certain names, attributes or labels from one
Custom Resource to another or to in-built resources. For application developers it is not easy to figure out how
to create this tie-in between different resources towards realizing specific workflows.

KubePlus approach to Platform-as-Code
In order to address above challenges and simplify Platform-as-Code experience for application developers, two
things are required:
1.
2.

Consistency across different CRDs/Operators.
Tooling that helps with discovery and binding of Kubernetes resources.

Consistency across CRDs/Operators
At CloudARK, we have created detailed guidelines to enable consistency and standardization across various
community CRDs/Operators. These guidelines describe how to design, implement, package and document
Kubernetes CRDs/Operators for enabling Platform-as-Code experience. We have analyzed over 100 open
source CRDs/Operators for their conformance to these guidelines. We also use these guidelines while curating
community Operators for our customers.

KubePlus API add-on
We have developed KubePlus API add-on to simplify discovery and use of Kubernetes Custom Resources

towards building Platform as Code workflows. Once installed in the Kubernetes cluster, this add-on offers new
API endpoints (e.g. ‘man’, ‘composition’) that enable application developers to query static and dynamic
information about various Custom Resources available in the cluster. Here is how KubePlus API add-on is used
when realizing Platform-as-Code workflows for the Moodle platform mentioned above (detailed steps):
•

Binding between Moodle and Mysql Custom Resources:
In order to bind to the underlying database provisioned by the MysqlCluster Custom Resource instance, the Moodle
Custom Resource YAML definition needs the name of Service resource corresponding to the MysqlCluster Custom
Resource instance to be specified in its 'mysqlServiceName’ Spec attribute. This information and the format of the Service
name can be found using the ‘man’ endpoint for Moodle and MysqlCluster Custom resources.
kubectl get --raw "/apis/platform-as-code/v1/man?kind=Moodle"
kubectl get --raw "/apis/platform-as-code/v1/man?kind=MysqlCluster"

•

Interdependency between Moodle, Restic and Deployment Resources:
In order to take backup of Moodle instance volume, the Deployment resource corresponding to that Moodle Custom
Resource instance needs to be given a label and that label needs to be used in the Restic Custom Resource label selector.
The ‘man’ and ‘composition’ endpoints help here. Use the ‘composition’ endpoint to find underlying Deployment
resource corresponding to a Moodle Custom Resource instance. The output of composition command will show all the
Kubernetes resources that are created by the Moodle Operator as part of handling ‘moodle1’ instance of Moodle Custom
Resource in the ‘ns1’ namespace. This output enables us to find the Deployment resource for ‘moodle1’ instance to which
label can be applied for taking its backup.
kubectl get --raw "/apis/platform-as-code/v1/man?kind=Restic"
kubectl get --raw "/apis/platform-as-code/v1/man?kind=Moodle"
kubectl get --raw "/apis/platform-as-code/v1/composition?kind=Moodle&instance=moodle1&namespace=ns1"

KubePlus API add-on currently provides information required to build interactions between Kubernetes
Custom and built-in resources manually and thus define Platform-as-Code workflows. We are now developing

additional automation in this add-on to enable automatic binding and resolution between Kubernetes
6 | Platform-as-Code eBook


www.cloudark.io
resources to realize different Platform-as-Code workflows. Here is the on-going work for this that uses
mutating web-hook admission controller to perform run-time discovery and binding between Custom
Resources required for a Platform-as-Code workflow.

Platform-as-Code tenets
The Platform-as-Code approach builds upon Kubernetes’s inherent declarative nature and leverages
Kubernetes YAMLs to define end-to-end platform workflows. Following are the key tenets of any Platform-asCode system. A platform layer that satisfies these can deliver Platform-as-Code experience to its end users.

Kubernetes Native
Kubernetes Native is about enabling all the key functionality under Kubernetes interfaces like kubectl or
Kubernetes YAMLs, instead of introducing new proprietary interfaces or abstraction wrappers.

Composable
Composability is about bringing in a level of consistency across Operators that simplifies consumption of
Operators in multi-Operator settings.

Declarative
Declarative nature is inherent to Kubernetes and its API constructs. From Platform-as-Code perspective it is
important to have Kubernetes Operators be designed in such a manner that they support various operations
on platform softwares as declarative actions.

Platform-as-Code benefits
Platform-as-Code approach enables enterprises to build Kubernetes-native, composable and declarative
platform stacks which in-turn enables platform engineering teams to deliver more value, faster, to their end
users. It also offers more flexibility to choose required platform elements compared to any pre-built platform

stack offering.
At a broad level, Platform-as-Code approach has the potential to influence following enterprise challenges and
create a level of impact that technologies like Terraform have created in pre-Kubernetes era.

Hybrid multi-cloud application portability
Due to composable and declarative nature of Platform-as-Code approach, enterprises can easily create their
platform stacks repeatably on any Kubernetes cluster. This significantly simplifies transfer or recreation of
workloads across hybrid and multi-cloud environments. It also enables the ability to lift and shift or burst
workloads between private and public clouds.

Figure 6: Hybrid multi-cloud application portability

7 | Platform-as-Code eBook


www.cloudark.io

Common language between Devs and Ops

Figure 7: Platform-as-Code bridging the gap between Devs & Ops

Platform-as-Code bridges the tooling gap between Developers and DevOps Engineers with Kubernetes YAML
becoming the common language between them. Over the years two types of tools have evolved – PaaSes and
Infrastructure-as-Code. PaaSes focused on delivering end-to-end application developer workflows.
Infrastructure-as-Code tools focused on simplifying Ops challenges in infrastructure provisioning. This created
tooling gap between development and DevOps teams due to adoption of different tools. Platform-as-Code has
the potential to bridge this gap by standardizing on Kubernetes YAML as a common language between Devs &
Ops, especially around platform level functionality. For instance, as mentioned in official Kubernetes case
studies, ThredUp, an ecommerce company, has managed to deprecate their 3200 Ansible scripts
(Infrastructure-as-Code) in favor of Kubernetes YAMLs/helm charts.

Conclusion
Kubernetes API Extensions (also known as CRDs/Operators), are paving the way for a new era in
platformization. Platform-as-Code approach brings in standardization in composing custom platform layer
using multiple CRDs/Operators. It builds upon Kubernetes’s inherent declarative and API driven nature and
leverages Kubernetes YAMLs to define end-to-end platform workflows. As a result, enterprises get a
Kubernetes-native, composable and declarative way of building their platform workflows. Platform-asCode approach is disrupting the traditional PaaS industry by bringing in composability and addressing the
inflexibility of pre-built platforms. This is especially attractive in today’s era of containerization and Kubernetes
where there is a large choice for building your platform technology stack.
At CloudARK, we offer purpose-built Kubernetes Native platforms to our customers based on the Platform-asCode approach. We also offer customized training on Kubernetes extensibility, which has already been availed
by some of the fortune 500 enterprises. Get in touch and we will be excited to help you in your Kubernetes
platformization journey.

8 | Platform-as-Code eBook