When a cockpit cluster is set up, subsequent clusters can be deployed and configured by means of Argo CD with GitOps. The important thing is the initial cluster. 

In this article we explore the chicken and egg problem encountered during the bootstrapping of a cockpit or management cluster, focusing on the initial cluster setup. As a starting tool, we’ll utilize Cluster.dev, an open-source product by SHALB, to try and integrate it into our bootstrapping process.

Introduction

Cluster.dev, an open-source tool, is designed to manage cloud-native infrastructure stack templates. It enables users to define their entire infrastructure cohesively and deploy it using a single tool, leveraging technologies like Terraform, Helm, and others. This approach simplifies the deployment of complex systems, with essential add-ons across diverse cloud environments. However, it is noteworthy to mention that Cluster.dev goes beyond mere templating; the tool is structured as a comprehensive project that incorporates interconnected objects called units.

Though Cluster.dev may appear as another templating language built upon Terraform and Helm,  that is not the case in practice. In the next section, I’ll illustrate why this is so and how you can utilize Cluster.dev to address typical challenges like the chicken and egg problem during bootstrapping. This practical demonstration will showcase the tool’s utility in real-world scenarios.

What is Cluster.dev?

Cluster.dev operates with units, akin to Terraform modules, allowing for efficient infrastructure deployment and management. These units function as building blocks, enabling seamless assembly of infrastructure components. However, Cluster.dev offers more than just deployment and management capabilities. It facilitates the deployment of infrastructure alongside operational software, including setups like Kubernetes, integrated with vital third-party tools like Cert-Manager and Argo CD, among others.

While Terraform technically offers the capability to achieve these tasks, my preference leans towards utilizing it primarily for provisioning infrastructure, supplemented by minimal cloud-init scripts. As a side note, deploying Helm charts with Terraform can seem somewhat counterintuitive as it doesn’t inherently feel suited to this task.

Cluster.dev enhances Terraform capabilities by enabling the definition of Stack templates. These templates can integrate diverse elements like Terraform modules, Kubernetes manifests, shell scripts, Helm charts, and Argo CD/Flux applications, alongside OPA policies and more. By synthesizing these components, Cluster.dev facilitates the deployment, testing, and distribution of a comprehensive set of elements with fixed versions. The Stack itself presents the values for the Stack template.

In this context, Units serve as the fundamental components. For example, Stack Templates utilize these building blocks to construct entire platforms for developers. The strength of Cluster.dev lies in effectively managing the integration of these elements, facilitated by pre and post hooks. This setup can tackle challenges such as sending notifications to a team once the Kubernetes cluster is ready, thereby eliminating the need for multiple pipelines with diverse steps. Instead, you define a single stack and leverage it within a single pipeline to fulfill most of your requirements.

Additionally, Cluster.dev enables the exchange of generated outputs among units, functioning akin to utilizing units as printers. As for secrets management, Cluster.dev seamlessly integrates with Secrets Managers, such as AWS Secrets Manager or SOPS Secrets, ensuring secure and efficient handling of sensitive data. Functions and Generators provide further flexibility for tailoring your specific requirements.

The diagram below delineates the relation of Cluster.dev components.

Cluster.dev principal diagram 

To better understand the Cluster.dev structure and the idea behind its components, refer to the Overview section of its documentation. 

Now let’s see how Cluster.dev differs from Terraform and other similar tools. 

Cluster.dev vs other technologies

Vs Helmfile

Cluster.dev and Helmfile both handle Kubernetes Helm charts, but serve distinctly different purposes. Cluster.dev provides a comprehensive infrastructure management approach, integrating various technologies beyond Helm, making it well-suited for large-scale and diverse cloud-native projects. In contrast, Helmfile specializes in Helm-centric environments, concentrating on advanced, detailed management of Helm charts, making it suitable for scenarios requiring consistent Helm chart deployments across multiple clusters or environments.

Vs Terraform

Cluster.dev elevates Terraform’s capabilities by providing a higher-level abstraction that addresses common issues such as complex interrelations and limited templating. By integrating various infrastructure tools into a cohesive management system, it offers a more streamlined, GitOps-first approach, facilitating precise and efficient infrastructure deployment and management.

Vs Pulumi and Crossplane

Cluster.dev sets itself apart from tools like Pulumi and Crossplane by orchestrating not just cloud resources but the entire infrastructure ecosystem. It integrates various tools such as Terraform, Pulumi, Bash, or Ansible into a cohesive workflow. Unlike being limited to a single tool with its specific language, Cluster.dev allows for a limitless combination of tools and workflows. Currently, it focuses on Terraform due to its widespread adoption and extensive community-developed modules.

Deploying Azure Kubernetes Service with addons using cdev

On the left side of the diagram below is depicted our current solution to the chicken and egg challenge. On its right side, you can see a potential single pipeline solution built with Cluster.dev.

Old way vs new way diagram

Let’s see how it works. 

Cluster.dev utilizes stack templates to instantiate users’ projects within a selected cloud environment. The Azure-AKS stack template is tailored to create and configure Kubernetes clusters on the Azure cloud using the Azure Kubernetes Service (AKS).

This page provides instructions for initiating an AKS cluster on Azure using the Azure-AKS stack template, one of the pre-configured samples provided by Cluster.dev. Executing the sample code will lead to the creation of the following resources:

Azure Infrastructure:

• Azure DNS Zone
• Azure Virtual Network

AKS Kubernetes cluster with add-ons:

• cert-manager
• ingress-nginx
• external-secrets (with Azure Key Vault backend)
• external-dns
• argocd

Prerequisites

• Terraform version 1.4+
• Azure account and a subscription
• Azure CLI installed and configured with your Azure account
• kubectl installed
• Cluster.dev client installed
• Parent Domain

Now let’s get into the process.

1. Clone the example repo:

git clone https://github.com/shalb/cdev-azure-aks.git && cd cdev-azure-aks/examples

It will look like this:

.
├── backend.yaml
├── project.yaml
└── stack-aks.yaml

2. Update project.yaml:

name: demo-project
kind: Project
backend: azure-backend
variables:
  location: westeurope
  domain_name: azure-cluster-dev.hpa-cloud.com
  resource_group_name: rg-cdev-aks-example
  state_storage_account_name: sacdevstates
  state_container_name: tfstate

3.  Create the Azure Storage Account and a container for Terraform backend:

az group create --name rg-cdev-aks-example --location westeurope
az storage account create --name sacdevstates --resource-group rg-cdev-aks-example --location westeurope --sku Standard_LRS
az storage container create --name tfstate --account-name sacdevstates

Edit variables in the example’s files, if necessary.

4. Run  cdev plan

5. Run cdev apply

Set up DNS delegation for the subdomain by creating NS records for the subdomain in the parent domain. Run cdev output

domain = demo.azure-cluster-dev.hpa-cloud.com.
name_servers = [
  "ns1-36.azure-dns.com.",
  "ns2-36.azure-dns.net.",
  "ns3-36.azure-dns.org.",
  "ns4-36.azure-dns.info."
]

6. Connect to AKS cluster. Run cdev output

az aks get-credentials --name snrp-aks-akscluster-re1-001 --resource-group snrp-rg-aks-re1 --overwrite-existing

7. Check if Argo CD is running:

kubectl get ingress -A 

If everything is ok you will see an Argo CD URL of your cluster. 

To clean up the setup, use the cdev destroy command. Also, be sure to remove NS records for the subdomain in the parent domain and delete Azure Storage Account and a container for Terraform backend:

az group delete --name cdevResourceGroup

This is a simple example but you can customize it to include your own Terraform modules that will be defined in the template.yaml.

Conclusion

Cluster.dev is a sophisticated and intelligent templating engine based on Golang, offering highly customizable solutions tailored to users’ needs.

The tool is not intended for beginners, though they may find the initial setup manageable. It requires prior knowledge of Terraform, Kubernetes, Helm, and cloud technologies to leverage the Cluster.dev capabilities to the fullest. 

In the AKS example, the AKS resources are initially provisioned, and corresponding Network Security Group Rules are established based on add-ons such as Argo CD. Following this, another module is loaded to provide the Helm charts for the tools. Subsequently, a DNS entry for Argo CD is created, enabling the utilization of the cluster along with the tools.

Achieving a productive setup within just 15 minutes is really impressive!

P.S. I believe it effectively addresses our chicken and egg problem in a straightforward manner, and we intend to integrate it into our bootstrapping workflow, albeit with our own Terraform modules.