Multi-Cluster Example
This is an example on how to build a multi-cluster environment having two separate cluster repos (i.e. ‘dev’ and ‘prod’). The steps will include building a GKE/GSM/DSN environment from scratch for both environments, and deploying projects to staging and the remote production. It will use DNS (jx3rocks.com
), TLS, Let’s Encrypt certificates. This example is intended for an audience already familiar with Jenkins X operability and focuses on an example of actual commands used to build a multi-cluster environment. Additional information regarding using Google as the provider for this example can be found under Google Cloud Platform Prequisites.
⚠️ Note to OSX users You may need to substitute
sed -i
commands withsed -i.bak
, taking note to delete the generated .bak file. Somesed -i
commands that are additive might not work but can easily be completed with a text editor.
Generate the Infra and Cluster Repos for Dev and Prod
Using a command-line based approach, the example employs a process modeled after the doc Setup Jenkins X on Google Cloud with GKE, and will use Google Secret Manger. It requires the installation of Git and Hub command line tools.
The following are the values used for the creation of the Infra and Cluster repos for both the Dev and Prod environments:
Repo Source: jx3-gitops-repositories
Prod Infra Repo: jx3-terraform-gke.prd
Prod Cluster Repo: jx3-gke-gsm.prd
Dev Infra Repo: jx3-terraform-gke.dev
Dev Cluster Repo: jx3-gke-gsm.dev
Git Organization: $JX3ORG
JX3 Root Directory: $JX3HOME
Building PROD INFRA
repo: https://github.com/${JX3ORG}/jx3-terraform-gke.prd
Building PROD CLUSTER
repo: https://github.com/${JX3ORG}/jx3-gke-gsm.prd
Building DEV INFRA
repo: https://github.com/${JX3ORG}/jx3-terraform-gke.dev
Buidling DEV CLUSTER
repo: https://github.com/${JX3ORG}/jx3-gke-gsm.prd
Initialize the Prod cluster repo
Prepare the remote prod cluster repo by using the out of the box (OOTB) config (i.e. dev, jx-staging, and jx-production environemnts). Also remove the default ‘-jx’ URL value and insert jx-production ‘-prd’ URL value (optional). Prior to building the prod infra repo, prepare the production cluster repo for use by removing unecessary components. The components to modify/remove in the designated remote prod environment are:
- Remove default ‘-jx.’ URL format (optional)
- Remove Non-used JX charts
- Remove Tekton pipelines
- Add jxgh/local-external-secrets chart (optional)
- Insert imagePullSecret in jx-global-variables.yaml (optional)
- Disable webhooks
⚠️ For cluster autoupdate support both the Lighthouse and jxboot-helmfile-resources charts must be removed.
Remote Prod Chart List
helmfiles/kuberhealthy/helmfile.yaml
- chart: jxgh/kh-tls-check
helmfiles/jx-staging/helmfile.yaml
- chart: jxgh/jx-verify
helmfiles/secret-infra/helmfile.yaml
- chart: external-secrets/kubernetes-external-secrets
- chart: jxgh/pusher-wave
helmfiles/jx/helmfile.yaml
- chart: bitnami/external-dns
- chart: jxgh/acme
- chart: jxgh/jenkins-x-crds
- chart: jenkins-x/nexus
- chart: stable/chartmuseum
- chart: jxgh/jx-kh-check
- chart: jxgh/local-external-secrets
helmfiles/jx-production/helmfile.yaml
- chart: jxgh/jx-verify
helmfiles/cert-manager/helmfile.yaml
- chart: jetstack/cert-manager
helmfiles/nginx/helmfile.yaml
- chart: ingress-nginx/ingress-nginx
helmfiles/tekton-pipelines/helmfile.yaml
- chart: cdf/tekton-pipeline
Build the prod infra with Terraform
The following TF_VAR environment variables are set prior to running Terraform commands:
TF_VAR_gcp_project=<prod google project>
TF_VAR_apex_domain_gcp_project=<dns google project>
TF_VAR_jx_bot_username=<git username>
TF_VAR_jx_bot_token=<git token>
TF_VAR_tls_email=mymail@jx3rocks.com
TF_VAR_apex_domain=jx3rocks.com
TF_VAR_subdomain=prd
TF_VAR_gsm=true
TF_VAR_cluster_name=jx3prd
TF_VAR_cluster_location=us-east4-c
TF_VAR_jx_git_url=https://github.com/${JX3ORG}/jx3-gke-gsm.prd.git
TF_VAR_lets_encrypt_production=true
TF_VAR_force_destroy=true
Additional details on Terraform settings can be found under Google Terraform Quickstart Template
Commands to build infrastructure:
Validate the prod certificate is active and health checks pass:
Initialize the Dev cluster repo
Prepare the dev cluster repo by using the out of the box (OOTB) config (i.e. dev, jx-staging, and jx-production environemnts). Also remove the default ‘-jx’ URL value and insert jx-production ‘-prd’ URL value (optional). Later on prior to importing the external steps will be used to remove the jx-prodcution environment (optional).
Dev Chart List (Initial)
helmfiles/kuberhealthy/helmfile.yaml
- chart: jxgh/kh-tls-check
helmfiles/jx-staging/helmfile.yaml
- chart: jxgh/jx-verify
helmfiles/secret-infra/helmfile.yaml
- chart: external-secrets/kubernetes-external-secrets
- chart: jxgh/pusher-wave
helmfiles/jx/helmfile.yaml
- chart: bitnami/external-dns
- chart: jxgh/acme
- chart: jxgh/jxboot-helmfile-resources
- chart: jxgh/jenkins-x-crds
- chart: jxgh/jx-pipelines-visualizer
- chart: jxgh/jx-preview
- chart: jenkins-x/lighthouse
- chart: jenkins-x/nexus
- chart: stable/chartmuseum
- chart: jxgh/jx-build-controller
- chart: jxgh/jx-kh-check
helmfiles/jx-production/helmfile.yaml
- chart: jxgh/jx-verify
helmfiles/cert-manager/helmfile.yaml
- chart: jetstack/cert-manager
helmfiles/nginx/helmfile.yaml
- chart: ingress-nginx/ingress-nginx
helmfiles/tekton-pipelines/helmfile.yaml
- chart: cdf/tekton-pipeline
Build the dev infra with Terraform
The following TF_VAR environment variables are set prior to running Terraform commands:
TF_VAR_gcp_project=<dev google project>
TF_VAR_apex_domain_gcp_project=<dns google project>
TF_VAR_jx_bot_username=<git username>
TF_VAR_jx_bot_token=<git token>
TF_VAR_tls_email=mymail@jx3rocks.com
TF_VAR_apex_domain=jx3rocks.com
TF_VAR_subdomain=dev
TF_VAR_gsm=true
TF_VAR_cluster_name=jx3dev
TF_VAR_cluster_location=us-east1-b
TF_VAR_jx_git_url=https://github.com/${JX3ORG}/jx3-gke-gsm.dev.git
TF_VAR_lets_encrypt_production=true
TF_VAR_force_destroy=true
Additional details on Terraform settings can be found under Google Terraform Quickstart Template
Commands to build intrastructure:
To validate the prod certificate is active and health checks pass:
Import Remote Prod Repo
Add the remote prod repo as your production target by importing. PLEASE NOTE: Prior to importing the remote repo make sure all updates to the dev repo have completed.
The ‘jx project import
’ command will begin to display the following:
we are now going to create a Pull Request on the development cluster git repository to setup CI/CD via GitOps
created file /tmp/jx-git-973407466/.jx/gitops/source-config.yaml
Created Pull Request: https://github.com/jx3rocks/jx3-gke-gsm.src/pull/1
we now need to wait for the Pull Request to merge so that CI/CD can be setup via GitOps
Waiting up to 20m0s for the pull request https://github.com/jx3rocks/jx3-gke-gsm.src/pull/1 to merge with poll period 20s....
Merge the outstanding pull request for
Below is an example of the URL for the PR:
Once the pull request is merged, it proceeds to wait for a trigger to be added to the lighthouse config:
Pull Request https://github.com/${JX3ORG}/jx3-gke-gsm.src/pull/1 was merged at sha a07df43fcad60df439dad087cf4502f00e002190 after waiting 12m7.719699579s
waiting up to 20m0s for a trigger to be added to the lighthouse configuration in ConfigMap config in namespace jx for repository: jx3rocks/jx3-gke-gsm-prd
you can watch the boot job to update the configuration via: jx admin log
for more information on how this works see: https://jenkins-x.io/docs/v3/about/how-it-works/#importing--creating-quickstarts
Eventually the process appears to time out waiting for the adding of the trigger.
WARNING: It looks like the boot job failed to setup this project.
You can view the log via: jx admin log
error: failed to wait for repository to be setup in lighthouse: failed to find trigger in the lighthouse configuration in ConfigMap config in namespace jx for repository: jx3rocks/jx3-gke-gsm-prd within 20m0s
error: failed to wait for the pipeline to be setup jx3rocks/jx3-gke-gsm-prd: failed to run 'jx pipeline wait --owner jx3rocks --repo jx3-gke-gsm-prd' command in directory '', output: ''
Adjust Prod Repo Promotion Type (optional)
The newly imported prod repo has a promotion type of ‘Auto’. To change this ‘Manual’ to provide greater deployment control make adjustments in the jx-requirements.yml file:
Remove jx-production environment (optional)
To have a single designated production environment remove the dev repo’s jx-production environment.
Environments
Both the dev and remote prod repos are now ready for deployments.
Set Dev Container Registry to Public
In order to deploy applications to the remote prod environment it is required that the dev container registry is publically available. The image below highlights how you can make the change using the Google console.
Or you can use the following command:
Deploy Quickstart Project
To ensure everything is working as intended, the following commands will create and deploy a simple NodeJS app to staging and the remote production.
Merge Pull Requests
For deployment to remote prod you must merge the auto generated pull request.
Project Deployments
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.