Cluster Rebuild Example

An example how to rebuild a existing environment back to its original state prior to shutdown.

This is an example on how to treat your clusters as cattle and not pets by rebuilding a JX3/GKE environment and restoring to it’s previous configuration. The steps will include building a JX3 Google/GSM environment from scratch and deploying projects to staging and production. It will utilize DNS (, TLS, Let’s Encrypt certificates. The example includes the recovery of the helm chart repository which is currently implemented with ChartMuseum . This example is inteded for an audience already familiar with Jenkins X operability and focuses on an example of actual commands used to build and maintain a recoverable environment. Additional information regarding using Google as the provider for this example can be found under Google Cloud Platform Prequisites.

If you wish to bypass the initial building of the cluster and project deployment steps, go to Prepare for outtage for the cluster rebuilding steps.

Initialize the Infra and Cluster Repos

Using a command-line based approach, the example employs a process modeled after the doc Setup Jenkins X on Google Cloud with GKE, and will use Google Secret Manger. It requires installation of Git and Hub command line tools.

The following are the values used for the creation of the Infra and Cluster repos:

Repo Source:		jx3-gitops-repositories
Infra Repo:		jx3-terraform-gke
Cluster Repo:		jx3-gke-gsm
Git Organization:	$JX3ORG
JX3 Root Directory:	$JX3HOME

Building infra repo:${JX3ORG}/jx3-terraform-gke

cd ${JX3HOME}
git clone jx3-terraform-gke
cd  jx3-terraform-gke
git remote set-url origin${JX3ORG}/jx3-terraform-gke.git
hub create -p ${JX3ORG}/jx3-terraform-gke
git commit -a -m "chore: Initial"
git push -u origin master

Buidling cluster repo:${JX3ORG}/jx3-gke-gsm

cd ${JX3HOME}
git clone jx3-gke-gsm
cd jx3-gke-gsm
git remote set-url origin${JX3ORG}/jx3-gke-gsm.git
hub create -p ${JX3ORG}/jx3-gke-gsm
git commit -a -m "chore: Initial"
git push -u origin master

Manage URL format

This example uses external DNS with subdomain and URLs will have a root of To avoid including the default string "-jx" in the URLs, the example blanks out the spec:ingress:namespaceSubDomain value (".") in the jx-requirements.yml file. To eliminate having non-unique URLs after doing this type of update, an additional spec:environment:ingress:namespaceSubDomain line item is added ("-stg.") for the staging environment.


  - key: dev
  - key: staging
       namespaceSubDomain: -stg.
  - key: production
    domain: ""
    externalDNS: false
    namespaceSubDomain: .

Push initial changes to cluster repo

All the customizations for the cluster repo are now propagated.

cd ${JX3HOME}/jx3-gke-gsm
git add .
git commit -a -m “chore: Jenkins example init”
git push	

Build the infrastucture with Terraform

The following TF_VAR environment variables are set prior to running Terraform commands:

TF_VAR_gcp_project=<google project>
TF_VAR_apex_domain_gcp_project=<google project>
TF_VAR_jx_bot_username=<git username>
TF_VAR_jx_bot_token=<git token>


Additional detail on Terraform settings can be found under Google Terraform Quickstart Template

Commands to build intrastructure:

cd ${JX3HOME}/jx3-gke-gsm
bin/                # Performs terraform init, plan and apply

Enable prod certificate

This example employs TLS and accesing Jenkins from a browser requires a valid production certificate. It should be noted that TLS is not supported with automated domains (i.e. The process will begin by installing a Let’s Encrypt test certicate. After the initial build completes and is successful, it’s best to ensure that the test certificate is ready (READY=True) before enabing the production certificate. You can find more information regarding this subject under TLS and DNS.

## Set credentials
gcloud container clusters get-credentials jx3web --zone us-east1-b 
## Set jx namespace
jx ns jx
## Check certificate
kubectl get cert -n jx
NAME                     READY   SECRET                   AGE
tls-web-jx3rocks-com-s   True    tls-web-jx3rocks-com-s   39m

You can also use Octant to check certificate status.

Enabling the production certificate will require updates to both infra (jx-terraform-gke) and cluster (jx3-gke-gsm) repos. The commands to create the prod certificate are:

cd ${JX3HOME}/jx3-gke-gsm
git pull         ## Make sure local cluster repo is up to date
cd ${JX3HOME}/jx3-terraform-gke
export TF_VAR_lets_encrypt_production=true
terraform apply -auto-approve 
cd ${JX3HOME}/jx3-gke-gsm
git commit --allow-empty -m "Dummy commit"
git push
jx admin log     ## Make sure it completes before proceeding
git pull         ## Once successful refresh local cluster repo

The ${JX3HOME}/jx3-gke-gsm/jx-requirements.yml file’s TLS production setting should now be true.

    externalDNS: true
    kind: ingress
    namespaceSubDomain: .
      enabled: true
      production: true

Once again, the example checks the certificates status till the production certifacte is ready before proceeding.

kubectl get cert -n jx
NAME                     READY   SECRET                   AGE
tls-web-jx3rocks-com-p   True    tls-web-jx3rocks-com-p   39m
tls-web-jx3rocks-com-s   True    tls-web-jx3rocks-com-s   86m

You can also use Octant to check the production certificate. Octant Display

Deploy project to Stage and Production

After production certificates become ready, the following commands will deploy a simple JSnode app to staging and production

# Using quickstart deploy app to staging
jx project quickstart --git-token ${TF_VAR_jx_bot_token} --git-username ${TF_VAR_jx_bot_user} --filter node-http --org ${JX3ORG} --batch-mode  --name node-http01

# For documentation simplicity, use the following command to promote to production
jx promote -v 0.0.1 -e production --app node-http01 --app-git-url${JX3ORG}/node-http01.git 

# Confirm deployments
jx get applications
APPLICATION STAGING PODS URL                                     PRODUCTION PODS URL
node-http01 0.0.1   1/1  http://node-http01-stg.two.${JX3ORG}.com 0.0.1     1/1  https://node-http01.two.${JX3ORG}.com

Prepare for Outtage

To prepare for shutting down your cluster, a backup of the helm repo is essential. The helm repo is located in the chartmuseum pod’s local storage and the following will copy the helm chart files to a separate backup location (i.e. $HOME/storage).

mkdir -p $HOME/storage
cd ${HOME}/storage
kubectl cp jx/`kubectl get pods -n jx --selector=app=chartmuseum -o jsonpath="{.items[0]}"`:/storage .

Helm Repo Backup

You should now see a copy of the chart and index file in the backup location. You can ignore the ‘lost+found’ directory.

cd ${HOME}/storage
index-cache.yaml  lost+found             node-http01-0.0.1.tgz  

Destroy Infrastructure

Commands to shutdown infrastructure. Make sure the same Terraform variables are set found under Building the infrastructure with Terraform

cd ${JX3HOME}/jx3-terraform-gke
terraform destroy -auto-approve

Cluster Down

Upon completion of the commands above, the Kubernetes resources will be completely freed up. You want to ensure your infra (jx3-terraform-gke), cluster (jx3-gke-gsm), and project repos, as well as your Terraform variables and helm repo backup, remain unchanged and available for when ready to rebuild..

Prepare Cluster Repo for Rebuild

Prior to bringing the infrastructure (jx3-terraform-gke) back online, prepare the cluster repo (jx3-gke-gsm) by removing the reference of the jx-staging and jx-production environments and the long term storage.

cd ${JX3HOME}/jx3-gke-gsm
git pull     ## Make sure you get any updates from repo
# Remove long term storage refrences
sed -i '/storage:/,/vault:/{/storage:/!{/vault:/!d}}' jx-requirements.yml 
sed -i '/storage:/d' jx-requirements.yml
# Remove staging and prod links
sed -i '/jx-staging/d' helmfile.yaml
sed -i '/jx-production/d' helmfile.yaml
jx gitops helmfile resolve
git commit -a -m "chore: Remove staging and prod links"
git push

Rebuild Infrastructure

Commands to rebuild infrastructure. Once again, make sure you have the same Terraform variables set found under Building the infrastructure with Terraform

cd ${JX3HOME}/jx3-terraform-gke
terraform apply -auto-approve

Rebuild Helm Repo

After Terraform creates the infrastruce, the cluster repo resources will be used to launch a new Kubernetes cluster. Once the boot process completes and is sucessful you can safely perform the following recovery commands:

# Restore backup files to new helm repo
# Make sure chartmuseum pod is running
kubectl get pods -n jx --selector=app=chartmuseum 
# When chartmuseum pod is running it is then safe to proceed to the next steps.
cd $HOME/storage
for i in *; do kubectl cp $i jx/`kubectl get pods -n jx --selector=app=chartmuseum -o jsonpath="{.items[0]}"`:/storage/$i; done

NOTE: You must wait until the chartmuseum is up and fully functioning. The chartmuseum pod can be running, but the best approach to determine if you can proceed is to access (successfully) the chartmuseum URL (i.e.

Rebuild Cluster Repo

Restore the jx-staging and jx-production links

cd ${JX3HOME}/jx3-gke-gsm
git pull	## Make sure to get latest updates to repo
sed -i '/helmfiles:/ a - path: helmfiles/jx-production/helmfile.yaml' helmfile.yaml
sed -i '/helmfiles:/ a - path: helmfiles/jx-staging/helmfile.yaml' helmfile.yaml
git commit -a -m "chore: Restore staging and prod links"
git push

Post Recovery Troubleshooting

Sometimes you will need to do some post recovery steps to bring your environment online. Below is a real life example of steps taken after cluster returned back online and needed some adjustments.

Reviewing the applications notice that ‘node-http04’ in staging does not show pods and is not accessible.

jx get applications
APPLICATION STAGING PODS URL                                     PRODUCTION PODS URL
node-http02 0.1.0   1/1 0.1.0      1/1
node-http03 0.0.1   1/1 0.0.1      1/1
node-http04 0.0.1 0.0.1      1/1
node-http05 0.0.1   1/1 0.0.1      1/1

Check to see if the host entry resolves

ping: Name or service not known

Check to see if DNS record set exists

gcloud dns record-sets list --zone=${ZONEIDSUB} --project ${PROJECTID} --filter="TYPE=A OR TYPE=TXT" --format 'table[no-heading](NAME,TYPE,TTL,DATA)' | awk '{print " --name=" $1 " --type=" $2 " --ttl=" $3 " " $4 }' | grep -- node-http04-stg --type=A --ttl=300 --type=TXT --ttl=300 "heritage=external-dns,external-dns/owner=default,external-dns/resource=ingress/jx-staging/node-http04"

Check pod status

kubectl get pods -n jx-staging
NAME                                       READY   STATUS             RESTARTS   AGE
jx-verify-gc-jobs-8gb9h                    0/1     Completed          0          32m
node-http02-node-http02-749d864b58-m5mvk   1/1     Running            0          32m
node-http03-node-http03-59f5f89f4-mq86h    1/1     Running            0          32m
node-http04-node-http04-56f5f757c4-75rhb   0/1     ImagePullBackOff   0          32m
node-http04-node-http04-b74787f78-jcd2c    0/1     ImagePullBackOff   0          32m
node-http05-node-http05-5d6ccf954-g9lrj    1/1     Running            0          32m
node-http05-node-http05-7bc47d4f74-6rs9f   0/1     ImagePullBackOff   0          32m

kubectl logs -f node-http04-node-http04-56f5f757c4-75rhb -n jx-staging
Error from server (BadRequest): container "node-http04" in pod "node-http04-node-http04-56f5f757c4-75rhb" is waiting to start: trying and failing to pull image

Try restarting the 3 pods with the ImagePullBackOff (not running) status.

delete pods --field-selector=status.phase!=Running -n jx-staging

Final check running applications

jx get applications
APPLICATION STAGING PODS URL                                     PRODUCTION PODS URL
node-http02 0.1.0   1/1 0.1.0      1/1
node-http03 0.0.1   1/1 0.0.1      1/1
node-http04 0.0.1   1/1 0.0.1      1/1
node-http05 0.0.1   1/1 0.0.1      1/1

Cluster Rebuilt

Following the completion of the previous commands the cluster has now been returned to it’s previous state and should be fully functioning. If you wish to repeat the rebuild process start from the Prepare for outtage step.