Jenkins X was the star of the show at DevOps World | Jenkins World 2019. In this article I will
share with you the dozen key things I learned about this exciting new cloud native CI/CD tool.
Beyond its capabilities as a CI/CD tool, Jenkins X also provides an excellent example of how to architect
a cloud native application on Kubernetes.
Jenkins X is a completely new CI/CD system that shares little but its name with the existing Jenkins. Jenkins X incorporates the best practices from the State of DevOps reports and the seminal book, Accelerate by Nicole Forsgren, Jez Humble and Gene Kim.
Terraform is recommended for
setting up the required Kubernetes cluster and storage buckets. As explained below, you may find it
useful to run jx boot from within Terraform.
By default, Terraform stores its state in local file terraform.tfstate. In an ephemeral cloud
environment, this state gets lost and you would create a new cluster each time you applied
Terraform. Remedy this by specifying a Terraform backend to store the state in more durable storage like Google Cloud Storage or
Presenters recommended nginx as an ingress controller and
cert-manager to manage TLS
(HTTPS SSL) certificates.
For introspection, navigation and object management of your Kubernetes cluster, try
VMWare’s Octant UI tool. It runs on your local client just like
kubectl. An advantage of Octant is that it authenticates the same way as kubectl: if kubectl works, octant works.
Static traditional Jenkins master with Jenkins pipelines. Use this if you want to continue
using your existing Jenkinsfiles.
Jenkins X pipelines based on Tekton pipelines.
This is now the default, and is recommended for the long term. This mode is controlled by
the new jenkins-x.yml file, whose syntax resembles the Jenkinsfile declarative pipeline
There are two interactive quick start commands. The older and presumably more stable is:
jx create quickstart
The new way to install, configure and upgrade Jenkins X is:
jx boot interactively queries the user for the required setup data, recording the responses in file jx-requirements.yml. It is re-entrant so you can run it repeatedly. Running jx boot from within Terraform is a useful technique.
Jenkins X evolves quickly, so jx boot records the Jenkinx X version to use in field
versionStream within jx-requirements.yml. This establishes the version to use on subsequent
invocations of jx boot. Update versionStream when you want to start using a newer version of
Get logs using:
jx get build logs
Track execution with:
jx get activity
List preview environments using:
jx get environments
Basic concepts in Jenkins X pipelines
At this point a little vocabulary lession is in order.
A step is a command that runs in a separate container, sharing a workspace with other steps. Once a step fails, subsequent steps will not run. Step names must be unique within a stage. There is also a loop step, which runs the same command for each value in a list.
All the usual Kubernetes container configuration of resources, limits, volume mounts and so on are available within steps.
A stage is a unit of work in a pipeline. A stage contains either steps or nested stages. Each stage with steps runs in its own pod. The workspace is copied from one stage’s pod to the next.
There are two types of pipelines:
Release pipelines merge into master, create a release and trigger promotion
As the name implies, pull request pipelines are used to create a merge request preview environment
A pipeline is controlled by a jenkins-x.yml file at the top of the source control repository.
Jenkins X has a bootstrapping problem: how to create a pipeline? The meta pipeline creates the real pipeline, sort of like Maven release:prepare.
There are three ways to define a Jenkins X pipeline:
Automatically via a build pack. The build pack automatically detects the source code language.
Specify a build pack and then override portions of it in jenkins-x.yml
Fully define an entirely new pipeline in jenkins-x.yml. This is only useful for a pipeline
that will not be reused. For reusable pipelines, define a build pack.
Build packs are standard, opinionated pipelines for languages. They consist of a predefined
sequence of steps that run in a consistent order. They are similar to stages in that they can be
overridden and extended.
Jenkins X is controlled by a jenkins-x.yml file that lives at the root of the Git repository.
In jenkins-x.yml, you can:
Override build packs
Run steps before existing step using setup
Use type: replace to replace a step
To add a step to the end of the stage, leave the step name unspecified
To apply an override to all pipelines, leave the pipeline name unspecified
Modify container configuration or environment variables. Override environment variables
using the same syntax as Kubernetes.
You can also define a completely new pipeline in jenkins-x.yml, but this is useful chiefly for debugging and testing. Again, make a pipeline reusable by defining it as a build pack.
Pull request and release pipelines are often very similar, so define common attributes for both
in a default pipeline.
Validating syntax and IDE autocompletion
Check your pipelines using:
jx step syntax validate
Pipelines are usually defined by multiple YAML files. See how they fit together in a single flat pipeline file:
jx step syntax effective
The Jenkins X YAML schema is uploaded to schemastore.org, where amazingly IntelliJ and the VS Code YAML Language Extension automatically pick it up.
GitOps, Prow and Lighthouse
GitOps is at the core of Jenkins X. GitOps extends Infrastructure as code by using pull requests to manage infrastructure changes. These pull requests might be created by developers, or they might be generated automatically by tooling.
Accordingly, Jenkins X creates a preview environment for each run of a pull request pipeline.
Jenkins X currently uses Prow as a webhook handler and ChatOps engine for pull requests. Importantly, at this time Prow supports only GitHub.
Since Jenkins X uses only a small portion of Prow, and since Prow supports only GitHub, the
Jenkins X team is developing Lighthouse as a
lightweight replacement. In this very experimental webhook handler, the Git provider is factored
out so that new providers can be easily added.
It can be difficult to avoid commiting secrets to Git. Thus separate file parameters.yml contains URL references to a secret. Also, Jenkins X runs Helm in a temporary directory.
The Helm chart values.yml files are separated into individual values.tmpl.yml files. These are templates so you can easily interpolate secrets into them.
Implementing HTTPS in preview environments
It is difficult to get HTTPS to work in preview environments because each preview environment gets a different URL. James Rawlings demonstrated the solution:
Specify that you want to use DNS for HTTPS in jx boot. This will cause external-dns to be installed automatically.
Add externalDNS in requirements.yml:
Then, create the domain (this example assumes you are using GKE):
jx create domain gke --domain rawlingsdemo.co.uk
Finally, go to your domain registrar and replace their name servers with Google’s name servers
Using Jenkins X in multiple clusters
Strictly speaking, Jenkins X does not require its own cluster, but things work better operationally if you use separate clusters for testing, staging and production.
You only need to install Jenkins X on the development cluster. Install the Jenkins X environment controller on the staging and production clusters. This is further explained in the article Multiple Clusters.
Upcoming Jenkins X features
Jenkins X is evolving rapidly. Upcoming features are:
Conditional execution of stages
Jenkins X apps to inject steps and stages
More advanced configuration of stage pods
More advanced solution for pipeline and stage sharing across repos
Learning Jenkins X
Many fine examples with source code on GitHub were presented:
Mauricio Salatino presented what was by far
the most extensive example.
Starting with a monolithic application, he decomposed it into microservices, built it with
Jenkins X and deployed it on Kubernetes.
For instruction on how to use Kubernetes itself, Mauricio recommends