Jenkins X Pipelines

cloud native serverless pipelines

In continuous delivery (CD) environments, a pipeline is a process (expressed as a collection of commands or plugins and a configuration file to express the development process) that automates the life cycle from repository source files to production deployment.

Jenkins X Pipelines is a serverless pipeline execution engine based on the Tekton Pipelines open source project. Tekton has been designed to be a modern cloud native solution for running pipelines.

Jenkins X pipelines are configured in YAML configuration files. The files can be found in two locations serving distinct purposes:

  • In the Jenkins X project repository, called jenkins-x.yml.
  • In the build packs for creating applications, if it is specified in the project repository pipeline.yaml file under buildPack.

Pipeline types

Each pipeline YAML file has a number of separate logical pipelines:

  • release for processing merges to the master branch which typically creates a new version and release then triggers promotion
  • pullRequest for processing Pull Requests
  • feature for processing merges to a feature branch. Consider using trunk based development which is a practice of high performing teams.

Lifecycles

Jenkins X has various steps in building, validating, and releasing your application through the development lifecycle. The lifecycle phases in the Jenkins X pipeline YAML configuration are:

  • setup - Steps to create the build environment, such as checking out code with git checkout or generating credentials files for Git provider authentication

  • preBuild - Steps to perform before a build occurs, such as ensuring a Docker image registry is available for building

  • build - Steps performed to build your application

  • postBuild - Steps performed after the build occurs, such as validating for Common Vulnerability Exposure (CVE) in any code changes.

  • promote - Shifting the state of an application (after build and validation) to another environment, such as Staging or Production.

Understanding Jenkins X pipelines

The Jenkins X cluster configuration process creates a YAML-based pipeline configuration file called jenkins-x.yml. This file configures the default development pipeline for building applications on kubernetes clusters with Jenkins X.

buildPack: none
pipelineConfig:
  pipelines:
    release:
      pipeline:
        agent:
          image: gcr.io/jenkinsxio/builder-go

buildPack specifies a build pack which contains a pipeline.yml file that supersedes the jenkins-x.yml file in the project directory. If none is specified, there is no build pack and Jenkins X uses the default pipeline configuration.

The configuration defines the pipeline agent, in this case a Google Container Registry image for the Go language build tools.

        environment:
          - name: DEPLOY_NAMESPACE
            value: jx

environment specifies environment variables used in the pipeline configuration. In this instance, the DEPLOY_NAMESPACE variable is used with a value of jx for the Jenkins X namespace.

        stages:
          - name: release
            steps:
              - name: verify-preintall
                dir: /workspace/source/env
                command: jx
                args: ['step','verify','preinstall']

stages are unique groups of steps (or nested stages sequentially run within a stage) that specify commands, directories, and arguments for a particular pipeline stage. In this instance, there is a step within the release stage called verify-preinstall that runs a jx command that verifies whether cloud infrastructure (such as the presence of the kubectl binary and the correct version of git is installed) was setup in the preinstallation process.

              - name: install-vault
                dir: /workspace/source/systems/vault
                command: jx
                args: ['step', 'boot','vault']

name calls out a unique step in the pipeline configuration that defines development steps to verify and apply arguments to various commands necessary for the stage under which it is nested. In this instance, install-vault installs the Hashicorp Vault tool for secrets management.

              - name: apply-repositories
                dir: /workspace/source/repositories
                command: jx
                args: ['step','helm','apply', '--name', 'repos']

This step creates and applies the Helm Package Manager for installation and management of helm kubernetes applications.

              - name: apply-pipeline-schedulers
                dir: /workspace/source/prowConfig
                command: jx
                args: ['step','scheduler','config', 'apply', '--direct=true']

This step allows the pipeline to work with a scheduler, which executes program jobs unattended in the background.

              - name: update-webhooks
                dir: /workspace/source/repositories
                command: jx
                args: ['update','webhooks','--verbose', '--warn-on-fail']

This step updates webhooks, which is a service that listens for GitHub activity and trigger jobs, send automated messages to chat clients such as Slack, and other configurable actions.

              - name: verify-install
                dir: /workspace/source/env
                command: jx
                args: ['step','verify','install', '--pod-wait-time', '30m']

This step verifies the project installation, downloading and installing or updating components when necessary.

    pullRequest:
      pipeline:
        agent:
          image: gcr.io/jenkinsxio/builder-go

pullRequest is a logical pipeline within the project pipeline that specifies how pull requests are managed when changes are made to the project repository in GitHub.

        stages:
          - name: release
            steps:
              - name: helm-build
                dir: /workspace/source/env
                command: make
                args: ['build']

The pullRequest pipeline contains a stage wherein steps can also be executed. In this instance, make is run to create a helm chart and validate that a build has been completed.

Extending pipelines

A pipeline YAML can extend another YAML file. You can reference a base pipeline YAML using the following methods:

  • Using file to reference a relative file path in the same build pack

    extends:
  file: ../jenkins-x.yaml

  • Using import to reference a YAML file:

    extends:
  import: classic
  file: maven/pipeline.yaml

which then refers to a named imported module via git:

modules:
- name: classic
  gitUrl: https://github.com/jenkins-x-buildpacks/jenkins-x-kubernetes.git
  gitRef: master

Overriding steps

Users can override steps in a pipeline YAML from a base pipeline YAML, similar to overriding classes in languages like Java. This allows users reuse the steps in a base pipeline’s lifecycle, then add additional steps.

By default any steps you define are added after the base pipeline YAML steps. For example:

extends:
  file: base-pipeline.yaml
pipelines:
  pullRequest:
    build:
      steps:
      - sh: export VERSION=$PREVIEW_VERSION && skaffold build -f skaffold.yaml

You can add steps before the base pipeline steps using the preSteps: property:

extends:
  file: base-pipeline.yaml
pipelines:
  release:
    setup:
      preSteps:
      - sh: echo BEFORE BASE SETUP
      steps:
      - sh: echo AFTER BASE SETUP
    build:
      replace: true
      steps:
      - sh: mvn clean deploy -Pmyprofile
        comment: this command is overridden from the base pipeline

If you want to completely replace all the steps from a base pipeline for a particular lifecycle you can use replace: true:

  replace: true
  steps:
  - sh: mvn clean deploy -Pmyprofile
    comment: this command is overridden from the base pipeline

Last modified August 29, 2020: fix: build pack pipeline file name (2be2618da7)