Pipeline Catalogs

Integration with Tekton Pipelines and Tekton Catalog

As part of the Tekton Catalog enhancement proposal we’ve improved support for Tekton in Jenkins X so that you can

Source changes

If you upgrade your cluster to the latest version stream then you will find if you create a new quickstart that:

Editing pipelines

You can now easily modify any of the Task, Pipeline or PipelineRun resources in any git repository - just look in each folder inside .lighthouse for the YAML files to edit.

e.g. for the default Jenkins X CI/CD pipelines edit either:

You can test out changes to the Pull Request pipeline by submitting changes in a Pull Request. Changes to a release only take place after merging the change to the main branch.

IDE support

If you use IntelliJ IDEA or Goland you might find the RedHat’s intellij-tekton plugin useful for editing pipelines with schema validation and completion.

If you use VS Code you may want to try Red Hat’s Tekton Pipelines Extension tekton

Adding Tasks from the Tekton Catalog

The new jx pipeline import command can be used to import Task resources from the Tekton Catalog and using them inside your project.

Here’s a demo of this in action:

The tekton Task resources are copied into your .lighthouse directory in a folder using kpt so that you can modify things locally if you need to and can upgrade your local copy with upstream changes via the jx gitops upgrade command described below.

Add new tasks/pipelines by hand

You can add new pipelines by hand into a new folder inside .lighthouse at any time.

To setup a trigger so that lighthouse will start your pipeline on a presubmits (i.e. for Pull Requests) or for postsubmits (i.e. releases on main branches) you need to also add a triggers.yaml file which uses the lighthouse trigger config file file format with this spec field.

You could look at the default .lighthouse/jenkins-x directory to see how all this works. The triggers.yaml file then refers to the tekton Task, Pipeline or PipelineRun files via the source: attribute in a presubmits or postsubmits entry.

Changing the triggers

You can modify the .lighthouse/*/triggers.yaml file to modify the presubmits and/or postsubmits entries to do things like:

  • customise the rerun_command or trigger ChatOps comments for presubmits
  • configure the branches patterns for postsubmit triggers
  • add new entries for new pipelines; or pipelines with different pipeline_run_params entries to parameterise existing PipelineRun files differently

Upgrading pipelines and helm charts

You can upgrade any git repository in the same way you upgrade your clusters git repository by running the jx gitops upgrade command inside a git checkout of your repository:

cd my-quickstart-thingy
jx gitops upgrade

This will then upgrade any helm charts or pipeline catalogs you are using in your git repository with the latest versions.

After running this command you will usually have some changes in git you can review. If you are happy with the changes commit them and create a Pull Request so that they can get applied on your cluster.

git add *
git commit -a -m "fix: upgrade pipeline catalog"
git push

It is possible that you can have merge conflicts.

You can follow the inline git helper messages to resolve conflicts - or use your IDE to help figure out the merge issues more easily.

Upgrading all repositories

You can now perform a batch of Pull Requests if you need to upgrade your pipelines on your repositories if the upstream pipeline catalogs have upgraded.

See: generate pull requests to upgrade pipelines

Diagnosing problems

If you edit pipelines or lighthouse trigger files and things don’t work there’s a couple of places the errors may show up.

We will hopefully add much better linting/error messages on Pull Requests soon to give you better and faster feedback.

Until then you could look in:

  • the lighthouse-webhooks-* pod(s) which take the webhooks from your git provider and convert them into lighthousejob resources
  • the lighthouse-tekton-controller-* pod(s) which watch for lighthousejob resources and create the Tekton PipelineRun resources
  • the tekton-controller-* pod(s) watches for Tekton PipelineRun resources and conver them into Kubernetes Pod resources

Any errors will usually be recorded in the status field of the resource that has issues (lighthousejob or pipelinerun).

Reference Guide

The following are the links to the various configuration file formats:

Tekton resources:

Lighthouse TriggerConfig:

  • presubmits for triggering pipelines on Pull Request
  • postsubmits for triggering pipelines on a push to a branch (e.g. releasing)

Parameters and Environment Variables

The following tekton parameters and environment variables are available inside the pipeline catalog. They are populated by lighthouse

  • BUILD_ID a unique long number for this build
  • JOB_NAME the name of the build which matches the name in the presubmit or postsubmit in your lighthouse triggers.yaml
  • JOB_SPEC is of the form type:presumit or type:postsubmit so you know what kind of job you are inside
  • PULL_BASE_REF the base branch name. e.g. master or main
  • PULL_BASE_SHA the base git SHA being built
  • REPO_NAME the name of the git repository
  • REPO_OWNER the owner of the git repository (a user or organisation)
  • REPO_URL the git URL to clone the repository being built

Pull Requests presubmit also have the following values:

  • PULL_NUMBER the number of the pull request
  • PULL_PULL_REF the git reference of the pull request; something like refs/pull/123/head
  • PULL_PULL_SHA the git SHA of the pull request

Additional environment variables

If your pipeline runs the jx gitops variables command it will lazily create the .jx/variables.sh script which will default a bunch more environment variables if they are not already populated in your git repository.

NOTE that these variables are dynamically created during the execution of the pipeline pod; so to access them you must source .jx/variables.sh inside your step.

So that your step looks something like this…

- image: gcr.io/jenkinsxio/jx-cli:latest
  name: my-step
  script: |
    #!/usr/bin/env bash
    source /workspace/source/.jx/variables.sh
    echo "now we can use variables like this: ${VERSION}"

Available variables:

  • APP_NAME the name of the application which defaults to the $REPO_NAME
  • BRANCH_NAME is really the pull request name so something like PR-123
  • BUILD_NUMBER the human readable short build number relative to the repository and branch. So builds start at 1 and go up incrementally
  • DOCKERFILE_PATH the location of the Dockerfile if it exists
  • DOCKER_REGISTRY the host name of the registry being used for image builds
  • DOCKER_REGISTRY_ORG the owner in the container registry (user name or organisation) to push images
  • JX_CHART_REPOSITORY the URL of the helm chart repository to use
  • PIPELINE_KIND the kind of pipeline being run pullrequest or release
  • VERSION the version number used for releases (and used to tag images and git etc) or the preview version for pull requests

If you want to define dynamic environment variables in one step for use in later steps you can append new variables to .jx/variables.sh and then add the source .jx/variables.sh later in your pipeline

Last modified December 4, 2020: fix: add batch migrate docs (ed87244347)