As we create more and more software we tend to get an explosion in the number of git repositories and microservices. Each repository needs automated CI and CD; but how do we manage the hundreds of pipelines we need - while also making it easy to share pipelines across repositories and allowing each repository to customize when required?
Jenkins X solves this as follows:
- the pipelines, tasks and steps are defined via Tekton YAML letting you use any tekton tooling such as IDE completion and validation
- we support an
image: uses:sourceURInotation that lets you inherit steps from a git repository without having to copy/paste the source code aross repositories.
For example if you create a new quickstart pipeline may look like this (slightly condensed)…
You may wonder what those
uses: strings mean.
Referencing a Task or Step
Rather than copy pasting task and step YAML between repositories we can refer to a
Task or a
Step in a Task as follows:
- refer to all the steps in a task by using
taskSpec: steps: - image: uses:sourceURI
- refer to a single named step from a task
taskSpec: stepTemplate: image: uses:sourceURI steps: - name: mystep
The source URI notation is enabled by a special
image prefix of uses: on step or if an image on a step is blank and the
stepTemplate: has an
image prefix of uses:
You can refer to the detailed documentation on how the step inheritence and overriding works.
For a github.com source URI we use the syntax:
- image: uses:owner/repository/pathToFile@version
This references the https://github.com repository for
owner/repository and @version can be a git tag, branch or SHA.
If you are not using github.com to host your git repositories you can access a pipeline task or step from your custom git serve use the uses:lighthouse: prefix before
- image: uses:lighthouse:owner/repository/pathToFile@version
We recommend you version everything with GitOps so you know exactly what versions are being used from git.
However you can use @HEAD to reference the latest version.
To use a locked down version based on the version stream of your cluster, you can use @versionStream which means use the git SHA for the repository which is configured in the version stream.
The nice thing about @versionStream is that the pipeline catalog you inherit tasks and steps from is locked down to an exact SHA in the version stream; but it avoids you having to go through every one of your git repositories whenever you upgrade a pipeline catalog.
file and URL syntax
If you want to reuse a Task or Step thats not easily accessible from the above git source URI syntax you can always use a http:// or https:// URL instead:
- image: uses:https://myserver.com/cheese.yaml
If there’s no @version on a uses string its interpreted as a local file:
- image: uses:some-file.yaml
How it looks
Also notice we don’t have to copy and paste the exact details of the images, commands, arguments, environment variables and volume mounts required for each step; we can just reference them via Git. Also each pipeline in each repository can reference different versions if required.
Adding your own steps
You can easily add your own steps in between the
uses: steps in your pipeline by adding a regular step which has a custom image.
my-prefix-step which has an explicit
image: value so isn’t inherited from the
Customizing an inherited step
You can edit the step in your IDE and add any custom properties such as
volumeMount - those values then override the inherited step.
e.g. you can then change any command line, add an environment variable or add a new volume mount without copy pasting the whole step. e.g. we change the
script value of the
jx-variables step below:
Any extra properties in the steps are used to override the underlying uses step.
Command line tools
To help understand the inheriance and overriding of steps we have a few command line tools to help:
Viewing the effective pipeline
To see the actual Tekton pipeline that would be executed from your local source directory you can run the jx pipeline effective command:
jx pipeline effective
If you want to open the effective pipeline in your editor, such as VS Code you can do:
jx pipeline effective -e code
jx pipeline effective -e idea
If you want to always view an effective pipeline in your editor then define the
JX_EDITOR environment variable…
export JX_EDITOR="code" # now we will always open effective pipelines inside VS Code jx pipeline effective
Overriding a pipeline step locally
If you want to edit a step that is inherited from a pipeline catalog just run the jx pipeline override command from a clone of your repository.
jx pipeline override
This will then prompt you to pick which pipeline and step that’s inherited via the
image: uses:sourceURI notation. When chosen the step will be inlined into your local file so you can edit any of the properties.
You can use the git compare to see the changes and remove any properties you don’t wish to override.
Overriding specific properties of a step locally
Rather than completely replacing the entire step with whatever is in the catalog; you can just specify the properties you wish to override.
e.g. usually you may just want to override the
script blog to modify, say, a command line argument passed into a command.
To do this just specify the -P with the name of the property.
jx pipeline override -P script
You will then just have a locally modified
script property in the Tekton YAML for the step you overrode. You can then locally modify to suit - or remove to go back to the inherited catalog properties.
The Tekton Catalog git repository defines a ton of Tekton pipelines you can reuse in your pipelines
Referencing Tasks or Steps from a Catalog
image: uses:sourceURI notation inside any pipeline file in your
.lighthouse/jenkins-x/mypipeline.yaml file like this:
steps: - image: uses:tektoncd/catalog/task/git-clone/0.2/git-clone.yaml@HEAD
This will then include the steps from the git-clone.yaml file
Custom Pipeline Catalog
See this section to configure your custom pipeline catalog.
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.