Moving Jenkins X v2 artifacts

Moving v2 artifacts

By James Rawlings | Thursday, August 26, 2021

ACTION REQUIRED

TL;DR - Jenkins X specific helm repositories and container registries hosted on GCP have been moved to GitHub. This will mainly affect jx v2 users but there is expected to be a small impact on v3 users too. Below describes the steps we believe are needed to keep Jenkins X installations working as normal but there will be some action needed.

Why the disruption?

When Jenkins X first started we made heavy use of GCP’s services for hosting the cloud infrastructure needed by users to install and run Jenkins X. This was great as we could use the same IAM to push and maintain content from our own hosted build infrastructure and ensure we were validating the same experience of using cloud provider hosted services wherever possible. As Jenkins X grew in popularity the cloud costs began to increase with the pricing model from GCP, specifically the networking costs of cross continent egress.

Given this, for jx3 we decided to see if switching to GitHub packages for container images and GitHub pages for helm repositories would be better, the result was it is better. In fact we have made it super easy for users to switch to using GitHub pages for releasing helm charts and using GitHub packages.

Now that we have validated GitHub is more cost effective for hosting public images and helm charts for the Jenkins X project, we want to switch to using GitHub for all v2 plus v3 users, then shutdown the GCP services which are causing unnecessary cost.

It is expected that v3 users will need a small change and v2 slightly more. Details for both will be described below but it is worth noting that there hasn’t been a v2 release in 9 months and v3 was GA in April earlier this year, so we aren’t expecting too many folks on v2. We are aiming to limit any disruption and help provide easy steps to handle the move.

We apologise for any extra work caused however, this is required to preserve the long running hosting of Jenkins X artifacts both past and present. If you experience issues that are not covered by the steps below please reach out to the community slack channel and we can help address and update this blog with details.

What is changing?

We will be shutting down a number of GCP projects that contain old helm charts plus container images and moving them to be hosted by GitHub packages and pages.

Helm

https://chartmuseum.build.cd.jenkins-x.io
http://chartmuseum.jenkins-x.io
https://storage.googleapis.com/chartmuseum.jenkins-x.io

have been moved to

https://jenkins-x-charts.github.io/v2

AND

https://storage.googleapis.com/jenkinsxio/charts

has been moved to

https://jenkins-x-charts.github.io/repo

Images

The most recently versioned images from gcr.io/jenkinsxio have been moved to ghcr.io/jenkins-x

Labs

There are some old labs images and helm charts which should not be in use as they are either deprecated or replaced with GA versions in the v3 helm repo or container registry.

v2 users

The Jenkins X own v2 build infrastructure was retired at the start of the year as no more releases were planned and to reduce costs. With that we are unable to perform a new release that automatically switches references to images from gcr.io/jenkinsxio to ghcr.io/jenkins-x. If you are still using v2 then please update your references to this container registry. An alternative which has not yet been verified is to use a image swap Kubernetes mutating admission controller which takes configuration to switch the registry on the fly. We have asked on slack for help validating the approach so if you do try it please share feedback and config used to help others in the channel, we can then update docs.
In your boot git repository, run a search for references of http://chartmuseum.jenkins-x.io and https://storage.googleapis.com/chartmuseum.jenkins-x.io replace with https://jenkins-x-charts.github.io/v2
Change the lighthouse-jx-controller deployment to use an environment variable

JX_DEFAULT_IMAGE=ghcr.io/jenkins-x/builder-maven:2.1.155-778-patch3

Environment controller (can be skipped if not using)

i) change the environment controller image to be ghcr.io/jenkins-x/builder-maven:2.1.155-778-patch3

ii) change the image used in the pipeline, needs to be changed in the jenkins-x.yaml of the enviromnet repo:
```
agent:
 container: ghcr.io/jenkins-x/builder-jx:2.1.155-778-patch3
```
iii) add this environment variable in the deployment of the environment-controller
```
 - name: BUILDER_JX_IMAGE
   value: ghcr.io/jenkins-x/builder-jx:2.1.155-778-patch3
```

In your boot git repository, set the following values for env/jenkins-x-platform deployment:

Value
expose.Image	ghcr.io/jenkins-x/exposecontroller
expose.ImageTag	2.3.118
cleanup.Image	ghcr.io/jenkins-x/exposecontroller
cleanup.ImageTag	2.3.118
controllerbuild.image.repository	ghcr.io/jenkins-x/builder-jx
controllerbuild.image.tag	2.1.155-778-patch3
controllerbuild.env.BUILDER_JX_IMAGE	ghcr.io/jenkins-x/builder-jx:2.1.155-778-patch3
postinstalljob.image.repository	ghcr.io/jenkins-x/builder-jx
postinstalljob.image.tag	2.1.155-778-patch3
postinstalljob.env.BUILDER_JX_IMAGE	ghcr.io/jenkins-x/builder-jx:2.1.155-778-patch3
controllerrole.image.repository	ghcr.io/jenkins-x/builder-jx
controllerrole.image.tag	2.1.155-778-patch3
controllerrole.env.BUILDER_JX_IMAGE	ghcr.io/jenkins-x/builder-jx:2.1.155-778-patch3
gcpreviews.image.repository	ghcr.io/jenkins-x/builder-jx
gcpreviews.image.tag	2.1.155-778-patch3
gcpreviews.env.BUILDER_JX_IMAGE	ghcr.io/jenkins-x/builder-jx:2.1.155-778-patch3
gcactivities.image.repository	ghcr.io/jenkins-x/builder-jx
gcactivities.image.tag	2.1.155-778-patch3
gcactivities.env.BUILDER_JX_IMAGE	ghcr.io/jenkins-x/builder-jx:2.1.155-778-patch3
gcpods.image.repository	ghcr.io/jenkins-x/builder-jx
gcpods.image.tag	2.1.155-778-patch3
gcpods.env.BUILDER_JX_IMAGE	ghcr.io/jenkins-x/builder-jx:2.1.155-778-patch3

In your boot git repository, set the following values for env/lighthouse-jx deployment:

Value
image.parentRepository	ghcr.io/jenkins-x
image.tag	0.0.164
jxcontroller.image.repository	ghcr.io/jenkins-x/lighthouse-jx-controller
jxcontroller.image.tag	0.0.164
env.JX_DEFAULT_IMAGE	ghcr.io/jenkins-x/builder-maven:2.1.155-778-patch3

In your boot git repository, set the following values for env/lighthouse deployment:

Value

image.parentRepository ghcr.io/jenkins-x

image.repository ghcr.io/jenkins-x/lighthouse

env.JX_DEFAULT_IMAGE ghcr.io/jenkins-x/builder-maven:2.1.155-778-patch3
In your boot git repository, update the jenkins-x.yml replacing any reference to image: gcr.io/jenkinsxio/builder-go:2.1.155-778 with image: ghcr.io/jenkins-x/builder-go:2.1.155-778-patch3
In your boot git repository, update the jenkins-x-release.yml replacing any reference to image: gcr.io/jenkinsxio/builder-go:2.1.155-778 with image: ghcr.io/jenkins-x/builder-go:2.1.155-778-patch3
In your boot git repository, update the systems/jxing/values.tmpl.yaml setting

Value
nginx-ingress.controller.image.repository	ghcr.io/jenkins-x/nginx-ingress-controller

In any environment managed repository (e.g. environment-*-staging | production) update env/values.yaml:

Value
expose.Image	ghcr.io/jenkins-x/exposecontroller
expose.ImageTag	2.3.118
cleanup.Image	ghcr.io/jenkins-x/exposecontroller
cleanup.ImageTag	2.3.118

In any environment managed repository (e.g. environment-*-staging | production) update env/requirements.yaml replacing any exposecontroller repository url with https://jenkins-x-charts.github.io/v2

- alias: expose
  name: exposecontroller
  repository: https://jenkins-x-charts.github.io/v2
  version: 2.3.118
- alias: cleanup
  name: exposecontroller
  repository: https://jenkins-x-charts.github.io/v2
  version: 2.3.118

In any applications repositories who has already import into jx, update the ./jenkins-x.yaml add or replace :
```
pipelineConfig:
  agent:
    image: `ghcr.io/jenkins-x/BUILDER_YOU_NEED:2.1.155-778-patch3`
```
Here is the list of the principal images who has migrated to new ghcr:

Value	Version
builder-jx	2.1.155-778-patch3
builder-nodejs	2.1.155-778-patch3
builder-nodejs12	2.1.155-778-patch3
builder-nodejs14	2.1.155-778-patch3
builder-maven-java11	2.1.155-778-patch3
builder-php5x	2.1.155-778-patch3
builder-php7x	2.1.155-778-patch3
builder-python37	2.1.155-778-patch3
builder-dotnet	2.1.155-778-patch3

Have we missed anything? Please contribute to this blog or feedback on the slack channel.

v3 users

There is not expected to be significant disruption to v3 users but if there is anything needed beyond the steps below then we are asking users to reach out asap and we can update this blog.

Run jx gitops upgrade to ensure you upgrade to the latest version stream with the old helm repository removed. If you are tracking the LTS version stream please delay until Wednesday 1st September to run this.
In your cluster git repository, run a search for references of https://storage.googleapis.com/jenkinsxio/charts replace with https://jenkins-x-charts.github.io/repo
In your cluster git repository, run a search for references of http://chartmuseum.jenkins-x.io and https://storage.googleapis.com/chartmuseum.jenkins-x.io replace with https://jenkins-x-charts.github.io/v2
Switch jx-verify helm chart repository for any application you have which is built by Jenkins X 3. This is under your applications git repository ./charts/preview/helmfile.yaml change https://storage.googleapis.com/jenkinsxio/charts to https://jenkins-x-charts.github.io/repo . Here is an example that changes the main pipeline catalog packs which are used when first creating or importing applications.
Switch any gcr.io/jenkinsxio images to ghcr.io/jenkins-x in your application git repo .lighthouse/*.yaml files if you have references there

Have we missed anything? Please contribute to this blog or feedback on the slack channel.

When will all this take place?

This blog is the initial notice which we will socialise, please help to raise awareness.
Friday 27th August - the labs project will be scheduled to shutdown, short notice because we believe no services should be used, if they are it is an easy switch to upgrade to GA versions. Labs efforts are never intended to be production grade and are used at risk.
Monday 6th September - make the GCP container registry and helm repository bucket private, during which time any image versions that have not been transferred to GitHub can be requested via the community slack channel. All helm versions have been moved to https://jenkins-x-charts.github.io/v2 as described above.
Monday 13th September - schedule for shutdown the two GCP projects hosting the container registry and helm repository.

Why are only the most recent versions v2 images copied to GitHub packages and not all versions?

There are 14 Terabytes of data that make up the jenkinsxio container registry on GCP, it would be costly and wasteful to transfer all this to GitHub so we picked the last known version of each image that was released last year. If there are specific images that you wish to use either pull / push them yourself to a container registry of your own or reach out and on a case by case effort, we can look to move them to GitHub while the read permissions are made private and before the project is shut down.

I’m on v2 and use a builder image which is not available on GitHub container registry, how do I build my own version to work with the new helm and image repositories?

The old v2 jx code lives on a branch you will need to:

fork this branch
apply search and replace changes for i) gcr.io/jenkinsxio to ghcr.io/jenkins-x example ii) https://storage.googleapis.com/chartmuseum.jenkins-x.io to https://jenkins-x-charts.github.io/v2 example
run make linux to build the updated jx binary
build and push the images you require, for example

docker build -f Dockerfile.builder-maven -t ghcr.io/jenkins-x/builder-maven:2.1.149-768-patch3 .
docker push ghcr.io/jenkins-x/builder-maven:2.1.149-768-patch3

once the builder-jx/go/maven/etc.. are pushed, you will need to configure the charts:

if a chart allows the image to be overridden using a value you can add that to your env/values.tmpl.yaml, replacing current gcr.io/jenkinsxio images with ghcr.io/jenkins-x e.g.

jenkins-x-platform:
  gcpreviews:
    image:
      repository: ghcr.io/jenkins-x/builder-jx
      tag: [your custom tag]

if you cannot override the image using a helm value you may need to:

download each needed chart/subchart needed to make jx work, exemple for jenkins-x-platform/, jxboot-resources/releases
for each charts/subcharts replace current gcr.io/jenkinsxio images with ghcr.io/jenkins-x if it exists or rebuild and host them on a private registry if not
host updated charts on a private chartmuseum
switch boot git repo to use custom charts

I’m getting a missing arg `--provider-values-dir` and helm repository https://jenkins-x-charts.github.io/v2 does not have an associated prefix in in the ‘charts/repositories.yml’ error

A few users have been hitting this error, it is related to the jx version used. This thread with the help of Francesco Capozzo contains both image tags and versions to use instead:

https://kubernetes.slack.com/archives/C9MBGQJRH/p1631112970450800

Images:

ghcr.io/jenkins-x/builder-jx:2.1.142-761-patch3
ghcr.io/jenkins-x/builder-maven:2.1.142-761-patch3
ghcr.io/jenkins-x/builder-go:2.1.142-761-patch3

Chart versions to use rather than relying on the version stream:

dependencies:
* name: jxboot-resources
  repository: https://jenkins-x-charts.github.io/v2
  version: 0.0.43
* alias: tekton
  name: tekton
  repository: https://jenkins-x-charts.github.io/v2
  version: 0.0.63
* alias: prow
  condition: prow.enabled
  name: prow
  repository: https://jenkins-x-charts.github.io/v2
  version: 0.0.1773
* alias: lighthouse
  condition: lighthouse.enabled
  name: lighthouse
  repository: https://jenkins-x-charts.github.io/v2
  version: 0.0.843
* alias: lighthouse-jx
  condition: lighthouse.enabled
  name: lighthouse-jx
  repository: https://jenkins-x-charts.github.io/v2
  version: 0.0.121
* alias: bucketrepo
  condition: bucketrepo.enabled
  name: bucketrepo
  repository: https://jenkins-x-charts.github.io/v2
  version: 0.1.42
* name: jenkins-x-platform
  repository: https://jenkins-x-charts.github.io/v2
  version: 2.0.2411
* name: jx-pipelines-visualizer
  repository: https://jenkins-x-charts.github.io/repo
  version: 1.7.3

←Previous

Value
image.parentRepository	ghcr.io/jenkins-x
image.repository	ghcr.io/jenkins-x/lighthouse
env.JX_DEFAULT_IMAGE	ghcr.io/jenkins-x/builder-maven:2.1.155-778-patch3