Jenkins Server Example

An example how to build Jenkins server with additional plugins and installers.

This is an example on building a JX3 Google/GSM environment from scratch which includes a Jenkins server and the import of a pipeline. It will utilize DNS (jx3rocks.com), TLS, Let’s Encrypt certificates, and additional Jenkins plugins and installer resources. It is inteded for an audience already familiar with Jenkins X operability and focuses on an example of actual commands required to build and operate the environment. Additional information regarding using Google as the provider for this example can be found under Google Cloud Platform Prequisites.

💡 This doc has been designed to build the Jenkins server under the Google cloud platform. The Jenkins server configuration steps have also been tested under Azure and Amazon cloud environments.

Initialize the Infra and Cluster Repos

Using a command-line based approach, the example employs a process modeled after the doc Setup Jenkins X on Google Cloud with GKE, and will use Google Secret Manger. It requires installation of Git and Hub command line tools.

The following are the values used for the creation of the Infra and Cluster repos:

Repo Source:		jx3-gitops-repositories
Infra Repo:		jx3-terraform-gke
Cluster Repo:		jx3-gke-gsm
Git Organization:	jx3rocksorg
JX3 Root Directory:	$JX3HOME                ## i.e. /home/user

Building infra repo: https://github.com/jx3rocksorg/jx3-terraform-gke

cd $JX3HOME
git clone https://github.com/jx3-gitops-repositories/jx3-terraform-gke.git jx3-terraform-gke
cd  jx3-terraform-gke
git remote set-url origin https://github.com/jx3rocksorg/jx3-terraform-gke.git
hub create -p jx3rocksorg/jx3-terraform-gke
git commit -a -m "chore: Initial"
git push -u origin main

Buidling cluster repo: https://github.com/jx3rocksorg/jx3-gke-gsm

cd $JX3HOME
git clone https://github.com/jx3-gitops-repositories/jx3-gke-gsm.git jx3-gke-gsm
cd jx3-gke-gsm
git remote set-url origin https://github.com/jx3rocksorg/jx3-gke-gsm.git
hub create -p jx3rocksorg/jx3-gke-gsm
git commit -a -m "chore: Initial"
git push -u origin main

Manage URL format

This example uses external DNS with subdomain and URLs will have a root of web.jx3rocks.com. To avoid including the default string "-jx" in the URLs, the example blanks out the spec:ingress:namespaceSubDomain value (".") in the jx-requirements.yml file. To eliminate having non-unique URLs after doing this type of update, an additional spec:environment:ingress:namespaceSubDomain line item is added ("-stg.") for the staging environment.

File:$JX3HOME/jx3-gke-gsm/jx-requirements.yml

  environments:
  - key: dev
  - key: staging
    ingress:
       namespaceSubDomain: -stg.
  - key: production
  ingress:
    domain: ""
    externalDNS: false
    namespaceSubDomain: .

Define Jenkins plugins and installers.

The default Jenkins server installation is a minimum configuration and generally requires additional installers and plugin components for the your specific Jenkins project to operate. The Jenkins components (i.e. plugins, installerss, etc.) are defined in 3 locations; Chart, System, and User configurations which get loaded in that order. During the build process settings get inherited and can be modified but depending on the end config how to make modifications may vary. This example takes into account the default configuration based upon the Chart and System settings. It will only require small additions to the configuration which is done at the User level. To make these config changes will require the modification of the User values.yaml file.

💡 Currently there is an issue where the Jenkins URL does not resolve to an IP address. To address this issue, it is imperative to have the ingressClassName: nginx included in the values.yaml file.

File: /tmp/values.yaml - Config Scripts

controller:
  ingress:             ## Required versionStream DNS/Ingress break fix
    ingressClassName: nginx
  JCasC:               ## Example of building an installer
    configScripts:
      mystuff: |
        tool:
          nodejs:
            installations:
            - name: "NodeJS"
              properties:
              - installSource:
                  installers:
                  - nodeJSInstaller:
                      id: "15.7.0"
                      npmPackagesRefreshHours: 72
  additionalPlugins:   ## Additional plugins for example
    - pipeline-stage-view:2.24
    - nodejs:1.5.1

Initialize Jenkins server configuration

jx gitops commands are used to create the Jenkins server configuration. More information can be found under Adding Jenkins Server into Jenkins X. The example uses the following command sequence:

cd $JX3HOME/jx3-gke-gsm                             ## cluster repo root
jx gitops jenkins add --name jx-jenkins             ## add Jenkins
jx gitops helmfile resolve --namespace jx-jenkins   ## resolve charts references (optional but helps later on in editing)
cp /tmp/values.yaml helmfiles/jx-jenkins            ## Update tool config

Push initial changes to cluster repo

All the customizations for the cluster repo are now propagated.

cd $JX3HOME/jx3-gke-gsm
git add .
git commit -a -m “chore: Jenkins example init”
git push	

Build the infrastucture with Terraform

The following TF_VAR environment variables are set prior to running Terraform commands:

TF_VAR_gcp_project=<google project>
TF_VAR_apex_domain_gcp_project=<google project>
TF_VAR_jx_bot_username=<git username>
TF_VAR_jx_bot_token=<git token>
TF_VAR_tls_email=mymail@jx3rocks.com

TF_VAR_apex_domain=jx3rocks.com
TF_VAR_subdomain=web

TF_VAR_gsm=true
TF_VAR_cluster_name=jx3web
TF_VAR_cluster_location=us-east1-b
TF_VAR_jx_git_url=https://github.com/jx3rocksorg/jx3-gke-gsm.src.git
TF_VAR_lets_encrypt_production=false
TF_VAR_force_destroy=true

Additional detail on Terraform settings can be found under Google Terraform Quickstart Template

Commands to build intrastructure:

cd $JX3HOME/jx3-terraform-gke
bin/create.sh                # Performs terraform init, plan and apply

Enable prod certificate

This example employs TLS and accesing Jenkins from a browser requires a valid production certificate. It should be noted that TLS is not supported with automated domains (i.e. nip.io). The process will begin by installing a Let’s Encrypt test certicate. After the initial build completes and is successful, it’s best to ensure that the test certificate is ready (READY=True) before enabing the production certificate. You can find more information regarding this subject under TLS and DNS.

## Set credentials
gcloud container clusters get-credentials jx3web --zone us-east1-b 
## Set jx namespace
jx ns jx
## Check certificate
kubectl get cert -n jx
NAME                     READY   SECRET                   AGE
tls-web-jx3rocks-com-s   True    tls-web-jx3rocks-com-s   39m

You can also use Octant to check certificate status.

Enabling the production certificate will require updates to both infra (jx-terraform-gke) and cluster (jx3-gke-gsm) repos. The commands to create the prod certificate are:

cd $JX3HOME/jx3-gke-gsm
git pull         ## Make sure local cluster repo is up to date
cd $JX3HOME/jx3-terraform-gke
export TF_VAR_lets_encrypt_production=true
terraform apply -auto-approve 
cd $JX3HOME/jx3-gke-gsm
git commit --allow-empty -m "Dummy commit"
git push
jx admin log     ## Make sure it completes before proceeding
git pull         ## Once successful refresh local cluster repo

The $JX3HOME/jx3-gke-gsm/jx-requirements.yml file’s TLS production setting should now be true.

  ingress:
    domain: web.jx3rocks.com
    externalDNS: true
    kind: ingress
    namespaceSubDomain: .
    tls:
      email: mymail@jx3rocks.com
      enabled: true
      production: true

Once again, the example checks the certificates status till the production certifacte is ready before proceeding.

kubectl get cert -n jx
NAME                     READY   SECRET                   AGE
tls-web-jx3rocks-com-p   True    tls-web-jx3rocks-com-p   39m
tls-web-jx3rocks-com-s   True    tls-web-jx3rocks-com-s   86m

You can also use Octant to check the production certificate. Octant Display

Sign on to Jenkins

After production certificates become ready, access to the Jenkins site: https://jenkins.web.jx3rocks.com is now available. The site has been configure with an admin user id and to obtain the admin user password the following commands are used:

kubectl get secret -n jx-jenkins jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode; echo

Deploy Jenkins Pipeline

To deploy a Jenkins Pipeline requires a Jenkinsfile file and can be created through either a jx project import or jx project quickstart command. You can find more detail in Importing project or creating quickstarts. This example creates a file Jenkinsfile in a new project directory (jx-example-node) and executes the jx project import command.

File: ~/jx-example-node/Jenkinsfile

pipeline {
    agent none
stages {
        stage ('First') {
            agent any
            steps {
                echo "First dummy stage"
            }
        }
        stage ('Input') {

            steps {
                script {
                    myStage = input message: 'What service type do you want to run now?', parameters: [choice(choices: 'Create\nUpdate', description: '', name: 'Stage')]
                }
                echo myStage
            }
        }
stage('Stage1') {
            when {
                expression { myStage == 'Create' }
            }
            steps {
                echo "Running Service: Create"
            }
        }
stage('Stage2') {
            when {
                expression { myStage == 'Update' }
            }
            steps {
                echo "Running Service:Update"
            }
        }
}

Import the project to deploy the pipeline

jx project import --git-token <token>  \
                  --git-username <git user> \
                  --org jx3rocksorg \
                  --jenkins jx-jenkins \
                  --dir ~/jx-example-node \
                  --name jx-example-node \
                  --batch-mode

Restarting Jenkins pod after pipeline deployment.

When the above pipeline is deployed it may be necessary to restart the Jenkins pod berfore you see it appear in the Jenkins dashboard.

kubectl delete pod jenkins-0 -n jx-jenkins            ## Restarts Jenkins pods

Adding Github Webhook (optional)

The current default configuration of JX3 for the import of Jenkins projects does not include the creation of a GitHub webhook for automated pipeline execution. You can use an GitHub API call to set up a webhook to provide a mechanism to kick off a pipeline when there’s an update to the project’s repo. Below is an example of the API call for this Jenkins configuration and import project example.

curl -X POST \
  https://api.github.com/repos/jx3rocksorg/jx-example-node/hooks \
  -H 'authorization: token <API_TOKEN>' \
  -H 'cache-control: no-cache' \
  -d '{ 
  "config": { 
    "url": "https://jenkins.web.jx3rocks.com/github-webhook/"
  }, 
  "events": [ 
    "issues", 
    "issue_comment", 
    "pull_request", 
    "pull_request_review", 
    "pull_request_review_comment", 
    "commit_comment", 
    "push", 
    "release", 
    "create", 
    "delete" 
  ] 
}'

Following this setup subsequent changes to the Jenkins project repo’s main, master, and PR-* branches should now automatically kick off a pipeline.

You could also set up a GitHub webhook for the Jenkins project using the GitHub UI. More details can be found at https://docs.github.com/en/developers/webhooks-and-events/creating-webhooks.

Removing the Jenkins server and repo

The Jenkins server definition and repos can be found in the$JX3HOME/.jx/gitops/source-config.yamlfile. This example’s file is the following:

apiVersion: gitops.jenkins-x.io/v1alpha1
kind: SourceConfig
metadata:
  creationTimestamp: null
spec:
  jenkinsServers:
  - groups:
    - owner: jx3rocksorg
      provider: https://github.com
      providerKind: github
      repositories:
      - name: jx-example-node
    server: jx-jenkins
  slack:
    channel: '#jenkins-x-pipelines'
    kind: failureOrNextSuccess
    pipeline: release

Commands to remove Jenkins resources and repo:

rm $JX3HOME/jx3-gke-gsm/.jx/gitops/source-config.yaml
rm -rf $JX3HOME/jx3-gke-gsm/helmfiles/jx-jenkins
sed -i '/jx-jenkins/d' $JX3HOME/jx3-gke-gsm/helmfile.yaml
hub delete -y jx3rocksorg/jx-example-node
git commit -a -m "chore: remove jenkins"
git push

Troubleshooting

Sometimes Jenkins requires attention due to invalid plugin versions and/or syntax. The following commands are helpful in debugging:

kubectl logs -f jenkins-0 -n jx-jenkins -c init       ## View init container
kubectl logs -f jenkins-0 -n jx-jenkins -c jenkins    ## View jenkins container
kubectl delete pod jenkins-0 -n jx-jenkins            ## Restarts Jenkins pods

Last modified May 31, 2022: Update example.md (c9e26d0684)