K3s

Setup Jenkins X on a K3s cluster running locally

NOTE

Ensure you are logged into GitHub else you will get a 404 error when clicking the links below

This guide will walk you though how to setup Jenkins X on your laptop using k3s

If you are on Mac OS, you can follow this guide to set up k3s. You do not need to install kalm for the rest of the tutorial.

Prerequisites

K3s

Make sure you have created a cluster using k3s.

If you dont have an existing k3s cluster, you can install one by running:

# We don't support Kubernetes 1.22+ yet
curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=v1.21 sh -s - --write-kubeconfig-mode 644
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/k3s-config
# Set it also in the bashrc or zshrc file, or you can flatten both of these configs into a single file
export KUBECONFIG=~/.kube/config:~/.kube/k3s-config

To verify that k3s has been installed successfully, and configured run:

kubectl get nodes
  • You will need to open multiple terminals later, so setting these env variables in the bashrc or zshrc might help you
nano ~/.bashrc
#go to the end of the file and paste the export command
export KUBECONFIG=~/.kube/config:~/.kube/k3s-config

Optional

  • If the above method didn’t work, copy the configurations to the ~/.kube/config instead (if you don’t have any other clusters, that should be fine)
sudo rm ~/.kube/k3s-config  #to make k3s uses ~/.kube/config
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
  • If still got permission denied
sudo chmod 777 ~/.kube/config #warning: this maybe vulnerable to multiple users 

This value of the node will be used later during installation and configuring of Jenkins X.

Check k3s install guide for more installation options but keep in mind we don’t support Kubernetes 1.22+ yet

Docker

You need to install docker and manage it as a non root user

Vault

Install vault cli. Refer to the vault docs on how to install vault for your platform.

Make sure you have vault running in a docker container with kubernetes auth enabled.

docker run --name jx-k3s-vault -d --cap-add=IPC_LOCK -e 'VAULT_DEV_ROOT_TOKEN_ID=myroot' -e 'VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200' --net host vault:latest

To verify if vault started properly use docker logs jx-k3s-vault.

Next enable kubernetes auth in vault.

export VAULT_ADDR='http://0.0.0.0:8200'
# you may want to set this at the end of the ~/.bashrc file or ~/.zshrc either to be accesible for all terminals you open like the way we did above
vault auth enable kubernetes

Note: If you get the error Error enabling kubernetes auth: Post "https://127.0.0.1:8200/v1/sys/auth/kubernetes": http: server gave HTTP response to HTTPS client, try the command vault login myroot.

Github

Jenkins-X

  • Make sure you have installed jx 3.x binary and put it on your $PATH as the jx admin operator will be used

Jenkins X v3 installation

  • Generate a cluster git repository from the jx3-k3s-vault template, by clicking here
  • Edit the value of the vault url in the jx-requirements.yaml file. Replace with "http://<replace with k3s node name>:8200"
  • Commit and push your changes:
git add .
git commit -m "fix: set vault url"
git push origin main
  • Set the GIT_USERNAME (bot username) and GIT_TOKEN (bot personal access token) env variable and run:
jx admin operator --username $GIT_USERNAME --token $GIT_TOKEN --url <url of the cluster git repo> --set "jxBootJobEnvVarSecrets.EXTERNAL_VAULT=\"true\"" --set "jxBootJobEnvVarSecrets.VAULT_ADDR=http://<replace with k3s node name>:8200"

Note: The first job will fail as it cannot authenticate against vault. The errors will be of the form error: failed to populate secrets: failed to create a secret manager for ExternalSecret. Once the secret-infra namespace has been created, we can configure vault. If you get an error connecting to the cluster, try running kubectl config view --raw >~/.kube/config as well as checking the permissions/owner of ~/.kube/config

Vault configuration

Install jq before running these commands.

Remember to run the following commands in a terminal where you have set the value of VAULT_ADDR

  • Create a vault config
export VAULT_ADDR='http://0.0.0.0:8200'
export VAULT_HELM_SECRET_NAME=$(kubectl -n secret-infra get secrets --output=json | jq -r '.items[].metadata | select(.name|startswith("kubernetes-external-secrets-token-")).name')
export TOKEN_REVIEW_JWT=$(kubectl -n secret-infra get secret $VAULT_HELM_SECRET_NAME --output='go-template={{ .data.token }}' | base64 --decode)
export KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)
export KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')

# you may want to set this at the end of the ~/.bashrc file or ~/.zshrc either to be accesible for all terminals you open like the way we did above

vault write auth/kubernetes/config \
        token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
        kubernetes_host="$KUBE_HOST" \
        kubernetes_ca_cert="$KUBE_CA_CERT" \
        disable_iss_validation=true
  • Create a vault role:
vault write /auth/kubernetes/role/jx-vault bound_service_account_names='*' bound_service_account_namespaces=secret-infra token_policies=jx-policy token_no_default_policy=true disable_iss_validation=true
  • Create a policy attached to vault role:
vault policy write jx-policy - <<EOF
path "secret/*" {
  capabilities = ["sudo", "create", "read", "update", "delete", "list"]
}
EOF

Once vault is configured, pull the changes commited to the cluster git repository by the bootjob, and push a dummy job

git pull
# Make a dummy change (change Readme file for example)
git add .
git commit -m "chore: dummy commit 1"
git push origin main

tail the logs of jx-git-operator pod in the jx-git-operator namespace.

not creating a Job in namespace jx-git-operator for repo jx-boot sha XXXXXX yet as there is an active job jx-boot-XXXXXXX

Kill that job.

kubectl delete job jx-boot-97dbb72f-4e4e-4b0e-8eb2-8908997b19f7 -n jx-git-operator

Once it’s killed a new boot job will be triggered. This job will create the secrets in vault which will be used by external secrets to create kubernetes secrets.

  • To verify the job succeeded, run jx admin log
  • To verfiy the secrets were created, run kubectl get es -A and jx secret verify
  • If this didn’t work try and repeat the steps but commit your dummy changes through github repository directly other than the git push origin main command

Set up ingress and webhook

  • Get the external IP of the traefik service (loadbalancer)
kubectl get svc -A | grep LoadBalancer
kube-system   traefik          LoadBalancer   <cluster-ip>    <external-ip>    80:31123/TCP,443:31783/TCP   40m
  • Edit the jx-requirements.yaml file by editing the ingress domain:
# There may be some changes committed by the jx boot job
git pull
jx gitops requirements edit --domain <external-ip>.nip.io
  • Next, download and install ngrok. Run this in a new terminal window/tab:
ngrok http 8080
  • Once this tunnel is open, paste the ngrok url (without http and https) in the hook field in the helmfiles/jx/jxboot-helmfile-resources-values.yaml file in the cluster git repository.
  • commit and push the changes.
git add .
git commit -m "chore: new ngrok ip"
git push origin main
  • In another terminal run the following command to enable webhooks via ngrok
jx ns jx
kubectl port-forward svc/hook 8080:80
  • Once the bootjob has succeeded, you should see:
HTTP Requests
-------------

POST /hook                     200 OK

Next steps


Last modified April 26, 2022: Update _index.md (b03229fed7)