Vault

Using Vault for your Secret storage

Installing Vault

Internal vault (vault created as part of Jenkins X installation)

If you are using Terraform with one of the Cloud Providers then your Vault will be installed automatically via Terraform.

Otherwise please see the On-Premises Vault Install Guide

whichever apporoach take you should have:

You can wait for the vault-0 pod in namespace jx-vault to be ready via jx secret vault wait command:

jx secret vault wait

External vault

Configure external vault to support kubernetes auth method. This step needs to be done before any Jenkins X related changes are made.

# Set VAULT_ADDR
export VAULT_ADDR='<vault url>'
vault auth enable kubernetes

We only support kubernetes auth method at kubernetes/ for now.

Add vault_url and boot_secrets to the main.tf file (replace https://external-vault.com with the actual vault url):

...
module "eks-jx" {
  source               = "jenkins-x/eks-jx/aws"
  ...
  vault_url            = "https://external-vault.com:8200"
  boot_secrets = [
    {
      name  = "jxBootJobEnvVarSecrets.EXTERNAL_VAULT"
      value = "true"
      type  = "string"
    },
    {
      name  = "jxBootJobEnvVarSecrets.VAULT_ADDR"
      value = "https://enter-your-vault-url:8200"
      type  = "string"
    }
  ]
}
....

This should create a secret jx-boot-job-env-vars in the jx-git-operator namespace, verify that this secret has EXTERNAL_VAULT set to true, and VAULT_ADDR set correctly.

This will prevent the terraform module from creating any vault resources in the kubernetes cluster and the cloud (AWS/GCP/Azure) account.

Now your vault can be used with Jenkins X.

Local jx-secret with External vault

jx-secret uses the JWT token type to authenticate with vault. The JWT token is only valid when used from inside the Kubernetes cluster and because of this we have to proxy the connection through a host inside the cluster.

Launch a pod in the jx cluster to act as the proxy host:

kubectl -n jx run vault-proxy --image=hpello/tcp-proxy --port=8200 -- vault.example.com 8200

Forward local port 8200 to the vault-proxy pod:

kubectl port-forward pods/vault-proxy 8200:8200&
jobid=`echo $!`

Setup your local environment:

unset VAULT_TOKEN
export VAULT_ADDR=https://localhost:8200/
export VAULT_SKIP_VERIFY=True
export JX_VAULT_MOUNT_POINT=kubernetes
export JX_VAULT_ROLE=jx-vault
export EXTERNAL_VAULT=true

Run jx-secret as you normally would:

jx secret edit -i

Cleanup:

kill $foo #stop the port-forward
kubectl -n jx delete pod vault-proxy #delete the pod

Configuration

To indicate that Vault is being used as the storage engine for your Secrets you need to configure vault via secretStorage: vault in your jx-requirements.yml. Note that this is usually done automatically for Cloud providers and Terraform:

cluster:
  provider: gke
environments:
  - key: dev
  - key: staging
  - key: production
kaniko: true
secretStorage: vault
webhook: lighthouse

Using Vault

To be able to import, export or edit secrets from your laptop you need to make sure you are running the jx secret vault portforward command to port forward the 8200 port on your laptop to the vault service:

jx secret vault portforward

This will allow the jx 3.x binary to access the Vault REST API.

You can now follow the instructions to edit secrets or import secrets.

Using the vault web UI

Once you are running the jx secret vault portforward command described above you can access the vault web UI at https://localhost:8200

Using the vault CLI directly

Someday we might have a nice jx secret vault shell command to automate all of this but until then…

Download the vault CLI binary and add it to your $PATH.

You can now setup a shell to access vault as follows:

export VAULT_TOKEN=$(kubectl get secrets vault-unseal-keys  -n jx-vault -o jsonpath={.data.vault-root} | base64 --decode)

# Tell the CLI that the Vault Cert is signed by a custom CA
kubectl get secret vault-tls -n jx-vault -o jsonpath="{.data.ca\.crt}" | base64 --decode > $PWD/vault-ca.crt
export VAULT_CACERT=$PWD/vault-ca.crt

# Tell the CLI where Vault is listening (the certificate has 127.0.0.1 as well as alternate names)
export VAULT_ADDR=https://127.0.0.1:8200

# Now we can use the vault CLI to list/read/write secrets...

# List all the current secrets
vault kv list secret

# Lets store a secret
vault kv put secret/mything foo=bar whatnot=cheese

Last modified December 6, 2021: refactor: add more information about boot secrets (29a072ae89)