Create EKS cluster

How to create an EKS cluster?

Prerequisites

The EKS Terraform module for Jenkins X allows you to create an EKS cluster for installation of Jenkins X. You need the following binaries locally installed and configured on your PATH:

  • terraform (~> 0.12.17)
  • kubectl (>=1.10)
  • aws-iam-authenticator
  • wget

Cluster provisioning

A default Jenkins X ready cluster can be provisioned by creating a main.tf file in an empty directory with the following content:

module "eks-jx" {
  source  = "jenkins-x/eks-jx/aws"
}

output "jx_requirements" {
   value = module.eks-jx.jx_requirements
}

output "vault_user_id" {
  value       = module.eks-jx.vault_user_id
  description = "The Vault IAM user id"
}

output "vault_user_secret" {
  value       = module.eks-jx.vault_user_secret
  description = "The Vault IAM user secret"
}

Due to the Vault issue 7450, this Terraform module needs for now to create a new IAM user for installing Vault. It also creates an IAM access key whose id and secret are defined in the output above. You need the id and secret for running jx boot.

The jx_requirements output is a helper for creating the initial input for jx boot.

If you do not want Terraform to create a new IAM user or you do not have permissions to create one, you need to provide the name of an existing IAM user.

module "eks-jx" {
    source  = "jenkins-x/eks-jx/aws"
}

output "jx_requirements" {
    value = module.eks-jx.jx_requirements
}

vault_user="<your_vault_iam_username>"
}

The IAM user does not need any permissions attached to it. For more information, refer to Configuring Vault for EKS in the Jenkins X documentation.

Once you have your initial configuration, you can apply it by running:

terraform init
terraform apply

This creates an EKS cluster with all possible configuration options defaulted.

You then need to export the environment variables VAULT_AWS_ACCESS_KEY_ID and VAULT_AWS_SECRET_ACCESS_KEY.

export VAULT_AWS_ACCESS_KEY_ID=$(terraform output vault_user_id)
export VAULT_AWS_SECRET_ACCESS_KEY=$(terraform output vault_user_secret)

If you specified vault_user you need to provide the access key id and secret for the specified user.

The following sections provide a full list of configuration in- and output variables.

Inputs

Name Description Type Default Required
apex_domain The main domain to either use directly or to configure a subdomain from string "" no
cluster_in_private_subnet Flag to enable installation of cluster on private subnets bool false no
cluster_name Variable to provide your desired name for the cluster. The script will create a random name if this is empty string "" no
cluster_version Kubernetes version to use for the EKS cluster. string "1.15" no
create_and_configure_subdomain Flag to create an NS record set for the subdomain in the apex domain’s Hosted Zone bool false no
desired_node_count The number of worker nodes to use for the cluster number 3 no
enable_external_dns Flag to enable or disable External DNS in the final jx-requirements.yml file bool false no
enable_key_rotation Flag to enable kms key rotation bool true no
enable_logs_storage Flag to enable or disable long term storage for logs bool true no
enable_nat_gateway Should be true if you want to provision NAT Gateways for each of your private networks bool false no
enable_node_group Flag to enable node group bool false no
enable_reports_storage Flag to enable or disable long term storage for reports bool true no
enable_repository_storage Flag to enable or disable the repository bucket storage bool true no
enable_spot_instances Flag to enable spot instances bool false no
enable_tls Flag to enable TLS in the final jx-requirements.yml file bool false no
enable_worker_group Flag to enable worker group bool true no
force_destroy Flag to determine whether storage buckets get forcefully destroyed. If set to false, empty the bucket first in the aws s3 console, else terraform destroy will fail with BucketNotEmpty error bool false no
map_accounts Additional AWS account numbers to add to the aws-auth configmap. list(string) [] no
map_roles Additional IAM roles to add to the aws-auth configmap.
list(object({
rolearn = string
username = string
groups = list(string)
}))
[] no
map_users Additional IAM users to add to the aws-auth configmap.
list(object({
userarn = string
username = string
groups = list(string)
}))
[] no
max_node_count The maximum number of worker nodes to use for the cluster number 5 no
min_node_count The minimum number of worker nodes to use for the cluster number 3 no
node_group_ami ami type for the node group worker intances string "AL2_x86_64" no
node_group_disk_size node group worker disk size string "50" no
node_machine_type The instance type to use for the cluster’s worker nodes string "m5.large" no
private_subnets The private subnet CIDR block to use in the created VPC list(string)
[
“10.0.4.0/24”,
“10.0.5.0/24”,
“10.0.6.0/24"
]
no
production_letsencrypt Flag to use the production environment of letsencrypt in the jx-requirements.yml file bool false no
public_subnets The public subnet CIDR block to use in the created VPC list(string)
[
“10.0.1.0/24”,
“10.0.2.0/24”,
“10.0.3.0/24"
]
no
region The region to create the resources into string "us-east-1" no
single_nat_gateway Should be true if you want to provision a single shared NAT Gateway across all of your private networks bool false no
spot_price The spot price ceiling for spot instances string "0.1" no
subdomain The subdomain to be added to the apex domain. If subdomain is set, it will be appended to the apex domain in jx-requirements-eks.yml file string "" no
tls_email The email to register the LetsEncrypt certificate with. Added to the jx-requirements.yml file string "" no
vault_url URL to an external Vault instance in case Jenkins X does not create its own system Vault string "" no
vault_user The AWS IAM Username whose credentials will be used to authenticate the Vault pods against AWS string "" no
vpc_cidr_block The vpc CIDR block string "10.0.0.0/16" no
vpc_name The name of the VPC to be created for the cluster string "tf-vpc-eks" no

Outputs

Name Description
cert_manager_iam_role The IAM Role that the Cert Manager pod will assume to authenticate
cluster_name The name of the created cluster
cm_cainjector_iam_role The IAM Role that the CM CA Injector pod will assume to authenticate
controllerbuild_iam_role The IAM Role that the ControllerBuild pod will assume to authenticate
external_dns_iam_role The IAM Role that the External DNS pod will assume to authenticate
jx_requirements The jx-requirements rendered output
jxui_iam_role The IAM Role that the Jenkins X UI pod will assume to authenticate
lts_logs_bucket The bucket where logs from builds will be stored
lts_reports_bucket The bucket where test reports will be stored
lts_repository_bucket The bucket that will serve as artifacts repository
tekton_bot_iam_role The IAM Role that the build pods will assume to authenticate
vault_dynamodb_table The Vault DynamoDB table
vault_kms_unseal The Vault KMS Key for encryption
vault_unseal_bucket The Vault storage bucket
vault_user_id The Vault IAM user id
vault_user_secret The Vault IAM user secret

Long Term Storage

You can choose to create S3 buckets for long term storage with enable_logs_storage, enable_reports_storage and enable_repository_storage.

During terraform apply the enabledS3 buckets are created, and the jx_requirements output will contain the following section:

storage:
  logs:
    enabled: ${enable_logs_storage}
    url: s3://${logs_storage_bucket}
  reports:
    enabled: ${enable_reports_storage}
    url: s3://${reports_storage_bucket}
  repository:
    enabled: ${enable_repository_storage}
    url: s3://${repository_storage_bucket}

If you just want to experiment with Jenkins X, you can set the variable force_destroy to true. This allows you to remove all generated buckets when running terraform destroy.

Vault

Vault is used by Jenkins X for managing secrets. Part of this module’s responsibilities is the creation of all resources required to run the Vault Operator. These resources are An S3 Bucket, a DynamoDB Table and a KMS Key.

You can also configure an existing Vault instance for the use with Jenkins X. In this case provide the Vault URL via the vault_url via the input variable and follow the Jenkins X documentation around the instllation of an external Vault instance.

ExternalDNS

You can enable ExternalDNS with the enable_external_dns variable. This modifies the generated jx-requirements.yml file to enable External DNS when running jx boot.

If enable_external_dns is true, additional configuration is required.

If you want to use a domain with an already existing Route 53 Hosted Zone, you can provide it through the apex_domain variable:

This domain will be configured in the jx_requirements output in the following section:

ingress:
  domain: ${domain}
  ignoreLoadBalancer: true
  externalDNS: ${enable_external_dns}

If you want to use a subdomain and have this module create and configure a new Hosted Zone with DNS delegation, you can provide the following variables:

subdomain: This subdomain is added to the apex domain and configured in the resulting jx-requirements.yml file.

create_and_configure_subdomain: This flag instructs the script to create a new Route53 Hosted Zone for your subdomain and configure DNS delegation with the apex domain.

By providing these variables, the script creates a new Route 53 HostedZone that looks like <subdomain>.<apex_domain>, then it delegates the resolving of DNS to the apex domain. This is done by creating a NS RecordSet in the apex domain’s Hosted Zone with the subdomain’s HostedZone nameservers.

This ensures that the newly created HostedZone for the subdomain is instantly resolvable instead of having to wait for DNS propagation.

cert-manager

You can enable cert-manager to use TLS for your cluster through LetsEncrypt with the enable_tls variable.

LetsEncrypt has two environments, staging and production.

If you use staging, you will receive self-signed certificates, but you are not rate-limited, if you use the production environment, you receive certificates signed by LetsEncrypt, but you can be rate limited.

You can choose to use the production environment with the production_letsencrypt variable:

You need to provide a valid email to register your domain in LetsEncrypt with tls_email.

Velero Backups

This module can set up the resources required for running backups with Velero on your cluster by setting the flag enable_backup to true.

Enabling backups on pre-existing clusters

If your cluster is pre-existing and already contains a namespace named velero, then enabling backups will initially fail with an error that you are trying to create a namespace which already exists.

Error: namespaces "velero" already exists

If you get this error, consider it a warning - you may then adjust accordingly by importing that namespace to be managed by Terraform, deleting the previously existing ns if it wasn’t actually in use, or setting enable_backup back to false to continue managing Velero in the previous manner.

The recommended way is to import the namespace and then run another Terraform plan and apply:

terraform import module.eks-jx.module.backup.kubernetes_namespace.velero_namespace velero

Running jx boot

A terraform output (jx_requirements) is available after applying this Terraform module.

terraform output jx_requirements

This jx_requirements output can be used as input to Jenkins X Boot which is responsible for installing all the required Jenkins X components into the cluster created by this module.

Execute:

terraform output jx_requirements > <some_empty_dir>/jx-requirements.yml
cd <some_empty_dir>
jx boot --requirements jx-requirements.yml

You are prompted for any further required configuration. The number of prompts depends on how much you have pre-configured via your Terraform variables.

More information about the boot process can be found in the Run Boot section.

Production cluster considerations

The configuration, as seen in Cluster provisioning, is not suited for creating and maintaining a production Jenkins X cluster. The following is a list of considerations for a production use case.

  • Specify the version attribute of the module, for example:
module "eks-jx" {
  source  = "jenkins-x/eks-jx/aws"
  version = "1.0.0"
  # insert your configuration
}

output "jx_requirements" {
  value = module.eks-jx.jx_requirements
}  

Specifying the version ensures that you are using a fixed version and that version upgrades cannot occur unintended.

  • Keep the Terraform configuration under version control by creating a dedicated repository for your cluster configuration or by adding it to an already existing infrastructure repository.

  • Setup a Terraform backend to securely store and share the state of your cluster. For more information refer to Configuring a Terraform backend.

Configuring a Terraform backend

A “backend” in Terraform determines how state is loaded and how an operation such as apply is executed. By default, Terraform uses the local backend, which keeps the state of the created resources on the local file system. This is problematic since sensitive information will be stored on disk and it is not possible to share state across a team. When working with AWS a good choice for your Terraform backend is the s3 backend which stores the Terraform state in an AWS S3 bucket. The examples directory of the terraform-aws-eks-jx repository contains configuration examples for using the s3 backed.

To use the s3 backend, you will need to create the bucket upfront. You need the S3 bucket as well as a Dynamo table for state locks. You can use terraform-aws-tfstate-backend to create these required resources.

Using spot instances

You can save up to 90% of cost when you use Spot Instances. You just need to make sure your applications are resilient. You can set the ceiling spot_price of what you want to pay then set enable_spot_instances to true.

EKS node groups

This module provisions self-managed worker nodes by default. If you want AWS to manage the provisioning and lifecycle of worker nodes for EKS, you can opt for managed node groups. They have the added benefit of running the latest Amazon EKS-optimized AMIs and gracefully drain nodes before termination to ensure that your applications stay available. In order to provision EKS node groups create a main.tf with the following content:

module "eks-jx" {
  source  = "jenkins-x/eks-jx/aws"
  enable_node_group   = true
  enable_worker_group = false
}

output "jx_requirements" {
  value = module.eks-jx.jx_requirements
}

output "vault_user_id" {
  value       = module.eks-jx.vault_user_id
  description = "The Vault IAM user id"
}

output "vault_user_secret" {
  value       = module.eks-jx.vault_user_secret
  description = "The Vault IAM user secret"
}

Last modified September 21, 2020: release 0.0.1895 (3145738)