Skip to content

Commit 6b7355f

Browse files
delterrdelterr
authored
Concourse pipeline (#8)
* init concourse * put quotes around variable declaration * remove concourse from branch check * use correct repo name * use correct directory * remove stray cd * refactor to use API for tagging * github_repo_archive_secrets -> repo_archive secrets * bump provider version to 6.2.0 * remove concourse conditional * update README * lint bash scripts * lint some more * remove shellcheck space * run prettier * Revert "run prettier" This reverts commit 873b10e. * add whitespace * remove shebang line * add sh-shebang and lint yaml * disable SC3040 * fix yaml file and add in SC2148 * fix yaml * remove container image from terraform_infra.sh
1 parent a53234c commit 6b7355f

7 files changed

Lines changed: 247 additions & 1 deletion

File tree

‎README.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -325,6 +325,48 @@ terraform destroy -var-file=env/dev/dev.tfvars
325325

326326
**Please Note:** Make sure to use the correct `.tfbackend` and `.tfvars` files for your environment.
327327

328+
### Deployments with Concourse
329+
330+
#### Allowlisting your IP
331+
332+
To setup the deployment pipeline with concourse, you must first allowlist your IP address on the Concourse
333+
server. IP addresses are flushed everyday at 00:00 so this must be done at the beginning of every working day
334+
whenever the deployment pipeline needs to be used. Follow the instructions on the Confluence page (SDP Homepage > SDP Concourse > Concourse Login) to
335+
login. All our pipelines run on sdp-pipeline-prod, whereas sdp-pipeline-dev is the account used for
336+
changes to Concourse instance itself. Make sure to export all necessary environment variables from sdp-pipeline-prod (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN).
337+
338+
#### Setting up a pipeline
339+
340+
When setting up our pipelines, we use ecs-infra-user on sdp-dev to be able to interact with our infrastructure on AWS. The credentials for this are stored on
341+
AWS Secrets Manager so you do not need to set up anything yourself.
342+
343+
To set the pipeline, run the following script:
344+
345+
```bash
346+
chmod u+x ./concourse/scripts/set_pipeline.sh
347+
./concourse/scripts/set_pipeline.sh github-repo-archive-script
348+
```
349+
350+
Note that you only have to run chmod the first time running the script in order to give permissions.
351+
This script will set the branch and pipeline name to whatever branch you are currently on. It will also set the image tag on ECR to the current commit hash at the time of setting the pipeline.
352+
353+
The pipeline name itself will usually follow a pattern as follows: `<repo-name>-<branch-name>`
354+
If you wish to set a pipeline for another branch without checking out, you can run the following:
355+
356+
```bash
357+
./concourse/scripts/set_pipeline.sh github-repo-archive-script <branch_name>
358+
```
359+
360+
If the branch you are deploying is "main" or "master", it will trigger a deployment to the sdp-prod environment. To set the ECR image tag, you must draft a Github release pointing to the latest release of the main/master branch that has a tag in the form of vX.Y.Z. Drafting up a release will automatically deploy the latest version of the main/master branch with the associated release tag, but you can also manually trigger a build through the Concourse UI or the terminal prompt.
361+
362+
#### Triggering a pipeline
363+
364+
Once the pipeline has been set, you can manually trigger a build on the Concourse UI, or run the following command:
365+
366+
```bash
367+
fly -t aws-sdp trigger-job -j github-repo-archive-script-<branch-name>/build-and-push
368+
```
369+
328370
## Linting and Testing
329371

330372
### GitHub Actions

‎concourse/ci.yml‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
---
2+
resources:
3+
- name: resource-repo
4+
type: git
5+
icon: github
6+
source:
7+
uri: https://github.com/ONS-Innovation/github-repository-archive-script
8+
username: ((github_access_token))
9+
password: x-oauth-basic
10+
branch: ((branch))
11+
12+
jobs:
13+
- name: build-and-push
14+
public: true
15+
plan:
16+
- get: resource-repo
17+
timeout: 5m
18+
- task: build-image
19+
privileged: true
20+
config:
21+
platform: linux
22+
image_resource:
23+
type: docker-image
24+
source:
25+
repository: hashicorp/terraform
26+
inputs:
27+
- name: resource-repo
28+
params:
29+
aws_account_id: ((aws_account_sdp_((env))))
30+
aws_role_arn: arn:aws:iam::((aws_account_sdp_((env)))):role/sdp-concourse-((env))
31+
secrets: ((sdp_((env))_github_repo_archive_secrets))
32+
run: # binary used to build the image
33+
path: sh
34+
args:
35+
- -cx
36+
- |
37+
apk add --no-cache aws-cli podman jq iptables curl
38+
39+
if [[ "((env))" == "prod" ]]; then
40+
tag=$(curl "https://api.github.com/repos/ONS-Innovation/github-repository-archive-script/releases" | jq -r '.[0].tag_name')
41+
export tag
42+
else
43+
export tag=((tag))
44+
fi
45+
git rev-parse --abbrev-ref HEAD
46+
chmod u+x ./resource-repo/concourse/scripts/assume_role.sh
47+
chmod u+x ./resource-repo/concourse/scripts/build_image.sh
48+
source ./resource-repo/concourse/scripts/assume_role.sh
49+
./resource-repo/concourse/scripts/build_image.sh
50+
timeout: 10m
51+
- task: terraform
52+
privileged: true
53+
config:
54+
platform: linux
55+
image_resource:
56+
type: docker-image
57+
source: { repository: hashicorp/terraform }
58+
inputs:
59+
- name: resource-repo
60+
params:
61+
secrets: ((sdp_((env))_github_repo_archive_secrets))
62+
github_access_token: ((github_access_token))
63+
run:
64+
path: sh
65+
args:
66+
- -cx
67+
- |
68+
apk add --no-cache jq curl
69+
70+
if [[ "((env))" == "prod" ]]; then
71+
tag=$(curl "https://api.github.com/repos/ONS-Innovation/github-repository-archive-script/releases" | jq -r '.[0].tag_name')
72+
export tag
73+
else
74+
export tag=((tag))
75+
fi
76+
chmod u+x ./resource-repo/concourse/scripts/terraform_infra.sh
77+
export env=((env))
78+
./resource-repo/concourse/scripts/terraform_infra.sh
79+
timeout: 30m

‎concourse/scripts/assume_role.sh‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# shellcheck disable=SC3040,SC2154,SC2148
2+
3+
set -euo pipefail
4+
5+
# shellcheck disable=SC3040,SC2154
6+
aws sts assume-role --output text \
7+
--role-arn "${aws_role_arn}" \
8+
--role-session-name concourse-pipeline-run \
9+
--query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
10+
| awk -F '\t' '{print $1 > ("AccessKeyId")}{print $2 > ("SecretAccessKey")}{print $3 > ("SessionToken")}'
11+
12+
# shellcheck disable=SC3040,SC2154
13+
AWS_ACCESS_KEY_ID="$(cat AccessKeyId)"
14+
AWS_SECRET_ACCESS_KEY="$(cat SecretAccessKey)"
15+
AWS_SESSION_TOKEN="$(cat SessionToken)"
16+
17+
export AWS_ACCESS_KEY_ID
18+
export AWS_SECRET_ACCESS_KEY
19+
export AWS_SESSION_TOKEN

‎concourse/scripts/build_image.sh‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# shellcheck disable=SC3040,SC2154,SC2148
2+
set -euo pipefail
3+
4+
export STORAGE_DRIVER=vfs
5+
export PODMAN_SYSTEMD_UNIT=concourse-task
6+
7+
# shellcheck disable=SC2154
8+
container_image=$(echo "$secrets" | jq -r .container_image)
9+
10+
# shellcheck disable=SC2154
11+
aws ecr get-login-password --region eu-west-2 | podman --storage-driver=vfs login --username AWS --password-stdin "${aws_account_id}".dkr.ecr.eu-west-2.amazonaws.com
12+
13+
# shellcheck disable=SC2154
14+
podman build -t "${container_image}":"${tag}" resource-repo
15+
16+
# shellcheck disable=SC2154
17+
podman tag "${container_image}":"${tag}" "${aws_account_id}".dkr.ecr.eu-west-2.amazonaws.com/"${container_image}":"${tag}"
18+
19+
# shellcheck disable=SC2154
20+
podman push "${aws_account_id}".dkr.ecr.eu-west-2.amazonaws.com/"${container_image}":"${tag}"

‎concourse/scripts/set_pipeline.sh‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# shellcheck disable=SC3040,SC2154,SC2148
2+
3+
repo_name=${1}
4+
5+
if [[ $# -gt 1 ]]; then
6+
branch=${2}
7+
git rev-parse --verify "${branch}"
8+
# shellcheck disable=SC2181
9+
if [[ $? -ne 0 ]]; then
10+
echo "Branch \"${branch}\" does not exist"
11+
exit 1
12+
fi
13+
else
14+
branch=$(git rev-parse --abbrev-ref HEAD)
15+
fi
16+
17+
if [[ ${branch} == "main" || ${branch} == "master" ]]; then
18+
env="prod"
19+
else
20+
env="dev"
21+
fi
22+
23+
if [[ ${env} == "dev" ]]; then
24+
tag=$(git rev-parse HEAD)
25+
else
26+
tag=$(git tag | tail -n 1)
27+
fi
28+
29+
fly -t aws-sdp set-pipeline -c concourse/ci.yml -p "${repo_name}"-"${branch}" -v branch="${branch}" -v tag="${tag}" -v env="${env}"

‎concourse/scripts/terraform_infra.sh‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# shellcheck disable=SC3040,SC2154,SC2148
2+
3+
set -euo pipefail
4+
5+
# shellcheck disable=SC2154
6+
aws_account_id=$(echo "$secrets" | jq -r .aws_account_id)
7+
aws_access_key_id=$(echo "$secrets" | jq -r .aws_access_key_id)
8+
9+
aws_secret_access_key=$(echo "$secrets" | jq -r .aws_secret_access_key)
10+
aws_secret_name=$(echo "$secrets" | jq -r .aws_secret_name)
11+
12+
env_name=$(echo "$secrets" | jq -r .env_name)
13+
lambda_name=$(echo "$secrets" | jq -r .lambda_name)
14+
15+
github_app_client_id=$(echo "$secrets" | jq -r .github_app_client_id)
16+
lambda_arch=$(echo "$secrets" | jq -r .lambda_arch)
17+
18+
github_org=$(echo "$secrets" | jq -r .github_org)
19+
20+
schedule=$(echo "$secrets" | jq -r .schedule)
21+
lambda_timeout=$(echo "$secrets" | jq -r .lambda_timeout)
22+
23+
lambda_memory=$(echo "$secrets" | jq -r .lambda_memory)
24+
25+
AWS_ACCESS_KEY_ID=$aws_access_key_id
26+
AWS_SECRET_ACCESS_KEY=$aws_secret_access_key
27+
28+
export AWS_ACCESS_KEY_ID
29+
export AWS_SECRET_ACCESS_KEY
30+
31+
# shellcheck disable=SC2086
32+
git config --global url."https://x-access-token:$github_access_token@github.com/".insteadOf "https://github.com/"
33+
34+
if [[ ${env} != "prod" ]]; then
35+
env="dev"
36+
fi
37+
38+
cd resource-repo/terraform/service
39+
40+
terraform init -backend-config=env/"${env}"/backend-"${env}".tfbackend -reconfigure
41+
42+
# shellcheck disable=SC2154
43+
terraform apply \
44+
-var "aws_account_id=$aws_account_id" \
45+
-var "aws_access_key_id=$aws_access_key_id" \
46+
-var "aws_secret_access_key=$aws_secret_access_key" \
47+
-var "aws_secret_name=$aws_secret_name" \
48+
-var "env_name=$env_name" \
49+
-var "lambda_version=${tag}" \
50+
-var "lambda_name=$lambda_name" \
51+
-var "lambda_arch=$lambda_arch" \
52+
-var "lambda_timeout=$lambda_timeout" \
53+
-var "github_app_client_id=$github_app_client_id" \
54+
-var "github_org=$github_org" \
55+
-var "lambda_memory=$lambda_memory" \
56+
-var "schedule=$schedule" \
57+
-auto-approve

‎terraform/service/providers.tf‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ terraform {
22
required_providers {
33
aws = {
44
source = "hashicorp/aws"
5-
version = "~> 5.0"
5+
version = "~> 6.2.0"
66
}
77
}
88
}

0 commit comments

Comments
 (0)