Skip to content

Commit 64d7f48

Browse files
delterrdelterr
authored
Merge pull request #19 from ONS-Innovation/concourse
Concourse pipeline
2 parents eea7e9a + 0b9da5e commit 64d7f48

6 files changed

Lines changed: 208 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -482,3 +482,38 @@ This returns your token, which you can use in testing the authentication on the
482482

483483
The /api/v1/verify route get's the client keys and redirect uri from the bucket.
484484

485+
### Deployments with Concourse
486+
487+
#### Allowlisting your IP
488+
To setup the deployment pipeline with concourse, you must first allowlist your IP address on the Concourse
489+
server. IP addresses are flushed everyday at 00:00 so this must be done at the beginning of every working day
490+
whenever the deployment pipeline needs to be used. Follow the instructions on the Confluence page (SDP Homepage > SDP Concourse > Concourse Login) to
491+
login. All our pipelines run on sdp-pipeline-prod, whereas sdp-pipeline-dev is the account used for
492+
changes to Concourse instance itself. Make sure to export all necessary environment variables from sdp-pipeline-prod (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN).
493+
494+
#### Setting up a pipeline
495+
When setting up our pipelines, we use ecs-infra-user on sdp-dev to be able to interact with our infrastructure on AWS. The credentials for this are stored on
496+
AWS Secrets Manager so you do not need to set up anything yourself.
497+
498+
To set the pipeline, run the following script:
499+
```bash
500+
chmod u+x ./concourse/scripts/set_pipeline.sh
501+
./concourse/scripts/set_pipeline.sh KEH-TAT-API
502+
```
503+
Note that you only have to run chmod the first time running the script in order to give permissions.
504+
This script will set the branch and pipeline name to whatever branch you are currently on. It will also set the image tag on ECR to the current commit hash at the time of setting the pipeline.
505+
506+
The pipeline name itself will usually follow a pattern as follows: `<repo-name>-<branch-name>`
507+
If you wish to set a pipeline for another branch without checking out, you can run the following:
508+
```bash
509+
./concourse/scripts/set_pipeline.sh KEH-TAT-API <branch_name>
510+
```
511+
512+
If the branch you are deploying is "main" or "master", it will trigger a deployment to the sdp-prod environment. To set the ECR image tag, you must draft a Github release pointing to the latest release of the main/master branch that has a tag in the form of vX.Y.Z. Drafting up a release will automatically deploy the latest version of the main/master branch with the associated release tag, but you can also manually trigger a build through the Concourse UI or the terminal prompt.
513+
514+
#### Triggering a pipeline
515+
Once the pipeline has been set, you can manually trigger a build on the Concourse UI, or run the following command:
516+
```bash
517+
fly -t aws-sdp trigger-job -j KEH-TAT-API-<branch-name>/build-and-push
518+
```
519+

‎concourse/ci.yml‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
resources:
2+
- name: resource-repo
3+
type: git
4+
icon: github
5+
source:
6+
uri: https://github.com/ONS-Innovation/keh-tech-audit-tool-api
7+
branch: ((branch))
8+
access_token: ((github_access_token))
9+
10+
jobs:
11+
- name: build-and-push
12+
public: true
13+
plan:
14+
- get: resource-repo
15+
timeout: 5m
16+
- task: build-image
17+
privileged: true
18+
config:
19+
platform: linux
20+
image_resource:
21+
type: docker-image
22+
source:
23+
repository: hashicorp/terraform
24+
inputs:
25+
- name: resource-repo
26+
params:
27+
aws_account_id: ((aws_account_sdp_((env))))
28+
aws_role_arn: arn:aws:iam::((aws_account_sdp_((env)))):role/sdp-concourse-((env))
29+
secrets: ((sdp_((env))_secrets_tat_api))
30+
run: # binary used to build the image
31+
path: sh
32+
args:
33+
- -cx
34+
- |
35+
apk add --no-cache aws-cli podman jq iptables curl
36+
37+
if [[ "((env))" == "prod" ]]; then
38+
tag=$(curl "https://api.github.com/repos/ONS-Innovation/keh-tech-audit-tool-api/releases" | jq -r '.[0].tag_name')
39+
export tag
40+
else
41+
export tag=((tag))
42+
fi
43+
git rev-parse --abbrev-ref HEAD
44+
chmod u+x ./resource-repo/concourse/scripts/assume_role.sh
45+
chmod u+x ./resource-repo/concourse/scripts/build_image.sh
46+
source ./resource-repo/concourse/scripts/assume_role.sh
47+
./resource-repo/concourse/scripts/build_image.sh
48+
timeout: 10m
49+
- task: terraform
50+
privileged: true
51+
config:
52+
platform: linux
53+
image_resource:
54+
type: docker-image
55+
source: {repository: hashicorp/terraform}
56+
inputs:
57+
- name: resource-repo
58+
params:
59+
secrets: ((sdp_((env))_secrets_tat_api))
60+
github_access_token: ((github_access_token))
61+
run:
62+
path: sh
63+
args:
64+
- -cx
65+
- |
66+
apk add --no-cache jq curl
67+
if [[ "((env))" == "prod" ]]; then
68+
tag=$(curl "https://api.github.com/repos/ONS-Innovation/keh-tech-audit-tool-api/releases" | jq -r '.[0].tag_name')
69+
export tag
70+
else
71+
export tag=((tag))
72+
fi
73+
chmod u+x ./resource-repo/concourse/scripts/terraform_infra.sh
74+
export env=((env))
75+
./resource-repo/concourse/scripts/terraform_infra.sh
76+
timeout: 30m
77+

‎concourse/scripts/assume_role.sh‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
set -euo pipefail
2+
3+
aws sts assume-role --output text \
4+
--role-arn "${aws_role_arn}" \
5+
--role-session-name concourse-pipeline-run \
6+
--query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
7+
| awk -F '\t' '{print $1 > ("AccessKeyId")}{print $2 > ("SecretAccessKey")}{print $3 > ("SessionToken")}'
8+
9+
10+
export AWS_ACCESS_KEY_ID="$(cat AccessKeyId)"
11+
export AWS_SECRET_ACCESS_KEY="$(cat SecretAccessKey)"
12+
export AWS_SESSION_TOKEN="$(cat SessionToken)"

‎concourse/scripts/build_image.sh‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
set -euo pipefail
2+
3+
export STORAGE_DRIVER=vfs
4+
export PODMAN_SYSTEMD_UNIT=concourse-task
5+
6+
container_image=$(echo "$secrets" | jq -r .ecr_repository)
7+
8+
aws ecr get-login-password --region eu-west-2 | podman --storage-driver=vfs login --username AWS --password-stdin ${aws_account_id}.dkr.ecr.eu-west-2.amazonaws.com
9+
10+
podman build -t ${container_image}:${tag} resource-repo/aws_lambda_script/
11+
12+
podman tag ${container_image}:${tag} ${aws_account_id}.dkr.ecr.eu-west-2.amazonaws.com/${container_image}:${tag}
13+
14+
podman push ${aws_account_id}.dkr.ecr.eu-west-2.amazonaws.com/${container_image}:${tag}

‎concourse/scripts/set_pipeline.sh‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
repo_name=${1}
2+
3+
if [[ $# -gt 1 ]]; then
4+
branch=${2}
5+
git rev-parse --verify ${branch}
6+
if [[ $? -ne 0 ]]; then
7+
echo "Branch \"${branch}\" does not exist"
8+
exit 1
9+
fi
10+
else
11+
branch=$(git rev-parse --abbrev-ref HEAD)
12+
fi
13+
14+
if [[ ${branch} == "main" || ${branch} == "master" ]]; then
15+
env="prod"
16+
else
17+
env="dev"
18+
fi
19+
20+
if [[ ${env} == "dev" ]]; then
21+
tag=$(git rev-parse HEAD)
22+
else
23+
tag=$(git tag | tail -n 1)
24+
fi
25+
26+
fly -t aws-sdp set-pipeline -c concourse/ci.yml -p ${repo_name}-${branch} -v branch=${branch} -v tag=${tag} -v env=${env}
27+

‎concourse/scripts/terraform_infra.sh‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
set -euo pipefail
2+
3+
aws_account_id=$(echo "$secrets" | jq -r .aws_account_id)
4+
aws_access_key_id=$(echo "$secrets" | jq -r .aws_access_key_id)
5+
6+
aws_secret_access_key=$(echo "$secrets" | jq -r .aws_secret_access_key)
7+
domain=$(echo "$secrets" | jq -r .domain)
8+
9+
service_subdomain=$(echo "$secrets" | jq -r .service_subdomain)
10+
ecr_repository=$(echo "$secrets" | jq -r .ecr_repository)
11+
12+
export AWS_ACCESS_KEY_ID=$aws_access_key_id
13+
export AWS_SECRET_ACCESS_KEY=$aws_secret_access_key
14+
15+
git config --global url."https://x-access-token:$github_access_token@github.com/".insteadOf "https://github.com/"
16+
17+
if [[ ${env} != "prod" ]]; then
18+
env="dev"
19+
fi
20+
21+
cd resource-repo/terraform/lambda
22+
23+
terraform init -backend-config=env/${env}/backend-${env}.tfbackend -reconfigure
24+
terraform apply \
25+
-var "aws_account_id=$aws_account_id" \
26+
-var "aws_access_key_id=$aws_access_key_id" \
27+
-var "aws_secret_access_key=$aws_secret_access_key" \
28+
-var "domain=$domain" \
29+
-var "service_subdomain=$service_subdomain" \
30+
-var "container_ver=${tag}" \
31+
-var "ecr_repository=$ecr_repository" \
32+
-auto-approve
33+
34+
cd ../api_gateway
35+
36+
terraform init -backend-config=env/${env}/backend-${env}.tfbackend -reconfigure
37+
terraform apply \
38+
-var "aws_account_id=$aws_account_id" \
39+
-var "aws_access_key_id=$aws_access_key_id" \
40+
-var "aws_secret_access_key=$aws_secret_access_key" \
41+
-var "domain=$domain" \
42+
-auto-approve
43+

0 commit comments

Comments
 (0)