diff --git a/README.md b/README.md index 87110d9..bf2ab9a 100644 --- a/README.md +++ b/README.md @@ -6,20 +6,20 @@ This repository provides a [Terraform module](https://learn.hashicorp.com/tutori This repository provides four submodules: -1. The [executors module](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.5.0/submodules/executors) provisions compute resources for executors. -2. The [docker-mirror module](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.5.0/submodules/docker-mirror) provisions a Docker registry pull-through cache. -3. The [networking module](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.5.0/submodules/networking) provisions a network to be shared by the executor and Docker registry resources. -4. The [credentials module](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.5.0/submodules/credentials) provisions credentials required by the Sourcegraph instance to enable observability and auto-scaling of executors. +1. The [executors module](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.6.0/submodules/executors) provisions compute resources for executors. +2. The [docker-mirror module](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.6.0/submodules/docker-mirror) provisions a Docker registry pull-through cache. +3. The [networking module](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.6.0/submodules/networking) provisions a network to be shared by the executor and Docker registry resources. +4. The [credentials module](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.6.0/submodules/credentials) provisions credentials required by the Sourcegraph instance to enable observability and auto-scaling of executors. -The [multiple-executors example](https://github.com/sourcegraph/terraform-aws-executors/blob/v6.5.0/examples/multiple-executors) uses the submodule directly to provision multiple executor resource groups performing different types of work. Follow this example if you are: +The [multiple-executors example](https://github.com/sourcegraph/terraform-aws-executors/blob/v6.6.0/examples/multiple-executors) uses the submodule directly to provision multiple executor resource groups performing different types of work. Follow this example if you are: 1. Provisioning executors for use with multiple features (e.g., both [auto-indexing](https://sourcegraph.com/docs/code_intelligence/explanations/auto_indexing) and [server-side batch changes](https://sourcegraph.com/docs/batch_changes/explanations/server_side)), or 2. Provisioning resources for multiple Sourcegraph instances (e.g., test, prod) -This repository also provides a [root module](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.5.0) combining the executors, network, and docker-mirror resources into an easier to use package. +This repository also provides a [root module](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.6.0) combining the executors, network, and docker-mirror resources into an easier to use package. -The [single-executor example](https://github.com/sourcegraph/terraform-aws-executors/blob/v6.5.0/examples/single-executor) uses the root module to provision a single executor type. Follow this example if you are deploying to a single Sourcegraph instance and using a single executors-backed feature. +The [single-executor example](https://github.com/sourcegraph/terraform-aws-executors/blob/v6.6.0/examples/single-executor) uses the root module to provision a single executor type. Follow this example if you are deploying to a single Sourcegraph instance and using a single executors-backed feature. ## Requirements @@ -28,7 +28,7 @@ The [single-executor example](https://github.com/sourcegraph/terraform-aws-execu - 4.2.0 and above: `>= 1.1.0, < 2.0.0` - [hashicorp/aws](https://registry.terraform.io/providers/hashicorp/aws) - 4.1.0 and below: `~> 3.0.0` - - 4.2.0 and above: `>= 3.0, < 6.5.0` + - 4.2.0 and above: `>= 3.0, < 6.6.0` ## Setup diff --git a/batch-change/release.yaml b/batch-change/release.yaml new file mode 100644 index 0000000..5ee2a8f --- /dev/null +++ b/batch-change/release.yaml @@ -0,0 +1,221 @@ +--- +meta: + productName: "terraform-aws-executors" + owners: + - "sourcegraph/release" + repository: "github.com/sourcegraph/terraform-aws-executors" +inputs: + - releaseId: server +requirements: + - name: "comby" + cmd: "which comby" + fixInstructions: "install comby" + - name: "GitHub CLI" + cmd: "which gh" + fixInstructions: "install GitHub cli" + - name: "prev_tag" + cmd: "[[ -n $PREV_TAG ]]" + fixInstructions: "Pass $PREV_TAG as an env var" +internal: + create: + steps: + minor: + - name: "env_var:prev_tag" + cmd: | + # get previous tag we need to replace + # There's no .git info for us to query in batch-changes + echo $PREV_TAG | sed 's/v//' > prev_tag + # convert the tag to the image family format which uses hipens and is only for `major-minor` + echo "{{tag}}" | grep -o -E "[[:digit:]]+\.[[:digit:]]+" | sed 's/\./-/' > family_tag + - name: "files(README.md)" + cmd: comby "$(cat prev_tag)" '{{tag}}' -i -f .md + - name: "files(tf)" + cmd: comby "\"$(cat prev_tag)\"" '"{{tag}}"' -i -f .tf -exclude providers.tf + - name: "family(name):docker-mirror" + cmd: comby '["sourcegraph-executors-docker-mirror-:[~\d+-\d+]-*"]' "[\"sourcegraph-executors-docker-mirror-$(cat family_tag)-*\"]" -i -f .tf + - name: "family(name):sourcegraph" + cmd: comby '["sourcegraph-executors-:[~\d+-\d+]-*"]' "[\"sourcegraph-executors-$(cat family_tag)-*\"]" -i -f .tf + - name: "cleanup" + cmd: | + # remove files we used in previous steps + rm -vf family_tag + rm -vf prev_tag + major: + - name: "env_var:prev_tag" + cmd: | + # get previous tag we need to replace + echo $PREV_TAG | sed 's/v//' > prev_tag + # convert the tag to the image family format which uses hipens and is only for `major-minor` + echo "{{tag}}" | grep -o -E "[[:digit:]]+\.[[:digit:]]+" | sed 's/\./-/' > family_tag + - name: "files(README.md)" + cmd: comby -in-place "$(cat prev_tag)" '{{tag}}' README.md + - name: "files(tf)" + cmd: comby "\"$(cat prev_tag)\"" '"{{tag}}"' -i -f .tf -exclude providers.tf + - name: "family(name):docker-mirror" + cmd: comby '["sourcegraph-executors-docker-mirror-:[~\d+-\d+]-*"]' "[\"sourcegraph-executors-docker-mirror-$(cat family_tag)-*\"]" -i -f .tf + - name: "family(name):sourcegraph" + cmd: comby '["sourcegraph-executors-:[~\d+-\d+]-*"]' "[\"sourcegraph-executors-$(cat family_tag)-*\"]" -i -f .tf + - name: "cleanup" + cmd: | + # remove files we used in previous steps + rm -vf family_tag + rm -vf prev_tag + finalize: + steps: + - name: "notifications" + cmd: | + set -eu + + branch="internal/release-{{version}}" + + # Post a comment on the PR. + cat << EOF | gh pr comment "$branch" --body-file - + - :green_circle: Internal release is ready for promotion! + - :warning: Do not close/merge the pull request or delete the associated branch if you intend to promote it. + EOF + # Post an annotation. + cat << EOF | buildkite-agent annotate --style info + Internal release is ready for promotion under the branch [\`$branch\`](https://github.com/sourcegraph/terraform-aws-executors/tree/$branch). + EOF + + # - name: 'Register on release registry' + # cmd: | + # echo "Registering internal terraform-aws-executors {{version}} release on release registry" + # body=$(wget --content-on-error -O- --header="Content-Type: application/json" --header="Authorization: ${RELEASE_REGISTRY_TOKEN}" --post-data '{ + # "name": "terraform-aws-executors", + # "version": "{{version}}", + # "git_sha": "'${COMMIT_SHA}'" + # }' "https://releaseregistry.sourcegraph.com/v1/releases") + # exit_code=$? + + # if [ $exit_code != 0 ]; then + # echo "❌ Failed to create release in release registry, got:" + # echo "--- raw body ---" + # echo $body + # echo "--- raw body ---" + # exit $exit_code + # else + # echo "Release created, see:" + # echo $body + # fi + +test: + steps: + - name: "changes:README" + cmd: | + count=$(grep -c "{{tag}}" --include "README.md" -r . | awk -F ":" '{sum=sum + $2} END {print sum;}') + expected=16 + if [[ ${count} -ne ${expected} ]]; then + echo "expected ${expected} new version tags of \"{{tag}}\" in README.md, got ${count}" + exit 1 + fi + - name: "changed:tf" + cmd: | + count=$(comby -match-only '"{{tag}}"' '' -f .tf | wc -l) + expected=6 + if [[ ${count} -ne ${expected} ]]; then + echo "expected ${expected} .tf files to be updated with \"{{tag}}\" but got ${count}" + exit 1 + fi + - name: changes:family(docker-mirror)"" + cmd: | + set -eu + + echo "{{tag}}" | grep -o -E "[[:digit:]]+\.[[:digit:]]+" | sed 's/\./-/' > family_tag + trap "rm family_tag" EXIT + + current_branch="${BUILDKITE_BRANCH:-$(git rev-parse --abbrev-ref HEAD)}" + + family_regex="\"sourcegraph-executors-docker-mirror-$(cat family_tag)-*\"" + if [[ $current_branch =~ ^(promote|release)/.* ]]; then + family_regex="\"sourcegraph-executors-docker-mirror-$(cat family_tag)-*\"" + fi + + count=$(comby -match-only "${family_regex}" '' -f .tf | wc -l) + expected=1 + if [[ ${count} -ne ${expected} ]]; then + echo "expected ${expected} .tf files to be updated with \"${family_regex}\" but got ${count}" + exit 1 + fi + - name: "changes:family(sourcegraph)" + cmd: | + set -eu + + echo "{{tag}}" | grep -o -E "[[:digit:]]+\.[[:digit:]]+" | sed 's/\./-/' > family_tag + trap "rm family_tag" EXIT + + current_branch="${BUILDKITE_BRANCH:-$(git rev-parse --abbrev-ref HEAD)}" + + family_regex="\"sourcegraph-executors-$(cat family_tag)-*\"" + if [[ $current_branch =~ ^(promote|release)/.* ]]; then + family_regex="\"sourcegraph-executors-$(cat family_tag)-*\"" + fi + + count=$(comby -match-only "${family_regex}" '' -f .tf | wc -l) + expected=1 + if [[ ${count} -ne ${expected} ]]; then + echo "expected ${expected} file to have the correct image family set but got but got ${count} files" + exit 1 + fi + +promoteToPublic: + create: + steps: + # Since we're promoting to public we need to remove the "internal" from the family + - name: "family(name):docker-mirror" + cmd: comby '["sourcegraph-executors-docker-mirror-:[family_tag_hole~\d+-\d+]-*"]' "[\"sourcegraph-executors-docker-mirror-:[family_tag_hole]-*\"]" -i -f .tf + - name: "family(name):sourcegraph" + cmd: comby '["sourcegraph-executors-:[family_tag_hole~\d+-\d+]-*"]' "[\"sourcegraph-executors-:[family_tag_hole]-*\"]" -i -f .tf + finalize: + steps: + # - name: 'Promote on release registry' + # cmd: | + # echo "Promoting terraform-aws-executors {{version}} release on release registry" + # body=$(wget --content-on-error -O- --header="Content-Type: application/json" --header="Authorization: ${RELEASE_REGISTRY_TOKEN}" --post-data '' "https://releaseregistry.sourcegraph.com/v1/releases/terraform-aws-executors/{{version}}/promote") + # exit_code=$? + + # if [ $exit_code != 0 ]; then + # echo "❌ Failed to promote release on release registry, got:" + # echo "--- raw body ---" + # echo $body + # echo "--- raw body ---" + # exit $exit_code + # else + # echo "Release promoted, see:" + # echo $body + # fi + - name: git:tag + cmd: | + set -eu + + # Branches + internal_branch="internal/release-{{version}}" + promote_branch="promote/release-{{version}}" + release_branch="release-{{version}}" + + # Create the final branch holding the tagged commit + git checkout "${promote_branch}" + git switch -c "${release_branch}" + git tag {{version}} + git push origin ${release_branch} --tags + + # Web URL to the tag + tag_url="https://github.com/sourcegraph/terraform-aws-executors/tree/{{version}}" + + # Annotate PRs + cat << EOF | gh pr comment "$internal_branch" --body-file - + - :green_circle: Release has been promoted, see tag: $tag_url. + - :no_entry: Do not under any circumstance delete the branch holding the tagged commit (i.e. \`$release_branch\`). + - :arrow_right: You can safely close the PR and delete its a associated branch. + EOF + + cat << EOF | gh pr comment "$promote_branch" --body-file - + - :green_circle: Release has been promoted, see tag: $tag_url. + - :no_entry: Do not under any circumstance delete the branch holding the tagged commit (i.e. \`$release_branch\`). + - :arrow_right: You can safely close that PR and delete its a associated branch. + EOF + + # Annotate build + cat << EOF | buildkite-agent annotate --style info + Promoted release is **publicly available** through a git tag at [\`{{version}}\`](https://github.com/sourcegraph/terraform-aws-executors/tree/{{version}}). + EOF diff --git a/examples/multiple-executors/README.md b/examples/multiple-executors/README.md index c179079..4d1bd62 100644 --- a/examples/multiple-executors/README.md +++ b/examples/multiple-executors/README.md @@ -1,6 +1,6 @@ # Multiple executor example -This example uses [networking](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.5.0/submodules/networking), [docker-mirror](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.5.0/submodules/docker-mirror), and [executors](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.5.0/submodules/executors) submodules that provision a network, a Docker registry mirror, and sets of resources running one or more types of executors. +This example uses [networking](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.6.0/submodules/networking), [docker-mirror](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.6.0/submodules/docker-mirror), and [executors](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.6.0/submodules/executors) submodules that provision a network, a Docker registry mirror, and sets of resources running one or more types of executors. The following variables must be supplied: @@ -10,4 +10,4 @@ The following variables must be supplied: If your deployment environment already has a Docker registry that can be used, only the `executor` submodule must be used (and references to the `networking` and `docker-mirror` modules can be dropped). The Docker registry mirror address can be supplied along with its containing VPC and subnet as pre-existing identifier literals. -All of these module's variables are defined in [modules/networking/variables.tf](https://github.com/sourcegraph/terraform-aws-executors/blob/v6.5.0/modules/networking/variables.tf), [modules/docker-mirror/variables.tf](https://github.com/sourcegraph/terraform-aws-executors/blob/v6.5.0/modules/docker-mirror/variables.tf), and [modules/executors/variables.tf](https://github.com/sourcegraph/terraform-aws-executors/blob/v6.5.0/modules/executors/variables.tf). \ No newline at end of file +All of these module's variables are defined in [modules/networking/variables.tf](https://github.com/sourcegraph/terraform-aws-executors/blob/v6.6.0/modules/networking/variables.tf), [modules/docker-mirror/variables.tf](https://github.com/sourcegraph/terraform-aws-executors/blob/v6.6.0/modules/docker-mirror/variables.tf), and [modules/executors/variables.tf](https://github.com/sourcegraph/terraform-aws-executors/blob/v6.6.0/modules/executors/variables.tf). \ No newline at end of file diff --git a/examples/multiple-executors/main.tf b/examples/multiple-executors/main.tf index ac30a65..d439d7f 100644 --- a/examples/multiple-executors/main.tf +++ b/examples/multiple-executors/main.tf @@ -6,7 +6,7 @@ locals { module "networking" { source = "sourcegraph/executors/aws//modules/networking" - version = "6.5.0" # LATEST + version = "6.6.0" # LATEST availability_zone = local.availability_zone randomize_resource_names = true @@ -14,7 +14,7 @@ module "networking" { module "docker-mirror" { source = "sourcegraph/executors/aws//modules/docker-mirror" - version = "6.5.0" # LATEST + version = "6.6.0" # LATEST vpc_id = module.networking.vpc_id subnet_id = module.networking.subnet_id @@ -25,7 +25,7 @@ module "docker-mirror" { module "executors-codeintel" { source = "sourcegraph/executors/aws//modules/executors" - version = "6.5.0" # LATEST + version = "6.6.0" # LATEST vpc_id = module.networking.vpc_id subnet_id = module.networking.subnet_id @@ -43,7 +43,7 @@ module "executors-codeintel" { module "executors-batches" { source = "sourcegraph/executors/aws//modules/executors" - version = "6.5.0" # LATEST + version = "6.6.0" # LATEST vpc_id = module.networking.vpc_id subnet_id = module.networking.subnet_id diff --git a/examples/private-single-executor/main.tf b/examples/private-single-executor/main.tf index e848d2d..c82aac9 100644 --- a/examples/private-single-executor/main.tf +++ b/examples/private-single-executor/main.tf @@ -5,7 +5,7 @@ locals { module "executors" { source = "sourcegraph/executors/aws" - version = "6.5.0" # LATEST + version = "6.6.0" # LATEST availability_zone = local.availability_zone executor_instance_tag = "codeintel-prod" diff --git a/examples/single-executor/README.md b/examples/single-executor/README.md index 7df31d5..2d1c8b5 100644 --- a/examples/single-executor/README.md +++ b/examples/single-executor/README.md @@ -1,6 +1,6 @@ # Single executor example -This example uses the [root module](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.5.0) that provisions a network, a Docker registry mirror, and a set of resources to run _one_ type of executor. To provision more than one type of executor (multiple queues or multiple environments), see the following `multiple-executors` example. +This example uses the [root module](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.6.0) that provisions a network, a Docker registry mirror, and a set of resources to run _one_ type of executor. To provision more than one type of executor (multiple queues or multiple environments), see the following `multiple-executors` example. The following variables must be supplied: @@ -12,4 +12,4 @@ The following variables must be supplied: - `executor_metrics_environment_label`: The name of the target environment (e.g., `staging`, `prod`). This value must be the same as the `EXECUTOR_METRIC_ENVIRONMENT_LABEL` environment variable as described in [Configuring auto scaling](https://docs.sourcegraph.com/admin/deploy_executors#aws). - `executor_instance_tag`: Compute instances are tagged by this value by the key `executor_tag`. We recommend this value take the form `{executor_queue_name}-{executor_metrics_environment_label}`. This value must be the same as `INSTANCE_TAG` as described in [Configuring observability](https://docs.sourcegraph.com/admin/deploy_executors#aws-1). -All of this module's variables are defined in [variables.tf](https://github.com/sourcegraph/terraform-aws-executors/blob/v6.5.0/variables.tf). \ No newline at end of file +All of this module's variables are defined in [variables.tf](https://github.com/sourcegraph/terraform-aws-executors/blob/v6.6.0/variables.tf). \ No newline at end of file diff --git a/examples/single-executor/main.tf b/examples/single-executor/main.tf index 8794b7e..11bb2c2 100644 --- a/examples/single-executor/main.tf +++ b/examples/single-executor/main.tf @@ -5,7 +5,7 @@ locals { module "executors" { source = "sourcegraph/executors/aws" - version = "6.5.0" # LATEST + version = "6.6.0" # LATEST availability_zone = local.availability_zone executor_instance_tag = "codeintel-prod" diff --git a/modules/docker-mirror/README.md b/modules/docker-mirror/README.md index d11c41f..a4d6a31 100644 --- a/modules/docker-mirror/README.md +++ b/modules/docker-mirror/README.md @@ -2,4 +2,4 @@ This module provides a hosted Docker registry pull-through cache to be used by [Sourcegraph executor](https://docs.sourcegraph.com/admin/executors). It is strongly recommended to deploy a Docker mirror as a cache to reduce rate limiting by the public [Docker Hub registry](https://hub.docker.com/). We have also seen deploying a Docker mirror in the same physical zone as the executors significantly decreased latencies during image pulls. -When using the sibling [executors module](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.5.0/submodules/executors), the `vpc_id` and `subnet_id` values must match and the executor module `docker_registry_mirror` value should match `"http://${static_ip}:5000"`. \ No newline at end of file +When using the sibling [executors module](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.6.0/submodules/executors), the `vpc_id` and `subnet_id` values must match and the executor module `docker_registry_mirror` value should match `"http://${static_ip}:5000"`. \ No newline at end of file diff --git a/modules/docker-mirror/main.tf b/modules/docker-mirror/main.tf index 1718d39..6ad8797 100644 --- a/modules/docker-mirror/main.tf +++ b/modules/docker-mirror/main.tf @@ -48,7 +48,7 @@ data "aws_ami" "latest_ami" { filter { name = "name" - values = ["sourcegraph-executors-docker-mirror-6-5-*"] + values = ["sourcegraph-executors-docker-mirror-6-6-*"] } filter { diff --git a/modules/executors/README.md b/modules/executors/README.md index 4e7d02b..9cd4774 100644 --- a/modules/executors/README.md +++ b/modules/executors/README.md @@ -1,6 +1,6 @@ # Executors module -This module provides the resources to provision [Sourcegraph executor](https://docs.sourcegraph.com/admin/executors) compute resources on AWS. For a high-level overview of the resources deployed by this module, see the [root module](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.5.0). This module includes the following resources: +This module provides the resources to provision [Sourcegraph executor](https://docs.sourcegraph.com/admin/executors) compute resources on AWS. For a high-level overview of the resources deployed by this module, see the [root module](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.6.0). This module includes the following resources: - AWS launch template - AWS autoscaler and autoscaling policy @@ -10,4 +10,4 @@ This module provides the resources to provision [Sourcegraph executor](https://d This module does **not** automatically create networking or Docker mirror resources. The `vpc_id`, `subnet_id`, and `docker_registry_mirror` variables must be supplied explicitly with resources that have been previously created. -This module is often used with the sibling modules that create [networking](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.5.0/submodules/networking) and [Docker mirror](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.5.0/submodules/docker-mirror) resources which can be shared by multiple instances of the executor module (listening to different queues or being deployed in a different environment). \ No newline at end of file +This module is often used with the sibling modules that create [networking](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.6.0/submodules/networking) and [Docker mirror](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.6.0/submodules/docker-mirror) resources which can be shared by multiple instances of the executor module (listening to different queues or being deployed in a different environment). \ No newline at end of file diff --git a/modules/executors/main.tf b/modules/executors/main.tf index 0d7ebad..6ace62e 100644 --- a/modules/executors/main.tf +++ b/modules/executors/main.tf @@ -153,7 +153,7 @@ data "aws_ami" "latest_ami" { filter { name = "name" - values = ["sourcegraph-executors-6-5-*"] + values = ["sourcegraph-executors-6-6-*"] } filter { diff --git a/modules/networking/README.md b/modules/networking/README.md index d1be80d..69a54e1 100644 --- a/modules/networking/README.md +++ b/modules/networking/README.md @@ -1,6 +1,6 @@ # Networking module -This module provides the networking glue between the sibling [executors](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.5.0/submodules/executors) and [docker-mirror](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.5.0/submodules/docker-mirror) modules. +This module provides the networking glue between the sibling [executors](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.6.0/submodules/executors) and [docker-mirror](https://registry.terraform.io/modules/sourcegraph/executors/aws/6.6.0/submodules/docker-mirror) modules. This module is very simple, creating only a network and a subnet by default.