You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A Python script that scrapes GitHub repositories and saves the data to an S3 bucket. This repository utilises the scheduled batch module to deploy the service as a batch job on AWS.
3
+
A Python script to collect GitHub repository statistics and store them within an S3 bucket.
4
+
This repository uses the scheduled batch module to deploy the service as a batch job on AWS.
4
5
5
-
This project utilises the [GitHub API Package](https://github.com/ONS-Innovation/github-api-package) GraphQL interface to get data from GitHub.
6
+
This tool is responsible for populating the Statistics page within the Digital Landscape ([View Repository](https://github.com/ONSdigital/keh-digital-landscape)).
6
7
7
-
The script is run from the command line using the following command:
8
+
## Disclaimer
8
9
9
-
### Prerequisites:
10
+
This repository is quite old and is due a fair bit of work to bring it up to the standards of our more recent repositories.
11
+
12
+
There are a few `TODOs` scattered throughout the README and codebase that will need to be addressed.
|`source_bucket`| The S3 bucket that will store the output of the script. |
72
+
|`source_key`| The key of the file that will store the output of the script. |
73
+
|`batch_size`| The number of repositories that will be scraped in each batch. |
74
+
|`environment`| Determines where to save the results. `development`: locally, `production`: to S3. |
75
+
76
+
3. Run the script:
77
+
78
+
```bash
79
+
make run
80
+
```
81
+
82
+
## Deployment
83
+
84
+
### Deployments with Concourse
85
+
86
+
#### Allowlisting your IP
87
+
88
+
To setup the deployment pipeline with concourse, you must first allowlist your IP address on the Concourse
89
+
server. IP addresses are flushed everyday at 00:00 so this must be done at the beginning of every working day
90
+
whenever the deployment pipeline needs to be used. Follow the instructions on the Confluence page (SDP Homepage > SDP Concourse > Concourse Login) to
91
+
login. All our pipelines run on sdp-pipeline-prod, whereas sdp-pipeline-dev is the account used for
92
+
changes to Concourse instance itself. Make sure to export all necessary environment variables from sdp-pipeline-prod (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN).
93
+
94
+
#### Setting up a pipeline
95
+
96
+
When setting up our pipelines, we use ecs-infra-user on sdp-dev to be able to interact with our infrastructure on AWS. The credentials for this are stored on
97
+
AWS Secrets Manager so you do not need to set up anything yourself.
Note that you only have to run chmod the first time running the script in order to give permissions.
107
+
This script will set the branch and pipeline name to whatever branch you are currently on. It will also set the image tag on ECR to 7 characters of the current branch name if running on a branch other than main. For main, the ECR tag will be the latest release tag on the repository that has semantic versioning(vX.Y.Z).
108
+
109
+
The pipeline name itself will usually follow a pattern as follows: `github-statistics-scraper-<branch-name>`for any non-main branch and `github-statistics-scraper`for the main/master branch.
110
+
111
+
#### Prod deployment
112
+
113
+
To deploy to prod, it is required that a Github Release is made on Github. The release is required to follow semantic versioning of vX.Y.Z.
114
+
115
+
A manual trigger is to be made on the pipeline name `github-statistics-scraper > deploy-after-github-release` job through the Concourse CI UI. This will create a github-create-tag resource that is required on the `github-statistics-scraper > build-and-push-prod` job. Then the prod deployment job is also through a manual trigger ensuring that prod is only deployed using the latest GitHub release tag in the form of vX.Y.Z and is manually controlled.
116
+
117
+
#### Triggering a pipeline
118
+
119
+
Once the pipeline has been set, you can manually trigger a dev build on the Concourse UI, or run the following commandfor non-main branch deployment:
**It is unlikely that you will need to destroy a pipeline, but the command is here if needed.**
140
+
141
+
**Note:** This will not destroy any resources created by Terraform. You must manually destroy these resources using Terraform.
142
+
143
+
### Manual Deployment
144
+
145
+
TODO: Add manual deployment instructions for deploying the script without Concourse.
146
+
147
+
## Testing
148
+
149
+
TODO: Add testing instructions.
150
+
151
+
TODO: Add tests.
152
+
153
+
## Linting and formatting
154
+
155
+
TODO: Update linting and formatting practices to bring the repo in line with common team practices.
56
156
57
157
Install dev dependencies:
58
158
```bash
@@ -78,38 +178,3 @@ Run black:
78
178
```bash
79
179
make black
80
180
```
81
-
82
-
### Deployments with Concourse
83
-
84
-
#### Allowlisting your IP
85
-
To setup the deployment pipeline with concourse, you must first allowlist your IP address on the Concourse
86
-
server. IP addresses are flushed everyday at 00:00 so this must be done at the beginning of every working day
87
-
whenever the deployment pipeline needs to be used. Follow the instructions on the Confluence page (SDP Homepage > SDP Concourse > Concourse Login) to
88
-
login. All our pipelines run on sdp-pipeline-prod, whereas sdp-pipeline-dev is the account used for
89
-
changes to Concourse instance itself. Make sure to export all necessary environment variables from sdp-pipeline-prod (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN).
90
-
91
-
#### Setting up a pipeline
92
-
When setting up our pipelines, we use ecs-infra-user on sdp-dev to be able to interact with our infrastructure on AWS. The credentials for this are stored on
93
-
AWS Secrets Manager so you do not need to set up anything yourself.
Note that you only have to run chmod the first time running the script in order to give permissions.
101
-
This script will set the branch and pipeline name to whatever branch you are currently on. It will also set the image tag on ECR to the current commit hash at the time of setting the pipeline.
102
-
103
-
The pipeline name itself will usually follow a pattern as follows: `<repo-name>-<branch-name>`
104
-
If you wish to set a pipeline for another branch without checking out, you can run the following:
If the branch you are deploying is "main" or "master", it will trigger a deployment to the sdp-prod environment. To set the ECR image tag, you must draft a Github release pointing to the latest release of the main/master branch that has a tag in the form of vX.Y.Z. Drafting up a release will automatically deploy the latest version of the main/master branch with the associated release tag, but you can also manually trigger a build through the Concourse UI or the terminal prompt.
110
-
111
-
#### Triggering a pipeline
112
-
Once the pipeline has been set, you can manually trigger a build on the Concourse UI, or run the following command:
0 commit comments