Archive Tool Improvements

.github/workflows/deploy_mkdocs.yml

@@ -0,0 +1,33 @@
+# A GitHub Actions workflow to deploy MkDocs documentation to GitHub Pages on push to the master or main branch.
+---
+name: Deploy MkDocs
+on:
+  push:
+    branches:
+      - master
+      - main
+permissions:
+  contents: write
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Configure Git Credentials
+        run: |
+          git config user.name github-actions[bot]
+          git config user.email 41898282+github-actions[bot]@users.noreply.github.com
+      - uses: actions/setup-python@v5
+        with:
+          python-version: 3.x
+      - run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
+      - uses: actions/cache@v4
+        with:
+          key: mkdocs-material-${{ env.cache_id }}
+          path: .cache
+          restore-keys: |
+            mkdocs-material-
+      - run: pip install mkdocs-material
+      - run: pip install poetry
+      - run: poetry install --only docs
+      - run: mkdocs gh-deploy --force

.mega-linter.yml

@@ -35,6 +35,9 @@ DISABLE_LINTERS:
   - TERRAFORM_TFLINT
   - TERRAFORM_TERRASCAN
 
+  # Disable due to poor configuration options
+  - ACTION_ACTIONLINT
+
 SHOW_ELAPSED_TIME: true
 
 FILEIO_REPORTER: false

Makefile

@@ -14,11 +14,13 @@ clean: ## Clean the temporary files.
 	rm -rf .coverage
 	rm -rf .ruff_cache
 	rm -rf megalinter-reports
+	rm -rf debug.log
+	rm -rf site
 
 .PHONY: format
 format:  ## Format the code.
-	poetry run black .
-	poetry run ruff check . --fix
+	poetry run black src
+	poetry run ruff check src --fix
 
 .PHONY: lint
 lint:  ## Run all linters (black/ruff/pylint/mypy).
@@ -42,6 +44,10 @@ install:  ## Install the dependencies excluding dev.
 install-dev:  ## Install the dependencies including dev.
 	poetry install
 
+.PHONY: install-docs
+install-docs:  ## Install only the documentation dependencies
+	poetry install --only docs
+
 .PHONY: megalint
 megalint:  ## Run the mega-linter.
 	docker run --platform linux/amd64 --rm \

README.md

@@ -8,6 +8,7 @@ A Python utility used to archive old, unused GitHub repositories from an organis
   - [Table of Contents](#table-of-contents)
   - [Prerequisites](#prerequisites)
   - [Makefile](#makefile)
+  - [Documentation](#documentation)
   - [Development](#development)
   - [Running the Project](#running-the-project)
     - [Containerised (Recommended)](#containerised-recommended)
@@ -18,6 +19,10 @@ A Python utility used to archive old, unused GitHub repositories from an organis
     - [Storing the Container on AWS Elastic Container Registry (ECR)](#storing-the-container-on-aws-elastic-container-registry-ecr)
     - [Deploying the Lambda](#deploying-the-lambda)
     - [Destroying / Removing the Lambda](#destroying--removing-the-lambda)
+    - [Deployments with Concourse](#deployments-with-concourse)
+      - [Allowlisting your IP](#allowlisting-your-ip)
+      - [Setting up a pipeline](#setting-up-a-pipeline)
+      - [Triggering a pipeline](#triggering-a-pipeline)
   - [Linting and Testing](#linting-and-testing)
     - [GitHub Actions](#github-actions)
     - [Running Tests Locally](#running-tests-locally)
@@ -41,6 +46,24 @@ This repository makes use of a Makefile to execute common commands. To view all
 make all
 ```
 
+## Documentation
+
+This project uses [MkDocs](https://www.mkdocs.org/) for documentation. The documentation is located in the `docs` directory. To view the documentation locally, you can run the following commands:
+
+1. Install MkDocs and its dependencies:
+
+    ```bash
+    make install-docs
+    ```
+
+2. Serve the documentation locally:
+
+    ```bash
+    mkdocs serve
+    ```
+
+3. Open your web browser and navigate to `http://localhost:8000`.
+
 ## Development
 
 To work on this project, you need to:
@@ -112,19 +135,21 @@ Before the doing the following, make sure your Daemon is running. If using Colim
     -e AWS_SECRET_NAME=<secret_name> \
     -e GITHUB_ORG=<org> \
     -e GITHUB_APP_CLIENT_ID=<client_id> \
+    -e S3_BUCKET_NAME=<bucket_name>\
     -e AWS_LAMBDA_FUNCTION_TIMEOUT=300
     github-repository-archive-script
     ```
 
     When running the container, you are required to pass some environment variable.
 
-    | Variable                    | Description                                                                               |
-    |-----------------------------|-------------------------------------------------------------------------------------------|
-    | GITHUB_ORG                  | The organisation you would like to run the tool in.                                       |
-    | GITHUB_APP_CLIENT_ID        | The Client ID for the GitHub App which the tool uses to authenticate with the GitHub API. |
-    | AWS_DEFAULT_REGION          | The AWS Region which the Secret Manager Secret is in.                                     |
-    | AWS_SECRET_NAME             | The name of the AWS Secret Manager Secret to get.                                         |
-    | AWS_LAMBDA_FUNCTION_TIMEOUT | The timeout time in seconds (Default: 300s / 5 minutes).                                  |
+    | Variable                    | Description                                                                                        |
+    |-----------------------------|----------------------------------------------------------------------------------------------------|
+    | GITHUB_ORG                  | The organisation you would like to run the tool in.                                                |
+    | GITHUB_APP_CLIENT_ID        | The Client ID for the GitHub App which the tool uses to authenticate with the GitHub API.          |
+    | AWS_DEFAULT_REGION          | The AWS Region which the Secret Manager Secret is in.                                              |
+    | AWS_SECRET_NAME             | The name of the AWS Secret Manager Secret to get.                                                  |
+    | AWS_BUCKET_NAME             | The name of the S3 bucket which has the cloud config in (Only used when `use_local_config=False`). |
+    | AWS_LAMBDA_FUNCTION_TIMEOUT | The timeout time in seconds (Default: 300s / 5 minutes).                                           |
 
     Once the container is running, a local endpoint is created at `localhost:9000/2015-03-31/functions/function/invocations`.
 
@@ -177,6 +202,7 @@ To run the Lambda function outside of a container, we need to execute the `handl
     export AWS_SECRET_ACCESS_KEY=<secret_access_key>
     export AWS_DEFAULT_REGION=eu-west-2
     export AWS_SECRET_NAME=<secret_name>
+    export S3_BUCKET_NAME=<bucket_name>
     export GITHUB_ORG=<org>
     export GITHUB_APP_CLIENT_ID=<client_id>
     ```

config/config.json

@@ -1,12 +1,13 @@
 {
   "features": {
-    "show_log_locally": true
+    "show_log_locally": false,
+    "use_local_config": false
   },
   "archive_configuration": {
     "archive_threshold": 365,
     "notification_period": 30,
     "notification_issue_tag": "Archive Notice",
-    "exemption_filename": "ArchiveExemption.txt",
+    "exemption_filename": ["ArchiveExemption.txt", "ArchiveExemption.md"],
     "maximum_notifications": 1
   }
 }

docs/documentation.md

@@ -0,0 +1,67 @@
+# Documentation
+
+This site uses MkDocs to build its documentation and GitHub Pages for hosting.
+
+## Format
+
+Documentation within this project follows the following pattern:
+
+- A `README.md` for each component
+- A `/docs` folder for the project
+
+Each `README.md` should contain:
+
+- A description of what the component is/does
+- A list of any prerequisites
+- Setup instructions
+- Execution instructions
+- Deployment instructions
+
+The `/docs` folder should contain:
+
+- A description of what the project is
+- An overview of how the everything fits together in the project
+- An explanation of the tech stack
+- Details of the underlying dataset
+
+A majority of the information should reside within the `/docs` directory over the `README`. The `README`s in this project should be kept for concise instructions on how to use each component. Any detailed explanation should be kept within `/docs`.
+
+## Getting MkDocs Setup
+
+In order to build an MkDocs deployment or serve the documentation locally, we need to install MkDocs and its dependencies.
+
+1. Navigate into the project's root directory.
+
+2. Install MkDocs and its dependencies.
+
+    ```bash
+    make install-docs
+    ```
+
+3. You can now use MkDocs. To see a list of commands run the following:
+
+    ```bash
+    mkdocs --help
+    ```
+
+## Updating MkDocs Deployment
+
+### GitHub Action to Deploy Documentation
+
+A GitHub Action is set up to automatically deploy the documentation to GitHub Pages whenever a commit is made to the `main` branch. This action is triggered by a push event to the `main` branch and runs the `mkdocs gh-deploy` command to build and deploy the documentation.
+
+### Manual Deployment
+
+If changes are made within `/docs`, the GitHub Pages deployment will need to be updated. Assuming you have already installed [MkDocs](https://www.mkdocs.org/getting-started/#installation) and [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/getting-started/#installation), do the following:
+
+1. Navigate to the projects root directory.
+
+2. Deploy the documentation to GitHub Pages.
+
+    ```bash
+    mkdocs gh-deploy
+    ```
+
+3. This will build the documentation and deploy it to the `gh-pages` branch of your repository. The documentation will be available at `https://ONS-Innovation.github.io/<repository-name>/`.
+
+**Please Note:** The `gh-deploy` command will overwrite the `gh-pages` branch and make the local changes available on GitHub Pages. Make sure that these changes are appropriate and have been reviewed before deployment.

docs/faq.md

@@ -0,0 +1,40 @@
+# Frequently Asked Questions (FAQs)
+
+## How does the Archive Process Work?
+
+- Each week, the archive tool will process each non-archived repository within ONSdigital.
+- For each repository, it will:
+  - Check if the repository hasn't been updated within the last year.
+  - If it hasn't been updated, it will create an issue in the repository to notify the maintainers. The issue will include details about how to avoid the repository being archived and how to archive the repository manually.
+  - If an issue already exists for the repository, it will check how long ago the issue was created.
+  - If the issue was created more than 30 days ago, the tool will archive the repository.
+
+In depth information on this process is available within the [Technical Documentation](./technical_documentation/the_process.md).
+
+## How Can I Prevent My Repository from Being Archived?
+
+To prevent your repository from being archived, you should push an update to the repository at least once a year. We recommend that an `ArchiveExemption.txt` or `ArchiveExemption.md` file is added to the root of the repository to indicate that it should not be archived. This file should contain a brief explanation of why the repository is still relevant and should not be archived.
+
+This file will need to be updated annually to ensure that the repository remains exempt from archiving. If you do not update this file, the repository will be archived after a year of inactivity.
+
+## What Happens if My Repository is Archived?
+
+When a repository is archived, it becomes read-only. This means that no further changes can be made to the repository, including issues or pull requests. The repository can still be viewed for reference, including any open issues or pull requests that were present at the time of archiving. If you need to make changes to an archived repository, you will need to unarchive it first.
+
+## How Do I Unarchive a Repository?
+
+In order to unarchive a repository, you need to have administrative access to the repository either being an organisation or repository owner. You can unarchive a repository by going to the repository settings and selecting the "Unarchive" option.
+
+To avoid the repository from being archived again, ensure that you push an update to the repository. This will make it exempt from archiving for another year. You may also want to add an `ArchiveExemption.txt` or `ArchiveExemption.md` file to the root of the repository to indicate that it should not be archived in the future.
+
+It is important to note that if a repository is without an owner or has no active admin users, an organisation admin will need to unarchive the repository.
+
+All repositories within ONSdigital should be maintained to have an up-to-date `CODEOWNERS` file and repository admins to ensure that access is available for unarchiving when necessary. This is specified within ONS' GitHub Usage Policy and must be adhered to.
+
+## Can I Archive My Own Repository?
+
+We encourage repository owners to archive their own repositories if they are no longer actively maintained or relevant. You can do this by going to the repository settings and selecting the "Archive" option. We recommend that you also add a notice to the repository's README file to inform users that the repository is archived and no longer maintained, along with closing any open issues or pull requests. This helps us align with GitHub's recommended practices for archiving repositories.
+
+## Further Questions
+
+For any further questions or concerns regarding the archiving process, please create an issue in the [Archive Tool repository](https://github.com/ONS-Innovation/github-repository-archive-script/issues) or contact an ONSdigital Owner.

docs/index.md

@@ -0,0 +1,15 @@
+# GitHub Repository Archive Script
+
+## Overview
+
+The Repository Archive Script is a Python script designed to automatically archive GitHub repositories that haven't been updated for a specified period of time. Archiving older repositories helps keep ONSdigital's GitHub organisation clean and organised, making it easier for users to find active projects, and reducing the organisation's footprint and maintenance overhead.
+
+## Frequently Asked Questions (FAQs)
+
+If you have any questions about the Repository Archive Script and its functionality, please refer to the [FAQs](./faq.md) section for more information.
+
+If this does not answer your question, please feel free to open an issue in the GitHub repository with the question label or contact an organisation owner.
+
+## Technical Documentation
+
+For information about the technical aspects of the Repository Archive Script, including its design, configuration, and logging, please refer to the [Technical Documentation](./technical_documentation/overview.md).