Skip to content

Commit f2f0882

Browse files
authored
Add Terraform configuration and IAM management scripts for GCP project (#35701)
* Add Terraform configuration and IAM management scripts for GCP project The idea is to setup the users and permissions for the GCP project using Terraform, allowing for easier management and automation of the infrastructure. This commit includes the following files: - `infra/.terraform.lock.hcl`: Lock file for Terraform dependencies. - `infra/README.md`: Documentation for the Terraform setup. - `infra/config.auto.tfvars`: Configuration variables for the Terraform project. - `infra/generate.py`: Python script to generate user configurations based on current users. - `infra/main.tf`: Main Terraform configuration file for the GCP project. - `infra/requirements.txt`: Python dependencies for the `generate.py` script. - `infra/users.tf`: Terraform configuration for managing GCP users. * Add GitHub Actions workflow to manage GCP User Roles with Terraform automatically * Move files to infra/iam directory * Update project variables and migrate * Discourage the use of generate.py script * Update readme after migration
1 parent d493275 commit f2f0882

9 files changed

Lines changed: 1043 additions & 0 deletions

File tree

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
# This workflow modifies the GCP User Roles when the infra/users.yml file is updated.
19+
# It applies the changes using Terraform to manage the IAM roles for users defined in the users.yml
20+
21+
name: Modify the GCP User Roles according to the infra/users.yml file
22+
23+
on:
24+
workflow_dispatch:
25+
# Trigger when the users.yml file is modified on the main branch
26+
push:
27+
branches:
28+
- main
29+
paths:
30+
- 'infra/iam/users.yml'
31+
32+
# This allows a subsequently queued workflow run to interrupt previous runs
33+
concurrency:
34+
group: '${{ github.workflow }} @ ${{ github.event.issue.number || github.sha || github.head_ref || github.ref }}-${{ github.event.schedule || github.event.comment.id || github.event.sender.login }}'
35+
cancel-in-progress: true
36+
37+
#Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
38+
permissions:
39+
contents: read
40+
41+
jobs:
42+
beam_UserRoles:
43+
name: Apply user roles changes
44+
runs-on: [self-hosted, ubuntu-20.04, main]
45+
timeout-minutes: 30
46+
steps:
47+
- uses: actions/checkout@v4
48+
- name: Setup gcloud
49+
uses: google-github-actions/setup-gcloud@v2
50+
- name: Install Terraform
51+
uses: hashicorp/setup-terraform@v3
52+
with:
53+
terraform_version: 1.12.2
54+
- name: Initialize Terraform
55+
working-directory: ./infra/iam
56+
run: terraform init
57+
- name: Terraform Plan
58+
working-directory: ./infra/iam
59+
run: terraform plan -out=tfplan
60+
- name: Terraform Apply
61+
working-directory: ./infra/iam
62+
run: terraform apply -auto-approve tfplan

infra/iam/.terraform.lock.hcl

Lines changed: 21 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

infra/iam/README.md

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# Infrastructure Permissions Management
21+
22+
This document outlines the structure of the Beam project control of infrastructure permissions and
23+
provides instructions on how to manage a user or role's permissions.
24+
25+
## Overview
26+
27+
### Managing User Roles
28+
29+
To manage user roles, edit the `users.yml` file. Add or modify entries under the `users` section to
30+
reflect the desired roles for each user. Remember to follow the YAML format:
31+
32+
```yaml
33+
users:
34+
- username: <username>
35+
email: <email>
36+
permissions:
37+
- role: <role>
38+
title: <title> (optional)
39+
description: <description> (optional)
40+
expiry_date: <expiry_date> (optional, format: YYYY-MM-DD)
41+
- role: <role> (optional, for multiple roles)
42+
```
43+
44+
> **Note**: `role/owner` roles are handled separately, adding them to the `users.yml` file will be ignored.
45+
46+
### Applying Changes
47+
48+
After modifying the `users.yml` file, open a Pull Request (PR) to the `infra/iam` directory.
49+
The changes will be reviewed and when approved, they will be merged into the main branch.
50+
51+
Once the PR is merged, [the GitHub Actions workflow](../../.github/workflows/beam_UserRoles.yml)
52+
will automatically trigger and apply the changes to the IAM policies in the GCP project using Terraform.
53+
54+
This will update the IAM policies in the GCP project based on the changes made in the `users.yml` file.
55+
56+
## Directory Structure
57+
58+
### Terraform Configuration
59+
60+
- **main.tf**: The main Terraform configuration file that defines the infrastructure resources and their permissions.
61+
- **config.auto.tfvars**: Contains the configuration variables for the Terraform project.
62+
- **users.tf**: Processes the `users.yml` file to associate users with their respective roles.
63+
- **users.yml**: A YAML file that contains the IAM policies and permissions for users and roles in the Beam project.

infra/iam/config.auto.tfvars

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
# This file is used for general configuration of the Terraform project,
19+
20+
# GCP Project ID
21+
project_id = "apache-beam-testing"

infra/iam/generate.py

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
# THIS IS NOT SUPPOSED TO RUN AFTER THE MIGRATION.
18+
# This script is used to export the IAM policy of a Google Cloud project to a YAML format.
19+
# It retrieves the IAM policy bindings, parses the members, and formats the output in a structured
20+
# YAML format, excluding service accounts and groups. The output includes usernames, emails, and
21+
# their associated permissions, with optional conditions for roles that have conditions attached.
22+
# You need to have the Google Cloud SDK installed and authenticated to run this script.
23+
24+
import argparse
25+
import datetime
26+
import yaml
27+
import logging
28+
from typing import Optional, List, Dict
29+
from google.cloud import resourcemanager_v3
30+
from google.api_core import exceptions
31+
32+
# Configure logging
33+
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
34+
logger = logging.getLogger(__name__)
35+
36+
def parse_member(member: str) -> tuple[str, Optional[str], str]:
37+
"""Parses an IAM member string to extract type, email, and a derived username.
38+
39+
Args:
40+
member: The IAM member string
41+
Returns:
42+
A tuple containing:
43+
- username: The derived username from the member string.
44+
- email: The email address if available, otherwise None.
45+
- member_type: The type of the member (e.g., user, serviceAccount, group).
46+
"""
47+
email = None
48+
username = member
49+
50+
# Split the member string to determine type and identifier
51+
parts = member.split(':', 1)
52+
member_type = parts[0] if len(parts) > 1 else "unknown"
53+
identifier = parts[1] if len(parts) > 1 else member
54+
55+
if member_type in ["user", "serviceAccount", "group"]:
56+
email = identifier
57+
if '@' in identifier:
58+
username = identifier.split('@')[0]
59+
else:
60+
username = identifier
61+
else:
62+
username = identifier
63+
email = None
64+
65+
return username, email, member_type
66+
67+
def export_project_iam(project_id: str) -> List[Dict]:
68+
"""Exports the IAM policy for a given project to YAML format.
69+
70+
Args:
71+
project_id: The ID of the Google Cloud project.
72+
Returns:
73+
A list of dictionaries containing the IAM policy details.
74+
"""
75+
76+
try:
77+
client = resourcemanager_v3.ProjectsClient()
78+
policy = client.get_iam_policy(resource=f"projects/{project_id}")
79+
logger.info(f"Successfully retrieved IAM policy for project {project_id}")
80+
except exceptions.NotFound as e:
81+
logger.error(f"Project {project_id} not found: {e}")
82+
raise
83+
except exceptions.PermissionDenied as e:
84+
logger.error(f"Permission denied for project {project_id}: {e}")
85+
raise
86+
except Exception as e:
87+
logger.error(f"An error occurred while retrieving IAM policy for project {project_id}: {e}")
88+
raise
89+
90+
members_data = {}
91+
92+
for binding in policy.bindings:
93+
role = binding.role
94+
95+
for member_str in binding.members:
96+
if member_str not in members_data:
97+
username, email_address, member_type = parse_member(member_str)
98+
if member_type == "serviceAccount":
99+
continue # Skip service accounts
100+
if member_type == "group":
101+
continue # Skip groups
102+
if not email_address:
103+
continue # Skip if no email address is found, probably a malformed member
104+
members_data[member_str] = {
105+
"username": username,
106+
"email": email_address,
107+
"permissions": []
108+
}
109+
110+
# Skip permissions that have a condition
111+
if "withcond" in role:
112+
continue
113+
114+
permission_entry = {}
115+
permission_entry["role"] = role
116+
117+
members_data[member_str]["permissions"].append(permission_entry)
118+
119+
output_list = []
120+
for data in members_data.values():
121+
data["permissions"] = sorted(data["permissions"], key=lambda p: p["role"])
122+
output_list.append({
123+
"username": data["username"],
124+
"email": data["email"],
125+
"permissions": data["permissions"]
126+
})
127+
128+
output_list.sort(key=lambda x: x["username"])
129+
return output_list
130+
131+
def to_yaml_file(data: List[Dict], output_file: str, header_info: str = "") -> None:
132+
"""
133+
Writes a list of dictionaries to a YAML file.
134+
Include the apache license header on the files
135+
136+
Args:
137+
data: A list of dictionaries containing user permissions and details.
138+
output_file: The file path where the YAML output will be written.
139+
header_info: A string containing the header information to be included in the YAML file.
140+
"""
141+
142+
apache_license_header = """#
143+
# Licensed to the Apache Software Foundation (ASF) under one or more
144+
# contributor license agreements. See the NOTICE file distributed with
145+
# this work for additional information regarding copyright ownership.
146+
# The ASF licenses this file to You under the Apache License, Version 2.0
147+
# (the "License"); you may not use this file except in compliance with
148+
# the License. You may obtain a copy of the License at
149+
#
150+
# http://www.apache.org/licenses/LICENSE-2.0
151+
#
152+
# Unless required by applicable law or agreed to in writing, software
153+
# distributed under the License is distributed on an "AS IS" BASIS,
154+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
155+
# See the License for the specific language governing permissions and
156+
# limitations under the License.
157+
"""
158+
159+
# Prepare the header with the Apache license
160+
header = f"{apache_license_header}\n# {header_info}\n# Generated on {datetime.datetime.now(datetime.timezone.utc).strftime('%Y-%m-%d %H:%M:%S')} UTC\n\n"
161+
162+
try:
163+
with open(output_file, "w") as file:
164+
file.write(header)
165+
yaml.dump(data, file, sort_keys=False, default_flow_style=False, indent=2)
166+
logger.info(f"Successfully wrote IAM policy data to {output_file}")
167+
except IOError as e:
168+
logger.error(f"Failed to write to {output_file}: {e}")
169+
raise
170+
171+
def main():
172+
"""
173+
Main function to run the script.
174+
175+
This function parses command-line arguments to either export IAM policies
176+
or generate permission differences for a specified GCP project.
177+
"""
178+
parser = argparse.ArgumentParser(
179+
description="Export IAM policies or generate permission differences for a GCP project."
180+
)
181+
parser.add_argument(
182+
"project_id",
183+
help="The Google Cloud project ID."
184+
)
185+
parser.add_argument(
186+
"output_file",
187+
help="Defaults to 'users.yml' if not specified. The file where the IAM policy will be saved in YAML format.",
188+
nargs='?',
189+
default="users.yml"
190+
)
191+
parser.add_argument(
192+
"--yes-i-know-what-i-am-doing",
193+
action="store_true",
194+
help="If set, the script will proceed"
195+
)
196+
197+
args = parser.parse_args()
198+
project_id = args.project_id
199+
output_file = args.output_file
200+
201+
if not args.yes_i_know_what_i_am_doing:
202+
logger.error("You must use the --yes-i-know-what-i-am-doing flag to proceed.")
203+
return
204+
205+
# Export the IAM policy for the specified project
206+
iam_data = export_project_iam(project_id)
207+
208+
# Write the exported data to the specified output file in YAML format
209+
to_yaml_file(iam_data, output_file, header_info=f"Exported IAM policy for project {project_id}")
210+
211+
if __name__ == "__main__":
212+
main()

0 commit comments

Comments
 (0)