Skip to content

Commit 978ad49

Browse files
Merge branch 'topic-release' into topic-release
2 parents 01962e9 + c90c8f3 commit 978ad49

28 files changed

Lines changed: 1529 additions & 115 deletions

README.md

Lines changed: 32 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -25,21 +25,36 @@ It provides a pre-built catalog of products which are ready to use out of the bo
2525

2626
| Virtual Machine Purpose | Virtual Machine Spec |
2727
| ---------------------------------- | ------------------------------------- |
28-
| Role: Portal | t2.large 2CPU, 8GB RAM, 100GB Disk |
28+
| Role: Portal | t3.large 2CPU, 8GB RAM, 100GB Disk |
2929
| Role: DB AWS DocumentDB | db.t3.large (dev) db.r5.large+ (prod) |
3030

31-
#### 2. Network Requirements
31+
#### 2. ACM or External Certificates for SSL
32+
33+
*Create a certificate for your domain*
34+
35+
To make the Research Gateway application available securely over SSL, you need a certificate issued by a public Certificate Authority (CA).
36+
If you already have a certificate for your domain issued by a third-party CA, you can import it into ACM. [See how](https://docs.aws.amazon.com/acm/latest/userguide/import-certificate.html).
37+
38+
If you already have a certificate for your domain issued by AWS in ACM, you can pass CertificateArn with launch stack ( #7 - #9) and skip the following step.
39+
40+
aws acm request-certificate --domain-name www.example.com --validation-method DNS --idempotency-token 1234 --options CertificateTransparencyLoggingPreference=DISABLED
41+
42+
You will need to validate your ownership of the domain either via DNS (recommended) or via email.
43+
44+
Fore more details on requesting a certificate, follow this [link](https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-request-public.html)
45+
46+
#### 3. Network Requirements
3247

3348
1. VPC
3449
2. 3 Public Subnets
3550
3. 3 Private Subnets
3651
4. IGW
3752
5. NAT Instance / NAT Gateway
38-
6. Bastion Hosts (optional)
53+
6. Bastion Hosts (Optional)
3954
7. Application Load Balancer
4055
8. Listener
41-
9. ACM or External Certificates for SSL
42-
10. Target Group
56+
9. Target Group
57+
10. Route 53
4358

4459
<!-- trunk-ignore(markdownlint/MD036) -->
4560
*Create #1 - #5 above using the following quick-start*
@@ -49,68 +64,36 @@ It provides a pre-built catalog of products which are ready to use out of the bo
4964

5065

5166
<!-- trunk-ignore(markdownlint/MD036) -->
52-
*Create an Application Load Balancer*
53-
54-
Deploying an Application Load Balancer as part of Research Gateway deployment helps in two ways:
55-
1. Isolates your portal from being directly exposed over the internet. The ALB allows only https(s) traffic through.
56-
2. Helps to serve the application on a secure port using SSL certificates stored in AWS ACM.
57-
58-
create an ALB security Group with the following inbound and outbound rules.
59-
- Inbound rules - HTTP 80 ,HTTPS 443,SSH 22 .
60-
- outbound Rules - All Traffic.
61-
62-
Use the AWS CLI to create an Application Load Balancer choosing all three public subnets created by the quickstart above.
63-
64-
aws elbv2 create-load-balancer --name research-gw-alb --subnets subnet-abcd1234 subnet-abcd5678 subnet-abcd9876 --security-groups sg-abcd1234 --region us-east-1
65-
The output of the command above will include the ARN of the loadbalancer. e.g. arn:aws:elasticloadbalancing:region:aws_account_id:loadbalancer/app/research-gw-alb/e5ba62739c16e642
66-
67-
*Create a target group*
68-
69-
aws elbv2 create-target-group --name tg-research-gw --protocol HTTP --port 80 --target-type instance --vpc-id vpc-abcd1234 --region us-east-1
70-
The output of the command will include the ARN of the target group. e.g. arn:aws:elasticloadbalancing:region:aws_account_id:targetgroup/tg-research-gw/209a844cd01825a4
7167

68+
<!-- trunk-ignore(markdownlint/MD036) -->
69+
*Create #7 - #9 above using the following quick-start*
7270

73-
*Create a certificate for your domain*
74-
75-
To make the Research Gateway application available securely over SSL, you need a certificate issued by a public Certificate Authority (CA).
76-
If you already have a certificate for your domain issued by a third-party CA, you can import it into ACM. [See how](https://docs.aws.amazon.com/acm/latest/userguide/import-certificate.html).
77-
If you already have a certificate for your domain issued by AWS in ACM, you can skip the following step.
78-
79-
aws acm request-certificate --domain-name www.example.com --validation-method DNS --idempotency-token 1234 --options CertificateTransparencyLoggingPreference=DISABLED
80-
81-
You will need to validate your ownership of the domain either via DNS (recommended) or via email.
82-
83-
Fore more details on requesting a certificate, follow this [link](https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-request-public.html)
84-
85-
*Create a listener on port 443 *
8671

87-
aws elbv2 create-listener --load-balancer-arn arn:aws:elasticloadbalancing:region:aws_account_id:loadbalancer/app/research-gw-alb/e5ba62739c16e642 --protocol HTTPS --port 443 --certificates CertificateArn=arn:aws:acm:us-east-1:aws_account_id:certificate/480cdfa8-bac6-4b99-977f-5f18441de49e --ssl-policy ELBSecurityPolicy-2016-08 --default-actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:region:aws_account_id:targetgroup/tg-research-gw/209a844cd01825a4 --region us-east-1
72+
[![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/create/review?templateURL=https://rglaunch-stack.s3.us-east-2.amazonaws.com/rg-alb-route53.yml)
8873

89-
*Create a listener on port 80 *
9074

91-
Note: This is only recommended for test setups and some of the features like secure links for resources will not work.
75+
<!-- trunk-ignore(markdownlint/MD036) -->
9276

93-
aws elbv2 create-listener --load-balancer-arn arn:aws:elasticloadbalancing:region:aws_account_id:loadbalancer/app/research-gw-alb/e5ba62739c16e642 --protocol HTTP --port 80 --default-actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:region:aws_account_id:targetgroup/tg-research-gw/209a844cd01825a4
77+
*Note* - Create #7 - #9 above using AWS CLI commands also .For that you refer to the rg_alb-tg-creation.md file from repo
9478

95-
<!-- trunk-ignore(markdownlint/MD036) -->
96-
*Set up Route 53 for your domain/sub-domain*
79+
*10.Set up Route 53 for your domain/sub-domain*
9780

9881
From the AWS Console, select Route 53 service. Create a hosted zone for your domain. If you are using an existing domain you can add a CNAME record and select the public DNS name of the ALB as the value. It is also possible to create a separate hosted zone for the sub-domain used for Research Gateway.
9982

10083
*Note* - It is possible to use Research Gateway without a domain-name but we do not recommend that for product workloads. To do so, use the public DNS name of the ALB as the URL of the Research Gateway in the setup scripts.
10184

102-
#### 3. Software Requirements
85+
#### 4. Software Requirements
10386

10487
| Virtual Machine Purpose | Software pre-requisites |
10588
| --------------------------- | ----------------------- |
10689
| Role: Portal | Python, Docker 20.04+ |
10790
| Role: DB (Option 1) MongoDB | MongoDB 3.6.23 |
108-
| | |
91+
10992

11093
The application software for Research Gateway will be made available to you as a docker image shared from Relevance Lab's Elastic Container Registry instance to your AWS account.
11194
As a part of this deployment, you will create an AMI for the portal EC2 instance which will have these softwares pre-deployed. Alternately, you can request the AMI to be shared with you by Relevance Lab and the software above will be available pre-deployed on the AMI shared with you.
11295

113-
#### 4. AWS Services required
96+
#### 5. AWS Services required
11497

11598
- AWS Cognito
11699
- Amazon S3
@@ -133,28 +116,9 @@ The following sofware needs to be installed on the Portal EC2 instance
133116
| jq | latest |
134117
| zip | latest |
135118

136-
For your convenience we have created packer scripts which allow you to create the AMI in your account.
137-
If an AMI with the pre-requisites has been shared with you, you can skip this section.
119+
For your convenience we have created packer scripts which allow you to create the AMI in your account For that you refer to the rg_AMI-creation.md file from repo
138120

139-
### Creating the AMI with pre-requisites
140-
141-
You can create the AMI with pre-requisites yourself by following these steps:
142-
143-
- [Install packer](https://learn.hashicorp.com/tutorials/packer/get-started-install-cli)
144-
- Export AWS Access Keys and Secret Keys
145-
146-
export AWS_ACCESS_KEY_ID="your_Access_Key"
147-
export AWS_SECRET_ACCESS_KEY="your_Secret_Key"
148-
export AWS_DEFAULT_REGION="Your_Region"
149-
150-
- Clone this repo on a machine.
151-
- Target Account number must be added in Admin Account-ECR and give permission to access image builds
152-
- Create a Role and attach a policy which permits ECR and EC2 actions and Replace the "iam-instance_profile" :"<your_rolename>" in builders section which is in the packer-rg.json.
153-
- Run packer build packer-rg.json
154-
155-
packer build -var 'awsRegion=your_region' -var 'vpcId=your_VPCID' -var 'subnetId=your_SubnetID' packer-rg.json
156-
- At Run time pass VPCID, SubnetID, AWSRegion as variables declared in packer-rg.json
157-
- Note that AMI id from the output
121+
If an AMI with the pre-requisites has been shared with you, you can skip this section
158122

159123
### Installing Research Gateway
160124

@@ -192,7 +156,7 @@ runid.json is created in the rgdeploy folder when you first run deploy.sh with p
192156

193157
The deployment creates EC2 Image Builder pipelines for building the RStudio and Nextflow AMIs that are used within Research Gateway. By default, these pipelines are set up to be manually triggered. You can change that in the AWS console if you wish to trigger them on a schedule.
194158

195-
steps to run pipelines: AWS console - Ec2imagebuilder – select image pipelines (Rstudio, Nextflow)-click on Actions-Run pipeline
159+
steps to run pipelines: AWS Console - EC2Imagebuilder – select image pipelines (Rstudio, Nextflow)-click on Actions- Run pipeline
196160

197161
Once a build is completed, the AMIs are automatically distributed to the regions supported by Research Gateway in your account. The AMI Ids need to be updated into your database before creating any projects.
198162

cft-templates/EFS.yaml

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
AWSTemplateFormatVersion: 2010-09-09
2+
3+
Description: Sets up EFS on AWS
4+
Metadata:
5+
AWS::CloudFormation::Interface:
6+
ParameterGroups:
7+
- Label:
8+
default: File system settings
9+
Parameters:
10+
- BackupPolicy
11+
- VpcId
12+
- SubnetId
13+
- CidrIp
14+
- Label:
15+
default: Lifecycle management
16+
Parameters:
17+
- TransitionIntoIA
18+
- TransitionToPrimaryStorageClass
19+
- PerformanceMode
20+
- ThroughputMode
21+
- ProvisionedThroughputInMibps
22+
- Label:
23+
default: Encryption
24+
Parameters:
25+
- Encrypted
26+
- KmsKeyId
27+
28+
Parameters:
29+
BackupPolicy:
30+
Description: Automatically backup your file system data with AWS Backup using recommended settings. Additional pricing applies.
31+
Type: String
32+
Default: DISABLED
33+
AllowedValues: [ENABLED,DISABLED]
34+
VpcId:
35+
Description: The ID of the VPC that the file system will be accessible from
36+
Type: AWS::EC2::VPC::Id
37+
SubnetId:
38+
Description: The subnet where to launch the service
39+
Type: AWS::EC2::Subnet::Id
40+
CidrIp:
41+
Description: The IP address range that can be used to access the file system.
42+
Type: String
43+
MinLength: '9'
44+
MaxLength: '18'
45+
Default: 0.0.0.0/0
46+
AllowedPattern: (\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})/(\d{1,2})
47+
ConstraintDescription: must be a valid IP CIDR range of the form x.x.x.x/x.
48+
TransitionIntoIA:
49+
Description: Migrates files that have not been accessed for a set period of time to the EFS Standard–Infrequent Access (Standard-IA) or One Zone–Infrequent Access (One Zone-IA) storage class, depending on your file system.
50+
Type: String
51+
Default: AFTER_60_DAYS
52+
AllowedValues: [AFTER_7_DAYS,AFTER_14_DAYS,AFTER_60_DAYS,AFTER_90_DAYS]
53+
TransitionToPrimaryStorageClass:
54+
Description: Automatically moves files back to the EFS Standard or EFS One Zone storage classes when the TransitionToPrimaryStorageClass lifecycle policy is set to On first access. This helps to eliminate the risk of unbounded access charges, while providing consistent low latencies.
55+
Type: String
56+
Default: NONE
57+
AllowedValues: [AFTER_1_ACCESS, NONE]
58+
PerformanceMode:
59+
Description: >-
60+
Amazon EFS delivers more than 10 gibibytes per second (GiBps) of throughput over 500,000 IOPS, and sub-millisecond or low single digit millisecond latencies.
61+
General Purpose mode supports up to 35,000 IOPS and has the lowest per-operation latency. File systems with EFS One Zone storage classes always use General Purpose performance mode.
62+
Max I/O mode supports 500,000+ IOPS and has higher per-operation latencies when compared to General Purpose mode.
63+
Type: String
64+
Default: generalPurpose
65+
AllowedValues: [generalPurpose, maxIO]
66+
ThroughputMode:
67+
Description: >-
68+
determines the throughput available to your file system.
69+
Bursting Throughput mode is a good fit for traditional applications that have a bursty throughput pattern.
70+
In Provisioned Throughput mode, you specify a level of throughput that the file system can drive independent of the file system's size or burst credit balance.
71+
Type: String
72+
Default: bursting
73+
AllowedValues: [bursting, provisioned]
74+
ProvisionedThroughputInMibps:
75+
Description: >-
76+
The amount of provisioned throughput, measured in MiB/s, for the file system. Valid for file systems using ThroughputMode set to provisioned.
77+
Valid range is 1-1024 MiB/s
78+
Type: Number
79+
Default: 1
80+
Encrypted:
81+
Description: Choose to enable encryption of your file system's data at rest. Uses the AWS KMS service key (aws/elasticfilesystem) by default
82+
Type: String
83+
Default: false
84+
AllowedValues: [true, false]
85+
KmsKeyId:
86+
Description: The ID of the AWS Key Management Service (AWS KMS) key used to encrypt Amazon EFS file system data.
87+
Type: String
88+
Default: NONE
89+
ResearcherName:
90+
Description: Researcher name that will be added as tag to the resources
91+
Type: String
92+
ProjectId:
93+
Description: Project Id that will be added as tag to the resources
94+
Type: String
95+
Namespace:
96+
Description: Name your file system. Name can include letters, numbers, and +-=._:/ symbols, up to 256 characters.
97+
Type: String
98+
99+
Conditions:
100+
burstingThroughputMode: !Equals [!Ref ThroughputMode ,bursting]
101+
KMSKeyNotProvided: !Equals [!Ref KmsKeyId, 'NONE']
102+
103+
Resources:
104+
EFSSecurityGroup:
105+
Type: "AWS::EC2::SecurityGroup"
106+
Properties:
107+
VpcId: !Ref VpcId
108+
SecurityGroupIngress:
109+
- CidrIp: !Ref CidrIp
110+
FromPort: 2049
111+
ToPort: 2049
112+
IpProtocol: tcp
113+
GroupDescription: EFS Security Group
114+
115+
FileSystem:
116+
Type: AWS::EFS::FileSystem
117+
Properties:
118+
BackupPolicy:
119+
Status: !Ref BackupPolicy
120+
Encrypted: !Ref Encrypted
121+
KmsKeyId: !If [KMSKeyNotProvided, !Ref 'AWS::NoValue', !Ref KmsKeyId]
122+
LifecyclePolicies:
123+
- TransitionToIA: !Ref TransitionIntoIA
124+
- TransitionToPrimaryStorageClass: AFTER_1_ACCESS
125+
PerformanceMode: !Ref PerformanceMode
126+
ThroughputMode: !Ref ThroughputMode
127+
ProvisionedThroughputInMibps: !If [burstingThroughputMode, !Ref 'AWS::NoValue', !Ref ProvisionedThroughputInMibps]
128+
129+
FileSystemTags:
130+
- Key: Name
131+
Value: !Ref Namespace
132+
- Key: cost_resource
133+
Value: !Sub ${AWS::StackName}
134+
- Key: researcher_name
135+
Value: !Ref ResearcherName
136+
- Key: project_name
137+
Value: !Ref ProjectId
138+
- Key: Name
139+
Value: !Join ['-', [Ref: Namespace, 'file-system']]
140+
141+
MountTarget:
142+
Type: AWS::EFS::MountTarget
143+
Properties:
144+
FileSystemId: !Ref FileSystem
145+
SubnetId: !Ref SubnetId
146+
SecurityGroups:
147+
- !Ref EFSSecurityGroup
148+
Outputs:
149+
EFS:
150+
Description: The created EFS filesystem ID
151+
Value: !Ref FileSystem
152+
MountTarget:
153+
Description: MountTarget for the created EFS filesystem.
154+
Value: !Ref MountTarget

0 commit comments

Comments
 (0)