Merge pull request #79 from RLOpenCatalyst/topic-release

1.11.5 Release updates

Author: RLIndia

README.md

@@ -35,7 +35,7 @@ It provides a pre-built catalog of products which are ready to use out of the bo
  3. 3 Private Subnets  
  4. IGW  
  5. NAT Instance / NAT Gateway  
- 6. Bastion Hosts  
+ 6. Bastion Hosts  (optional)
  7. Application Load Balancer  
  8. Listener  
  9. ACM or External Certificates for SSL  
@@ -55,6 +55,10 @@ Deploying an Application Load Balancer as part of Research Gateway deployment he
 1. Isolates your portal from being directly exposed over the internet. The ALB allows only https(s) traffic through.
 2. Helps to serve the application on a secure port using SSL certificates stored in AWS ACM.
 
+create an ALB security Group with the following inbound and outbound rules.
+ - Inbound rules - HTTP 80 ,HTTPS 443,SSH 22 .
+ - outbound Rules - All Traffic.
+
 Use the AWS CLI to create an Application Load Balancer choosing all three public subnets created by the quickstart above.
 
     aws elbv2 create-load-balancer --name research-gw-alb --subnets subnet-abcd1234 subnet-abcd5678 subnet-abcd9876 --security-groups sg-abcd1234 --region us-east-1
@@ -115,6 +119,8 @@ As a part of this deployment, you will create an AMI for the portal EC2 instance
 - AWS ImageBuilder
 - AWS EC2
 - AWS IAM
+- AWS service catalog
+- Elastic container Registery
 
 ## Installing the required 3rd party software
 
@@ -142,6 +148,7 @@ You can create the AMI with pre-requisites yourself by following these steps:
       export AWS_DEFAULT_REGION="Your_Region"
 
 - Clone this repo on a machine.
+- Target Account number must be added in Admin Account-ECR and give permission to access image builds
 - Create a Role and attach a policy which permits ECR and EC2 actions and Replace the "iam-instance_profile" :"<your_rolename>" in builders section which is in the packer-rg.json.
 - Run packer build packer-rg.json
 
@@ -152,7 +159,15 @@ You can create the AMI with pre-requisites yourself by following these steps:
 ### Installing Research Gateway
 
 Clone this repo on a machine that has AWS CLI configured with Default output format as JSON.
-Run deploy.sh with the following parameters
+Run deploy.sh with the following parameters.
+
+ *Note* : Check aws configure before running script
+  - $aws configure
+      - AWS Access Key ID:"your_Access_Key"
+      - AWS Secret Access Key :"your_Secret_Key"
+      - Default region name:"Your_Region"
+      - Default output format : json
+
 
 | Parameter# | Purpose                                                                                    |
 | ---------- | ------------------------------------------------------------------------------------------ |
@@ -177,6 +192,8 @@ runid.json is created in the rgdeploy folder when you first run deploy.sh with p
 
 The deployment creates EC2 Image Builder pipelines for building the RStudio and Nextflow AMIs that are used within Research Gateway. By default, these pipelines are set up to be manually triggered. You can change that in the AWS console if you wish to  trigger them on a schedule.
 
+steps to run pipelines: AWS console - Ec2imagebuilder – select image pipelines (Rstudio, Nextflow)-click on Actions-Run pipeline
+
 Once a build is completed, the AMIs are automatically distributed to the regions supported by Research Gateway in your account. The AMI Ids need to be updated into your database before creating any projects. 
 
 - Note down the names of the two pipelines created for RStudio and Nextflow_Advanced. They will be of the format:
@@ -186,6 +203,8 @@ The runid will be the random 4-character string generated for your instance duri
 - In the rgdeploy folder, cd to products folder. You will find an img-builder-config.json file there. Edit it and set the pipeline names according to the ones deployed in your account. Save the file.
 - Run the script make-amilist.sh. You may have to run chmod +x make-amilist.sh if execute permissions are not set on the file.
 
+- Note: we need to wait untill ec2image builder pipeline distribution complete,without builds complete by running below command shows Error
+
       ./make-amilist.sh > new-ami-list.json
 - Next run the following command to update your DB.
 

cft-templates/Rstudio.yml

@@ -19,15 +19,15 @@ Parameters:
       the environment instance, including any bootstrap scripts
   InstanceType:
     Type: String
-    Description: Instance type for RStudio. Default is t2.medium.
+    Description: Instance type for RStudio. Default is t3.medium.
     AllowedValues:
       - t2.medium
       - t2.large
       - t3.medium
       - t3.large
 
-    ConstraintDescription: Valid instance type in the t2 families
-    Default: t2.medium
+    ConstraintDescription: Valid instance type in the t3 families
+    Default: t3.medium
   InitialUser:
     Type: String
     Description: User Name for RStudio. Do not use 'root' and 'ec2-user'

cft-templates/cromwell-advanced.yaml

@@ -82,8 +82,8 @@ Parameters:
   InstanceType:
     Description: Head Node EC2 instance type
     Type: String
-    Default: t2.small
-    AllowedValues: [t2.nano, t2.micro, t2.small, t2.medium]
+    Default: t3.small
+    AllowedValues: [t3.nano, t3.micro, t3.small, t3.medium]
     ConstraintDescription: Must be a valid EC2 instance type.
   WorkerNodeInstanceType:
     Description: Specify the instance types to be used to carry out the computation. You can specify one or more family or instance type. The option 'optimal' chooses the best fit of M4, C4, and R4 instance types available in the region.
@@ -932,4 +932,6 @@ Outputs:
       EC2 Launch Template ID to use when creating AWS Batch compute environments
     Value: !Ref EC2LaunchTemplate
   WorkDataLocation:
-    Value: !Sub ${S3Bucket}
+    Value: !Sub ${S3Bucket}
+  BatchComputeEnvironment:
+    Value: !Ref SpotComputeEnv

cft-templates/ec2-EIP.yml

@@ -21,10 +21,10 @@ Parameters:
       An S3 URI (starting with "s3://") that specifies the location of files to be copied to
       the environment instance, including any bootstrap scripts
   InstanceType:
-    Description: Choose the instance type for this instance. e.g. t2.small
+    Description: Choose the instance type for this instance. e.g. t3.small
     Type: String
-    Default: t2.small
-    AllowedValues: [t2.nano, t2.micro, t2.small, t2.medium]
+    Default: t3.small
+    AllowedValues: [t3.nano, t3.micro, t3.small, t3.medium]
     ConstraintDescription: must be a valid EC2 instance type.
   KeyPair:
     Description: Name of an existing EC2 KeyPair to enable SSH access to the instance. If no key pairs exist, please create one from the button next to the dropdown. Please contact your Administrator if you are unable to create one.

cft-templates/ec2-linux-docker-mysql.yml

@@ -0,0 +1,171 @@
+Metadata: 
+  License: Apache-2.0
+AWSTemplateFormatVersion: '2010-09-09'
+Description: 'AWS CloudFormation Template to create an EC2 instance
+  **WARNING** This template creates an Amazon EC2 instance and an Elastic IP Address.
+  You will be billed for the AWS resources used if you create a stack from this template.'
+
+Parameters:
+  Namespace:
+    Type: String
+    Description: An environment name that will be prefixed to resource names
+  S3Mounts:
+    Type: String
+    Description: A JSON array of objects with name, bucket, and prefix properties used to mount data
+  IamPolicyDocument:
+    Type: String
+    Description: The IAM policy to be associated with the launched workstation
+  EnvironmentInstanceFiles:
+    Type: String
+    Description: >-
+      An S3 URI (starting with "s3://") that specifies the location of files to be copied to
+      the environment instance, including any bootstrap scripts
+  InstanceType:
+    Description: Choose the instance type for this instance. e.g. t3.small
+    Type: String
+    Default: t3.small
+    AllowedValues: [t3.nano, t3.micro, t3.small, t3.medium]
+    ConstraintDescription: must be a valid EC2 instance type.
+  KeyPair:
+    Description: Name of an existing EC2 KeyPair to enable SSH access to the instance. If no key pairs exist, please create one from the button next to the dropdown. Please contact your Administrator if you are unable to create one.
+    Type: AWS::EC2::KeyPair::KeyName
+    ConstraintDescription: must be the name of an existing EC2 KeyPair.
+  AllowedSSHLocation:
+    Description: The IP address range that can be used to SSH to the EC2 instances
+    Type: String
+    MinLength: '9'
+    MaxLength: '18'
+    Default: 0.0.0.0/0
+    AllowedPattern: (\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})/(\d{1,2})
+    ConstraintDescription: must be a valid IP CIDR range of the form x.x.x.x/x.
+  LatestAmiId:
+    Type:  'AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>'
+    Default: '/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id'
+  MasterPassword:
+    NoEcho: "true"
+    Description : "The database master password"
+    Type: "String"
+    MinLength: "8"
+    MaxLength: "100"
+    AllowedPattern : "[a-zA-Z0-9]{8,100}"
+    ConstraintDescription : "must contain only alphanumeric characters."
+  ConnectFromPort:
+    Type: Number
+    Description: Required Port mappings
+  ConnectToPort:
+    Type: Number
+    Description: Required Port mappings 
+
+Conditions:
+  IamPolicyEmpty: !Equals [!Ref IamPolicyDocument, '{}']
+
+Resources:  
+  IAMRole:
+    Type: 'AWS::IAM::Role'
+    Properties:
+      RoleName: !Join ['-', [Ref: Namespace, 'ec2-role']]
+      Path: '/'
+      AssumeRolePolicyDocument:
+        Version: '2012-10-17'
+        Statement:
+          - Effect: 'Allow'
+            Principal:
+              Service:
+                - 'ec2.amazonaws.com'
+            Action:
+              - 'sts:AssumeRole'
+      Policies:
+        - !If
+          - IamPolicyEmpty
+          - !Ref 'AWS::NoValue'
+          - PolicyName: !Join ['-', [Ref: Namespace, 's3-studydata-policy']]
+            PolicyDocument: !Ref IamPolicyDocument
+
+  InstanceProfile:
+    Type: 'AWS::IAM::InstanceProfile'
+    Properties:
+      InstanceProfileName: !Join ['-', [Ref: Namespace, 'ec2-profile']]
+      Path: '/'
+      Roles:
+        - Ref: IAMRole
+
+  EC2Instance:
+    Type: AWS::EC2::Instance
+    CreationPolicy:
+      ResourceSignal:
+        Timeout: PT5M
+    Properties:
+      UserData:
+        Fn::Base64: !Sub |
+          #!/usr/bin/env bash
+          # pull mysql version 8
+          sudo yum install zip -y
+          sudo yum install unzip -y
+          # Install AWS CLI version2
+          curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
+          unzip awscliv2.zip
+          sudo ./aws/install
+          # Install Mysql Shell
+          sudo yum install mysql -y
+          # Pull Mysql Image
+          docker pull mysql:8
+          mkdir docker
+          mkdir docker/mysql
+          mkdir docker/mysql/conf
+          mkdir docker/mysql/data
+          docker run -d -p 3306:3306 \
+          -v /home/ec2-user/docker/mysql/conf/:/etc/mysql/conf.d \
+          -v /home/ec2-user/docker/mysql/data/:/var/lib/mysql \
+          -e MYSQL_ROOT_PASSWORD=${MasterPassword} \
+          --restart always \
+          --name docker_mysql mysql:8 \
+          # Download and execute shell script
+          cd /home/ec2-user
+          aws s3 cp "${EnvironmentInstanceFiles}/alter_rootpassword.sh" "alter_rootpassword.sh"
+          chmod +x alter_rootpassword.sh
+          ./alter_rootpassword.sh ${MasterPassword}
+          # Install cfn
+          yum install -y aws-cfn-bootstrap
+          # Download and execute shell script
+          aws s3 cp "${EnvironmentInstanceFiles}/get_bootstrap_mysql.sh" "/tmp"
+          chmod 500 "/tmp/get_bootstrap_mysql.sh"
+          /tmp/get_bootstrap_mysql.sh "${EnvironmentInstanceFiles}" '${S3Mounts}'
+          # Signal result to CloudFormation
+          /opt/aws/bin/cfn-signal --exit-code 0 --resource EC2Instance --region ${AWS::Region} --stack ${AWS::StackName}
+      InstanceType: !Ref 'InstanceType'
+      SecurityGroups: [!Ref 'InstanceSecurityGroup']
+      KeyName: !Ref 'KeyPair'
+      ImageId: !Ref 'LatestAmiId'
+      IamInstanceProfile: !Ref InstanceProfile
+      Tags:
+        - Key: Name
+          Value: !Join ['-', [Ref: Namespace, 'ec2-linux']]
+        - Key: Description
+          Value: EC2 workspace instance
+  
+  InstanceSecurityGroup:
+    Type: AWS::EC2::SecurityGroup
+    Properties:
+      GroupDescription: Enable SSH access
+      SecurityGroupIngress:
+      - IpProtocol: tcp
+        FromPort: '22'
+        ToPort: '22'
+        CidrIp: !Ref 'AllowedSSHLocation'
+      - IpProtocol: tcp
+        FromPort: !Ref 'ConnectFromPort'
+        ToPort: !Ref 'ConnectToPort'
+        CidrIp: !Ref 'AllowedSSHLocation'
+Outputs:
+  InstanceId:
+    Description: InstanceId of the newly created EC2 instance
+    Value: !Ref 'EC2Instance'
+  InstanceIPAddress:
+    Description: IP address of the newly created EC2 instance
+    Value: !GetAtt [EC2Instance, PublicIp]
+  InstanceDNSName:
+    Description: DNS name of the newly created EC2 instance
+    Value: !GetAtt [EC2Instance, PublicDnsName]
+  InstancePrivateIPAddress:
+    Description: Private IP address of the newly created EC2 instance
+    Value: !GetAtt [EC2Instance, PrivateIp]