Example to create multiple instances

[vsoch]

Describe the Feature

I'd like to know how to modify this example https://github.com/cloudposse/terraform-aws-ec2-instance/blob/master/examples/complete/main.tf for multiple EC2 instances

Expected Behavior

NA

Use Case

Bringing up a small networked cluster

Describe Ideal Solution

A similar example in the examples folder for multiple networked EC2 instances from custom AMI.

Alternatives Considered

No response

Additional Context

No response

[jamengual]

If you need instances not on an autoscaling group, you can use a for_each or count on the module.

if you can put them on an ASG then use the asg module.

[vsoch]

okay let me put this together - so I'd take this part of that spec:

module "ec2_instance" {
  source = "../../"

  ssh_key_pair                = module.aws_key_pair.key_name
  vpc_id                      = module.vpc.vpc_id
  subnet                      = module.subnets.private_subnet_ids[0]
  security_groups             = [module.vpc.vpc_default_security_group_id]
  assign_eip_address          = var.assign_eip_address
  associate_public_ip_address = var.associate_public_ip_address
  instance_type               = var.instance_type
  security_group_rules        = var.security_group_rules
  instance_profile            = aws_iam_instance_profile.test.name
  tenancy                     = var.tenancy

  context = module.this.context
}

and reading about for_each I would do something like this?

module "ec2_instance" {
  source = "../../"
  # This would be a 4 node cluster
  for_each = toset( ["0", "1", "2", "3"] )
 
 # Where would I specify a name or a hostname?
  name = "instance-${each.key}"
  ssh_key_pair                = module.aws_key_pair.key_name
  vpc_id                      = module.vpc.vpc_id
  subnet                      = module.subnets.private_subnet_ids[0]
  security_groups             = [module.vpc.vpc_default_security_group_id]
  assign_eip_address          = var.assign_eip_address
  associate_public_ip_address = var.associate_public_ip_address
  instance_type               = var.instance_type
  security_group_rules        = var.security_group_rules
  instance_profile            = aws_iam_instance_profile.test.name
  tenancy                     = var.tenancy

  context = module.this.context
}

Or should I just set instance count to the number that I want? In which case, how would I be able to know there hostnames in advance? Thanks for the help! Sorry I'm new to this.

[jamengual]

the hostname is something you will have to setup since the instances will pick the instance id as hostname i-dsad3424dsfds

if you are building a cluster that may or may not autoscale I will just use the asg module not this one. https://github.com/cloudposse/terraform-aws-ec2-autoscale-group

if you have the requirement to have separated and unique instances that are not that mutable then you could use this module for that with the for_each or count.

[vsoch]

The cluster will have our job manager, Flux framework, and the different node hostnames need to be known at creation time (and we don't currently support any concept of scaling) so that's why I was looking to this config! The instances can be separated and not unique, but I do need to figure out how to, for example, get all the hostnames in the cluster for the main broker instance. As an example, when I do this in Kubernetes I use an Indexed Job, and then I know the hostname is something like:

flux-sample-0.flux-service.flux-operator.svc.cluster.local

And then the broker config just needs to know the shared DNS network (flux-service.flux-operator.svc.cluster.local) and then the range of hosts (e.g., flux-sample[0-4]). When we set this up in terraform with GCP (and I didn't create these recipes so I understand them superficially) I think we set the hostnames via a variable that goes into metadata for the instance https://github.com/GoogleCloudPlatform/scientific-computing-examples/blob/b6995a84ba084bd55e08d3e09d9a1b8e6715db65/fluxfw-gcp/tf/modules/compute/main.tf#L72-L80 and then during startup we can ping that API to get the exact value. Does that make sense? I'm looking for something similar / simple here - I basically just need a set number of instances (no auto-scaling) that I can somehow retrieve the hostnames for to write into the cluster configs.

[vsoch]

What about this recipe / setup under this same org? I see there is a DNS -> hostname https://github.com/clouddrove/terraform-aws-ec2/blob/edcac308fa17b8135b7813e643c2f7448306c01e/main.tf#L193

[jamengual]

it all depends on what you are doing, I have no idea about your requirements so I can only assume.

if the software you are using requires a valid domain name record to point to the instance then you can use that.
if it uses the hostname then you can add a userdata script that sets the hostname in a predictable format.

#cloud-config
instanceid=$(curl -s http://169.254.169.254/latest/meta-data/instance-id | sed 's/i-//g')
hostnamectl set-hostname "cassandra-$${instanceid}.${namespace}-${environment}"
echo "cassandra-$${instanceid}.${namespace}-${environment}" > /etc/hostname
hostname -F /etc/hostname

[vsoch]

okay I've been able to use the recipe to deploy two nodes, and I'm looking at the metadata above!

instanceid=$(curl -s http://169.254.169.254/latest/meta-data/instance-id | sed 's/i-//g')
[rocky@ip-172-16-215-227 ~]$ echo $instanceid 
0b5058dd4d3be6142

Stupid question - where did you get http://169.254.169.254 from (because it works for me too). I'm assuming the above would be able to set a hostname based on the instance id (which I can't control?) from within a single node, but I would not be able to request getting a set of instance ids? And if I were to setup the DNS section of the config (that I linked to above) that would be done manually on S3 first, and then do the instances get a predictable name?

I think the requirements are fairly loose - I just need an ip address / hostname that one instance can see for the other. I am fairly indifferent about how that is accomplished. If we could walk through the logic of one simple way, I'd be really appreciative!

For the above - if we know the instance id (or some other metadata variable) that is unique to the instance across all instances, this would actually be perfect. I would set up a user data section that can run on each to generate an updated hostname and add the known hostnames to the other nodes generated. They would be added to /etc/hosts. If we created the equivalent of a headless service, then I would still need to know the hostnames for the broker, but wouldn't need to add to /etc/hosts because they would be discovered via DNS.

[vsoch]

Maybe I could try setting user-data and seeing if it shows up in this list:

And then part of the user data could (somehow) be the count (the for_each part?) although I don't fully understand how that relates to the explicit instance_count (or maybe I could pipe that index in?)

[jamengual]

okay I've been able to use the recipe to deploy two nodes, and I'm looking at the metadata above!

instanceid=$(curl -s http://169.254.169.254/latest/meta-data/instance-id | sed 's/i-//g')
[rocky@ip-172-16-215-227 ~]$ echo $instanceid 
0b5058dd4d3be6142

Stupid question - where did you get http://169.254.169.254 from (because it works for me too). I'm assuming the above would be able to set a hostname based on the instance id (which I can't control?) from within a single node, but I would not be able to request getting a set of instance ids? And if I were to setup the DNS section of the config (that I linked to above) that would be done manually on S3 first, and then do the instances get a predictable name?

I think the requirements are fairly loose - I just need an ip address / hostname that one instance can see for the other. I am fairly indifferent about how that is accomplished. If we could walk through the logic of one simple way, I'd be really appreciative!

For the above - if we know the instance id (or some other metadata variable) that is unique to the instance across all instances, this would actually be perfect. I would set up a user data section that can run on each to generate an updated hostname and add the known hostnames to the other nodes generated. They would be added to /etc/hosts. If we created the equivalent of a headless service, then I would still need to know the hostnames for the broker, but wouldn't need to add to /etc/hosts because they would be discovered via DNS.

you can template the userdata to pass whatever string to it instead of the instanceid, I use that in my case.

do you need the ips as immutable?

[vsoch]

do you need the ips as immutable?

They only need to be consistent for the lifecycle of a single cluster deployment - the general design is we bring it up, use it (and knowing the ips for a single broker in the cluster allows them to see one another) and then we throw away. We can bring up another one later with completely different ones.

[jamengual]

I will say use the asg module instead if that is the case

[vsoch]

Can you tell me how to get the index of the instance for the launch script? I tried setting for each, but it doesn't seem to use it - it tries to create the same one twice (and then tells me it already exists). I found this example https://www.middlewareinventory.com/blog/terraform-aws-ec2-user_data-example/ that has a count.index that it starts at 1 but I don't understand how that's working.

[vsoch]

E.g., I can see there are indices here: module.ec2["2"].aws_iam_instance_profile.default[0]

[vsoch]

Even if I could get an index though, I don't know how to get the instances to see one another. E.g., I manually set one of two to flux-1, but flux-1 cannot see flux-0

[rocky@flux-0 ~]$ ping flux-1
PING flux-1(flux-1 (fe80::10f9:24ff:feb0:836b%eth0)) 56 data bytes
64 bytes from flux-1 (fe80::10f9:24ff:feb0:836b%eth0): icmp_seq=1 ttl=64 time=0.040 ms
64 bytes from flux-1 (fe80::10f9:24ff:feb0:836b%eth0): icmp_seq=2 ttl=64 time=0.037 ms
64 bytes from flux-1 (fe80::10f9:24ff:feb0:836b%eth0): icmp_seq=3 ttl=64 time=0.034 ms
^C
--- flux-1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2057ms
rtt min/avg/max/mdev = 0.034/0.037/0.040/0.002 ms
[rocky@flux-0 ~]$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

[rocky@flux-0 ~]$ ping flux-0
ping: flux-0: Name or service not known