Deploy taking incredibly long suddenly.

[digitlninja]

How can I troubleshoot why the deploy is taking incredibly long all of a sudden?

my task definition

{
  "ipcMode": null,
  "executionRoleArn": "arn:aws:iam::185944984862:role/ecsTaskExecutionRole",
  "containerDefinitions": [
    {
      "dnsSearchDomains": null,
      "environmentFiles": null,
      "logConfiguration": {
        "logDriver": "awslogs",
        "secretOptions": null,
        "options": {
          "awslogs-group": "/ecs/identity-backend",
          "awslogs-region": "eu-west-2",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "entryPoint": null,
      "portMappings": [
        {
          "hostPort": 3001,
          "protocol": "tcp",
          "containerPort": 3001
        }
      ],
      "command": null,
      "linuxParameters": null,
      "cpu": 1024,
      "environment": [],
      "resourceRequirements": null,
      "ulimits": null,
      "dnsServers": null,
      "mountPoints": [],
      "workingDirectory": null,
      "secrets": [
        {
          "valueFrom": "xxx",
          "name": "IoTBackend-Staging"
        }
      ],
      "dockerSecurityOptions": null,
      "memory": null,
      "memoryReservation": null,
      "volumesFrom": [],
      "stopTimeout": null,
      "image": "185944984862.dkr.ecr.eu-west-2.amazonaws.com/identity:4f177c4240adda5b3bf8f5f83f7b766e490e2775",
      "startTimeout": null,
      "firelensConfiguration": null,
      "dependsOn": null,
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": true,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": null,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "identity-backend"
    }
  ],
  "placementConstraints": [],
  "memory": "2048",
  "taskRoleArn": "arn:aws:iam::185944984862:role/ecsTaskExecutionRole",
  "compatibilities": [
    "EC2",
    "FARGATE"
  ],
  "taskDefinitionArn": "arn:aws:ecs:eu-west-2:185944984862:task-definition/identity-backend:3",
  "family": "identity-backend",
  "requiresAttributes": [
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.execution-role-awslogs"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.ecr-auth"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.secrets.asm.environment-variables"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.task-iam-role"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.execution-role-ecr-pull"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.task-eni"
    }
  ],
  "pidMode": null,
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "networkMode": "awsvpc",
  "cpu": "1024",
  "revision": 3,
  "status": "ACTIVE",
  "inferenceAccelerators": null,
  "proxyConfiguration": null,
  "volumes": []
}

[allisaurus]

@digitlninja as a first step, you can check the ECS service events in the AWS ECS web console to see whether your tasks are flapping (coming up/down) or some specific task (like becoming healthy in the load balancer) is taking a while to stabilize. Or, if you have CloudWatch logs enabled, you can check the task logs in the ECS or CloudWatch logs web console to see if specific containers are hanging on any particular start up step. Also, if your container image recently increased in size or changed location (like a diff region) it might be taking longer to pull?

Sorry these are rather generic advice, but I can't think of a specific reason this action would cause deployments to take longer. If it just recently started happening w/o any changes on your end we could continue to investigate. LMK if any of the above it helpful.

[allisaurus]

Closing due to lack of response, but feel free to reopen if you find reason to believe this action is effecting your deployment times.

[JefferyHus]

I can confirm this. The ECS task is running & the new updates have been applied, but the GitHub action is still loading. It takes up to 10 minutes to finish and sometimes more.

[SunnGHubX]

Same with me while deploying Task definitions via ecs using github actions, it hangs to passing all the way to 25 mins. I had to stop it. I had investigated this issue and verify that from the Events tab, old ones are still not shutting down when new are trying to deploy.. meaning this is a safety secure feature as green-->blue deployment just incase new deployment is not good.
But for me my deployment didn't error out just hung up .
Is there a fixed for this?

• Also verified if its because of image, but my image is a *-alpine so its not even that big.. I am running a frontend+backend react service..
  Is there a workaround, I can't seem to find answers...
  ....

[allisaurus]

@JefferyHus can you tell me more about that status of your service when the GH action is hanging? Has a new revision successfully been deployed and stabilized, or has a rollback occurred? Any output you have from the service's events (tab visible in ECS console or via ecs:describe-services) or errors from the GH action event itself would be helpful

@SunnGHubX what you describe sounds similar to #113 (comment) . Can you let me know if adjusting the deployment preferences fixes your issue?

[JefferyHus]

@allisaurus No errors whatsoever cominng from both the ECS logs or GH actions. GH action just hangs there loading for minutes before moving to the next & final step, the build howevere is successful. The only thing that I could think of is that the GH plugin is waiting a status & ECS doesn't return a status till fargate switches the old container with the new one.

[mcsrk]

Im facing the same situation using GH actions. In my case, it happened like this:

I used ECS to deploy a django image from ECR. Simple Fargate cluster and simple task definition and NOT using load balancer. When I was messign around to test my deployment I triggered several times my actions an the average time was 4 min. (90% of the time it was consumed by the "Deploy Amazon ECS task definition" step (shown in mage).

Due to my requirements, my implementation must have a static ip, so I did my research and restructured the whole thing, so I added to the ECS Service a Network Load Balancer that uses an Elastic IP. So I was doing my multiples commits to test the result and this time the "Deploy Amazon ECS task definition" was taking 12 minutes each time I ran the actions. I don´t fully understand why it might take more time by adding a Load Balancer.

PS: Apart from the LB, the thign i made different the second time was creating a ecs service from the "task def." tab in the ECS console, instead of go into a cluster and click "Create Service" or "Run task" which was the way I did on my 1st try.

[damusix]

I get timed out at 30 minutes. Everything deploys, but the GH Actions runs until failure.

[baranberkay96]

Is there any update?

[samlachance]

I am experiencing this as well. The container appears to be deployed and functioning but github actions just spins. I also use a load balancer for what it's worth.

[chihiros]

I too take a long time to deploy.
How can I shorten it?

[amalsgit]

It takes around 12 mins for me to update my Fargate service 😢