Vijay T - Technical Blog

Secure ECS Container Config: Replace Environment Variables with Secrets Manager

Learn how to migrate environment variables to AWS Secrets Manager for ECS tasks using JSON, CloudFormation, CDK, and Terraform, with IAM, rotation, tips.

Introduction

When you run containers on Amazon ECS (Fargate or EC2) you normally inject configuration through the environment block of a task definition.
Those key‑value pairs are stored in plain text inside the task definition, which is fine for non‑sensitive data but not ideal for passwords, API keys, or database credentials.

AWS Secrets Manager solves this problem by keeping secrets encrypted at rest and giving you fine‑grained IAM control over who can read them. ECS can pull secrets from Secrets Manager and expose them to the container as ordinary environment variables – without any code changes in the container itself.

Below is a step‑by‑step guide that shows:

  1. How to create the secret and grant permissions.
  2. How to replace the environment block with a secrets block in various IaC formats (raw JSON, CloudFormation, CDK, Terraform).
  3. What the container sees at runtime.
  4. Optional patterns (runtime SDK access, rotation).
  5. A checklist and common pitfalls.

1️⃣ Prerequisites

1.1 Create a Secrets Manager secret

{
  "DB_USER": "my-db-user",
  "DB_PASSWORD": "super-secret-pwd",
  "API_KEY": "abcd1234"
}

Secret name: myapp/prod/creds
ARN example:

arn:aws:secretsmanager:us-east-1:123456789012:secret:myapp/prod/creds-ABC123

1.2 Grant the ECS task‑execution role permission to read the secret

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue"],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:myapp/prod/creds-*"
    },
    {
      "Effect": "Allow",
      "Action": ["kms:Decrypt"],
      "Resource": "*"
    }
  ]
}

Attach this policy to the role that you use as executionRoleArn (the default ecsTaskExecutionRole works fine).

Note: The task role (taskRoleArn) is only needed if you intend to call Secrets Manager from inside the container; for plain‑env‑var injection the execution role is sufficient.

2️⃣ Replacing environment with secrets

2.1 Raw JSON task definition (CLI / aws ecs register-task-definition)

Before – plain text

{
  "family": "my-app",
  "containerDefinitions": [
    {
      "name": "web",
      "image": "...",
      "environment": [
        {"name": "DB_USER", "value": "my-db-user"},
        {"name": "DB_PASSWORD", "value": "super-secret-pwd"},
        {"name": "API_KEY", "value": "abcd1234"}
      ],
      "portMappings": [{"containerPort": 8080}]
    }
  ],
  "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "256",
  "memory": "512"
}

After – secrets injection

{
  "family": "my-app",
  "containerDefinitions": [
    {
      "name": "web",
      "image": "...",
      "secrets": [
        {
          "name": "DB_USER",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:myapp/prod/creds-ABC123:DB_USER::"
        },
        {
          "name": "DB_PASSWORD",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:myapp/prod/creds-ABC123:DB_PASSWORD::"
        },
        {
          "name": "API_KEY",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:myapp/prod/creds-ABC123:API_KEY::"
        }
      ],
      "portMappings": [{"containerPort": 8080}]
    }
  ],
  "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "256",
  "memory": "512"
}

Key details

  • name → the env‑var name that will appear inside the container.
  • valueFrom → full Secrets Manager ARN plus the JSON key (:DB_PASSWORD:) and the required trailing ::.

The secret value is fetched just before the container starts and injected as a normal environment variable. No code change in the container is required.

2.2 CloudFormation (YAML)

Resources:
  MyTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: my-app
      Cpu: "256"
      Memory: "512"
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      ExecutionRoleArn: !GetAtt TaskExecutionRole.Arn
      ContainerDefinitions:
        - Name: web
          Image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest
          PortMappings:
            - ContainerPort: 8080
          Secrets:
            - Name: DB_USER
              ValueFrom: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:myapp/prod/creds-ABC123:DB_USER::"
            - Name: DB_PASSWORD
              ValueFrom: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:myapp/prod/creds-ABC123:DB_PASSWORD::"
            - Name: API_KEY
              ValueFrom: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:myapp/prod/creds-ABC123:API_KEY::"

  TaskExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: ecsTaskExecutionRole
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
      Policies:
        - PolicyName: AccessMySecrets
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action: secretsmanager:GetSecretValue
                Resource: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:myapp/prod/creds-*"

2.3 AWS CDK (TypeScript)

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager';

export class EcsWithSecretsStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Import an existing secret
    const mySecret = secretsmanager.Secret.fromSecretNameV2(
      this,
      'MyAppCreds',
      'myapp/prod/creds'
    );

    // Execution role with permission to read the secret
    const execRole = new iam.Role(this, 'TaskExecRole', {
      assumedBy: new iam.ServicePrincipal('ecs-tasks.amazonaws.com'),
      managedPolicies: [
        iam.ManagedPolicy.fromAwsManagedPolicyName('service-role/AmazonECSTaskExecutionRolePolicy')
      ]
    });
    mySecret.grantRead(execRole);   // adds secretsmanager:GetSecretValue

    // Task definition
    const taskDef = new ecs.FargateTaskDefinition(this, 'TaskDef', {
      executionRole: execRole,
      cpu: 256,
      memoryLimitMiB: 512,
    });

    // Container that consumes the secret
    const container = taskDef.addContainer('Web', {
      image: ecs.ContainerImage.fromRegistry('123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest'),
      logging: ecs.LogDrivers.awsLogs({ streamPrefix: 'my-app' })
    });

    container.addSecret('DB_USER', ecs.Secret.fromSecretsManager(mySecret, 'DB_USER'));
    container.addSecret('DB_PASSWORD', ecs.Secret.fromSecretsManager(mySecret, 'DB_PASSWORD'));
    container.addSecret('API_KEY', ecs.Secret.fromSecretsManager(mySecret, 'API_KEY'));

    container.addPortMappings({ containerPort: 8080 });
  }
}

What CDK does for you

  • ecs.Secret.fromSecretsManager(secret, jsonKey) automatically builds the correct valueFrom ARN.
  • mySecret.grantRead(execRole) adds the necessary IAM policy.
  • No hand‑written JSON; CDK synthesizes CloudFormation for you.

2.4 Terraform

# Existing secret
data "aws_secretsmanager_secret" "my_app" {
  name = "myapp/prod/creds"
}

# Execution role
resource "aws_iam_role" "ecs_task_exec" {
  name = "ecsTaskExecutionRole"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "ecs-tasks.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy_attachment" "ecs_exec_policy" {
  role       = aws_iam_role.ecs_task_exec.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

# Permission to read the secret
resource "aws_iam_policy" "secrets_access" {
  name   = "MyAppSecretAccess"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["secretsmanager:GetSecretValue"]
      Resource = "${data.aws_secretsmanager_secret.my_app.arn}*"
    }]
  })
}

resource "aws_iam_role_policy_attachment" "secrets_attach" {
  role       = aws_iam_role.ecs_task_exec.name
  policy_arn = aws_iam_policy.secrets_access.arn
}

# Task definition using the secret
resource "aws_ecs_task_definition" "my_app" {
  family                   = "my-app"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = "256"
  memory                   = "512"
  execution_role_arn       = aws_iam_role.ecs_task_exec.arn

  container_definitions = jsonencode([{
    name      = "web"
    image     = "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest"
    essential = true
    portMappings = [{ containerPort = 8080 }]

    secrets = [
      {
        name      = "DB_USER"
        valueFrom = "${data.aws_secretsmanager_secret.my_app.arn}:DB_USER::"
      },
      {
        name      = "DB_PASSWORD"
        valueFrom = "${data.aws_secretsmanager_secret.my_app.arn}:DB_PASSWORD::"
      },
      {
        name      = "API_KEY"
        valueFrom = "${data.aws_secretsmanager_secret.my_app.arn}:API_KEY::"
      }
    ]
  }])
}

3️⃣ What the container sees at runtime

ECS fetches the secret once, just before the container starts, and injects each entry as a regular environment variable.

# Bash
echo "DB user = $DB_USER"
echo "Password = $DB_PASSWORD"

# Python
import os
db_user = os.getenv('DB_USER')
db_pwd  = os.getenv('DB_PASSWORD')

No extra SDK calls or code modifications are required. The secret is treated exactly like a plain‑text env var, only the source (Secrets Manager) is different.

4️⃣ Optional patterns

Use‑case How to implement
On‑demand secret refresh without task restart Give the task role (taskRoleArn) secretsmanager:GetSecretValue permission and call the AWS SDK from inside the container. Keep the secret out of the secrets block for that key.
Partial exposure Use secrets for the values you want as env vars, and SDK calls for the rest.
Automatic rotation Enable rotation in Secrets Manager (e.g., Lambda‑based). A new task launch will automatically receive the fresh value.
SSM Parameter Store instead of Secrets Manager The same valueFrom syntax works; just point to an SSM ARN (arn:aws:ssm:…).

5️⃣ Checklist before you redeploy

✅ Item Why it matters
Secret exists and is correctly formatted (JSON or plain string). Otherwise valueFrom will return an empty string or error.
Execution role has secretsmanager:GetSecretValue (and kms:Decrypt if custom CMK). Prevents AccessDeniedException at task start.
Task definition uses secrets instead of environment. Guarantees the secret is pulled at launch time.
valueFrom includes the trailing :: when referencing a JSON key. Missing :: makes ECS treat the whole ARN as a literal string.
Service / task points to the new revision of the task definition. Old revisions keep using the plain‑text values.
Log/monitor: Verify that the env var appears (but never log the secret itself). Quick sanity check that injection succeeded.
Rotation (if enabled) → a new task launch picks up the new value. Confirms rotation works end‑to‑end.
IAM Access Analyzer / CloudTrail → confirm only intended role can read the secret. Security best practice.

6️⃣ Common pitfalls and how to fix them

Symptom Typical cause Fix
Container crashes with “Unable to read environment variable” valueFrom ARN malformed (missing :: or wrong JSON key). Verify the ARN syntax: arn:aws:secretsmanager:region:account:secret:name‑suffix:JSON_KEY::
AccessDeniedException in ECS logs Execution role lacks secretsmanager:GetSecretValue. Attach the policy shown in §1.2.
Secret appears empty The secret’s JSON does not contain the requested key, or the secret is a binary value. Check the secret in the console; adjust the key name.
Secret rotation does not take effect until you redeploy Secrets are fetched only at task start. Redeploy the service or run a new task. If you need instant refresh, use SDK calls from inside the container.
Want to use Parameter Store but get “Invalid ARN” Using the wrong ARN prefix (ssm: vs secretsmanager:). Build the ARN with the arn:aws:ssm: prefix and reference the parameter name.

7️⃣ Quick one‑liner for the CLI

If you already have a secret myapp/prod/creds and want to register a task definition that only injects DB_PASSWORD:

aws ecs register-task-definition \
  --family my-app \
  --requires-compatibilities FARGATE \
  --cpu 256 --memory 512 \
  --execution-role-arn arn:aws:iam::123456789012:role/ecsTaskExecutionRole \
  --container-definitions "$(cat <<EOF
[
  {
    "name": "web",
    "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest",
    "portMappings": [{ "containerPort": 8080 }],
    "secrets": [
      {
        "name": "DB_PASSWORD",
        "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:myapp/prod/creds-ABC123:DB_PASSWORD::"
      }
    ]
  }
]
EOF
)"

Run a new task or update the service, and echo $DB_PASSWORD inside the container will print the secret value.

8️⃣ Wrap‑up

By moving sensitive configuration from plain‑text environment variables to AWS Secrets Manager you gain:

  • Encryption at rest and automatic rotation.
  • Fine‑grained IAM control over which tasks can read which secrets.
  • Zero‑code changes in most cases – the secret just becomes another env var.
  • Consistent IaC patterns across JSON, CloudFormation, CDK, and Terraform.

Implement the changes described above, redeploy your service, and enjoy a more secure ECS deployment pipeline. Happy containerizing! 🚀

Made with chatblogr.com