Migrating Amazon EKS Nodes to Amazon Linux 2023 (AL2023) from Amazon Linux 2 (AL2) Using CloudFormation and Troposphere

Husna Hariz
Ronard Luna

Introduction

In this post, we’ll explain how we extended our existing AWS CloudFormation infrastructure, built using Troposphere, to support the use of Amazon Linux 2023 (AL2023) for Amazon EKS (Elastic Kubernetes Service) nodes. This migration was necessary to ensure compatibility ahead of our planned upgrade to Kubernetes version 1.33.

Our existing EKS nodes, deployed several years ago, were originally provisioned with the default Amazon Linux 2 (AL2) images. AWS now flags these images as deprecated, with the following warning:

AWS AL2 Deprecation Warning

To stay ahead of deprecation timelines and ensure compatibility with future EKS versions, we proceeded to update our infrastructure code to support explicit configuration changes for AMI Ids and AMI types.

Background: Troposphere and Our CloudFormation Setup

We use Troposphere, a Python library for generating CloudFormation templates. In our infrastructure codebase, the EKS cluster resources are defined in a module named eks.py.

By default, AWS provides optimized AMIs for EKS nodes, and typically, one doesn’t need to specify a custom image. Therefore, support for configuring custom AMIs was never implemented.

However, to migrate to AL2023 based images we needed to introduce support for custom AMIs and explicitly defined image types. To maintain backward compatibility for existing deployments, these parameters were configured to be optional, ie. we added conditional checks that would determine if they would be injected into the templates.

Migration - First Attempt by Passing a Custom AMI ID only

Step 1: Adding Support for Custom AMIs

To begin, we added support for specifying a custom AMI Id via CloudFormation parameters. This allowed us to override the default behavior and pass in a specific image id.

custom_eks_ami = Ref(template.add_parameter(
    Parameter(
        "CustomEKSAMI",
        Description="Custom AMI ID for the EKS node group. It is recommended not to set this value unless needed, as AWS will automatically select the most optimized image when CustomAMIImageType is specified.",
        Type="String",
        Default="",
    ),
    group="Elastic Kubernetes Service (EKS)",
    label="Custom EKS AMI",
))

use_custom_ami = "UseCustomAMI"
template.add_condition(
    use_custom_ami,
    Not(Equals(custom_eks_ami, ""))
)

The use_custom_ami condition ensures that the custom AMI is only used when explicitly defined, keeping the template backward-compatible.

When specified, the custom AMI Id is injected into the EKS nodegroup launch template as ImageId:

nodegroup_launch_template = ec2.LaunchTemplate(
    "NodegroupLaunchTemplate",
    template=template,
    LaunchTemplateData=ec2.LaunchTemplateData(
        ImageId=If(use_custom_ami, custom_eks_ami, Ref("AWS::NoValue")),
        BlockDeviceMappings=[
            ec2.LaunchTemplateBlockDeviceMapping(
                DeviceName="/dev/xvda",
                Ebs=ec2.EBSBlockDevice(
                    DeleteOnTermination=True,
                    Encrypted=use_aes256_encryption,
                    KmsKeyId=If(use_cmk_arn, Ref(cmk_arn), Ref("AWS::NoValue")),
                    VolumeType="gp3",
                    VolumeSize=container_volume_size,
                ),
            ),
        ],
        InstanceType=container_instance_type,
        MetadataOptions=ec2.MetadataOptions(
            HttpTokens="required",
            HttpPutResponseHopLimit=3,
        ),
    )
)

Step 2: Finding the Appropriate AL2023 AMI

To find the correct AMI, we used the AWS CLI:

aws ec2 describe-images \
  --owners 602401143452 \  # Official AWS EKS AMI account
  --filters "Name=name,Values=amazon-eks-node-al2023-x86_64-standard-1.33-*" \
  --region <region> \
  --query "Images[*].[ImageId,Name]" \
  --output table

This returned a list of available AL2023 images for Kubernetes 1.33. For example:

|  ami-003ad84a1e3a4614f |  amazon-eks-node-al2023-x86_64-standard-1.33-v20250704  |
|  ami-00848331bb38314ea |  amazon-eks-node-al2023-x86_64-standard-1.33-v20250807  |
|  ami-014f71cc7221992de |  amazon-eks-node-al2023-x86_64-standard-1.33-v20250519  |
|  ami-0408802da7988486a |  amazon-eks-node-al2023-x86_64-standard-1.33-v20250804  |
|  ami-03db9f32eaa4c3c75 |  amazon-eks-node-al2023-x86_64-standard-1.33-v20250715  |
|  ami-030da889991638ab6 |  amazon-eks-node-al2023-x86_64-standard-1.33-v20250821  |
|  ami-0090e12bb58007b26 |  amazon-eks-node-al2023-x86_64-standard-1.33-v20250819  |
|  ami-0543d51a21420448f |  amazon-eks-node-al2023-x86_64-standard-1.33-v20250801  |
|  ami-0e26ca27395ce58ad |  amazon-eks-node-al2023-x86_64-standard-1.33-v20250627  |
|  ami-0ab8b9aed40e77e9e |  amazon-eks-node-al2023-x86_64-standard-1.33-v20250620  |
|  ami-0cf8cd7cb46ff26a0 |  amazon-eks-node-al2023-x86_64-standard-1.33-v20250610  |
|  ami-0fa03f713b55841a2 |  amazon-eks-node-al2023-x86_64-standard-1.33-v20250813

We selected the latest version available at the time and passed it as a parameter, CustomEKSAMI, during stack deployment.

Step 3: Failed Deployment

Once we had our configurations in place, we ran the deployment, which initially appeared successful. A new nodegroup and associated Auto Scaling Group (ASG) were created. However, the ASG would immediately terminate the nodes it attempted to launch.

CloudFormation eventually rolled back the entire deployment.

After further investigation, we discovered that defining the correct AmiType is not optional when attempting to do a migration. So despite selecting an image compatible with EKS, AWS still required, the Image Type to be defined especially for managed node groups with a custom launch template as explained in this official guide.

If you’re using managed node groups with either the standard launch template or with a custom launch template that doesn’t specify the AMI ID, you’re required to upgrade using a blue/green strategy. A blue/green upgrade is typically more complex and involves creating an entirely new node group where you would specify AL2023 as the AMI type. The new node group will need to then be carefully configured to ensure that all custom data from the AL2 node group is compatible with the new OS. Once the new node group has been tested and validated with your applications, Pods can be migrated from the old node group to the new node group. Once the migration is completed, you can delete the old node group.

At this point, we realised defining the AMI type is sufficient and passing an AMI Id isn’t necessary as AWS would be able to select an optimised image id for EKS.

CloudFormation handled the blue/green upgrade process behind the scenes. There was no need to manually create a new node group or manually delete the old AL2-based nodegroup. These steps were automatically managed as part of the upgrade process.

Migration - Second Attempt by Passing a Custom AMI Type only

Step 4: Introducing Support for Custom AMI Types

To address the failed deployment, we added another parameter: CustomAMIImageType. This explicitly defines the image type expected by the EKS node group.

custom_ami_image_type = Ref(template.add_parameter(
    Parameter(
        "CustomAMIImageType",
        Description="The image type to match the custom AMI. E.g., AL2023_x86_64_STANDARD, AL2_x86_64",
        Type="String",
        Default="",
    ),
    group="Elastic Kubernetes Service (EKS)",
    label="Custom AMI Image Type",
))

use_custom_ami_type = "UseCustomAMIType"
template.add_condition(
    use_custom_ami_type,
    Not(Equals(custom_ami_image_type, ""))
)

The use_custom_ami_type condition is used to determine whether to include the AmiType in the EKS nodegroup resource, or use AWS’s default:

eks.Nodegroup(
    "Nodegroup",
    template=template,
    DependsOn=[cluster],
    ClusterName=Ref(cluster),
    NodeRole=GetAtt(container_instance_role, "Arn"),
    LaunchTemplate=eks.LaunchTemplateSpecification(
        Id=Ref(nodegroup_launch_template),
        Version=GetAtt(nodegroup_launch_template, "LatestVersionNumber"),
    ),
    ScalingConfig=eks.ScalingConfig(
        DesiredSize=desired_container_instances,
        MaxSize=max_container_instances,
        MinSize=2,
    ),
    Subnets=[Ref(private_subnet_a), Ref(private_subnet_b)],
    AmiType=If(use_custom_ami_type, custom_ami_image_type, Ref("AWS::NoValue"))
)

Before initiating the deployment, we set the CustomEKSAMI parameter to an empty string and specified only the CustomAMIImageType, which was configured as AL2023_x86_64_STANDARD.

With this configuration, new AL2023-based nodes were successfully provisioned and scaled as expected.

Conclusion

To migrate from AL2 to AL2023 nodes in an EKS cluster, it is sufficient to set the desired AMI type using the CustomAMIImageType parameter. AWS will automatically select an optimized AMI ID based on this specification.

However, we retained support for launching a specific AL2023 AMI by allowing the use of a custom AMI ID via the CustomEKSAMI parameter, should there be a need for this in the future.