Import existing resources into CloudFormation template

Posted on by

Aviatrix developed Migration Toolkit to help customer migrate from existing AWS/ Azure environment to Aviatrix Transit and Spoke Multi-Cloud Networking Architecture (MCNA). I have discussed the process in blog: Migrate from Azure vNet hub and spoke architecture to Aviatrix Transit. The AWS migration process is similar, where the toolkit make copies of existing route tables, when Aviatrix Spoke is attached to Aviatrix Transit, we are using these copied route tables, hence no traffic interruption would happen. During the traffic switching phrase, subnets will be associated with the copied route table, and in TGW we disable the migrating VPC router advertisement, so the traffic would swing over to Aviatrix Spoke/Transit.

Some of our customers are using CloudFormation to manage the deployment of their environment, while Aviatrix Controller will handle bulk of the work such as populating RFC1918 and/or default route in the route table and/or non-RFC1918 routes from External connections, they still would like to have the ability to continue to use CloudFormation to manage endpoint routes. This created a split brain scenario, how do we handle this?

If you are familiar with Terraform, you would understand anything declared in .tf file is called desired state, when you run terraform apply and found current state in the cloud isn’t matching the desired state, terraform will try to correct current state to match desired state declared. If things are not declared in the terraform file, terraform will not manage these and will not make corrections. The same concept applies to CloudFormation template.

For example, let’s use following CloudFormation template create a VPC. The VPC have one public subnet and one private subnet, where the public subnet route table has 0.0.0.0/0 point to internet gateway.

Resources:
  MyVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: "10.0.0.0/16"
      Tags:
        - Key: Name
          Value: my-vpc

  PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref MyVPC
      CidrBlock: "10.0.1.0/24"
      AvailabilityZone:
        Fn::Select: 
        - 0
        - Fn::GetAZs: ""
      Tags:
        - Key: Name
          Value: public-subnet

  PrivateSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref MyVPC
      CidrBlock: "10.0.2.0/24"
      AvailabilityZone: 
        Fn::Select: 
        - 0
        - Fn::GetAZs: ""
      Tags:
        - Key: Name
          Value: private-subnet

  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref MyVPC
      Tags:
        - Key: Name
          Value: public-route-table

  PrivateRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref MyVPC
      Tags:
        - Key: Name
          Value: private-route-table

  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: my-igw

  GatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref MyVPC
      InternetGatewayId: !Ref InternetGateway

  PublicRoute:
    Type: AWS::EC2::Route
    DependsOn: GatewayAttachment
    Properties:
      RouteTableId: !Ref PublicRouteTable
      DestinationCidrBlock: "0.0.0.0/0"
      GatewayId: !Ref InternetGateway

  PublicSubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet
      RouteTableId: !Ref PublicRouteTable

  PrivateSubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PrivateSubnet
      RouteTableId: !Ref PrivateRouteTable

Public Route table shown in AWS Console

Explore resources created by the CloudFormation template:

Assume we created another public route table in AWS Console, simulating the copied route table via Aviatrix Migration Toolkit

Assume we create a default route in new-public-route-table point to IGW in AWS Console

Let’s see how we import the new-public-route-table

For import to work, all resources in the CloudFormation template must have DeletionPolicy set.

First add following resource declaration into existing CloudFormation template, this is to tell CloudFormation to manage a new route table of a Logic Resource ID of NewPublicRouteTable

  NewPublicRouteTable:
    Type: AWS::EC2::RouteTable
    DeletionPolicy: Delete
    Properties:
      VpcId: !Ref MyVPC
      Tags:
        - Key: Name
          Value: new-public-route-table

The new-public-route-table has a unique Resource Identifier of: rtb-0ebd453646fdff442

We need to tell CloudFormation how to link the Logical Resource ID and the unique Resource Identifier, by creating a json file:

[
  {
      "ResourceType":"AWS::EC2::RouteTable",
      "LogicalResourceId":"NewPublicRouteTable",
      "ResourceIdentifier": {
        "RouteTableId":"rtb-0ebd453646fdff442"
      }
  }
]

The structure of the json file is described in this article

[
  {
    "ResourceType": "string",
    "LogicalResourceId": "string",
    "ResourceIdentifier": {"string": "string"
      ...}
  }
  ...
]

To get the format of ResourceType, LogicalResourceId and ResourceIdentifier, run following command, note the highlighted section.

aws cloudformation get-template-summary --stack-name myvpc

{
    "Parameters": [],
    "ResourceTypes": [
        "AWS::EC2::InternetGateway",
        "AWS::EC2::VPC",
        "AWS::EC2::RouteTable",
        "AWS::EC2::RouteTable",
        "AWS::EC2::VPCGatewayAttachment",
        "AWS::EC2::Subnet",
        "AWS::EC2::RouteTable",
        "AWS::EC2::Subnet",
        "AWS::EC2::Route",
        "AWS::EC2::SubnetRouteTableAssociation",
        "AWS::EC2::Route",
        "AWS::EC2::SubnetRouteTableAssociation"
    ],
    "Version": "2010-09-09",
    "ResourceIdentifierSummaries": [
        {
            "ResourceType": "AWS::EC2::VPC",
            "LogicalResourceIds": [
                "MyVPC"
            ],
            "ResourceIdentifiers": [
                "VpcId"
            ]
        },
        {
            "ResourceType": "AWS::EC2::RouteTable",
            "LogicalResourceIds": [
                "PublicRouteTable",
                "PrivateRouteTable",
                "NewPublicRouteTable"
            ],
            "ResourceIdentifiers": [
                "RouteTableId"
            ]
        },
        {
            "ResourceType": "AWS::EC2::SubnetRouteTableAssociation",
            "LogicalResourceIds": [
                "PrivateSubnetRouteTableAssociation",
                "PublicSubnetRouteTableAssociation"
            ],
            "ResourceIdentifiers": [
                "Id"
            ]
        },
        {
            "ResourceType": "AWS::EC2::InternetGateway",
            "LogicalResourceIds": [
                "InternetGateway"
            ],
            "ResourceIdentifiers": [
                "InternetGatewayId"
            ]
        },
        {
            "ResourceType": "AWS::EC2::Subnet",
            "LogicalResourceIds": [
                "PrivateSubnet",
                "PublicSubnet"
            ],
            "ResourceIdentifiers": [
                "SubnetId"
            ]
        }
    ]
}

The new template would looks like this now:

Resources:
  MyVPC:
    Type: AWS::EC2::VPC
    DeletionPolicy: Delete
    Properties:
      CidrBlock: "10.0.0.0/16"
      Tags:
        - Key: Name
          Value: my-vpc

  PublicSubnet:
    Type: AWS::EC2::Subnet
    DeletionPolicy: Delete
    Properties:
      VpcId: !Ref MyVPC
      CidrBlock: "10.0.1.0/24"
      AvailabilityZone:
        Fn::Select: 
        - 0
        - Fn::GetAZs: ""
      Tags:
        - Key: Name
          Value: public-subnet

  PrivateSubnet:
    Type: AWS::EC2::Subnet
    DeletionPolicy: Delete
    Properties:
      VpcId: !Ref MyVPC
      CidrBlock: "10.0.2.0/24"
      AvailabilityZone: 
        Fn::Select: 
        - 0
        - Fn::GetAZs: ""
      Tags:
        - Key: Name
          Value: private-subnet

  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    DeletionPolicy: Delete
    Properties:
      VpcId: !Ref MyVPC
      Tags:
        - Key: Name
          Value: public-route-table

  PrivateRouteTable:
    Type: AWS::EC2::RouteTable
    DeletionPolicy: Delete
    Properties:
      VpcId: !Ref MyVPC
      Tags:
        - Key: Name
          Value: private-route-table

  InternetGateway:
    Type: AWS::EC2::InternetGateway
    DeletionPolicy: Delete
    Properties:
      Tags:
        - Key: Name
          Value: my-igw

  GatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    DeletionPolicy: Delete
    Properties:
      VpcId: !Ref MyVPC
      InternetGatewayId: !Ref InternetGateway

  PublicRoute:
    Type: AWS::EC2::Route
    DeletionPolicy: Delete
    DependsOn: GatewayAttachment
    Properties:
      RouteTableId: !Ref PublicRouteTable
      DestinationCidrBlock: "0.0.0.0/0"
      GatewayId: !Ref InternetGateway

  PublicSubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    DeletionPolicy: Delete
    Properties:
      SubnetId: !Ref PublicSubnet
      RouteTableId: !Ref PublicRouteTable

  PrivateSubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    DeletionPolicy: Delete
    Properties:
      SubnetId: !Ref PrivateSubnet
      RouteTableId: !Ref PrivateRouteTable

  NewPublicRouteTable:
    Type: AWS::EC2::RouteTable
    DeletionPolicy: Delete
    Properties:
      VpcId: !Ref MyVPC
      Tags:
        - Key: Name
          Value: new-public-route-table

Upload both modified template and the json file to CloudShell

Run following command for the change set

aws cloudformation create-change-set --stack-name myvpc --change-set-name ImportChangeSet --change-set-type IMPORT --resources-to-import file://import.json --template-body file://vpc.yaml

Validate change set in AWS Console, then Execute change set

Upon completion, validate Resources under stack and find the route table is now been managed

This is great, then how about import the routes? In above screenshot, we can see that the PublicRoute have a physical id of myvpc-Publi-1O66C3YL5SCJE

It can be obtained via cloudformation cli:

[cloudshell-user@ip-10-4-89-56 ~]$ aws cloudformation describe-stack-resources --stack-name myvpc --query 'StackResources[?ResourceType==`AWS::EC2::Route`]'
[
    {
        "StackName": "myvpc",
        "StackId": "arn:aws:cloudformation:us-east-1:<account-id>:stack/myvpc/99dde040-e06f-11ed-b816-12c48b848691",
        "LogicalResourceId": "PublicRoute",
        "PhysicalResourceId": "myvpc-Publi-1O66C3YL5SCJE",
        "ResourceType": "AWS::EC2::Route",
        "Timestamp": "2023-04-21T18:10:02.288000+00:00",
        "ResourceStatus": "CREATE_COMPLETE",
        "DriftInformation": {
            "StackResourceDriftStatus": "NOT_CHECKED"
        }
    }
]

But it appears to be impossible to obtain this physical ID for a route created outside of CloudFormation, as each route doesn’t really have a unique ID:

[cloudshell-user@ip-10-4-89-56 ~]$ aws ec2 describe-route-tables --route-table-id rtb-0bfc1da00ea922888
{
    "RouteTables": [
        {
            "Associations": [
                {
                    "Main": false,
                    "RouteTableAssociationId": "rtbassoc-002eb5d86d30c4067",
                    "RouteTableId": "rtb-0bfc1da00ea922888",
                    "SubnetId": "subnet-0a792e7973a2fbad7",
                    "AssociationState": {
                        "State": "associated"
                    }
                }
            ],
            "PropagatingVgws": [],
            "RouteTableId": "rtb-0bfc1da00ea922888",
            "Routes": [
                {
                    "DestinationCidrBlock": "10.0.0.0/16",
                    "GatewayId": "local",
                    "Origin": "CreateRouteTable",
                    "State": "active"
                },
                {
                    "DestinationCidrBlock": "0.0.0.0/0",
                    "GatewayId": "igw-082f5b8803aa503e2",
                    "Origin": "CreateRoute",
                    "State": "active"
                }
            ],
            "Tags": [
                {
                    "Key": "aws:cloudformation:logical-id",
                    "Value": "PublicRouteTable"
                },
                {
                    "Key": "Name",
                    "Value": "public-route-table"
                },
                {
                    "Key": "aws:cloudformation:stack-id",
                    "Value": "arn:aws:cloudformation:us-east-1:<account-id>:stack/myvpc/99dde040-e06f-11ed-b816-12c48b848691"
                },
                {
                    "Key": "aws:cloudformation:stack-name",
                    "Value": "myvpc"
                }
            ],
            "VpcId": "vpc-0023fa5f554d32f7c",
            "OwnerId": "<account-id>"
        }
    ]
}

Turned out not route cannot be imported

An error occurred (ValidationError) when calling the CreateChangeSet operation: ResourceTypes [AWS::EC2::Route] are not supported for Import

If tried to append following in the CloudFormation template:

  NewPublicRoute:
    Type: AWS::EC2::Route
    DependsOn: GatewayAttachment
    Properties:
      RouteTableId: !Ref NewPublicRouteTable1
      DestinationCidrBlock: "0.0.0.0/0"
      GatewayId: !Ref InternetGateway

Run stack update, and it will fail as the default route already exist

Based on above testing, Customer need to perform the task of make copy of the route table, then Aviatrix would utilize the copied route table for traffic switching.

Once the traffic switch is completed, the copied route table would be associated with the subnet, need to be able to update the CloudFormation to reflect this association change.

Edit public subnet association in AWS Console

Switch from public-route-table to new-public-route-table1

Update CloudFormation template, from:

  PublicSubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    DeletionPolicy: Delete
    Properties:
      SubnetId: !Ref PublicSubnet
      RouteTableId: !Ref PublicRouteTable

To: 

  PublicSubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    DeletionPolicy: Delete
    Properties:
      SubnetId: !Ref PublicSubnet
      RouteTableId: !Ref NewPublicRouteTable

Stack -> Update and replace current template:

Note: If selected preserve successfully provisioned resources in Stack failure options

The update would fail, as it was trying to replace the deployed association

Leave a Reply

Your email address will not be published. Required fields are marked *