Recently we were helping customer to launch Spoke Gateways in their AWS account, after 10 minutes launching the gateway, the gateway creation were reverted and following errors generated

Error: [AVXERR-TRANSIT-0119] Failed to launch gateway test. Instance i-0005da0797da40ae8 could not be started. Delete the gateway test to clean up resources and try again. It is possible that gateway size t3.small is not supported in the region us-east-1 or EBS encryption KMS CMK Key policy Key administrators and users are not updated with your Aviatrix APP role and Aviatrix EC2 role.

Behaviors

• Aviatrix Controller is in AWS. Controller have been assigned aviatrix-role-ec2 role, and assumes aviatrix-role-app role, which in turn uses aviatrix-app-policy policy.
• Right after you start to create the gateway, if you switch over to AWS console, you can observe the instance immediately gets shutdown and terminated.

• Within CloudTrail, you cannot find any events with Event name equals TerminateInstances in last 30 minutes.

• After 10 minutes Aviatrix controller would report the error and roll back gateway deployment

• This issue doesn’t happen if Aviatrix controller is using IAM based authentication, where the IAM user have been granted AdministratorAccess policy

The causes

While as indicated in the error message, two potential causes of the issue:

1. The size of the gateway you are launching isn’t available in the target AZ. You may try to launch another instance with the same size in the same AZ manually from AWS console to confirm if this was the cause
2. The second issue was what we ran into: Customer configured EBS (Elastic Block Storage) encryption for the region where we wanted to launch the gateway, also they selected to CMK (Customer managed keys) instead of AWS managed keys for the encryption. And our aviatrix-role-app wasn’t added as CMK key user.
3. Some customer have AWS Config defined tag policy that declined the gateway deployment due to incorrect tag key parameters.

Inspection

Go to EC2 Dashboard, select the correct region, on right side find EBS encryption

Verify if EBS encryption was Enabled, and copy down the KMS key ID, eg: 597160bf-b8e6-4968-ba13-bb27456653cf

Go to KMS services,

If customer is using the AWS managed keys, eg the aws/ebs key highlighted, this won’t be an issue. You know it’s not the key been used, as the Key ID didn’t match 597160bf-b8e6-4968-ba13-bb27456653cf noted before

If you click on the aws/ebs key, in Key policy section, you will note that all AWS principles (*) will have access to this key for encrypt/decrypt etc. This is why when the AWS managed keys been used you should not have an issue.

Now switch over to Customer managed keys, you should find the key ID we recorded earlier, eg: 597160bf-b8e6-4968-ba13-bb27456653cf

Click on the CMK key. In my example, I’ve purposely removed any key administrators, and key users

If you prefer JSON format, you can click on switch to policy view. Note the root account could have full KMS access, which explained why IAM user with AdministratorAccess policy worked fine.

Our controller is trying to assume role aviatrix-role-app which doesn’t have permission to use the CMK key to encrypt the EBS volume, hence the gateway creation would fail.

The fix

In CMK Policy view, make sure to add the aviatrix-role-app ARN to both “Allow use of the key” and “Allow attachment of persistent resources” sections.

{
    "Id": "key-consolepolicy-3",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<aws_account_id>:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "Allow use of the key",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<aws_account_id>:role/<some_other_role>",
                    "arn:aws:iam::<aws_account_id>:role/aviatrix-role-app"
                ]
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow attachment of persistent resources",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<aws_account_id>:role/<some_other_role>",
                    "arn:aws:iam::<aws_account_id>:role/aviatrix-role-app"
                ]
            },
            "Action": [
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant"
            ],
            "Resource": "*",
            "Condition": {
                "Bool": {
                    "kms:GrantIsForAWSResource": "true"
                }
            }
        }
    ]
}

Alternatively, if you don’t like to use JSON, you may also want to switch to Default view. In the Key users section, click on Add

Search aviatrix-role-app and add

Now the gateway launch should be successful. If you switch over to AWS console, for the newly launched gateway, the status should be Running, and if switch over to Storage tab, and you can check the KMS key ID should match the CMK key, that you have granted key user permission for aviatrix-role-app

If the controller would be properly delete the gateway, in CloudTrail, Event name equals TerminateInstances would show the as following: