Terraform init failed to download providers

One of our customers are looking to use Aviatrix to automatic their self-service process for AWS China region.

The issue they are running into, was the terraform init would fail 50% of time. Is there anything we can do to help in this situation?

What are providers?

Terraform uses plug-ins called providers to translate terraform definition to API calls to AWS/Azure/GCP or Aviatrix controller. The providers are compiled binary files for specific platform.

These binaries are published on https://releases.hashicorp.com
You may find familiar providers for AWS/Azure as highlighted below, as well as Aviatrix

Once you click on the aviatrix link, you will be presented to various different versions of aviatrix providers. Aviatrix is constantly improving it’s product and adding new features, as such new APIs are added/modified/removed, which resulting different versions of terraform providers. Refer to this link for the relationship between Aviatrix provider and Aviatrix controller. It’s important to match the Aviatrix provider to version of your controller.

Understanding with newer version of terraform (v0.13 and above), the required provider section will define which version of providers terraform will use

terraform {
  required_providers {
    aviatrix = {
      source  = "AviatrixSystems/aviatrix"
      version = "~> 2.20.1"
    }
  }
}

It’s also important to understand the effect of version = “” portion

version = “2.21.1-6.6.ga” will download *exact* version: 2.21.1-6.6.ga
version = “~> 2.20.1” Only the rightmost version can be incremented. In the screenshot above, we have version 2.20.0, 2.20.1, 2.20.2, and 2.20.3. As it cannot change the left portion: 2.20, it will download latest version: 2.20.3
version = “>= 2.18.0, < 2.20.0” will download latest version that will immediately lower than 2.0.0 AND higher or equal to 1.2.0 (Can you figure out which version this would be for Aviatrix? Read further to find answer)

If you further click through a version in release.hashicorp.com page, you will see list of .zip files ended with various platform names. As you can see some are for freebsd, and some are for linux and some are for windows, and we have 386/arm/am64 architecture etc.

What does terraform init do in the background?

By default, terraform will look at your configuration, find out what provider and which version you intended to use, also figure out what OS and CPU architecture you are on, then it will download specific zip file and extract it under your terraform projects folder.

Eg: I’m running Ubuntu 64 bits under WSL2 and I have this version restriction: version = “>= 2.18.0, < 2.20.0”

Initializing provider plugins...
- Finding aviatrixsystems/aviatrix versions matching ">= 2.18.0, < 2.20.0"...
- Installing aviatrixsystems/aviatrix v2.19.5...
- Installed aviatrixsystems/aviatrix v2.19.5 (signed by a HashiCorp partner, key ID 6105FDDA096C7419)

Under current terraform folder “project1” it created following structure

/project1.terraform/providers/registry.terraform.io/aviatrixsystems/aviatrix/2.19.5/linux_amd64

This is called Unpacked layout, as the zip file was extracted, in the format of: HOSTNAME/NAMESPACE/TYPE/VERSION/TARGET
Where:
HOSTNAME = registry.terraform.io
NAMESPACE = aviatrixsystems
TYPE = aviatrix
VERSION = 2.19.5
TARGET = linux_amd64

Structure of the folder:

Running tcpdump while doing terraform init shows:

Query against registry.terraform.io
Query against github.com
Query against objects.githubusercontent.com
As well as consequent HTTPs connections against them

sudo tcpdump -i eth0 -s 65535 -w /mnt/c/gitrepos/project1/project1.pcap

Given the users are in China and certain DNS query and HTTPs connections may not work as desired, we need to find a way to enable customers to accomplish their self-service portal.

Configure Terraform to use pre-downloaded providers

Use Terraform to download providers

In terraform v0.13 and above, a new command has been added to help download the required providers.

For China users, they will need to create a jumpbox that have no problem talking to resolve registry.terraform.io and github.com and objects.githubusercontent.com, and consequently connect to them via HTTPS.

On this jumpbox, create a folder with terraform code like this:

terraform {
  required_providers {
    aviatrix = {
      source  = "AviatrixSystems/aviatrix"
      version = "2.20.1"
    }
  }
}

Within the folder, run following command, following example is in Linux environment, and assuming that the jumpbox as the same OS/CPU Architecture as production terraform instance.

terraform providers mirror [options] <target-dir>

Eg:
terraform providers mirror /mnt/c/gitrepos/providers

You will observe the folder structure created under /mnt/c/gitrepos/providers looks like:

C:\gitrepos\providers\registry.terraform.io\aviatrixsystems\aviatrix
HOSTNAME/NAMESPACE/TYPE/terraform-provider_VERSION_TARGET_TYPE.zip

Where:
HOSTNAME = registry.terraform.io
NAMESPACE = aviatrixsystems
TYPE = aviatrix
VERSION_TARGET_TYPE = 2.19.5_linux_amd64

This is called Packed layout, as the providers are download as ZIP not extracted.

I’ve modified my version and ran it three times, note, it downloaded 2.19.5, 2.20.1, 2.22.1, and each have corresponding .json file. Content of 2.19.5.json:

{
  "archives": {
    "linux_amd64": {
      "hashes": [
        "h1:9MN1U/vRiPWBnUNwnF12uSR4pgnVpWoilNahBTWdaLM="
      ],
      "url": "terraform-provider-aviatrix_2.19.5_linux_amd64.zip"
    }
  }
}

Content of inde.json, acting as a lookup index for all versions:

{
  "versions": {
    "2.19.5": {},
    "2.20.1": {},
    "2.22.1": {}
  }
}

terraform providers mirror command will manage the download of provider ZIP files and these .json files.

Note: If the jumpbox is in different OS/Architecture as the instance running terraform, you can use -platform=OS_ARCH switch to force terraform to download specific OS/Architecture binary to target folder.

Eg:

terraform providers mirror -platform linux_amd64 /mnt/c/gitrepos/providers

Configure terraform to use the download providers

Assume you have downloaded all necessary providers under /mnt/c/gitrepos/providers, you have to tell terraform to use this folder instead of trying to download provider binaries from Internet:

Easier approach: modify terraform CLI configuration file

For Windows: %APPDATA%\terraform.rc file
For other systems: Home directory (linux ~) of the user running terraform, eg: ~/.terraformrc

Explicit tell terraform location of the downloaded provider binaries

provider_installation {
  filesystem_mirror {
    path    = "/mnt/c/gitrepos/providers"
    include = ["AviatrixSystems/aviatrix"]
  }
}

Under filesystem_mirror section, you tell terraform where the downloaded binaries are, and which provider to use local downloaded binary. In above example, when terraform would use an Aviatrix provider, it will look under /mnt/c/gitrepos/providers. For any other providers, it will try to download from Internet.

If you would run terraform init after this, it will goes very fast and wireshare will show zero packet left the instance while running terraform init.

- Finding aviatrixsystems/aviatrix versions matching "2.22.1"...
- Installing aviatrixsystems/aviatrix v2.22.1...
- Installed aviatrixsystems/aviatrix v2.22.1 (unauthenticated)

Read more about this explicit configuration.

cloudlearning365

Cloud Learning 365