Detailed explanation of Terraform Provider

Preface

Terraform is designed as a multi-cloud infrastructure orchestration tool. Unlike CloudFormation, which is bound to the AWS platform, Terraform can orchestrate resources of various cloud platforms or other infrastructures at the same time. Terraform's approach to multi-cloud orchestration is the Provider plug-in mechanism

Terraform uses the go-plugin library (https://github.com/hashicorp/go-plugin) developed by HashiCorp. In essence, each Provider plug-in is an independent process, which is called with the Terraform process through rpc. The Terraform engine first reads and analyzes the Terraform code written by the user to form a graph (Graph) composed of data and resources, and then calls the Provider plug-ins corresponding to these data and resources through rpc; The plug-in framework defines various data and resources, and implements the corresponding CRUD methods; when implementing these CRUD methods, you can call the SDK provided by the target platform, or directly operate the target platform by calling the Http(s) API.

1. Download Provider

First we performed a terraform init. terraform init will analyze the Provider used in the code, and try to download the Provider plug-in to the local, we will find a .terraform folder, the UCloud Provider plug-in we use is downloaded and installed in it

.terraform
└── plugins
    ├── registry.terraform.io
    │ └── ucloud
    │ └── ucloud
    │ └── 1.22.0
    │ └── darwin_amd64 -> /Users/byers/.terraform.d/plugin-cache/registry.terraform.io/ucloud/ucloud/1.22.0/darwin_amd64
    └── selections.json

Sometimes downloading some providers is very slow, or there are many Terraform projects in the development environment, each project has its own independent plugin folder, which is very wasteful of disk, then we can use plugin cache

There are two ways to enable plugin caching:

The first method is to configure the environment variable TF_PLUGIN_CACHE_DIR:

export TF_PLUGIN_CACHE_DIR="$HOME/.terraform.d/plugin-cache"

The second method is to use the CLI configuration file. Under Windows, a file named "terraform.rc" is created in the %APPDATA% directory of the relevant user. For Macos and Linux users, a file named ".terraformrc" is created under the user's home. Configure in the file as follows

plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"

When the plugin cache is enabled, whenever the terraform init command is executed, the Terraform engine will first check whether the desired plugin already exists in the cache folder, and if so, it will copy the cached plugin to the . inside the terraform folder. If the plugin does not exist, Terraform will still download the plugin as before, save it in the plugins folder first, and then copy it from the plugins folder to the .terraform folder in the current working directory. To try to avoid saving the same plugin multiple times, Terraform will use symlinks instead of actually copying from the plugin cache directory to the working directory as long as the operating system supports it.

The Terrafom engine will never actively delete plugins in the cache folder. The size of the cache folder may grow to a very large size over time, in which case manual cleaning is required.

2. Search Provider

To find out which providers are officially accepted, there are two ways:

The first method is to visit the official Terraform Provider documentation, which lists the mainstream providers

The second method is to go to registry.terraform.io to search:

It is currently recommended to search for providers in the registry, because a large number of providers developed by the community are registered there.

Generally speaking, how to declare the relevant Provider, as well as the usage instructions of the relevant data and resources, can be found in the relevant documents on the registry.

registry.terraform.io can not only query Providers, but also publish Providers;

3. Provider declaration

For a set of Terraform code to be executed, the associated Provider must be declared in the code. Many Providers need to pass in some key information to be used when they are declared. For example, in the example in Chapter 1, the access key and the desired UCloud region (Region) information must be given.

terraform {
  required_providers {
    ucloud = {
      source = "ucloud/ucloud"
      version = ">=1.24.1"
    }
  }
}

provider "ucloud" {
  public_key = "your_public_key"
  private_key = "your_private_key"
  project_id = "your_project_id"
  region = "cn-bj2"
}

In this Provider declaration, it is first declared in the required_providers of the terraform section that this code must be executed by a Provider named ucloud. The line source = "ucloud/ucloud" declares the source address of the ucloud plugin (Source Address ). A source address is globally unique and instructs Terraform how to download the plugin. A source address consists of three parts:

[<HOSTNAME>/]<NAMESPACE>/<TYPE>

HostName is optional, the default is the official registry.terraform.io, readers can also build their own private Terraform repository. Namespace is the organization name in the Terraform repository, which represents the organization or individual that publishes and maintains the plugin. Type is a short name representing a plugin, and Type must be unique under a specific HostName/Namespace.

The plugin declaration in required_providers also declares the version constraints of the plugins required by the source code, in this case version = ">=1.24.1". The version number of the Terraform plug-in adopts the semantic format of MAJOR.MINOR.PATCH. The version constraint usually uses the operator and the version number to express the constraint condition. The conditions can be spliced ​​with commas to express the AND association, such as ">= 1.2.0, < 2.0.0". The available operators are

• = (or without =, use the version number directly): Only a specific version number is allowed, and it is not allowed to be used in combination with other conditions
• !=: a specific version number is not allowed

• ,>=,<,<=: Compare with a specific version number, which can be greater than, greater than or equal to, less than, less than or equal to

• ~>: Lock MAJOR and MINOR, allow PATCH number greater than or equal to a specific version number, for example, ~>0.9 is equivalent to >=0.9, <1.0, ~>0.8.4 is equivalent to >=0.8.4, <0.9

Terraform will check the current working environment or the plugin cache to see if there is a plugin that satisfies the version constraint. If not, Terraform will try to download it. If Terraform cannot obtain any plugins that satisfy the version constraints, then it refuses to proceed with any subsequent operations.

Preview versions can be declared by adding a suffix, for example: 1.2.0-beta. The preview version can only be specified by the "=" operator (or the vacancy operator) followed by an explicit version number, and cannot be used in conjunction with >=, ~>, etc.

It is recommended to use the ">=" operator to constrain the minimum version. If you are writing module code that is intended to be reused by others, avoid using the "~>" operator, even if you know that the module code will be incompatible with newer versions of the plugin.

5. Built-in Provider

Most providers are distributed separately in the form of plug-ins, but there is currently one provider built into the main Terraform process, that is terraform_remote_state data source. Since the Provider is built-in, it is not necessary to declare required_providers in terraform when using it. The source address of this built-in Provider is terraform.io/builtin/terraform

6. Multiple Provider instances

The provider section declares the various configurations required by the ucloud provider. In the above code example, provider "ucloud" and ucloud in the ucloud = {...} block in required_providers are both the Local Name of the Provider, and a Local Name is a unique identifier for a Provider in a module.

We can also declare multiple providers of the same type and give different Local Names

terraform {
  required_version = ">=0.13.5"
  required_providers {
    ucloudbj = {
      source = "ucloud/ucloud"
      version = ">=1.24.1"
    }
    ucloudsh = {
      source = "ucloud/ucloud"
      version = ">=1.24.1"
    }
  }
}

provider "ucloudbj" {
  public_key = "your_public_key"
  private_key = "your_private_key"
  project_id = "your_project_id"
  region = "cn-bj2"
}

provider "ucloudsh" {
  public_key = "your_public_key"
  private_key = "your_private_key"
  project_id = "your_project_id"
  region = "cn-sh2"
}

data "ucloud_security_groups" "default" {
  provider = ucloudbj
  type = "recommend_web"
}

data "ucloud_images" "default" {
  provider = ucloudsh
  availability_zone = "cn-sh2-01"
  name_regex = "^CentOS 6.5 64"
  image_type = "base"
}

For example, in the above example, we have declared two UCloud Providers, which are located in the Beijing area and the Shanghai area respectively. We explicitly specify the provider's Local Name in the following data declaration, which allows us to operate resources in different regions and accounts at the same time in a set of configuration files.

We can also use alias aliases to distinguish different instances of the same Provider:

terraform {
  required_version = ">=0.13.5"
  required_providers {
    ucloud = {
      source = "ucloud/ucloud"
      version = ">=1.24.1"
    }
  }
}

provider "ucloud" {
  public_key = "your_public_key"
  private_key = "your_private_key"
  project_id = "your_project_id"
  region = "cn-bj2"
}

provider "ucloud" {
  alias = "ucloudsh"
  public_key = "your_public_key"
  private_key = "your_private_key"
  project_id = "your_project_id"
  region = "cn-sh2"
}

data "ucloud_security_groups" "default" {
  type = "recommend_web"
}

data "ucloud_images" "default" {
  provider = ucloud.ucloudsh
  availability_zone = "cn-sh2-01"
  name_regex = "^CentOS 6.5 64"
  image_type = "base"
}

Compared to multiple Local Names, using aliases allows us to distinguish between different instances of the provider. Only ucloud is declared once in the required_providers of the terraform section, and ucloud.ucloudsh is passed in when the provider is specified in data. Use aliases for multi-instance providers.

Every provider declaration without the alias attribute is a default provider declaration. The data and resource that do not explicitly specify the provider use the provider corresponding to the first word of the default resource name. For example, the default provider corresponding to the data of ucloud_images is ucloud, and the default provider corresponding to the resource aws_instance is aws.

If all explicitly declared providers in the code have aliases, then the Terraform runtime will construct a default provider with all configurations empty. If the provider has required fields, and another resource uses the default provider, Terraform will throw an error complaining that the default provider is missing the required fields