»Terraform Cloud Agents

Hands-on: Endeavor the Manage Private Environments with Terraform Cloud Agents tutorial on HashiCorp Larn.

Note: Terraform Cloud Agents are a paid feature, available as function of the Terraform Deject for Business upgrade package. Learn more about Terraform Cloud pricing here. The number of agents you lot are eligible to deploy is determined by the number of concurrent runs your organisation is entitled to.

Terraform Cloud Agents allow Terraform Cloud to communicate with isolated, private, or on-bounds infrastructure. By deploying lightweight agents within a specific network segment, y'all can constitute a uncomplicated connexion betwixt your environment and Terraform Cloud which allows for provisioning operations and management. This is useful for on-premises infrastructure types such as vSphere, Nutanix, OpenStack, enterprise networking providers, and anything yous might have in a protected enclave.

The agent architecture is pull-based, so no inbound connectivity is required. Any agent you lot provision will poll Terraform Cloud for work and carry out execution of that piece of work locally.

»Before Install

»Supported Operating Systems

Agents currently only support x86_64 chip Linux operating systems. You can as well run the amanuensis inside Docker using our official Terraform Agent Docker container.

»Supported Terraform Versions

Agents support Terraform versions 0.12 and above. Workspaces configured to utilize Terraform versions beneath 0.12 will not be able to select the agent-based execution mode.

»Hardware Requirements

The host running the agent will have varying resource requirements depending on the workspace. A host can be a dedicated or shared deject instance, virtual machine, bare metal server, or a container. Y'all should monitor and adjust memory, CPU, and disk space based on each workspace's usage and performance. The name of your instance blazon may vary depending on your deployment environs.

Y'all can apply the specs beneath as a reference:

  • At least 4GB of complimentary deejay space
    • Each run requires the amanuensis to temporarily store local copies of the tarred repository, extracted repository, state file, any providers or modules, and the Terraform binary itself.
  • At least 2GB of system memory

»Networking Requirements

In gild for an agent to office properly, it must be able to make outbound requests over HTTPS (TCP port 443) to the Terraform Cloud application APIs. This may require perimeter networking every bit well as container host networking changes, depending on your surround. The IP ranges are documented in the Terraform Cloud IP Ranges documentation.

Additionally, the agent must likewise be able to communicate with any services required past the Terraform code information technology is executing. This includes the Terraform releases distribution service, releases.hashicorp.com (supported past Fastly), as well every bit any provider APIs. The services which run on these IP ranges are described in the table beneath.

Hostname Port/Protocol Directionality Purpose
app.terraform.io tcp/443, HTTPS Outbound Polling for new workloads, providing condition updates, and downloading individual modules from Terraform Deject'south Individual Module Registry
registry.terraform.io tcp/443, HTTPS Outbound Downloading public modules from the Terraform Registry
releases.hashicorp.com tcp/443, HTTPS Outbound Updating amanuensis components and downloading Terraform binaries
archivist.terraform.io tcp/443, HTTPS Outbound Blob Storage

»Operational Considerations

The agent is distributed as a standalone binary which tin be run on whatsoever supported system. By default, the agent will run in the foreground equally a long-running procedure that will continuously poll for workloads from Terraform Cloud. Nosotros strongly recommend pairing the agent with a process supervisor to ensure that it is automatically restarted in case of an error.

Agents do not guarantee a clean working environment per Terraform execution. Each execution is performed in its own temporary directory with a make clean surround, but references to absolute file paths or other machine land may cause interference between Terraform executions. We strongly recommend that you write your Terraform code to be stateless and idempotent. You may also want to consider using single-execution mode to ensure your agent simply runs a unmarried workload.

»Updating

By default, the agent will automatically update itself to the latest pocket-sized version. Administrators are required to update the host operating organisation and all other installed software.

To customize this update behavior, pass the flag -car-update or set the surroundings variable TFC_AGENT_AUTO_UPDATE. The valid options are presented in the table beneath.

Update Setting Behavior
small Matches the default behavior, automatically update the agent to the latest small-scale version.
patch The agent will only be updated to the newest patch version, new small-scale versions will crave a manual update.
disabled Disables automated updates, all updates volition be manual.

»Security Considerations

Agents should be considered a global resource within an organization. Once configured, whatsoever workspace owner may configure their workspace to target the organization'south agents. This may allow a malicious workspace to admission state files, variables, or code from other workspaces targeting the same agent, or access sensitive information on the host running the agent. For this reason, nosotros recommend carefully considering the implications of enabling agents inside an organization, and restricting access to your organization to only trusted parties.

»Limitations

Agents allow you to run Terraform operations from a Terraform Cloud workspace on your private infrastructure. Agents do non back up:

  • Connecting to individual infrastructure from Picket policies using the http import.
  • Connecting Terraform Cloud workspaces to VCS instances that practice not allow access from the public internet. For example, you cannot utilise agents to connect to a GitHub Enterprise Server instance that requires access to your VPN.

For these use cases, we recommend you leverage the data provided past the IP Ranges documentation to permit directly communication from the appropriate Terraform Cloud service to your internal infrastructure.

Organizations are express to 20 pools each.

»Terraform Enterprise

Terraform Enterprise supports Terraform Deject Agents; see Terraform Cloud Agents on TFE for TFE-specific documentation and requirements.

»Managing Amanuensis Pools

Agents are organized into pools, which can be managed in Terraform Cloud'due south UI. Each workspace tin specify which amanuensis puddle should run its workloads.

Notation: You must exist a member of the "Owners" team inside your organization in order to manage an system's agents in Terraform Cloud. (More most permissions.)

»Create a new Agent Pool

  1. Navigate to Organization Settings > Agents and click "New amanuensis pool".

    Screenshot: The Agents page with no pools

  2. Give your pool a proper name, then click "Continue". This proper name will be used to distinguish your pools when changing the settings of a workspace.

    Screenshot: Step 1 of pool creation, naming the agent pool

  3. Give your token a clarification and click "Create Token".

    Note: Your token data will not be displayed once more. Make sure to save it appropriately before moving to the final footstep.

    Screenshot: Step 2 of pool creation, token creation

  4. Click "Terminate".

»Delete an Agent Puddle

  1. Navigate to Organization Settings > Agents and click on the proper name of the pool y'all would like to delete.

  2. Click "Delete agent pool".

    Screenshot: Deleting an agent pool

  3. Ostend the deletion of the pool past clicking "Yeah, delete agent pool".

    Screenshot: Confirmation to delete an agent pool

    Important: Agent pools which are however associated with a workspace are unable to exist deleted. To delete these pools, first ensure the related workspace has completed all in progress runs, and remove the clan to the agent pool in Workspace Settings > General Settings.

    Screenshot: Unable to delete an agent pool with attached workspace

»Revoke an Agent Token

Yous may revoke an issued token from your agents at any fourth dimension.

Revoking a token will cause the agents using it to exit. For agents to go on servicing workspace jobs, they must be reinitialized with a new token. Under normal circumstances, it may be desirable to generate a new token first, initialize the agents using it, so revoke the old token one time no agents are using it. Agent tokens display information nigh the final fourth dimension they were used to assistance you determine whether they are safe to revoke.

  1. Navigate to Organization Settings > Agents, then click on the agent puddle y'all would like to manage.

  2. Click "Revoke Token" for the token you would like to revoke.

    Screenshot: The Agent tokens list with the "Revoke Token" link highlighted

  3. Confirm the deletion of the token by clicking "Yes, delete token".

    Screenshot: The "Revoke agent token" confirmation modal

»Managing Agents

The amanuensis software runs on your own infrastructure. Amanuensis pool membership is determined past which token you provide when starting the agent.

»Download and Install the Agent

  1. Download the latest amanuensis release, the associated checksum file (.SHA256sums), and the checksum signature (.sig).
  2. Verify the integrity of the downloaded annal, equally well as the signature of the SHA256SUMS file using the instructions available on HashiCorp's security page.
  3. Extract the release annal. The unzip utility is bachelor on most Linux distributions and may be invoked equally unzip <archive file>. Two private binaries volition be extracted (tfc-amanuensis and tfc-agent-cadre). These binaries must reside in the same directory for the agent to part properly.

»Outset the Amanuensis

Using the Agent token you copied earlier, set the TFC_AGENT_TOKEN and TFC_AGENT_NAME environment variables.

export TFC_AGENT_TOKEN=your-token                             consign TFC_AGENT_NAME=your-amanuensis-proper name                             ./tfc-agent                          
                                                                            export TFC_AGENT_TOKEN=your-token                      export TFC_AGENT_NAME=your-amanuensis-name                      ./tfc-agent

Note: The TFC_AGENT_NAME variable is optional. If you do not specify a name hither, i will not be displayed. These names are for your reference only, and the agent ID is what volition announced in logs and API requests.

In one case consummate, your agent volition appear on the Agents page and brandish its current status.

Screenshot: The Agents page with one connected agent in the idle state

»Optional Configuration: Running an Amanuensis using Docker

Alternatively, you can use our official agent Docker container to run the Amanuensis.

docker pull hashicorp/tfc-agent:latest                             docker run -e TFC_AGENT_TOKEN=your-token -eastward                             TFC_AGENT_NAME=your-agent-name hashicorp/tfc-amanuensis                          
                                                                            docker pull hashicorp/tfc-agent:latest                      docker run -e TFC_AGENT_TOKEN=your-token -e                      TFC_AGENT_NAME=your-agent-name hashicorp/tfc-amanuensis

This Docker epitome executes the tfc-agent process every bit the non-root tfc-agent user. For some workflows, such every bit those that require the power to install software via apt-get during local-exec scripts, yous may need to build a customized version of the agent Docker epitome for your internal employ.

                              FROM                hashicorp/tfc-agent:latest                                                          USER                root                                                            RUN                apt-get -y install sudo                                                           RUN                repeat                'tfc-agent ALL=NOPASSWD: /usr/bin/apt-get , /usr/bin/apt'                >> /etc/sudoers.d/50-tfc-agent                                                          USER                tfc-agent            
                                                                                                                              FROM                          hashicorp/tfc-agent:latest                                                                                                                                          USER                          root                                                                                                                # Install sudo. The container runs as a non-root user, only people may rely on                                                                    # the ability to apt-go install things.                                                                                              RUN                          apt-get -y install sudo                                                                                                                # Permit tfc-agent to use sudo apt-get commands.                                                                                              RUN                          echo                          'tfc-agent ALL=NOPASSWD: /usr/bin/apt-get , /usr/bin/apt'                          >> /etc/sudoers.d/50-tfc-amanuensis                                                                                                                                          USER                          tfc-amanuensis

An image customized in this fashion will permit installation of additional software via sudo apt-get.

»Stopping the Agent

The amanuensis maintains a registration and a liveness indicator within Terraform Cloud during the entire course of its runtime. When an agent is to exist retired, it must deregister itself from Terraform Cloud. The agent performs deregistration automatically as part of its shutdown procedure in the following scenarios:

  • If using an interactive concluding, Ctrl-C is pressed.
  • One of SIGINT, SIGTERM, or SIGQUIT is sent to the agent procedure ID. It is important to ship only 1 indicate. If a second signal is received past the agent, it is interpreted as a forceful termination signal and volition cause the amanuensis to exit immediately.

In both cases, after initiating a graceful shutdown, the last user or parent program should wait for the agent to exit. The amount of time this takes depends on the agent's electric current workload. The agent will expect for whatsoever current operations to complete before deregistering and exiting.

It is highly recommended that the amanuensis is only terminated using one of the higher up methods. Abruptly terminating an amanuensis by forcefully killing the process, power cycling the host, etc., will non provide the agent the opportunity to deregister, and will result in an Unknown amanuensis condition. This may cause further capacity issues, as outlined beneath in Agent Capacity Usage.

»Optional Configuration: Single-execution mode

The Amanuensis tin also be configured to run in single-execution fashion, which will ensure that the Agent just runs a single workload, then terminates. This can be used in combination with Docker and a process supervisor to ensure a make clean working environs for every Terraform run.

To employ single-execution mode, starting time the amanuensis with the -single command line statement.

»Configuring Workspaces to use the Agent

Note: You must have "Admin" access to the workspace you are configuring to alter the execution mode. (More virtually permissions.)

Important: Changing your workspace'southward execution manner after a run has already been planned will cause the run to error when it is practical. To minimize the number runs that error, you should disable auto-apply, complete any runs that are no longer in the pending stage, and lock your workspace before changing the execution way.

To configure a workspace to execute runs using an agent:

  1. Open the workspace from the chief "Workspaces" view, then navigate to "Settings > Full general" from the dropdown menu.
  2. Select Amanuensis equally the execution mode, as well equally the agent puddle this workspace should use.
  3. Click "Save Settings" at the bottom of the folio.

Screenshot: The Workspace General settings page, with agent selected for execution mode

»Run Details

Runs which are candy past an amanuensis volition accept additional information virtually that agent in the details section of the run:

Screenshot: Run details with agent information

Note: Different agents may be used for the plan and use operations, depending on agent availability inside the pool.

»Running Multiple Agents

You lot may choose to run multiple agents inside your network, up to the organization's purchased amanuensis limit. If in that location are multiple agents available inside an arrangement, Terraform Cloud volition select the first bachelor agent within the target pool.

Each agent process will run a unmarried Terraform run at a time. Multiple agent processes can be concurrently run on a single case, license limit permitting.

»Resilience

Information technology is possible that an amanuensis process could be terminated unexpectedly (due to killing the procedure forcefully, power cycling the host motorcar, etc.). We strongly recommend pairing the amanuensis with a process supervisor to ensure that it is automatically restarted in case of an error.

(Meet Agent Capacity Usage below).

»Troubleshooting

»Viewing Agent Statuses

Screenshot: Possible agent statuses

Agent status appear on the Organization Settings > Agents page and will contain one of these values:

  • Idle: The agent is running usually and waiting for jobs to be available.
  • Busy: The agent is running normally and currently executing a job.
  • Unknown: The agent has not reported whatever condition for an unexpected menstruum of time. The agent may yet recover if the agent's situation is temporary, such as a brusque-lived network partitioning.
  • Errored: The agent encountered an unrecoverable error or has been in an Unknown state for long enough that Terraform Cloud considers it errored. This may indicate that the amanuensis procedure was interrupted, has crashed, a permanent network sectionalization exists, etc. If the amanuensis was in the process of running an functioning (such equally a plan or apply), that functioning has been marked equally errored. If the current agent process does manage to recover, it will be instructed to get out immediately.
  • Exited: The agent exited usually, and has successfully informed Terraform of it doing then.

»Agent Chapters Usage

Agents are considered active and count towards the system's purchased agent chapters if they are in the Idle, Decorated, or Unknown state. Agents which are in the Errored or Exited state do not count towards the arrangement's total agent capacity.

The number of active agents you are eligible to deploy is determined by the number of Concurrent Runs your organization is entitled to. Agents are available as part of the Terraform Cloud Business tier.

Agents in the Unknown state continue to be counted against the organization's total agent allowance, as this status is typically an indicator of a temporary advice outcome betwixt the amanuensis and Terraform Cloud. Unknown agents which do not reply after a flow of 2 hours volition automatically transition to an Errored country, at which point they will not count confronting the amanuensis allowance.

Agents may have an Unknown condition if they are terminated without gracefully exiting. Agents should always be shut downwards according to the Stopping the Agent department to permit them to deregister from Terraform Cloud. We strongly recommend ensuring that any process supervisor, application scheduler, or other runtime manager is configured to follow this procedure to minimize Unknown agent statuses.

»Viewing Agent Logs

Output from the Terraform execution will be visible on the run's folio within Terraform Cloud. For more in-depth debugging, you may wish to view the amanuensis'southward logs, which are sent to stdout and configurable via the -log-level command line statement. By default, these logs are not persisted in any mode. It is the responsibility of the operator to collect and store these logs if they are needed.