Consul security considerations

This guide provides a high-level overview of each security consideration and links to additional resources. With Consul, there are several important considerations to consider when planning for security. You need to manage three distinct types of secrets within a Consul deployment; TLS certificates, ACL tokens, and gossip encryption keys.

TLS certificates

• Certificate Authority (Root & Intermediate)

  • Consul contains two different types of built-in Certificate Authorities:

    • Consul Agent CA (also referred to as “built-in” CA) - this is optional for both Consul server agents and Consul client agents. If used, this will provide RPC encryption on server agents and client agents. The CA and certificates are created with the CLI as shown below.

      $ consul tls ca create && consul tls cert create -server && \consul tls cert create -client

    • Consul service mesh CA - this is used for mTLS authentication within a service mesh and the auto-encrypt feature for Consul client agents. Auto-encrypt cannot be used for Consul server agent certificates.

  • You can use your own CA to sign and issue both server and client certificates, however, you must include the following subject alternative names to the server certificates (without these you will get bad certificate errors and the Consul agent will not start):

    • DNS: server.dc1.consul
    • DNS: localhost
    • IP address: 127.0.0.1

  Note

  HashiCorp Vault can be used as a Certificate Authority for Consul and can handle both generation and renewal of dynamic X.509 certificates.

• Server Certificates

  • Can use built-in Consul Agent CA
  • Can use 3rd party CA

• Client Certificates

  • Can use built-in Consul Agent CA
  • Can use 3rd party CA
  • Can use auto_encrypt (Connect CA) or auto_config options to automatically distribute client certificates

Review the TLS certificates tutorial for step-by-step setup guidance.

Certificate summary

TLS best practices

The following is a list of recommended practices for TLS encryption.

• Ensure that certificates have an expiration period that aligns with your company’s guidelines.

• Enable either auto_encrypt or auto_config to automatically distribute client certificates.

• For automatic key rotation and certificate renewals, the use of a secrets management solution such as HashiCorp Vault is strongly recommended. The Generate mTLS Certificates for Consul with Vault tutorials has step-by-step guidance.

• Avoid the use of wildcard certificates to reduce possible attack surface. Instead, leverage named certificates with the use of subject alternative name (SAN).

• Verify the server hostname setting verify_server_hostname is set to ensure that the client only accepts the certificate files from a server that presents a certificate matching “server.[dc].consul”.

• Secure API/UI/CLI with HTTPS.

  • If there is a requirement to expose the API/CLI/UI outside of the Consul cluster, the use of HTTPS is strongly encouraged.

    Note

    Firewall/Security group rules will need to be adjusted to provide access to these services.

  • In order to leverage HTTPS with the API/UI/CLI, a separate certificate should be generated. Review the Secure Consul Agent Communication with Encryption and Certificates tutorial to learn how to create and distribute these certificates.

  • Once HTTPS has been enabled, HTTP should be disabled by leveraging the ports stanza in the server and client configuration.

    HCLJSON

    ports = {   https = 8501   http = -1}

    "ports": {   "https": 8501   "http" : -1}

  • Unless you have configured environment variables or include specific parameters in consul commands, the CLI and API will be inaccessible after enabling HTTPS. The following environment variables and CLI commands can be used to configure the appropriate values.

    Environment VariablesCLI

    $ export CONSUL_HTTP_ADDR=https://server.dc1.consul:8501$ export CONSUL_CACERT=consul-agent-ca.pem$ export CONSUL_CLIENT_CERT=cli.client.dc1.consul.crt$ export CONSUL_CLIENT_KEY=cli.client.dc1.consul.key

    $ consul members \ -http-addr="https://server.dc1.consul:8501" \     -ca-file="consul-agent-ca.pem" \-client-cert="cli.client.dc1.consul.crt" \ -client-key="cli.client.dc1.consul.key"

Consul ACL tokens

Tokens are tied to policies that grant permission to various API actions. It's important to consider that ACL tokens do not expire. They must be removed by other processes.

You can use the Consul CLI to create ACL policies.

$ consul acl policy create -name <policy_name> -rules @policy.hcl

You can use the Consul CLI to create ACL tokens.

$ consul acl token create -description "description" -policy-name <policy_name>

Review the ACL production tutorial for step-by-step guidance.

Intitial management token

Consul ACLs are not enabled until they have been “bootstrapped”. The bootstrap process enables ACLs in the Consul cluster and provides an initial output that contains the bootstrap token in it. The command consul acl bootstrap will generate an ACL token, the bootstrap token, that is tied to the Global Management policy, granting full privileges to the Consul datacenter, and that can be used initially to configure the ACL system.

$ consul acl bootstrap

{    "CreateIndex": 18,    "ModifyIndex": 18,    "AccessorID": "2268e5dc-17fc-02d5-7af9-da8584f16cfe",    "SecretID": "af410085-8bf6-80d6-08ad-2bfaa6526256",    "Description": "Bootstrap Token (Global Management)",    "Policies": [        {            "ID": "00000000-0000-0000-0000-000000000001",            "Name": "global-management"        }    ],    "Local": false,    "CreateTime": "---",    "Hash": "X2AgaFhnQGRhSSF/h0m6qpX1wj/HJWbyXcxkEM/5GrY="}

ACL node identity

An ACL node identity is an ACL policy template for expressing a link to a policy suitable for use as an Consul agent token. They are usable on both tokens and roles and are composed of the following elements:

• Node Name - The name of the node to grant access to.
• Datacenter - The datacenter that the node resides within.

They allow you to generate specific node tokens without the need to manually generate the respective policies.

You can use the Consul CLI to create ACL node tokens.

$ consul acl token create -description "description" -node-identity=<nodename:datacenter>

ACL service identity

This is similar to the Consul node token specified above, but would be specific to each service that is managed by Consul (e.g. web, database, etc.)

An ACL service identity is an ACL policy template for expressing a link to a policy suitable for use in Consul service mesh. They are usable on both tokens and roles and are composed of the following elements:

• Service Name - The name of the service.
• Datacenters - A list of datacenters the effective policy is valid within. (Optional)

You can use the Consul CLI to create ACL node tokens.

$ consul acl token create -description "description" -service-identity=<servicename:datacenter>

ACL token best practices

The following is a list of recommended practices for ACLs.

• The default policy should be set to “deny” on a new Consul cluster. On an existing Consul cluster, it should be set to “allow”, until ACL tokens have been distributed to all nodes.

• ACLs should be created for either every server role type or every agent node through the use of rules & policies. Each customer will need to evaluate the most appropriate approach based on their security requirements.

  • For example, using an exact match resource rule is the most secure. It grants the least privileges necessary to accomplish the task. Generally, creating policies and tokens with the least privileges will result in more policy definitions. Alternatively, for a simplified process, the prefix resources rules can apply to zero-to-many objects (eg web servers, database servers, etc.) The trade-off of a less complicated token creation process is the potential for wider blast radius on the token or possible workload compromise. For additional details, refer to the Understand Access Control Privileges tutorial.

• Use of ACL tokens associated with the Global Management policy, granting unrestricted permission to the datacenter resources, should be restricted to the initial bootstrap configuration. Once the initial config is complete and additional ACL tokens have been created, including a new emergency Global Management token, the original bootstrap token should be deleted. Additionally, this new Global Management token should be rotated on a regular basis in alignment with a customer’s security requirements.

• Leverage Vault to manage ACLs within the environment.

• If more granular access controls are required, Consul Enterprise supports the use of Sentinel for more fine-grained and logic-based policies. Sentinel policies extend the ACL system in Consul beyond static "read", "write", and "deny" policies to support full conditional logic and integration with external systems.

Gossip encryption key

Gossip is unencrypted by default and should be encrypted on both new and existing Consul clusters. When gossip encryption is enabled, a line similar to the following should appear in the startup logs.

Encrypt: Gossip:true, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false

For additional information on how to generate gossip encryption keys for both new and existing Consul clusters, review the Secure Gossip Communication with Encryption tutorial.

Gossip encryption best practices

We recommend that you regularly rotate the gossip encryption key to improve the overall security of Consul.

• Rotate Gossip Encryption Keys in Consul

A tutorial is also available to help you automate the rotation of gossip keys through the usage of Vault.

• Leverage Vault to automatically rotate encryption keys

Additional security best practices

Enabling script checks via the -enable-script-checks configuration setting may introduce a remote execution vulnerability that is known to be targeted by malware. We strongly recommend -enable-local-script-checks instead. See this blog post for more details.

Configure client agents to bind to localhost for the HTTP, HTTPS, DNS, and gRPC APIs. This ensures these APIs are not exposed outside the node where the client agent is running. Consul client agents bind to localhost by default if the addresses and client_addr config options are not specified.

Verify that the version of Consul deployed within the environment does not have any active CVEs that are relevant by visiting the following page: Hashicorp Discuss: Consul Security

Leverage Consul intentions within a service mesh for defining rules which allow service-to-service communication.

Access summary

Each of the secret types listed above provides security for a specific part of the system. The table below details the security features and the relevant components of the system that is being secured.

Additional references

• Consul Security
• Consul Agent Configuration & TLS Encryption
• Consul ACLs
• Consul Intentions
• Secure Agent Communication with Existing Certificate Authority
• Consul Service Mesh Security
• Consul Service Mesh CA
• Day 1: Consul Security
• Consul Datacenter Security Operations
• Automate Consul Agent Security with Auto Config
• Enforce a Zero-trust Network with Consul Service Mesh
• Administer Consul Access Control Tokens with Vault
• Automatically Rotate Gossip Encryption Keys Secured in Vault
• Generate mTLS Certificates for Consul with Vault
• HashiCorp Sentinel