hashicorp / syntax

TextMate grammars for highlighting HCL, HCL-based languages and Sentinel.

HashiCorp Syntax

TextMate grammars for highlighting configuration used by HashiCorp software, primarily HCL (HashiCorp Configuration Language) and HCL-based languages.

How do I use this?

As end-user

Generally you should never need to interact with this repository directly.

Instead you can use the grammar through editors, editor extensions, or highlighting libraries which in turn may use this repository, for example:

• Terraform VS Code Extension
• HCL VS Code Extension
• Sentinel VS Code Extension

As maintainer enabling end-users

If your editor or other syntax highlighting engine supports TextMate, you can introduce a build-time dependency along these lines:

VERSION=0.1.0
URL="https://github.com/hashicorp/syntax/raw/v${VERSION}/syntaxes/hcl.tmGrammar.json"
curl -fL $URL > ./hcl.tmGrammar.json

We may consider distributing the grammars via a packaging system in the future.

Grammar Mapping

Most highlighting engines match files based on filename patterns. Matched files are then mapped to language IDs. With that in mind we recommend the following mapping.

Pattern	Language ID	Language
*.hcl	hcl	HCL
*.tf	terraform	Terraform (configuration)
*.tfvars	terraform-vars	Terraform (variables)
*.sentinel	sentinel	Sentinel (policies)

Language IDs are also often used in highlighting Markdown code blocks. Picking the right language IDs is key to enable accurate and consistent highlighting everywhere and to enable IntelliSense (such as completion) in editors.

As explained below you can use the HCL grammar to highlight e.g. Terraform files but you should still assign schema-specific language IDs (such as terraform or terraform-vars) to these files.

How do I contribute?

See instructions in DEVELOPMENT.md.

Non-TextMate grammars

We do not accept contributions of non-TextMate grammars at this point as we do not have any established strategy for testability and sustainable maintenance of multiple grammars.

In the meantime you can use other available grammars and highlighting solutions, e.g.

• mitchellh/tree-sitter-hcl
• PrismJS
• Rouge

You may take inspiration from the regular expressions used in our TextMate grammar, assuming your highlighting engine is regex-based.

Product X grammar

HashiCorp Products

We welcome contributions of other product-based grammars (such as Packer) and fixes for any existing grammars.

Bear in mind that TextMate grammars may generally not provide perfectly accurate product-specific highlighting given the complexity of how HCL schemas are applied in various products. Editors are expected to use LSP semantic tokens to compensate for this. We also generally aim to reflect the latest version of the product as TextMate grammars have no way of communicating product versions.

3rd Parties

This repository is intended for grammars of HashiCorp-maintained products. Therefore we do not accept contributions of grammars for HCL-based languages used by 3rd party products. These can be maintained in their own repositories.

Maintainers of such 3rd party grammars are however welcomed to use the provided HCL grammar or take inspiration from any provided product grammar to build their own. Namely we recommend the use of "include": "source.hcl#pattern-name" in such a 3rd party product grammar to take advantage of the HCL grammar and treat it as an upstream dependency.

Also make sure to pick a unique language ID (that is different from the ones above) where applicable.

HCL vs (Terraform / Packer / ...)

In Short: Can I use grammar X for file Y?

• ❌ Do not use product grammar (e.g. Terraform) for other products (e.g. Packer)
• ❌ Do not use product grammar (e.g. Terraform) for "generic" HCL
• ✅ Do use HCL grammar to highlight any HCL-based language (incl. Terraform or Packer) where no alternatives exist.

Long version

HCL is a schema-based language. Schema is an integral part of any HCL-based language as it declares the names of blocks, names of attributes, expected expressions and data types etc. You can think of it as similar to JSON schemas in JSON.

The language itself (HCL) cannot and should not imply any schema, i.e. does not imply any block or attribute names. Schema is what makes the language product-specific.

To put it simply:

HCL + Schema (e.g. Terraform or Packer) = Product-specific language (Terraform language or Packer language).

To provide any product-specific feature, either more precise product-specific highlighting or any IntelliSense, such as completion, hover, go-to-definition etc., editors need to know what schema to use.

For example, to understand the meaning of resource block in Terraform when user hovers over it, the editor needs to know the file is a Terraform configuration file. Unlike terraform, hcl does not convey that information.

Schemas may also differ even in the context of a single product (e.g. terraform vs terraform-vars). This is why it's essential to use documented language IDs.

Why does the TextMate grammar use scope X instead of Y?

Picking the right scope is a balancing act between the following:

1. TextMate's own documented naming conventions
2. meaningful semantic mapping between naming conventions and DSL (HCL) constructs
3. resulting UX when chosen scopes are rendered via common/default themes in editors (which may or may not reflect 1. and 2.)

This has a potential downside for authors of custom themes who may want to be more specific, or generally expect the scopes to more intuitively reflect names of HCL constructs, such as "block", "attribute" etc.

We plan to address these needs via custom semantic tokens and modifiers within VSCode (see hashicorp/vscode-terraform#958 for more details) and remain open to feedback how this works in other editors and contexts.

What about HCL JSON?

We recommend HCL JSON configuration files to be highlighted as plain JSON at this time, for which there should be existing grammars maintained by the community. We do not have an established convention for language IDs for these files as it is unclear whether and how language servers should interact with these files.

LSP Semantic Token Based Highlighting

LSP (Language Server Protocol) 3.16 introduced language server-driven highlighting. Language server is typically better equipped to provide more contextual and accurate highlighting as it can parse the whole AST, unlike a TextMate grammar operating on regex-basis.

Is LSP the way forward?

LSP 3.17 does support use cases where semantic highlighting is the only way to highlight a file (through augmentsSyntaxTokens client capability). However in the context of HCL we recommend semantic highlighting to be used as an addition to highlight product-specific pieces of an HCL-based language (such as Terraform) and improve any inaccuracies of product-specific static grammars, rather than to semantically highlight generic HCL.

Note there are also environments outside of editors in which LSP is not a suitable way of highlighting code - e.g. code rendered on websites such as GitHub or terraform.io.

syntax's Issues

Add map to sentinel keyword highlighting

      "match": "\\b(if|case|for|any|all|filter|map|break|continue|return)\\b"

https://docs.hashicorp.com/sentinel/language/spec#quantifier-expressions-any-all-filter-map

Originally posted by @radeksimko in #10 (comment)

Double curly braces incorrectly highlighted in strings

This came up in hashicorp/vscode-hcl#86

Vault has templated ACL policies that use {{ and }} as templating delimiters. Since they're used within a string, we should treat them like any other character.

Incorrect highlight

Mark repo as public

When the vscode-terraform and vscode-hcl repos have been successfully setup to pull grammar from this repo, mark public

syntax: Broken highlighting of nested complex types within `object({ ... })`

Issue Description

variable "test" {
  type = object({
    test_object = map( # here
      object({ # here
        test_map = string
      })
    )
  })
}

Currently the above snippet fails to highlight the nested map and object types.

Interestingly, nested types under map are highlighted correctly:

Environment Information

Terraform Information

Version: 1.1.5
Platform: darwin_arm64
Outdated: true

Visual Studio Code

Name	Version
Operating System	Darwin arm64 21.3.0
VSCode	1.64.2

Visual Studio Code Extensions

Visual Studio Code Extensions(Click to Expand)

Extension	Author	Version
azure-account	ms-vscode	0.10.0
codetour	vsls-contrib	0.0.58
esbuild-problem-matchers	connor4312	0.0.2
go	golang	0.31.1
json	ZainChen	2.0.2
remote-containers	ms-vscode-remote	0.217.4
terraform-preview	hashicorp	2022.3.115
vscode-docker	ms-azuretools	1.20.0
vscode-eslint	dbaeumer	2.2.2

Extension Logs

Find this from the first few lines of the relevant Output pane:
View -> Output -> 'HashiCorp Terraform'

Highlight static segments of address (e.g. `var`) differently

Hi,

Terraform extension v2.13.0

See the screenshot, note how sometimes the text var is in blue and not white.

I think this happens when the terraform VSC extension knows the value should be a boolean or number rather than a string.

I am sure this has only started to happen in the last few days/weeks.

Variables with no space between them break syntax highlighting

Raised via hashicorp/vscode-terraform#1040

Example code

resource "aws_db_parameter_group" "main" {
  name_prefix = var.name
  family      = "${var.engine}${var.version}"
}

Build POC generator of grammar via templates/configs

• pool of patterns / a la TM pattern repository
• from the pool we generate HCL TM grammar
• from the pool we generate Terraform TM grammar

Copy HCL grammar patterns to Terraform grammar

The goal is to mainly ensure that we don't "downgrade" users from the existing https://github.com/hashicorp/vscode-terraform/blob/main/syntaxes/terraform.tmGrammar.json

Specifically the copy pasting is intending to address

• function calls highlighted correctly within blocks
• #19
• hashicorp/vscode-terraform#8
• hashicorp/vscode-terraform#9

and unblock any automation downstream in vscode-terraform and continued releasing of the extension with improved syntax, while we work on #TODO

Problem parsing block with dash in name

If the name of a block contains a dash, the parsing is incorrect.

For instance, using Packer post-processor block, the name isn't well parsed.

build {
  sources = ["source.docker.sysbox"]

  post-processor "docker-tag" {
    repository = var.repository
    tags       = var.tags
  }
}

See the current snapshot for the post-processor block

>post-processor "docker-tag" {}
#^^^^ source.hcl meta.block.hcl entity.name.type.hcl
#    ^^^^^^^^^^ source.hcl meta.block.hcl variable.other.enummember.hcl
#              ^ source.hcl meta.block.hcl
#               ^^^^^^^^^^^^ source.hcl meta.block.hcl variable.other.enummember.hcl
#                           ^ source.hcl meta.block.hcl
#                            ^ source.hcl meta.block.hcl punctuation.section.block.begin.hcl
#                             ^ source.hcl meta.block.hcl punctuation.section.block.end.hcl

The parsing should be the following.

>post-processor "docker-tag" {}
#^^^^^^^^^^^^^^ source.hcl meta.block.hcl entity.name.type.hcl
#              ^ source.hcl meta.block.hcl
#               ^^^^^^^^^^^^ source.hcl meta.block.hcl variable.other.enummember.hcl
#                           ^ source.hcl meta.block.hcl
#                            ^ source.hcl meta.block.hcl punctuation.section.block.begin.hcl
#                             ^ source.hcl meta.block.hcl punctuation.section.block.end.hcl

Improve highlighting of built-in functions

The current version of the Terraform language grammar contains a list of built-in functions. However, as we don't know which version of Terraform a user uses, it is highly possible that the list is either incomplete or contains removed functions.

https://github.com/hashicorp/vscode-terraform/blob/562c7cd7b27b5599b4fb1d4ae21b4fc63639dd91/syntaxes/terraform.tmGrammar.json#L750

To improve the syntax highlighting of built-in functions, we should remove the static list from the grammar file and use semantic token highlighting via the LS. The language server has a better understanding of which Terraform version is used and which functions are available in this version.

Wrong highlighting after `==` and a template expression

This only affects the HCL grammar and didn't make it's way into the Terraform grammar yet.

Example

assert {
  condition     = aws_s3_bucket == "${run}-aws-s3-website-test"
  error_message = "Invalid bucket name"
}

Expected

Support provider-defined functions syntax

• Part of hashicorp/vscode-terraform#1674

Background

We currently only have a static list of valid function names in the Terraform grammar. Provider-defined functions introduce a new syntax for function calls, provider::aws::noop(). Also, it's impossible to compile all function names as a static list in the grammar, so we'll just assume that any function call with this new snytax is valid.

Proposal

• Update Terraforms functions grammar block, e.g.

  functions:
    begin: ([:\w]+)(\()
    name: meta.function-call.hcl
    comment: Built-in and provider defined function calls
    beginCaptures:
      "1":
        patterns:
          - match: \b(abs|abspath|...|zipmap)\b
            name: support.function.builtin.terraform
          - match: \b(provider::\w+::\w+)\b
            name: support.function.provider
      "2":
        name: punctuation.section.parens.begin.hcl
    end: \)

• Add some tests for the new syntax

Document use of grammars in template/expression context

History

The main context in which we expect HCL/Terraform grammars to be used is to highlight whole HCL "bodies". For example

attr = "foobar"
block "label" {
  another = 42
}
ref = data.aws_instance.foo.tags["noot"]

The "expression" on the RHS after the = is part of the existing grammars and we happen to treat standalone expressions, such as

42
"foobar"
data.aws_instance.foo.tags["noot"]

The decision to treat standalone expressions as valid was mostly pragmatic, where it was easier to do it than not to, due to the way TextMate grammars and nesting inside works.

To date however, this was a rather "hidden feature".

Potential

There are context in which this can be genuinely useful. For example:

• rendering full object structures in highlighted hover data (markdown), e.g. { foo = string, bar = set(number) }
• template files, which are effectively the inner part of a template expression, i.e. the ... in attr = "${...}" - see hashicorp/vscode-terraform#496 / hashicorp/vscode-terraform#636

Proposal

• Ensure we have this use case covered by tests
• Document this use case clearly in the Readme

First segment of references and object contents mis-highlighted

Extension Version

v2.26.20230511

VS Code Version

Version: 1.79.1 (user setup)
Commit: 4cb974a7aed77a74c7813bdccd99ee0d04901215
Date: 2023-06-12T16:14:05.102Z
Electron: 22.5.7
Chromium: 108.0.5359.215
Node.js: 16.17.1
V8: 10.8.168.25-electron.0
OS: Windows_NT x64 10.0.19044

Operating System

Windows 10

Steps to Reproduce

Setup files as below - note the colours in the main.pkr.hcl file for:

    repo          = var.GITHUB_REPOSITORY
    branch        = var.REF_NAME
    event_type    = var.EVENT_NAME

Expected Behavior

Colours should be correct.

Actual Behavior

Colours incorrect on lines 12/13/14.

Configuration

file.pkr.hcl

source "amazon-ebs" "nat-ubuntu" {
  ami_name      = "digital-images-nat-ubuntu-${local.timestamp}"
  source_ami    = data.amazon-ami.nat-ubuntu.id
  instance_type = "t4g.micro"
  region        = "eu-west-1"
  ssh_username  = "ubuntu"
  ami_regions   = var.ami_regions

  tags = {
    base_ami_id   = "{{ .SourceAMI }}"
    base_ami_name = "{{ .SourceAMIName }}"
    repo          = var.GITHUB_REPOSITORY
    branch        = var.REF_NAME
    event_type    = var.EVENT_NAME
    production    = var.REF_NAME == "main" ? "true" : "false"
  }

  vpc_filter {
    filters = {
      "isDefault" : "false"
    }
  }
  subnet_filter {
    filters = {
      "tag:Name" : "primary*"
    }
    random = true
  }

  ami_users = var.accounts
}

variable.pkr.hcl

variable "accounts" {
  type = list(string)

  default = []
}

variable "ami_regions" {
  type = list(string)

  default = [
    "us-east-1",
    "eu-central-1",
    "eu-west-2",
    "sa-east-1",
  ]
}

variable "GITHUB_REPOSITORY" {
  description = "GitHub repo"

  type = string

  default = "Hogarth-Worldwide/terraform-aws-amis-digital"
}

variable "REF_NAME" {
  description = "GitHub branch"

  type = string

  default = "local_running"
}

variable "EVENT_NAME" {
  description = "GitHub event name"

  type = string

  default = "local_running"
}

Switch GitHub highlighting to `hashicorp/syntax`

Background

Currently GitHub uses https://github.com/alexlouden/Terraform.tmLanguage for highlighting HCL:

https://github.com/github/linguist/blob/cddf7476af4c95d1572956ffc5c0cb84f7e431c5/lib/linguist/languages.yml#L2384-L2399

That project is primarily used for highlighting in Sublime Text and has also limited maintenance capacity as per SublimeText/Terraform#36

Proposal

• #34
• Release v0.1 of https://marketplace.visualstudio.com/items?itemName=hashicorp.hcl
• Wait for a few weeks to gather any feedback from VS Code users
• Bring over syntax tests from https://github.com/alexlouden/Terraform.tmLanguage (and remember to give credit to Pat Gaffney)
• Ensure the configuration under test produces comparable or better highlighting to https://github.com/alexlouden/Terraform.tmLanguage

Raise PR to github/linguist to:

• switch GitHub's highlighting of HCL to hcl grammar in this repo
• add GitHub's highlighting of Sentinel via grammar in this repo

Adjust highlighting of "constants" like `var` and `locale`

The current grammar defines keywords for referencing other configuration parts as constants.

https://github.com/hashicorp/vscode-terraform/blob/59174bd3e9a66c177a0f2de5751595d3f1cec838/syntaxes/terraform.tmGrammar.json#L39-L43

For example, the grammar highlights the var keyword different from the .variable_name.

When the language server is done processing all files, we have more information about the available variables and semantic tokens adjust highlighting accordingly.

This behavior is confusing, as seen in hashicorp/vscode-terraform#938

It is proposed to update the grammar to highlight the keywords with the same scope as the following names. Then in a separate language server issue (hashicorp/terraform-ls#805), improve the highlighting of invalid references with the help of semantic tokens.

Related to #25

Mis-highlighting after `for` expression

While comparing different grammars, I discovered that we don't handle multi-line for-expressions correctly.

After line 3, the highlighting breaks (theme: GitHub Dark Default)

Bad highlighting when resource dot number notation is used in template string

Server Version

0.27.0

Terraform Version

Terraform v1.1.9
on windows_amd64

Client Version

VS Code

Terraform Configuration Files

locals {
  a = ["foo"]
}
output "foo" {
  value = {
    foo = "${local.a.0}a"
    bar = "This is not highlighted as a string"
  }
}

Log Output

https://gist.github.com/orgads/9e2edc3d4ab85de5beaeba113077ae63

Expected Behavior

a after the closing brace should be highlighted as a string, and bar should be highlighted as a symbol

Actual Behavior

Steps to Reproduce

Open VS Code and paste the code above.

Notice that changing local.a.0 to local.a[0] solves the issue.

Migrate terraform grammar to syntax repo

• Copy terraform.tmGrammar.json from vscode-terraform to syntax repo
• Copy tmgrammar tests from vscode-terraform to syntax repo
• Copy tmgrammar GitHub Action test yaml to syntax repo
• Release artifact

Extract HCL2 grammar from terraform to new hcl grammar file

• Create hcl.tmGrammar.json
• Copy existing grammar to hcl.tmGrammar.json with exception of terraform specific rules
• Modify terraform.tmGrammar.json to inherit from hcl.tmGrammar.json

Support forward slashes and other characters in block label string literals

Previously a block label with a forward slash would not highlight correctly. This was found in Sentinel Configuration file syntax example;

mock "tfplan/v2" {
 ....
}

The HCL Language Spec says the block labels are Identifiers or String Literals, where a string literal can contain basically all valid UTF8 characters. An Identifier has a mich stricter set of allowed characters.

Update Prism HCL syntax for 0.12 features

Maybe just add a TF specific component?

https://github.com/PrismJS/prism/blob/master/components/prism-hcl.js

Improve highlighting of primitive type declarations

Problem Statement

This is to follow up on hashicorp/terraform-ls#827 which introduces semantic highlighting of type declarations.

Type declarations are highlighted by the server as keyword.

Our grammar currently highlights them as storage.type which seems to result in a different association in the default VSCode themes.

https://github.com/hashicorp/vscode-terraform/blob/c8b0154b8d2cc4cd67b4e68065ad6b85f8e133a8/syntaxes/terraform.tmGrammar.json#L49-L53

Grammar (only) Highlighting

Semantic Highlighting

Expected User Experience

Grammar and semantic highlighting are more aligned, i.e. type declarations are highlighted as keywords in the grammar too.

Proposal

Update terraform_type_keywords matcher?

Wrong highlighting for name-spaced function with object as first argument

A name-spaced function that has an object as its first argument is incorrectly highlighted. It looks like it was recognized as a block instead.

Example

provider::terraform::encode_tfvars({
  example, "Hello!",
)

Incorrect highlighting for `=>`

We highlight => with two different scopes when it's not part of a single-line for-expression

Proposal: Introduce => as separate keyword.operator.hcl

Reflect 1.4 / 1.5 Terraform language changes in grammar

New blocks may need to be introduced here

syntax/src/terraform.yml

Line 18 in b30903f

- match: \bresource|provider|variable|output|locals|module|data|terraform\b

and new functions here

syntax/src/terraform.yml

Line 48 in b30903f

- match: abspath|abs|ceil|floor|log|max|min|pow|signum|chomp|formatlist|indent|join|lower|regexall|regex|replace|split|strrev|substr|title|trimspace|upper|chunklist|coalescelist|coalesce|compact|concat|contains|distinct|element|flatten|index|keys|length|lookup|matchkeys|merge|range|reverse|setintersection|setproduct|setunion|slice|sort|transpose|values|zipmap|base64decode|base64encode|base64gzip|csvdecode|jsondecode|jsonencode|urlencode|yamldecode|yamlencode|dirname|pathexpand|basename|fileexists|fileset|filebase64|templatefile|formatdate|timeadd|timestamp|base64sha256|base64sha512|bcrypt|filebase64sha256|filebase64sha512|filemd5|filemd1|filesha256|filesha512|md5|rsadecrypt|sha1|sha256|sha512|uuidv5|uuid|cidrhost|cidrnetmask|cidrsubnet|tobool|tolist|tomap|tonumber|toset|tostring|file|format

Rewrite guidance for 3rd party grammars

https://github.com/hashicorp/syntax#3rd-parties

With generator this now seems outdated. We may want to update it to provide more accurate guidance at some point.

Create an accurate Terraform & HCL grammar for PrismJS

Background

HCL and Terraform languages evolve over time and the existing community-maintained PrismJS grammar doesn't accurately capture the state of these languages or relationship between HCL and Terraform.

In particular they seem to be treated as the same language, where any HCL config (such as Packer) highlights Terraform keywords:
https://github.com/PrismJS/prism/blob/c2f6a64426f44497a675cb32dccb079b3eff1609/components/prism-hcl.js

Proposal

• Publish a PrismJS grammar for HCL, based on existing TextMate grammar
• Publish a PrismJS grammar for Terraform, based on existing TextMate grammar
• (Unknown) Find out how existing websites can switch to that and/or whether we can get PrismJS/prism to consume our grammars
  • Terraform - e.g. https://www.terraform.io/language
  • Packer - e.g. https://www.packer.io/docs/templates/hcl_templates
  • Nomad - e.g. https://www.nomadproject.io/docs/configuration
  • Waypoint - e.g. https://www.waypointproject.io/docs/waypoint-hcl#template-waypoint-hcl
  • Consul - e.g. https://www.consul.io/docs/agent/config/config-files
  • Vault - e.g. https://www.vaultproject.io/docs/configuration
  • Boundary - e.g. https://www.boundaryproject.io/docs/configuration

Decouple Readme based on use cases

I'm not tied to filenames, but from your response we appear to be in agreement that this README should be a landing page and not a catalogue of all the different sets of knowledge required to use the files in this repo.

Extrapolating from your thoughts, I think we need:

• README.md with sections briefly explaining the repo and the different types of uses cases. Each with links to get users/contributors where they need to go.
• USAGE.md document outlining expected ways to consume these files
• CONTRIBUTING.md document outlining what content we accept.

Originally posted by @jpogran in #11 (comment)

Consider highlighting grammar for function signatures

Context

As part of hashicorp/terraform-ls#37 we discovered that there's need for rendering function signatures in a markdown hover popup. Because signatures aren't technically valid HCL, the existing grammar(s) do not highlight it particularly well. Specifically, the grammar cannot distinguish between type declaration and parameter name.

Proposal

• Explore the option of a separate small grammar to highlight just function signatures
• Discuss and agree on a stable/predictable way of rendering the signatures - e.g. how to present types (set(string) vs set of string; any single type vs dynamic etc.)

Wrong highlighting for function that splits over multiple lines inside a string

Originally reported in hashicorp/vscode-terraform#1756

Example

locals {
  list = [
    "1", "2", "3"
  ]
  function = [
    for i in local.list : "Out: ${join("-", [i, "join"]
    )}" if i != "2"
  ]
}

output "test" {
  value = local.function
}

It breaks some time after the 0.4.2 release