hashicorp / hil

HIL is a small embedded language for string interpolations.

HIL

HIL (HashiCorp Interpolation Language) is a lightweight embedded language used primarily for configuration interpolation. The goal of HIL is to make a simple language for interpolations in the various configurations of HashiCorp tools.

HIL is built to interpolate any string, but is in use by HashiCorp primarily with HCL. HCL is not required in any way for use with HIL.

HIL isn't meant to be a general purpose language. It was built for basic configuration interpolations. Therefore, you can't currently write functions, have conditionals, set intermediary variables, etc. within HIL itself. It is possible some of these may be added later but the right use case must exist.

Why?

Many of our tools have support for something similar to templates, but within the configuration itself. The most prominent requirement was in Terraform where we wanted the configuration to be able to reference values from elsewhere in the configuration. Example:

foo = "hi ${var.world}"

We originally used a full templating language for this, but found it was too heavy weight. Additionally, many full languages required bindings to C (and thus the usage of cgo) which we try to avoid to make cross-compilation easier. We then moved to very basic regular expression based string replacement, but found the need for basic arithmetic and function calls resulting in overly complex regular expressions.

Ultimately, we wrote our own mini-language within Terraform itself. As we built other projects such as Nomad and Otto, the need for basic interpolations arose again.

Thus HIL was born. It is extracted from Terraform, cleaned up, and better tested for general purpose use.

Syntax

For a complete grammar, please see the parser itself. A high-level overview of the syntax and grammar is listed here.

Code begins within ${ and }. Outside of this, text is treated literally. For example, foo is a valid HIL program that is just the string "foo", but foo ${bar} is an HIL program that is the string "foo " concatened with the value of bar. For the remainder of the syntax docs, we'll assume you're within ${}.

• Identifiers are any text in the format of [a-zA-Z0-9-.]. Example identifiers: foo, var.foo, foo-bar.

• Strings are double quoted and can contain any UTF-8 characters. Example: "Hello, World"

• Numbers are assumed to be base 10. If you prefix a number with 0x, it is treated as a hexadecimal. If it is prefixed with 0, it is treated as an octal. Numbers can be in scientific notation: "1e10".

• Unary - can be used for negative numbers. Example: -10 or -0.2

• Boolean values: true, false

• The following arithmetic operations are allowed: +, -, *, /, %.

• Function calls are in the form of name(arg1, arg2, ...). Example: add(1, 5). Arguments can be any valid HIL expression, example: add(1, var.foo) or even nested function calls: add(1, get("some value")).

• Within strings, further interpolations can be opened with ${}. Example: "Hello ${nested}". A full example including the original ${} (remember this list assumes were inside of one already) could be: foo ${func("hello ${var.foo}")}.

Language Changes

We've used this mini-language in Terraform for years. For backwards compatibility reasons, we're unlikely to make an incompatible change to the language but we're not currently making that promise, either.

The internal API of this project may very well change as we evolve it to work with more of our projects. We recommend using some sort of dependency management solution with this package.

Future Changes

The following changes are already planned to be made at some point:

• Richer types: lists, maps, etc.

• Convert to a more standard Go parser structure similar to HCL. This will improve our error messaging as well as allow us to have automatic formatting.

• Allow interpolations to result in more types than just a string. While within the interpolation basic types are honored, the result is always a string.

hil's Issues

Escaped quotes not handled correctly by parser

In the new hand-written scanner/parser, the scanner is correctly handling escaped quotes in quoted sequences -- ensuring that they don't prematurely terminate the string token -- but the parser is missing the rule to replace an escaped quote with a literal quote.

This was reported in hashicorp/terraform#10236 .

Conversions from string expose raw Go library error messages

As reported in hashicorp/terraform#10854, when HIL implicitly converts from string to one of the other types it calls into one of the conversion functions in the Go standard library, and if the conversion fails HIL just passes through verbatim the Go-oriented error message. For example:

* __builtin_StringToBool: strconv.ParseBool: parsing "sg-0a784a77": invalid syntax in:

${var.sg_id ? var.sg_id : aws_security_group.main.id}

Several things wrong with this error message:

• It exposes the implementation detail that conversion is implemented via builtin functions (__builtin_StringToBool)
• It mentions strconv.ParseBool, which is a Go library function and not something anyone using HIL is expected to be familiar with.
• The phrasing is confusing and can be misunderstood that the syntax error is in the interpolation expression ${var.sg_id ? var.sg_id : aws_security_group.main.id} rather than in the value "sg-0a784a77".
• The line and column number of the errant expression are not included. (Currently including this would not actually help much in Terraform since it doesn't retain HCL position context when parsing HIL, but I hope to fix that later...)

Proposed better error message (with similar variants for other types converting to string):

* can't convert string "sg-0a784a77" to bool at foo.tf:12:2; must be either "true" or "false"

This addresses the three things I think should be included in all good error messages: "What's the problem (in user-facing terminology)?" "Where is the problem?" "How might the problem be corrected?"

Add examples of custom variable and function lookups

From the conversation in #8

It would be very useful to implement lookup functions for variables and functions without modifying HIL package code.

One example use-case would be reading arbitrary environment variables, eg.

greeting = "Hi ${env.NAME}"

This initially appeared possible by passing something that implements the Scope interface to the EvalConfig function, however other parts of the code explicitly reference the BasicScope implementation making this not possible.

If this is already possible it would be great to document some examples in the README. If not would it be possible to allow other implementations of the Scope interface?

Prefer promotion from int to float, vs. the converse

(Originally reported in hashicorp/terraform#10778.)

Currently when type-checking an arithmetic operation we take the type from the first operand and then attempt to implicitly convert the remaining operands to match. This causes a surprising result when dealing with the numeric types:

• 0.5 * 100 => 50.0 because integer 100 is converted to float 100.0
• 100 * 0.5 => 0 because float 0.5 is converted to int 0

The automatic conversion to int in the second case is lossy and there's no significant use-case where this behavior is desirable, so we should instead be smarter about our conversions and prefer to convert to float if any of the arguments are floats, regardless of argument order.

Cannot evaluate arithmetic inside of array index

Trying to evaluate something like ${foo[2-1]}, where foo is just an array with at least two members, fails with (formatting mine):

Error: node doesn't support evaluation: *ast.Arithmetic{
    Op:2, Exprs:[]ast.Node{
        *ast.LiteralNode{
            Value:2,
            Typex:0x8,
            Posx:ast.Pos{Column:7, Line:1}
        },
        *ast.LiteralNode{
            Value:1, 
            Typex:0x8, 
            Posx:ast.Pos{Column:9, Line:1}
        }
    },
    Posx:ast.Pos{Column:7, Line:1}
}

Where can syntax support be found for issues?

As the title states, we can individuals turn for assistance in syntax issues in their terraform templates when the issue is not a defect/bug.

Wrong result type for function of return type ast.TypeBool

Hi,

The following snippet doesn't return ast.TypeBool anymore:

https://play.golang.org/p/fc9xgAcIvp

Instead it returns:

Type: TypeString
Value: true

It should return TypeBool

Backslash escapes not working properly at the end of quoted strings

@ziggythehamster pointed out via Terraform's Gitter chat that there's still some quirkiness with backslash escapes in HIL.

Consider this example:

resource "null_resource" "foo" {
  triggers = {
    thing = "${replace(var.thing, "\\", "/")}"
  }
}

This produces a syntax error in the HIL parser:

Error loading config: Error loading try.tf: Error reading config for null_resource[foo]: parse error at 1:31: expected "${" but found invalid sequence ")}"

It seems that the \\" sequence is escaping the quote, rather than escaping the backslash as expected. Adding a space before the quote mark seems to get past the parse error. Presumably this is a logic error in the escape handling.

Using hil as a predicate DSL

For now, hil always generate string, with hil.Parse and hil.Eval.

hil uses types, but some magic implicit conversion to string happens.

If you spy here : https://github.com/hashicorp/hil/blob/master/eval.go#L407 v.Exprs contains a chain of calls (and their arguments), something like Call(__builtin_BoolToString, Call(__builtin_IntCompare, Literal(TypeInt, 12), Literal(TypeInt, 2), Literal(TypeInt, 1))) for ${2 > 1} or Call(__builtin_IntToString, Literal(TypeInt, 42)) for ${ 42 }. The string conversion is in the executable AST tree.

I read all the steps, from scanner to parser to AST to eval, puts a lots of ugly fmt.Println, but I can't find where the implicit conversion to string happens.

Can you add some options for using hil as a simple script language that return typed result ?

Go 1.10: eval_test.go:2048: Fatalf format %#s has unrecognized flag #

fa9f258 does not pass unit tests with Go 1.10. At least:

+ GOPATH=/builddir/build/BUILD/hil-fa9f258a92500514cc8e9c67020487709df92432/_build:/usr/share/gocode
+ go test -buildmode pie -compiler gc -ldflags '-extldflags '\''-Wl,-z,relro  '\'''
# github.com/hashicorp/hil
./eval_test.go:2048: Fatalf format %#s has unrecognized flag #
./eval_test.go:2051: Fatalf format %#s has unrecognized flag #
FAIL    github.com/hashicorp/hil [build failed]

Conditional operator doesn't short-circuit evaluation based on result

To avoid extensive refactoring of the evaluator while implementing the conditional syntax, we accepted the limitation that it would evaluate both "sides" and then discard the inappropriate one when given syntax like this:

foo ? bar : baz

This is in most cases just a small waste of compute to evaluate a result that will never be used, but it's troublesome in cases where one of the expressions can be evaluated without error only if its associated condition state is true. For example:

length(foo) > 0 ? foo[0] : "default"

This doesn't work as intended because we fail on trying to index into the empty list foo in spite of the conditional.

The correct behavior is for the evaluator to evaluate the boolean expression first and then visit only one of the given expressions.

GitHub Actions - deprecated warnings found - action required!

Workflow Name: hil
Branch: main
Run URL: https://github.com/hashicorp/hil/actions/runs/4693804324

save-state deprecation warnings: 0
set-output deprecation warnings: 1
node12 deprecation warnings: 1

Please review these deprecation warnings as soon as possible and merge in the necessary updates.

GitHub will be removing support for these commands and plan to fully disable them on 31st May 2023. At this time, any workflow that still utilizes these commands will fail. See https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/.

GitHub have not finalized a date for deprecating node12 yet but have indicated that this will be summer 2023. So it is advised to switch to node16 asap. See https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/.

If you need any help, please reach out to us in #team-rel-eng.

Missing unary plus and minus operators support

They was introduced in Terraform 0.6.9, see changelog and pull-request.
But during migration to separate package/repository support was lost alongside with tests.
Recently @mitchellh added support for unary minus operator, but it would be nice to use original code.

Why this happened:

• hashicorp/hel (previous name) was branched on 05.12.2015.
• PR-3621 was merged into hashicorp/terraform/config/lang on 18.12.2015
• Terraform switched to hashicorp/hil on 03.02.2016. At this stage code in hashicorp/terraform/config/lang was dropped, without merging into hashicorp/hil)

Feature request: Trim mode

Consider the following block:

variable "one" {
  type    = "string"
  default = "one"
}

variable "two" {
  type    = "string"
  default = ""
}

variable "three" {
  type    = "string"
  default = "three"
}

output "output" {
  value = <<EOS
${var.one}
${var.two}
${var.three}
EOS
}

This gives:

one

three

What would be nice, is if we could do something like:

output "output" {
  value = <<EOS
${var.one}
${var.two -}
${var.three}
EOS
}

And get instead:

one
three

I'd love to do the work if someone tells me where to start.

I'm guessing it would be an extension to the scanner to locate -} (and maybe ${- for good measure), adding a TrimEnd and maybe TrimStart to hil.EvaluationResult, and then writing the resulting code in Terraform to handle the trimming of the heredoc or "multiline" string (ie: trim whitespace to next non-whitespace character, or all previous whitespace back to and including last newline).

This is new territory for me, so my logic may be off. Again, would love to know where to look. :)

Single-quote strings

I'd love to use hil for a project but double-quotes are not very JSON friendly making the highlighting look all wonky, any chance of getting single-quoted strings in?

Identifiers starting with digits are no longer supported after scanner rewrite

(As reported in hashicorp/terraform#10597 .)

As of the scanner rewrite, it's now treating the sequence var.1x1 as (IDENTIFIER, DOT, INTEGER, IDENTIFIER), which then causes a parse error. This is a change from the previous scanner, most likely because the old scanner treated this entire sequence (including the intermediate period) as a single token.

I suppose that either the code that deals with integers needs to "scan ahead" to see if any letters follow, falling back on the identifier scanner instead if so, or the parser needs to get a bit smarter and tolerate an integer immediately followed by an identifier as a valid part of a variable name, similarly to how we tolerate integers in foo.1.baz sequences. At first glance the first option seems better since the latter would accidentally tolerate weird sequences like var.1 x1 as a single variable name.

Passing an Empty List Confuses Configuration

Not sure if this is the right spot. I think this is a HIL issue and not a Terraform issue.

Consider this snippet of TF code used in a module:

variable security_groups {
    type = "list"
    default = []
}

resource "aws_network_interface" "nic" {
    # ...
    security_groups = ["${var.security_groups}"]
}

When creating the module with explicit security_groups everything works fine. However the default case will complain about The security group '' does not exist

Ctrl+Q no-longer displaying link to documentation

While it would be more useful to have the documentation embedded in the CTRL+Q and CTRL+P functionality, it was still extremely useful having a direct link to the online docs, even though it was displayed on resource instances, rather than resource types.
In one of the later versions, this functionality has been removed, though autocomplete displays doc information for individual properties. Will CTRL+Q/P documentation display be re-implemented?

please add license and copyright statement