microsoft / rego-cpp

A C++ interpreter for the OPA policy language Rego

rego-cpp

This project is an effort to create a C++ interpreter for the OPA policy language, Rego. Our goal is to build both a standalone executable and a library such that programmers who are working in C++ can interpret Rego programs natively. To achieve this we are building our interpreter on top of the experimental term rewriter Trieste.

Warning While this project has progressed to the point that we support full Rego language (see Language Support below) we do not support all built-ins. That said, we have verified compliance with the OPA Rego test suite. Even so, it should still be considered experimental software and used with discretion.

Getting Started

Start by installing CMake in the way appropriate for your environment.

Linux

Create a build directory and initialize the cmake project:

mkdir build
cd build
cmake .. --preset release-clang

You can then build and run the tests using:

ninja install
ctest

Windows

Create a build directory and initialize the cmake project:

mkdir build
cd build
cmake .. --preset release

You can then build and run the tests using:

cmake --build . --config Release --target INSTALL
ctest -C Release

Using the rego CLI

The interpreter tool will be located at build/dist/bin/rego. Here are some example commands using the provided example files and run from the suggested dist install directory:

./bin/rego -d examples/scalars.rego -q data.scalars.greeting
"Hello"

./bin/rego -d examples/objects.rego -q data.objects.sites[1].name
"smoke1"

./bin/rego -d examples/data0.json examples/data1.json examples/objects.rego -i examples/input0.json  -q "[data.one, input.b, data.objects.sites[1]]"
[{"bar": "Foo", "baz": 5, "be": true, "bop": 23.4}, "20", {"name": "smoke1"}]

./bin/rego -q "5 + (2 - 4 * 0.25) * -3 + 7.4"
9.4

./bin/rego -d examples/bodies.rego -i examples/input1.json -q data.bodies.e
{"one": 15, "two": 15}

You can run the test driver from the same directory:

./bin/rego_test tests/regocpp.yaml

Using the rego Library

See the examples directory for examples of how to use the library from different langauages.

Language Support

At present we support v0.55.0 of Rego as defined by OPA, with the following grammar:

module          = package { import } policy
package         = "package" ref
import          = "import" ref [ "as" var ]
policy          = { rule }
rule            = [ "default" ] rule-head { rule-body }
rule-head       = ( ref | var ) ( rule-head-set | rule-head-obj | rule-head-func | rule-head-comp )
rule-head-comp  = [ assign-operator term ] [ "if" ]
rule-head-obj   = "[" term "]" [ assign-operator term ] [ "if" ]
rule-head-func  = "(" rule-args ")" [ assign-operator term ] [ "if" ]
rule-head-set   = "contains" term [ "if" ] | "[" term "]"
rule-args       = term { "," term }
rule-body       = [ "else" [ assign-operator term ] [ "if" ] ] ( "{" query "}" ) | literal
query           = literal { ( ";" | ( [CR] LF ) ) literal }
literal         = ( some-decl | expr | "not" expr ) { with-modifier }
with-modifier   = "with" term "as" term
some-decl       = "some" term { "," term } { "in" expr }
expr            = term | expr-call | expr-infix | expr-every | expr-parens | unary-expr
expr-call       = var [ "." var ] "(" [ expr { "," expr } ] ")"
expr-infix      = expr infix-operator expr
expr-every      = "every" var { "," var } "in" ( term | expr-call | expr-infix ) "{" query "}"
expr-parens     = "(" expr ")"
unary-expr      = "-" expr
membership      = term [ "," term ] "in" term
term            = ref | var | scalar | array | object | set | membership | array-compr | object-compr | set-compr
array-compr     = "[" term "|" query "]"
set-compr       = "{" term "|" query "}"
object-compr    = "{" object-item "|" query "}"
infix-operator  = assign-operator | bool-operator | arith-operator | bin-operator
bool-operator   = "==" | "!=" | "<" | ">" | ">=" | "<="
arith-operator  = "+" | "-" | "*" | "/"
bin-operator    = "&" | "|"
assign-operator = ":=" | "="
ref             = ( var | array | object | set | array-compr | object-compr | set-compr | expr-call ) { ref-arg }
ref-arg         = ref-arg-dot | ref-arg-brack
ref-arg-brack   = "[" ( scalar | var | array | object | set | "_" ) "]"
ref-arg-dot     = "." var
var             = ( ALPHA | "_" ) { ALPHA | DIGIT | "_" }
scalar          = string | NUMBER | TRUE | FALSE | NULL
string          = STRING | raw-string
raw-string      = "`" { CHAR-"`" } "`"
array           = "[" term { "," term } "]"
object          = "{" object-item { "," object-item } "}"
object-item     = ( scalar | ref | var ) ":" term
set             = empty-set | non-empty-set
non-empty-set   = "{" term { "," term } "}"
empty-set       = "set(" ")"

Definitions:

[]     optional (zero or one instances)
{}     repetition (zero or more instances)
|      alternation (one of the instances)
()     grouping (order of expansion)
STRING JSON string
NUMBER JSON number
TRUE   JSON true
FALSE  JSON false
NULL   JSON null
CHAR   Unicode character
ALPHA  ASCII characters A-Z and a-z
DIGIT  ASCII characters 0-9
CR     Carriage Return
LF     Line Feed

Builtins

At the moment only support a few builtins, but are actively working on adding all the standard builtins. The following builtins are currently supported:

• aggregates
• arrays
• bits
• casts
• encoding
• graphs
• numbers
• objects
• regex
• semver
• sets
• strings
• types
• units
• miscellaneous
  • opa.runtime
  • print
  • time.now_ns

Compatibility with the OPA Rego Go implementation

Our goal is to achieve and maintain full compatibility with the reference Go implementation. We have developed a test driver which runs the same tests and validates that we produce the same outputs. At this stage we pass all the non-builtin specific test suites, which we clone from the OPA repository. To build with the OPA tests available for testing, use one of the following presets:

• release-clang-opa
• release-opa

At present, we are NOT passing the following test suites in full:

• crypto*
• glob*
• graphql
• invalidkeyerror
• json* (except jsonbuiltins)
• jwt*
• net*
• planner-ir
• providers-aws
• time
• walkbuiltin

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

rego-cpp's Issues

`json` built-ins

`if` keyword

`else` keyword

Array/Set comprehensions

`semver` Built-ins

• semver.compare
• semver.is_valid

`bin-infix` expressions

Policy is not evaluated correctly when statements are not in order

package test

x = y {
 y = z
 z = 12
}

x is evaluated to {} instead of 12.

Arithmetic expressions which are split over lines do not parse

Repro:

a := 1 + 2 +
  3

regopy does not work on ARM

Installing regopy on ARM architecture works, but using it throws an error.

▶ pip install regopy  
Collecting regopy
  Using cached regopy-0.3.10-cp311-cp311-macosx_11_0_arm64.whl.metadata (7.9 kB)
Using cached regopy-0.3.10-cp311-cp311-macosx_11_0_arm64.whl (7.8 MB)
Installing collected packages: regopy
Successfully installed regopy-0.3.10

▶ python3                        
Python 3.11.6 (main, Dec  4 2023, 18:49:48) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from regopy import Interpreter
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/matthias/Documents/Work/someeng/resoto/venv/lib/python3.11/site-packages/regopy/__init__.py", line 5, in <module>
    from ._regopy import (
ImportError: dlopen(/Users/matthias/Documents/Work/someeng/resoto/venv/lib/python3.11/site-packages/regopy/_regopy.cpython-311-darwin.so, 0x0002): tried: '/Users/matthias/Documents/Work/someeng/resoto/venv/lib/python3.11/site-packages/regopy/_regopy.cpython-311-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/matthias/Documents/Work/someeng/resoto/venv/lib/python3.11/site-packages/regopy/_regopy.cpython-311-darwin.so' (no such file), '/Users/matthias/Documents/Work/someeng/resoto/venv/lib/python3.11/site-packages/regopy/_regopy.cpython-311-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

`aggregates` built-ins

• count
• max
• min
• product
• sort
• sum

rego-cpp throws when opa has no issues

I'm using a locally built rego cpp interpreter but it throws an error instead of returning the expected value false for a given query.

Example

• policy.demo.rego:

  package demo
  import future.keywords.if
  default allow := false
  allow if {
      count(trim_space(input.foo)) > 0
  }
  

• input.demo.json:

  { "bar": "baz" }
  

The following query would throw:

./bin/rego_interpreter -d policy.demo.rego -i input.demo.json -q data.demo.allow

rego_interpreter: /home/ubuntu/demo/src/unifier.cc:1248: std::optional<RankedNode> rego::UnifierDef::resolve_rulefunc(const trieste::Node &, const trieste::Nodes &): Assertion `rulefunc->type() == RuleFunc' failed.
Aborted

Whilst the regular https://play.openpolicyagent.org/ has no issues with evaluating, equally the opa cli works with the following results:

./opa eval data.demo.allow --data policy.demo.rego --input input.demo.json

{
  "result": [
    {
      "expressions": [
        {
          "value": false,
          "text": "data.demo.allow",
          "location": {
            "row": 1,
            "col": 1
          }
        }
      ]
    }
  ]
}

The expectation is that the query should not fail due to a missing key (most likely) in the input.

Ability to specify path to clang-format?

When building on a machine that has multiple variants of clang-format, which is an unfortunate necessity sometimes because they tend to produce different results for the same rules, there is no single clang-format in the path, instead there is clang-format-10, clang-format-15 etc.

Running cmake produces the following output:

CMake Warning at build/_deps/snmalloc-src/CMakeLists.txt:158 (message):
  Not generating clangformat target, no clang-format tool found
Call Stack (most recent call first):
  build/_deps/snmalloc-src/CMakeLists.txt:520 (clangformat_targets)

This does not prevent the build from taking place, but it presumably means that a user facing this situation would not easily be able to format their potential changes easily. Would it be possible to let users specify the path to clang-format, as they already can for CC, CXX etc?

`yaml` built-ins

Object comprehensions

`casts` built-ins

`arrays` built-ins

• array.concat
• array.reverse
• array.slice

`time` built-ins

Add compile caching for Interpreter

At the moment the interpreter restarts compilation for each query. It should instead cache the step immediately before unification so that everything but the input and query are already good to go.

Notes

• This will not work for more complex queries, or rather they may still need to pass through compilation.
• Input will need to be parsed as well (this will need to be done by the standalone JSON parser from #60)

Build instructions in the README confused this user

Following the steps listed under https://github.com/microsoft/rego-cpp/blob/main/README.md?plain=1#L32 seem to lead to an error:

$ mkdir build
$ cd build
$ CXX=`which clang++-15` CC=`which clang-15` cmake -GNinja .. -DCMAKE_INSTALL_PREFIX=dist -DREGOCPP_BUILD_TOOLS=1
$ ninja # presumed necessary before building the tests, but does not affect result of next step
$ cmake --build . --config Debug --target INSTALL
[0/2] Re-checking globbed directories...
ninja: error: unknown target 'INSTALL'

Following the alternative steps proposed for ninja does not seem to run the test successfully either:

$ ninja install
$ ctest -C Debug
Test project /home/amchamay/rego-cpp/build
No tests were found!!!

Partial rule construction

Support rules of the form:

a["one"] {
}

b[2] := "foo" {
}

Support refhead rules

Interpreter does not support rules with ref names, i.e.:

package example

fruit.apple.seeds = 12

`contains` keyword

Rust binding

`graphs` built-ins

`numbers` built-ins

• abs
• ceil
• floor
• rand.intn
• round

`objects` built-ins

• json.filter
• json.match_schema
• json.patch
• json.remove
• json.verify_schema
• object.filter
• object.get
• object.keys
• object.remove
• object.subset
• object.union
• object.union_n

`types` built-ins

Term nodes for static rule values

All rule values are currently rewritten to UnifyBody nodes and unified, even if they are static terms. This can be optimized out via better rewrite rules.

SIGSEGV when accessing invalid members of input/data files

Comprehensions do not handle variables from parent scope correctly

Given the following policy:
scope.rego

package test

x = y {
	a = [1, 2, 3]
	y = {z | z = a[_]}
}

rego-cpp/build/dist/bin/rego -d scope.rego data.test.x produces [].
It is supposed to produce a set {1, 2, 3}. See https://play.openpolicyagent.org/p/AGT70depTf

.NET binding

Better use of Trieste well-formed node queries

At the moment rewrites rely a lot on positional child node retrieval, and the system would be more robust if it used well-formed node queries instead.

Eval a rego rule on multiple input

Hello
I have rego modules and lots of rules inside and I would like to evaluate a specific rule on different inputs.

What is the rego/rego-cpp way to do that?

If I try to do that:

  let rego = Interpreter::new();
  rego.add_module("module1", module1);
  rego.add_module("module2", module2);
  // ...
  rego.add_module("module30", module30);

  for input in inputs {
     rego.set_input_json(input);
     rego.query("data.modulex.rulex");
  }

For each query, all the rules within all modules are evaluated and the result of rulex is fetched, it can work I suppose but not in a optimized way.

Regards

Sets built-ins

• x & y
• intersection
• x - y
• x | y
• union

OPA Test Driver

Write a driver that processes the OPA YAML-encoded test cases.

C API

SEGV on framework.rego

Running the rego program on the framework.rego policy causes a segmentation fault.

$ dist/bin/rego -d ../tests/aci/framework.rego data
rego 0.3.10 (main:a24f12c, Tue, 2 Jan 2024 10:14:34 +0000)[AppleClang 15.0.0.15000100] on macos
Segmentation fault: 11

`bits` built-ins

`with` keyword

Full package name support

package foo
package foo.bar
package foo.bar.baz
package foo["bar.baz"].qux

`import` statements for other modules

sprintf() doesn't return a quoted string

Hello

value() removes the " character at the beginning and ending of a NodeKind::String.

the return of sprintf (and possibly all builtin string-functions?) doesn't seem to be a quoted string.

which result of this kind of behavior:

match interpreter.query(r#"x := {"bar": sprintf("%s", ["foo"])}"#) {
  Ok(value) => {
    let x = value.binding("x").expect("cannot get x");
    if let NodeValue::String(bar) = x
     .lookup("bar")
     .expect("bar key missing")
     .value()
     .expect("bar value missing")
     {
       assert_eq!(bar, "foo");
     }
  },

thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `"o"`,
 right: `"foo"`

Failed test for main

The following test fails on main:

 ./rego_trieste test -f --seed 3109230158

Uncovered in my PR (https://github.com/microsoft/rego-cpp/actions/runs/6131481980/job/16641498709?pr=70#step:5:64), but this failed on main when I tried it.

`every` keyword

`glob` built-ins

`in` keyword

`strings` built-ins

• concat
• contains
• endswith
• format_int
• indexof
• indexof_n
• lower
• replace
• split
• sprintf
• startswith
• strings.any_prefix_match
• strings.any_suffix_match
• strings.replace_n
• strings.reverse
• substring
• trim
• trim_left
• trim_prefix
• trim_right
• trim_space
• trim_suffix
• upper

wrappers-c CI doesn't work for PRs from forks

The CI assumes that the main repo contains the commit hash, but that is not the case if the PR is from a fork. See:

https://github.com/microsoft/rego-cpp/actions/runs/6131481980/job/16641498869?pr=70#step:4:31

Python binding

`regex` Built-ins

• regex.find_all_string_submatch_m
• regex.find_n
• regex.globs_match
• regex.is_valid
• regex.match
• regex.replace
• regex.split
• regex.template_match