projectcontour / integration-tester Goto Github PK

View Code? Open in Web Editor NEW

3.0 3.0 5.0 304 KB

Integration Tester for Kubernetes

License: Other

Makefile 1.71% Go 94.24% Shell 2.78% Open Policy Agent 1.26%

integration-tester's People

Contributors

Stargazers

Watchers

Forkers

jpeach stevesloka sunjaybhatia

integration-tester's Issues

Policy modules need builtins loaded before validation

What steps did you take and what happened:

If you add a policy file that references a builtin package, it errors on startup:

$ ./run-test-case.sh --trace=rego --preserve ./httpproxy/015-http-health-checks.yaml
integration-tester: 2 errors occurred:
/Users/jpeach/upstream/contour/_integration/testsuite/policies/contour-expect.rego:22: rego_type_error: undefined function data.builtin.result.Passf
/Users/jpeach/upstream/contour/_integration/testsuite/policies/contour-expect.rego:24: rego_type_error: undefined function data.builtin.result.Errorf

This happens in loadPolicies where we validate that the Rego policy files compile before building the option to pass them down into the test runner. We need to load the builtins first (and not export them to the option) (the runner does this, so we could consolidate that code).

Switch to OPA v0.19.2

The http.send updates we need were released in v0.19, so we can switch to the latest release and stop pinning to my forked repository.

Kubernetes informers log errors on GKE

What steps did you take and what happened:

Running Contour tests on GKE.

What did you expect to happen:

They should pass (they do on kind).

Anything else you would like to add:

It looks like something goes wrong when we may the GroupVersionResource and start an informer.

TAP version 13
# reading document from ./httpproxy/002-header-condition-match.yaml
# decoding document with 14 parts from ./httpproxy/002-header-condition-match.yaml
# decoded part 0 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:1-22)
# decoded part 1 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:24-32)
# decoded part 2 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:34-42)
# decoded part 3 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:44-52)
# decoded part 4 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:54-62)
# decoded part 5 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:64-72)
# decoded part 6 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:74-82)
# decoded part 7 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:84-92)
# decoded part 8 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:94-102)
# decoded part 9 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:104-112)
# decoded part 10 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:114-158)
# decoded part 11 as Rego (lines ./httpproxy/002-header-condition-match.yaml:160-172)
# decoded part 12 as Rego (lines ./httpproxy/002-header-condition-match.yaml:174-192)
# decoded part 13 as Rego (lines ./httpproxy/002-header-condition-match.yaml:194-272)
ok 1 - validating document "./httpproxy/002-header-condition-match.yaml"
E1006 14:55:00.197755   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
# test run ID is 7c80c72e-a5cb-4182-bee2-2ce0b3a28f7c
ok 2 - compiling test document
# hydrated apps/v1:Deployment object 'default/echo-header-present'
ok 3 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:1-22
ok 4 - matching anonymous Kubernetes object
# performing update operation on Deployment 'default/echo-header-present'
ok 5 - updating Kubernetes object
# checking update of Deployment 'default/echo-header-present'
ok 6 - running object update check
# hydrated v1:Service object 'default/echo-header-present'
ok 7 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:24-32
ok 8 - matching anonymous Kubernetes object
# performing update operation on Service 'default/echo-header-present'
ok 9 - updating Kubernetes object
# checking update of Service 'default/echo-header-present'
ok 10 - running object update check
# hydrated apps/v1:Deployment object 'default/echo-header-contains'
ok 11 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:34-42
ok 12 - matching anonymous Kubernetes object
# performing update operation on Deployment 'default/echo-header-contains'
E1006 14:55:01.372665   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
ok 13 - updating Kubernetes object
# checking update of Deployment 'default/echo-header-contains'
ok 14 - running object update check
# hydrated v1:Service object 'default/echo-header-contains'
ok 15 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:44-52
ok 16 - matching anonymous Kubernetes object
# performing update operation on Service 'default/echo-header-contains'
ok 17 - updating Kubernetes object
# checking update of Service 'default/echo-header-contains'
ok 18 - running object update check
# hydrated apps/v1:Deployment object 'default/echo-header-notcontains'
ok 19 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:54-62
ok 20 - matching anonymous Kubernetes object
# performing update operation on Deployment 'default/echo-header-notcontains'
ok 21 - updating Kubernetes object
# checking update of Deployment 'default/echo-header-notcontains'
ok 22 - running object update check
# hydrated v1:Service object 'default/echo-header-notcontains'
ok 23 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:64-72
ok 24 - matching anonymous Kubernetes object
# performing update operation on Service 'default/echo-header-notcontains'
ok 25 - updating Kubernetes object
# checking update of Service 'default/echo-header-notcontains'
ok 26 - running object update check
# hydrated apps/v1:Deployment object 'default/echo-header-exact'
ok 27 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:74-82
ok 28 - matching anonymous Kubernetes object
# performing update operation on Deployment 'default/echo-header-exact'
ok 29 - updating Kubernetes object
# checking update of Deployment 'default/echo-header-exact'
ok 30 - running object update check
# hydrated v1:Service object 'default/echo-header-exact'
ok 31 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:84-92
ok 32 - matching anonymous Kubernetes object
# performing update operation on Service 'default/echo-header-exact'
E1006 14:55:02.572764   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
ok 33 - updating Kubernetes object
# checking update of Service 'default/echo-header-exact'
ok 34 - running object update check
# hydrated apps/v1:Deployment object 'default/echo-header-notexact'
ok 35 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:94-102
ok 36 - matching anonymous Kubernetes object
# performing update operation on Deployment 'default/echo-header-notexact'
ok 37 - updating Kubernetes object
# checking update of Deployment 'default/echo-header-notexact'
ok 38 - running object update check
# hydrated v1:Service object 'default/echo-header-notexact'
ok 39 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:104-112
ok 40 - matching anonymous Kubernetes object
# performing update operation on Service 'default/echo-header-notexact'
ok 41 - updating Kubernetes object
# checking update of Service 'default/echo-header-notexact'
ok 42 - running object update check
# hydrated projectcontour.io/v1:HTTPProxy object 'default/header-conditions'
ok 43 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:114-158
ok 44 - matching anonymous Kubernetes object
# performing update operation on HTTPProxy 'default/header-conditions'
ok 45 - updating Kubernetes object
# checking update of HTTPProxy 'default/header-conditions'
ok 46 - running object update check
ok 47 - running Rego check lines ./httpproxy/002-header-condition-match.yaml:160-172
E1006 14:55:04.091891   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
ok 48 - running Rego check lines ./httpproxy/002-header-condition-match.yaml:174-192
E1006 14:55:05.140502   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:06.208101   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:07.258947   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:08.311610   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:09.362543   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:10.409447   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:11.457482   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:12.504765   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:13.554265   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:14.611691   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:15.708549   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:16.759884   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:17.805455   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:18.850454   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:20.008406   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:21.057146   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
ok 49 - running Rego check lines ./httpproxy/002-header-condition-match.yaml:194-272
E1006 14:55:22.104704   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:23.234507   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:24.284648   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:25.387649   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:26.507108   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:27.559167   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:28.611710   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:29.666883   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:30.718077   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
ok 50 - deleting test objects
1..50

Add xUnit reporting

Add a --report argument that will write a test result report (possibly in a user-specified format). I don't know of any widely used, well-documented standard format, but xUnit (aka JUnit) and Cucumber both have suitable structured outputs.

I wasn't able to find a spec for cucumber JSON, but there are a few tools out there that generate nice HTML reports from that format.

I don't think that JUnit has a defined spec either, though like Cucumber there are a number fo implementations (e.g. this one in sonobuoy. Everyone mostly seems to just roll their own.

Need to check into how we can push results into the Github PR. There's no built-in Github Action support for test reporting. We could hit the GitHub API directly, or write an adjunct tool to publish the xUnit results.

Investigate OPA constraint framework

Take a look at the constraint framework to see whether it could be used to relieve people of having to write low level Rego code.

Builtins are missing from initial policy loader

What steps did you take and what happened:

If you add a policy file that references a builtin package, it errors on startup:

$ ./run-test-case.sh --trace=rego --preserve ./httpproxy/015-http-health-checks.yaml
integration-tester: 2 errors occurred:
/Users/jpeach/upstream/contour/_integration/testsuite/policies/contour-expect.rego:22: rego_type_error: undefined function data.builtin.result.Passf
/Users/jpeach/upstream/contour/_integration/testsuite/policies/contour-expect.rego:24: rego_type_error: undefined function data.builtin.result.Errorf

Consistent Versioning

We should make the versioning consistent across the projectcontour repos. The latest release currently is v4. It would be nice to follow what contour does (e.g. v4.0.0)

Improve Rego error location reporting

integration-tester creates implicit Rego packages for every document fragment that is detected as Rego, then compiled the whole conglomeration together in order to preflight compilation and syntax errors before making any cluster changes.

Since the Rego parser only ever sees the Rego language as strings, however, line numbers for error messages are relative to the fragment where the error is, not relative to the test document as a whole. This makes it awkward to correlate errors back to the original source (requires a bit of guesswork).

For example, this error is in part 1, but because it is a duplicate variable definition, we don't know until we compile the whole thing. Since the Rego compiler doesn't have the right line offsets to start with, it can't emit the right error location. The Rego parser has API to pass in the filename of a string to compile, but no API to set the initial line offsets.

$ ./run-test-case.sh ./httpproxy/012-https-fallback-certificate.yaml
TAP version 13
# reading document from ./httpproxy/012-https-fallback-certificate.yaml
# decoding document with 8 parts from ./httpproxy/012-https-fallback-certificate.yaml
# decoded part 0 as Kubernetes (lines 1-10)
# decoded part 1 as Rego (lines 12-42)
# decoded part 2 as Kubernetes (lines 44-121)
# decoded part 3 as Kubernetes (lines 123-131)
# decoded part 4 as Kubernetes (lines 133-141)
# decoded part 5 as Kubernetes (lines 143-158)
# decoded part 6 as Rego (lines 160-186)
# decoded part 7 as Rego (lines 188-214)
ok 1 - validating document "./httpproxy/012-https-fallback-certificate.yaml"
# test run ID is 52f423e6-504e-4e41-97ec-aa8c6349f05a
# Fatal - 1 error occurred: internal/check/ACnGakFqooos:28: rego_compile_error: var msg assigned above
not ok 2 - compiling test document
  ---
  - Message: '1 error occurred: internal/check/ACnGakFqooos:28: rego_compile_error:
      var msg assigned above'
    Severity: Fatal
    Timestamp: "2020-08-03T14:25:30.585828+10:00"

  ...
# skipping
ok 3 - deleting test objects
1..3

Investigate OPA decision logs

Please describe the problem you have

OPA has a "decision logging" feature that can be used to emit events describing Rego evaluations. Investigate whether it makes sense to try to use this to improve debugging and test failure triage.

https://github.com/open-policy-agent/opa/blob/master/plugins/logs/plugin.go#L411

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs

Jooble