redhat-appstudio / e2e-tests

Repository for covering appstudio by e2e tests.

Red Hat AppStudio E2E Tests and Testing Framework

Testing framework and E2E tests are written in Go using Ginkgo and Gomega frameworks to cover Red Hat AppStudio. It is recommended to install AppStudio in E2E mode, but the E2E suite can be also usable in development and preview modes.

Features

• Instrumented tests with Ginkgo 2.0 framework. You can find more information in Ginkgo documentation.
• Ability to run the E2E tests everywhere: locally(CRC/OpenShift local), OpenShift Cluster, OSD...
• Writes tests results in JUnit XML/JSON file to a custom directory by using --ginkgo.junit(or json)-report flag.
• Ability to run the test suites separately.

Start running E2E tests

All the instructions about installing RHTAP locally/CI and running tests locally/CI can be found in this Documentation, which contains also information about how to pair Pull Requests when breaking changes are introduced.

Start developing tests

To develop new tests in RHTAP consider first to reading some tips for a better experience:

• Basic tips to write readable tests. Documentation.
• How to auto generate tests. Documentation.

Start debugging CI

To onboard a new component in Openshift CI follow this Documentation. To debug CI jobs follow this Documentation.

HAPPY TESTING!

e2e-tests's Issues

Contributors must use `infra-deployments` as the name of their fork

When contributors make changes to the App Studio deployment, they currently must use a fork named infra-deployments. This is not always true - developers are free to use their own repo names when forking. Some developers (me!) like to use verbose repo names in their forks to avoid name collisions.

Page not Found in link

In section "The e2e-appstudio command" a word is linked with link but it redirect to 404 web page

Error: invalid argument "client" for "--dry-run" flag: strconv.ParseBool: parsing "client": invalid syntax

Followed https://github.com/redhat-appstudio/e2e-tests#install-appstudio-in-e2e-mode, but got the below errors. Could you help have a look? Or am I missing something? Thanks!

MacBook-Pro:e2e-tests jianzhang$ make local/cluster/prepare
./mage -v local:prepareCluster
Running target: Local:PrepareCluster
exec: which "jq"
/opt/homebrew/bin/jq
exec: which "kubectl"
/usr/local/bin/kubectl
exec: which "oc"
/usr/local/bin/oc
exec: which "yq"
/opt/homebrew/bin/yq
exec: which "git"
/usr/bin/git
exec: go "install" "-mod=mod" "github.com/onsi/ginkgo/v2/ginkgo"
I0120 16:49:52.547757   95146 request.go:682] Waited for 1.044926375s due to client-side throttling, not priority and fairness, request: GET:https://api.qe-daily-413-0120.qe.azure.devcluster.openshift.com:6443/apis/policy/v1?timeout=32s
W0120 16:49:54.027569   95146 install.go:128] folder /Users/jianzhang/e2e-tests/tmp/infra-deployments already exists... removing
I0120 16:49:54.028738   95146 install.go:138] cloning 'https://github.com/redhat-appstudio/infra-deployments' with git ref 'refs/heads/main'
Enumerating objects: 9062, done.
Counting objects: 100% (30/30), done.
Compressing objects: 100% (23/23), done.
Total 9062 (delta 13), reused 17 (delta 7), pack-reused 9032

Installing the OpenShift GitOps operator subscription:
subscription.operators.coreos.com/openshift-gitops-operator created

Waiting for default project (and namespace) to exist: ......................................OK

Waiting for OpenShift GitOps Route: OK

Patching OpenShift GitOps ArgoCD CR
argocd.argoproj.io/openshift-gitops patched
argocd.argoproj.io/openshift-gitops patched
argocd.argoproj.io/openshift-gitops patched

Add Role/RoleBindings for OpenShift GitOps:
clusterrole.rbac.authorization.k8s.io/appstudio-openshift-gitops-argocd-application-controller created
clusterrole.rbac.authorization.k8s.io/appstudio-openshift-gitops-argocd-server created
clusterrolebinding.rbac.authorization.k8s.io/appstudio-openshift-gitops-argocd-application-controller created
clusterrolebinding.rbac.authorization.k8s.io/appstudio-openshift-gitops-argocd-server created

Setting secrets for GitOps
Error: invalid argument "client" for "--dry-run" flag: strconv.ParseBool: parsing "client": invalid syntax


Aliases:
namespace, ns

Examples:
  # Create a new namespace named my-namespace
  kubectl create namespace my-namespace

Options:
      --allow-missing-template-keys=true: If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.
      --dry-run=false: If true, only print the object that would be sent, without sending it.
      --generator='namespace/v1': The name of the API generator to use.
  -o, --output='': Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file.
      --save-config=false: If true, the configuration of current object will be saved in its annotation. Otherwise, the annotation will be unchanged. This flag is useful when you want to perform kubectl apply on this object in the future.
      --template='': Template string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].
      --validate=true: If true, use a schema to validate the input before sending it

Usage:
  kubectl create namespace NAME [--dry-run] [options]

Use "kubectl options" for a list of global command-line options (applies to all commands).

error: no objects passed to apply
Error: error when bootstrapping cluster: exit status 1
make: *** [local/cluster/prepare] Error 1

The go and oc info

MacBook-Pro:e2e-tests jianzhang$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.13.0-0.nightly-2023-01-17-152326   True        False         9h      Cluster version is 4.13.0-0.nightly-2023-01-17-152326
MacBook-Pro:e2e-tests jianzhang$ go version
go version go1.18.5 darwin/amd64
MacBook-Pro:e2e-tests jianzhang$ oc version 
Client Version: 4.12.0
Kustomize Version: v4.5.7
Server Version: 4.13.0-0.nightly-2023-01-17-152326
Kubernetes Version: v1.25.2+7dab57f

Junit XML Report is NOT a well formed XML document when it contains a Failure element

I was demo-ing validating the xunit file generated by the e2e-test framework for this run

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/redhat-appstudio_e2e-tests/120/pull-ci-redhat-appstudio-e2e-tests-main-redhat-appstudio-e2e/1537469058608271360/artifacts/redhat-appstudio-e2e/redhat-appstudio-e2e/artifacts/e2e-report.xml

I noticed I got a bunch of errors just running it through a basix xml validator like these two

https://www.w3schools.com/xml/xml_validator.asp
https://www.freeformatter.com/xpath-tester.html

The cause was this Failure element's message= attribute

<failure message="Unexpected error: <*errors.StatusError | 0xc000f463c0>: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: "ApplicationSnapshot.appstudio.redhat.com \"snapshot\" is invalid: spec.images: Required value", Reason: "Invalid", Details: { Name: "snapshot", Group: "appstudio.redhat.com", Kind: "ApplicationSnapshot", UID: "", Causes: [ { Type: "FieldValueRequired", Message: "Required value", Field: "spec.images", }, ], RetryAfterSeconds: 0, }, Code: 422, }, } ApplicationSnapshot.appstudio.redhat.com "snapshot" is invalid: spec.images: Required value occurred" type="failed">/go/src/github.com/redhat-appstudio/e2e-tests/tests/release/release.go:66 github.com/redhat-appstudio/e2e-tests/tests/release.glob..func1.3.1() /go/src/github.com/redhat-appstudio/e2e-tests/tests/release/release.go:66 +0xa3</failure>

The cause was two fold

• I needed to escape any <|> characters properly in the *errors.StatusError of the message string
• I need to remove the double quotes that wrapped the message string and wrap it in a single quotes

Here is how it looks after I made the changes

<failure message='Unexpected error: &lt;*errors.StatusError | 0xc000f463c0&gt;: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: "ApplicationSnapshot.appstudio.redhat.com \"snapshot\" is invalid: spec.images: Required value", Reason: "Invalid", Details: { Name: "snapshot", Group: "appstudio.redhat.com", Kind: "ApplicationSnapshot", UID: "", Causes: [ { Type: "FieldValueRequired", Message: "Required value", Field: "spec.images", }, ], RetryAfterSeconds: 0, }, Code: 422, }, } ApplicationSnapshot.appstudio.redhat.com "snapshot" is invalid: spec.images: Required value occurred' type="failed">/go/src/github.com/redhat-appstudio/e2e-tests/tests/release/release.go:66 github.com/redhat-appstudio/e2e-tests/tests/release.glob..func1.3.1() /go/src/github.com/redhat-appstudio/e2e-tests/tests/release/release.go:66 +0xa3</failure>

We should most likely sanitizing any error messages that we plan to write out so that we don't have issues parsing the junit xml report for later use in reporting of results.

https://www.advancedinstaller.com/user-guide/xml-escaped-chars.html

Failed PipelineRuns are waiting for timeout

When pipelineRun fails then it's waiting on timeout instead of immediate fail.

https://github.com/redhat-appstudio/e2e-tests/blob/2c13a2031925cdb973269de3234a5b93b2d1398e/pkg/tests/has/devfile_source.go#L86-L98

Please add corresponding needed entries into "Type of change" category in the PR page

Please categories as per the Conventional Commits PR Check CI tests
For example: test