adrianriobo / qe-eventmanager Goto Github PK

Event handler to integrate corporate messaging mechanism with CI/CD

Makefile 2.23% Go 93.76% Dockerfile 0.48% Shell 3.53%

ci-cd messaging openshift-v4 tekton-pipelines amqp github-api

qe-eventmanager's Introduction

qe-eventmanager

Roles

The app can act as a manager handling integrations based on flow definitions, or it can act as a tool to interact within the providers configured.

Actioner

As an actioner the cli allows to run single actions:

Send an UMB menssage

./qe-eventmanager umb send -p providers.yaml \
                           -m message.json \
                           -d VirtualTopic.sample

Create / update a repository status on github

./qe-eventmanager github status -p providers.yaml \
                           -s success \
                           -o sample-owner \
                           --repo sample-repo \
                           --ref SHA_COMMIT

Manager

As a manager integrate providers based on flow definitnions:

./qe-eventmanager start -p providers.yaml \
                        -f flow1.yaml,flow2.yaml

A simple overview on an umb-tekton integration

Configuration

The eventmanager requires a set of information around the providers on which it can act upon
and a set or flows defining the integrations and the actions to be executed.

Providers

umb:
  consumerID: foo
  driver: amqp
  brokers: broker1:5556,broker2:5556
  userCertificate: XXXXXXX # encoded as base64
  userKey: XXXXX # encoded as base64
  certificateAuthority: XXXXXX # encoded as base64
tekton:
  namespace: myNamespace
  workspaces:
  - name: workspace1
    pvc: pvc1
  - name: workspace2
    pvc: pvc2
  kubeconfig: XXXXXX # encoded as base64. This value is optional is used to connect to remote cluster
                     # Otherwise eventmanager can rely on RBAC when running inside the cluster
github: # Can be configured to auth based on pat or as a github app, bot require read public repos and read-write status rights 
  token: github_pat_token 
  appID: 1 
  appInstallationID: 99
  appKey: XXXXXX # encoded as base64

Flows

name:  sample-flow
input:
  umb:
    topic: topic-to-consume
    filters:
      - $.node1.node2[?(@.field1=='value1')].field1
      - $.node1.node2[?(@.field2=='value2')].field1
  ack:
    github:
      status:
        ref: $.node1.node2.sha
        owner: sample-owner
        repo: sample-repo # Configured github providers require rights on this repo
        status: pending
action:
  tektonPipeline:
    name: XXX
    params:
    - name: foo
      value: $.node1.node2[(@.field=='foo')].id # $. jsonpath expression function
    - name: bar
      value: bar # constant string 
  success:
    umb:
      topic: topic-to-produce
      eventSchema: message-schema-to-send
      eventFields:
      - name: foo
        value: $(pipeline.results.result1) # Pick value from pipeline results result1 
      - name: bar
        value: bar # constant string
  error:
    github:
      status:
        ref: $.node1.node2.sha
        owner: sample-owner
        repo: sample-repo # Configured github providers require rights on this repo
        status: error

Build

Cli

make clean
make build

Container

make container-build

qe-eventmanager's People

Contributors

Stargazers

Watchers

Forkers

jsliacan

qe-eventmanager's Issues

[Feature] Extend filtering to headers

Currently filters are only applied to message content, some events sent through UMB add content on headers, in order to properly get the messages it is required to extend the filtering mechanism to the headers

[Feature] Rely on triggers to improve decoupling and event definition standardization

[Enhacement] Handler aggregation

This issue is a following of #49 to track the implementation for the option with best perfomance

Review informers

Currently informers for tekton utilization is not optimal, need to move to:

filtered NewFilteredPipelineRunInformer
Factory

informers "github.com/tektoncd/pipeline/pkg/client/informers/externalversions"

sharedInfomer := informers.NewSharedInformerFactoryWithOptions(client, 0)
pipelineRunsInformer := sharedInfomer.Tekton().V1beta1().PipelineRuns()
sharedInfomer := informers.NewSharedInformerFactoryWithOptions(client, 0)
	pipelineRunsInformer := sharedInfomer.Tekton().V1beta1().PipelineRuns()
	pipelineRunsInformer.Informer().AddEventHandler(cache.FilteringResourceEventHandler{FilterFunc: func(obj interface{}) bool {
		
	}})
...

[Documentation] Include steps on how to deploy

Add steps to README on how to deploy qe-eventmanager

Subsription specs vs programatically

The idea is define subscription and actions based on yamls, initial idea would be something like:

destinations:
  inTopic:
    topicName: topic1
    filters:
    - jsonpath: $.artifact.products[?(@.nvr=='codeready_containers-1')].nvr   
  outTopic: topic2
  errorTopic: topic3
actions:
  pipeline:
    name: pipeline-name
    params:
    - name: rhel-version
      jsonpath: $.artifact.products[(@.product=='rhel')].version
    results:
       event: event-name
       fields:
       - eventfieldName: event-name
         pipelineResultName: result-name

https://play.golang.com/p/Ewby5vYaVrn
https://golangrepo.com/repo/spyzhov-ajson-go-json

[bug] Health check is not working for remote errors

We are seeing

:"Error reading from topic: VirtualTopic.qe.ci.product-scenario.build.complete. remote error: tls: user canceled","time":"2023-12-01T18:11:38Z"}
{"level":"debug","msg":"Finalize consumer for subscription VirtualTopic.qe.ci.product-scenario.build.complete","time":"2023-12-01T18:11:38Z"}
{"level":"debug","msg":"Service will shutdown gracefully","time":"2023-12-01T18:11:38Z"}
{"level":"error","msg":"remote error: tls: user canceled","time":"2023-12-01T18:11:38Z"}
{"level":"info","msg":"Unsubscribing VirtualTopic.qe.ci.product-scenario.build.complete","time"

It is expected the health check reports unhealthy on that situation as so the pod would be restarted. This behavior is not being honored

[Enhancement] Github app

Create a github app to manage the upstream downstream integration, currently the auth is based on PAT which is linked to an user... we want to decouple the auth management from personal users

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

dockerfile

images/builder/Dockerfile

golang 1.18

github-actions

.github/workflows/go.yaml

actions/checkout v3

actions/setup-go v3

gomod

go.mod

go 1.18

github.com/Azure/go-amqp v0.17.5

github.com/go-stomp/stomp/v3 v3.0.5

github.com/joshdk/go-junit v0.0.0-20210226021600-6145f504ca0d@6145f504ca0d

github.com/sirupsen/logrus v1.8.1

github.com/spf13/cobra v1.3.0

github.com/spf13/pflag v1.0.5

github.com/spf13/viper v1.10.0

github.com/spyzhov/ajson v0.6.0

github.com/tektoncd/pipeline v0.35.1

golang.org/x/exp v0.0.0-20231127185646-65229373498e@65229373498e

k8s.io/api v0.23.5

k8s.io/apimachinery v0.23.5

k8s.io/client-go v0.23.5

k8s.io/utils v0.0.0-20231127182322-b307cd553661@b307cd553661

knative.dev/pkg v0.0.0-20220329144915-0a1ec2e0d46c@0a1ec2e0d46c

github.com/bradleyfalzon/ghinstallation/v2 v2.0.4

github.com/google/go-github/v45 v45.0.0

golang.org/x/oauth2 v0.0.0-20220309155454-6242fa91716a@6242fa91716a

Check this box to trigger a request for Renovate to run again on this repository

Add retry mechanism on sending umb messages

[Enhancement] add gh integration

Add support for gh integration

https://github.com/cloudposse/github-status-updater

[BUG] Handle ERROR message:connection closed

On long run the subscriptions can have connectivity issues in that case we need to identify if the connection was closed due to an error or it was closed on purpose due to graceful stop

2021/05/26 05:28:34 Subscription 1: XXXX: ERROR message:connection closed
{"level":"debug","msg":"Read message from inactive subscription XXXX","time":"2021-05-26T05:28:34Z"}
{"level":"debug","msg":"Finalize consumer for subscription XXXX"

[Feature] UMB avoid missing messages

Query Datagrepper for messages which should be handle. Then queries the handle history source of truth. Pick those unattended and handle them

Create a configuration file structure

umb:
  consumerID: foo
  driver: amqp
  brokers: broker1:5556,broker2:5556
  userCertificate-data: XXXXXXX
  userKey-data: XXXXX
  certificateAuthority-data: XXXXXX
tekton:
  namespace: myNamespace
  workspaces:
  - name: workspace1
    pvc: pvc1
  - name: workspace2
    pvc: pvc2
  kubeconfig-data: XXXXXX #This value is optional, eventmanager can run based on RBAC inside the cluster

[Feature] github status as single action

Review config file

Currently it is only linked to umb, new integrations are expected (as an example #24) need to review the configuration file structure to accommodate this.

Also probably add validation on spec...(validate provider for input / output for spec) is defined.

[Bug] no route host control

"level":"error","msg":"Error reading from topic: VirtualTopic.qe.ci.product-scenario.build.complete. read tcp 172.23.36.243:56632-\u003e10.31.2.24:5671: read: no route to host","time":"2022-08-08T05:00:13Z"}

[Enhacement] Add metrics

https://prometheus.io/docs/guides/go-application/

[Feature] Add support for rhel interop events

Add support for rhel interop events

Avoid using same subscription ID on mulitple consumer queues

https://source.redhat.com/groups/public/enterprise-services-platform/it_platform_wiki/umb_appendix#queues-topics-and-virtualtopics

[Enhacement] Add health checks

https://blog.gopheracademy.com/advent-2017/kubernetes-ready-service/

[Enhancement] Add eventTemplate to define result messages within flow specs

Avoid having fixed event schemas on code.... allow on fly definitions

[Bug] tls: user canceled

{"level":"error","msg":"Error reading from topic: VirtualTopic.qe.ci.product-scenario.ariobolo.build.complete. remote error: tls: user canceled","time":"2022-06-22T22:04:39Z"}
{"level":"debug","msg":"Finalize consumer for subscription VirtualTopic.qe.ci.product-scenario.ariobolo.build.complete","time":"2022-06-22T22:04:39Z"}
{"level":"error","msg":"Error reading from topic: VirtualTopic.qe.ci.product-scenario.build.complete. remote error: tls: user canceled","time":"2022-06-22T22:04:39Z"}
{"level":"debug","msg":"Finalize consumer for subscription VirtualTopic.qe.ci.product-scenario.build.complete","time":"2022-06-22T22:04:39Z"}
{"level":"error","msg":"Error reading from topic: VirtualTopic.qe.ci.product-scenario.build.complete. remote error: tls: user canceled","timerror: tls: user canceled","time":"2022-06-22T2e":"2022-06-22T22:04:39Z"}
{"level":"debug","msg":"Finalize consumer for subscription VirtualTopic.qe.ci.product-scenario.build.complete","time":"2022-06-22T22:04:39Z"}

Externalize consumerID value

Design how to deal with output messages composition

The income message does not really need a fixed structure defined....as the idea is access direct field with jsonpath expression and use them to accommodate the actions.

On the other hand the outcome will transmit a message to a target. So it is needed to think the best approach for handle different message schemas.

Do they need to be pre defined on code?
Do we include a set of them depending on the integrations?. How do we make users aware of them?
Would the spec support inline template definition?
In this case how to reuse to avoid flooding specs with duplications?

[QE] Adding declarative tests

Add tests using kuttl

xunit util

A new util class was created to support certain actions on xunit files, this tool could be expanded in multiple ways:

Validate xunit and resume content
Integrate xunit file/s with report portal
Integrate xunit file/s with polarion *Add validate to xunit to support that integration

[Bug] artifact and system nodes from interop events are missing

During the refactor some bug was introduced, now the response is not showing the artifact and system nodes anymore

[BUG] Workspaces bindings not added to pipelinerun spec

As result pipelineruns fail with:

Warning  InternalError  17m (x2 over 17m)  PipelineRun  1 error occurred:
           * pipeline requires workspace with name...

[Bug] Success not executed on pipeline with skipped tasks

Currently the informer on pipelinerun will check for state PipelineRunReasonSuccessful from doc:

// PipelineRunReasonSuccessful is the reason set when the PipelineRun completed successfully
PipelineRunReasonSuccessful PipelineRunReason = "Succeeded"
// PipelineRunReasonCompleted is the reason set when the PipelineRun completed successfully with one or more skipped Tasks
PipelineRunReasonCompleted PipelineRunReason = "Completed"

So it is required to include PipelineRunReasonCompleted as success execution management

[Feature] Add optional action on input ack

The idea would be define some action as an ack for input reception, sample

name: gh-pr-updater
input: 
  umb: 
    topic: VirtualTopic.external.github.code-ready.crc
    filters:
    - $.action=='synchronize'
  ack:
    github:
      commitStatus:
        owner: XXXX
        repo: XXXX
        commit: XXXXX
        status: pending

Standarization

The possible action would be define the contracts for the pipelines (input parameters, output parameters) as an example we can use OPA to ensure the specs.

Base pipeline can be re used and only specific steps would be replace per project.

If contract is pre-defined do not need to rely on triggers as the invocation would be the same to all products. We style require specific steps (again with some contract handle with OPA) and move the jsonpath from triggers to umb read messagges.

[BUG] Error stopping the qe-eventmanager

When stopping the qe-eventmanager with ctrl+c a panic error is shown:

^Cpanic: sync: negative WaitGroup counter

goroutine 24 [running]:
sync.(*WaitGroup).Add(0xc0000404c4, 0xffffffffffffffff)
	/usr/lib/golang/src/sync/waitgroup.go:74 +0x139
sync.(*WaitGroup).Done(0xc0000404c4)
	/usr/lib/golang/src/sync/waitgroup.go:99 +0x34
github.com/adrianriobo/qe-eventmanager/pkg/services/messaging/umb.consume(0xc0000a25a0, 0xc0000404e0)
	/workspace/pkg/services/messaging/umb/client.go:85 +0x1c3
created by github.com/adrianriobo/qe-eventmanager/pkg/services/messaging/umb.Subscribe
	/workspace/pkg/services/messaging/umb/client.go:60 +0x244

Remove hacky code for interop ocp integration

Previous version includes some hacky code for interop ocp testing with fixed values for results urls. Now the system gets the information direcly from pipeline results.

[Feature] Design queue mechanism for PR

Under certain circumstances it is possible new synchronized events re trigger the same action. This new event invalidate the previous one so does the previous execution.

The idea would be to map (persist) the PR with the action, and on new events for the PR stop previous action execution and start a new execution.

Include cmd to send messages through UMB

[Bug] Multiple handler for same topic

Currently each flow definition will create its own subscription, the subscription consumer id are created the same...so only one handler will get the message.

Quick fix can be created adding some extra suffix or prefix to the consumer id per flow definition.

The best approach although (performance) would be parse the flow definitions and aggregate the handling to avoid multiple subscriptions to same topic, but having one subscription with multiple handlers

[Spike] amqp support for UMB

https://github.com/streadway/amqp
https://github.com/rabbitmq/amqp091-go

Include validators

On spec the user has the freedom to define linkages with components.

One of them at the moment would be tekton pipelines and params on a specific pipeline.

It will be required to run some validations around them.

Get pipeline spec and check params and results
Create some OPA template and run against pipeline spec?

[Bug] Run informers on temporary lost connectivity

{"level":"debug","msg":"Created pipelinerun crc-builder-custom-gz5c7 with params [{ocp-index-url {string https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/4.11.0-fc.3 []}}]","time":"2022-06-28T13:14:27Z"}
{"level":"debug","msg":"Added informer for pipelinerun crc-builder-custom-gz5c7","time":"2022-06-28T13:14:27Z"}
W0629 02:25:50.986082       1 reflector.go:442] pkg/services/cicd/tekton/tekton.go:109: watch of *v1beta1.PipelineRun ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0629 02:26:22.189954       1 reflector.go:324] pkg/services/cicd/tekton/tekton.go:109: failed to list *v1beta1.PipelineRun: Get "https://172.30.0.1:443/apis/tekton.dev/v1beta1/namespaces/codeready-container/pipelineruns?resourceVersion=2824531277": dial tcp 172.30.0.1:443: i/o timeout
I0629 02:26:22.190146       1 trace.go:205] Trace[1297033098]: "Reflector ListAndWatch" name:pkg/services/cicd/tekton/tekton.go:109 (29-Jun-2022 02:25:52.185) (total time: 30004ms):
Trace[1297033098]: ---"Objects listed" error:Get "https://172.30.0.1:443/apis/tekton.dev/v1beta1/namespaces/codeready-container/pipelineruns?resourceVersion=2824531277": dial tcp 172.30.0.1:443: i/o timeout 30004ms (02:26:22.189)
Trace[1297033098]: [30.004583446s] [30.004583446s] END
E0629 02:26:22.190186       1 reflector.go:138] pkg/services/cicd/tekton/tekton.go:109: Failed to watch *v1beta1.PipelineRun: failed to list *v1beta1.PipelineRun: Get "https://172.30.0.1:443/apis/tekton.dev/v1beta1/namespaces/codeready-container/pipelineruns?resourceVersion=2824531277": dial tcp 172.30.0.1:443: i/o timeout
W0629 02:26:54.448830       1 reflector.go:324] pkg/services/cicd/tekton/tekton.go:109: failed to list *v1beta1.PipelineRun: Get "https://172.30.0.1:443/apis/tekton.dev/v1beta1/namespaces/codeready-container/pipelineruns?resourceVersion=2824531277": dial tcp 172.30.0.1:443: i/o timeout
I0629 02:26:54.448927       1 trace.go:205] Trace[63718591]: "Reflector ListAndWatch" name:pkg/services/cicd/tekton/tekton.go:109 (29-Jun-2022 02:26:24.446) (total time: 30002ms):
Trace[63718591]: ---"Objects listed" error:Get "https://172.30.0.1:443/apis/tekton.dev/v1beta1/namespaces/codeready-container/pipelineruns?resourceVersion=2824531277": dial tcp 172.30.0.1:443: i/o timeout 30002ms (02:26:54.448)
Trace[63718591]: [30.002162328s] [30.002162328s] END
E0629 02:26:54.448946       1 reflector.go:138] pkg/services/cicd/tekton/tekton.go:109: Failed to watch *v1beta1.PipelineRun: failed to list *v1beta1.PipelineRun: Get "https://172.30.0.1:443/apis/tekton.dev/v1beta1/namespaces/codeready-container/pipelineruns?resourceVersion=2824531277": dial tcp 172.30.0.1:443: i/o timeout
I0629 02:34:53.364901       1 trace.go:205] Trace[1751069271]: "Reflector ListAndWatch" name:pkg/services/cicd/tekton/tekton.go:109 (29-Jun-2022 02:34:40.633) (total time: 12731ms):
Trace[1751069271]: ---"Objects listed" error:<nil> 12731ms (02:34:53.364)
Trace[1751069271]: [12.731582346s] [12.731582346s] END
E0629 03:58:08.831856       1 reflector.go:138] pkg/services/cicd/tekton/tekton.go:109: Failed to watch *v1beta1.PipelineRun: the server has asked for the client to provide credentials (get pipelineruns.tekton.dev)
I0629 03:58:22.568620       1 trace.go:205] Trace[1996325786]: "Reflector ListAndWatch" name:pkg/services/cicd/tekton/tekton.go:109 (29-Jun-2022 03:58:10.103) (total time: 12464ms):
Trace[1996325786]: ---"Objects listed" error:<nil> 12464ms (03:58:22.568)
Trace[1996325786]: [12.464914408s] [12.464914408s] END

[Enhacement] re design flow model implementation

Currently the app has grown since initial use case as umb tekton integration.. The initial desing include the action and results as part of handling on the umb manager side...

Now it is required to decouple it and add a chan mechanism to communicate the different stages of the flow.

The final idea is to handle stages by the general manager and not by the umb manager (legacy design)