attx-project / distribution-component Goto Github PK

Description

We need to come up with a way to actually implement scenarios described in
https://attx-project.github.io/ServiceDiscovery-Implementation.html using ElasticSearch.

DoD

Application level metadata can be queries from ServiceDiscovery.

Testing

Integration tests.

What are the components and configurations needed to setup ES with ATTX API?

Description

There is a need to have a "Big Brother" like component for the logs from our components in order to check for status, errors and possible warnings.

A possible implementation would be logstash: https://www.elastic.co/products/logstash or the ELK stack
A tutorial in Czech (although google translate FTW) is: https://www.zdrojak.cz/clanky/sber-logu-z-kontejneru/
https://www.katacoda.com/courses/docker-production/launch-elk-aggregate-container-logs

Alternative solutions:

• http://www.fluentd.org/ with tutorial: http://www.fluentd.org/guides/recipes/docker-logging

DoD:

Logs collected and visible in the Distribution Component (TBD?) Elasticsearch (TBD?)
Possible changes to different components on logs format need to be made.
Provenance should be tied to the logs.

Testing:

Peer Review Unit test

Use cris publication from UHOD.

DoD:
We have a set of json documents that contain nested data structure.

Testing:
Peer review

Description

Persistent messaging is required so that we can use the MS as kind of a shared storage between DPUs working within a single pipeline execution.

DoD

Working image + configuration for docker compose.

Testing

Peer-review

Description
What if the new versions of ES have the features we need in order to index them.
DoD:
Wikipage documenting motivation and the test data, test queries and indicators of performance (speed + accuracy).

Implement tests for running them and extracting the indicators.
small HTML file as output

Testing:
Peer review

Refactoring
Unit testing

• is it running?
• were all the plugins installed successfully?
• are the queries working?

DoD:

Two instances of Elastic search

• elasticsearch v 1.3.4 + siren plugin
• latest elasticseach + custom config

Containarize instances with the data.

Testing:
Peer Review