elastic / elastic-agent-inputs Goto Github PK

Document how run and test inputs.

Make sure to include:

• Development conventions and guidelines
• Requirements for creating a PR (all lint/checks, code generation/etc)
• Any specific testing setup. E.g: how to connect to a local Shipper, how to connect to Elastic-Agent, etc
• Any debugging setup. E.g: How to setup delve when running under Elastic-Agent. Any timeouts or other settings that need to be changed in order to use the debugger, etc.

1. #24
2. #5
3. #25
4. #26
5. #27
6. #28
7. #29
8. #3
9. #30
10. #31
11. #32
12. #33
13. #35
14. #34
15. elastic/elastic-agent#592

The goal is to make sure we can run unit tests with our automation and that they are run on CI/PRs

Acceptance criteria:

• At least one unit tests (it can be a very simple one or a dummy one)
• Create the necessary mage targets
• Create lint targets
• Integrate on CI
• Make sure PRs require the correct checks before merges

Create the load generation integration and have it released at least as a snapshot package.

The goal is to be able to run under Elastic-Agent and communicate with it to receive config and send status. Only the bare bones of the connection/communication needs to be implemented.

Acceptance criteria:

• Make sure the Elastic-Agent has got everything needed to start and run the load generation input
• Bare bones of control protocol is implemented:
  • Integrate the shipper client and control client (if not integrated already)
  • Elastic-Agent can start input and connect to it
  • Input can send metrics/report errors

The goal is to implement the core functionality of a load generator to test the shipper and be the firs implemented input.

Design

• Look into the current documentation about the shipper performance testing: elastic/elastic-agent-shipper#57
• Investigate if https://github.com/elastic/stream can be used as our core

Acceptance criteria

• Configurable throughput
• Set the number of events or a constant rate/throughput
• Timestamp increments can be configured independently from throughput (e.g: 100 events with a timestamp delay of 1s between each other can be generated/sent as fast as possible)
• Random data generated, the first implementation can have a single format
• Unique ID on every message (can be something log friendly like xid)

Nice to have:

• Configurable data format, ideally based on Elastic-Agent integrations

The goal is to make sure the Elastic-Agent can fully control an input, the load generation input is going to be the first one.

Acceptance criteria:

• Receive/validate/parse config and return errors when needed
• The process starts and data starts to be sent to the shipper
• Stopping the process/input
• Reload configuration

The goal of this task is to provide a suite for testing inputs to validate that it can be controlled by Elastic Agent and can forward events to the shipper. The test suite should rely on the testing package of Golant.

Requirements

• The tester must be able to test an input regardless of its language (e.g. input can be written in Python)
• The tester must check if the input understands the control protocol
• The tester must check if the input can ship events using the shipper protocol

Test cases

Control protocol

• Reports OK if correct configuration is sent
• Reports Error if incorrect configuration is sent
• Can start with correct configuration
• Can be stopped when running in healthy state
• Can be killed if running in degraded state

Shipper

• Can publish events to the shipper
• Can handle publishing errors

Update Go version to 1.18.x (the latest at the time of implementation).

Acceptance criteria:

• Input/generated binary work
• All tests pass, no flakiness are introduced
• Build & Package work
• Lint works

To support developers who are creating Go based inputs we should create a code generator.

• identify parts of the code that can be reused in all inputs
• write the generator (it can use template, go gen, whatever necessary)
• add basic generator job to CI (similar to the old beat generator job)

The goal of the issue is to prepare a repository for developers to contribute to.

TODO

• Create a new repository named elastic-agent-inputs
• Transfer tickets from the meta ticket to the new repo
• Add licence
• Add CHANGELOG
• Add the new repo to the CI
  • add access to @elasticmachine
  • run onboarder job: https://beats-ci.elastic.co/job/robots/job/onboarding-ci-pipeline/
• Run linter from elastic-agent-libs (example: elastic/beats#30896 elastic/beats#30985)
• Enable robots automation for labels creation

What is an Elastic Input?

Elastic Inputs are binaries that are distributed with and controlled by Elastic Agent. As the name implies they collect data from and on endpoints where they are deployed.

Goal

The goal of this project is to design a developer friendly inputs platform. The system should help people to write new inputs and test them easily. This ticket lists the requirements and defines the different development phases.

You can read about the motivation in the design document below.

Requirements

Input

• Each input must speak the control protocol of Elastic Agent
• Each input must provide a manifest file describing its features
• Each input must run in a separate process
• Each input must log to stdout and stderr in JSON format
• Each input must be able to test a configuration and report errors without starting data collection
• Each input must be able to send events to the shipper
• Each input must provide a reference configuration to aid integration developers
• Each input must accept a logger on init and enrich log events with metadata so the issues can be tracked down elastic/beats#9177

Developer experience

• Testing framework
  • to check if the input understands the control protocol
  • to check if the input can report its own state, metrics and logs
• Input code generator to let users work on production logic and not waste time on boilerplate
• Write a guide for Elastic developers for migrating existing inputs

Development plan

Phase I.

The goal of the first phase is to lay the foundation for the new developer tooling by developing the first new agent input. The first input developed will be the agent load generator described in elastic/elastic-agent-shipper#57. Developing a completely new input will allow this phase to focus on the basic input structure, developer tooling, and release process.

TODO

TBD.

• Input development to support elastic/elastic-agent-shipper#57
• elastic/beats#2
• Use the new V2 protocol status reporting features to support: elastic/elastic-agent#100
• elastic/beats#3
• elastic/beats#5

Dependencies

• elastic/elastic-agent#189
  • elastic/elastic-agent-client#29
  • elastic/elastic-agent#592
  • elastic/elastic-agent#221
  • elastic/elastic-agent#222

Packaging

Packaging is done with mage. Every binary is crossbuilt in a Docker container. Then the resulting binaries and generated files are moved to a common folder.

A package contains inputs in separate binaries with their manifest files.

$ tar tvf elastic-agent-inputs-linux-arm64.tar.gz
inputs/
  filestream/
    filestream
    manifest.yml
  journald/
    journald
    manifest.yml
...

If someone wants to create their own inputs package, it must follow the pattern above. Then this package has to be extracted into the folder where Agent can find it.

inputs/
  {input_name}/
    {input_name}
    manifest.yml

The packages could be installed manually by moving the inputs to the appropriate folder. However, we should also provide DEB, RPM and other packages that can be installed along with Elastic Agent.

More details here: elastic/elastic-agent#222

Phase II.

In this phase we focus on making input migration and development accessible to both internal and external developers. In this phase the filestream and system metrics inputs will be migrated to the new framework.

TODO

• elastic/beats#6
• elastic/beats#4
• elastic/beats#7
• Write documentation for creating new inputs
• Write documentation for migrating existing inputs

Dependencies

• elastic/elastic-agent#222

Phase III.

Phase III is going to be the first phase for moving real inputs to the new system. The inputs selected for this phase are easy to move because they already use the inputs v2 architecture. We could delegate/ask other teams to help with inputs under x-pack/filebeat.

• Move inputs that have adopted the input management v2 architecture in filebeat
  • filestream
  • journald
  • kafka
  • unix
  • winlog
• Move inputs that have adopted the input management v2 architecture in x-pack/filebeat
  • awss3
  • awscloudwatch
  • cloudfoundry
  • http_endpoint
  • o365audit

Phase IV.

This is the last phase of the development. I expect collaboration from all teams as our team does not have the bandwidth to migrate all existing inputs. In this phase the Data plane team should focus on supporting other developers by reviewing their work, adjusting documentation if needed, etc.

TODO

• Move remaining inputs...
• Document how to add independent inputs written in arbitrary languages

Documentation

• Document how to collect monitoring information from inputs
• Document how to add new Golang inputs to our input manager
• Document how to add independent inputs written in arbitrary programming languages

Related

• elastic/elastic-agent#1237

The goal is to have the few first integration tests as well as the infrastructure necessary to run them and implement new ones.

Acceptance criteria:

• Use the testing framework to create the first integration tests (it can be a very simple one like: “input can start and stop)
• Create the necessary mage targets (align with shipper/beats/agent)
• Include it on CI/PR checks, etc.

Connect to the Shipper and send data.

Acceptance criteria:

• Implement the V2 control protocol
• Abide to all defined configuration formats
• Support a simple console/file output that can be used for debugging development independently from the shipper

Currently this repository is lacking some of the configuration/automation other repositories like beats have. The goal is to bring this repository to the same standards.

Acceptance criteria

• main branch is protected
• Follow current standards for issue/PR creation
• Only allow "squash and merge" for PRs

Implement status/error reporting following the V2 specification.

Acceptance criteria:

• Can collect and report metrics and logs
• Can report errors when they happen (push model)
• Follows the V2 specification

The goal is to be able to receive a configuration, parse, test, and use it. At the moment the focus is on reading a configuration YAML, but the implementation should be easily pluggable into the control protocol V2.

Acceptance criteria:

• Read/parse a YAML config file.
• Implement config validation
• Provide examples/reference
• Logging configuration (if not part of the control protocol must also be included)
• Basic input configuration (id, etc)
• Refuses to start if config is invalid

The newly created inputs must be packaged so Elastic Agent can run them. In the first round, it does not have to be perfect.

Related elastic/elastic-agent#222