elastic / stream

Test utility for streaming data via udp/tcp/tls/webhook/GCP Pub-Sub/Kafka.

stream

stream is a test utility for streaming data via:

• UDP
• TCP
• TLS
• Webhook
• GCP Pub-Sub
• Kafka
• Lumberjack
• HTTP Mock Server
• Azure Blob Storage
• Google Cloud Storage
• Azure Event Hub

Input data can be read from:

• log file - Newline delimited files are streamed line by line.
• pcap file - Each packet's transport layer payload is streamed as a packet. Useful for replaying netflow and IPFIX captures.

HTTP Server mock reference

stream can also serve logs setting up a complete HTTP mock server.

Usage:

stream http-server --addr=":8080" --config="./config.yml"

The server can be configured to serve specific log files on certain routes. The config should be defined in a yaml file of the following format:

---
    rules:
    - path: "/path1/test"
      methods: ["GET"]

      user: username
      password: passwd
      query_params:
        p1: ["v1"]
      request_headers:
        accept: ["application/json"]

      responses:
      - headers:
          x-foo: ["test"]
        status_code: 200
        body: |-
          {"next": "http://{{ hostname }}/page/{{ sum (.req_num) 1 }}"}
    - path: "/page/{pagenum:[0-9]}" params.
      methods: ["POST"]

      responses:
      - status_code: 200
        body: "{{ .request.vars.pagenum }}"
        headers:
          content-type: ["text/plain"]

The rules will be defined in order, and will only match if all criteria is true for a request. This means that you need to define the more restrictive rules on top.

Options

• as_sequence: if this is set to true, the server will exit with an error if the requests are not performed in order.
• rules: a list of rules. More restrictive rules need to go on top.
• path: the path to match. It can use gorilla/mux parameters patterns.
• methods: a list of methods to match with the rule.
• user and password: username and password for basic auth matching.
• query_params: Key-Value definitions of the query parameters to match. It can use gorilla/mux parameters patterns for the values. Web form params will also be added and compared against this for simplicity. If a key is given an empty value, requests with this parameter will not satisfy the rule.
• request_headers: Key-Value definitions of the headers to match. Any headers outside of this list will be ignored. The matches can be defined as regular expressions.
• request_body: a string defining the expected body to match for the request. If the string is quoted with slashes, the leading and trailing slash are stripped and the resulting string is interpreted as a regular expression.
• responses: a list of zero or more responses to return on matches. If more than one are set, they will be returned in rolling sequence. If as_sequence is set to true, they will only be able to be hit once instead of in rolling sequence.
• status_code: the status code to return.
• headers: Key-Value list of the headers to return with the response. The values will be evaluated as Go templates.
• body: a string defining the body that will be returned as a response. It will be evaluated as a Go template.

When using Go templates as part of the response.headers or response.body, some functions and data will be available:

• hostname: function that returns the hostname.
• env KEY: function that returns the KEY from environment.
• sum A B: function that returns the sum of numbers A and B (only for integers).
• file PATH: function that returns the contents of the file at PATH.
• .req_num: variable containing the current request number, auto incremented after every request for the rule.
• .request.vars: map containing the variables received in the request (both query and form).
• .request.url: the url object. Can be used as per the Go URL documentation.
• .request.headers the headers object. Can be used as per the Go http.Header documentation.

Lumberjack Output Reference

Lumberjack is the protocol used between Elastic Beats and Logstash. It is implemented using the elastic/go-lumber library. stream sends data using version 2 of the Lumberjack protocol. Each log line is sent as its own batch containing a single event. The output blocks until the batch is ACKed.

When using the Lumberjack output the address flag value (--addr) can indicate when to send via TLS. Format the address as a URL with a tls scheme (e.g. tls://127.0.0.1:5044) to use TLS. If a scheme isn't specified then a TCP connection is used (i.e. localhost:5044 implies tcp://localhost:5044).

By default, Lumberjack batches contain one event with a message field.

[
  {
    "message": "{{ input_data }}"
  }
]

If --lumberjack-parse-json is used then the input data is parsed as JSON and the resulting data is sent as a batch.

GCS Output Reference

The GCS output is used to collect data from the configured source, create a GCS bucket, and populate it with the incoming data. When specifying a (--addr) which should be a combination of both host and port, usually pointing to a locally running emulator, the client will be overriding the configured API endpoint, which defaults to the public google storage API, towards the emulator instead. The emulator does not require authentication.

Options

• gcs-bucket: The name of the GCS bucket that should be created, should not already exist.
• gcs-object: The name of the GCS object that will be populated with the collected data, using the configured GCS bucket.
• gcs-projectid: The related projectID used when creating the bucket, this is required to be changed from the default value when not using an emulator.

Azure Event Hub Output Reference

The Azure Event Hub output is used to collect data from the azure event hub resource When specifying a --azure-event-hub-connection-string, it should be retrieved as described here.

Sample config:

version: '2.3'
services:
  azure-event-hub:
    image: docker.elastic.co/observability/stream:v0.12.0
    volumes:
      - ./sample_logs:/sample_logs:ro
    command:
      - log
      - --retry=30
      - -p=azureeventhub
      - --azure-event-hub-connection-string="Endpoint=sb://test-eventhub-stream-seis.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SharedAccessKey"
      - /sample_logs/testdata.log

Options

• azure-event-hub-connection-string: The connection string to connect to the Event Hub.
• azure-event-hub-namespace: The fully qualified domain name of the Event Hubs namespace. This it the Event Hubs namespace followed by servicebus.windows.net (e.g. myeventhub.servicebus.windows.net).
• azure-event-hub-name: The name of the Event hub.

stream's Issues

Add option to http mock to fail if it gets a request that does not match

Currently the default behavior is to return a 404 if it gets a request for an unmatched rule.

We want to add a flag that exits 1 if this happens.

Add a GCP pub/sub output

Create an output that creates a GCP pub/sub topic if it does not exist and publishes messages to the topic. This could be used with the GCP pub/sub emulator for testing. I would expect the output to write to a single topic for simplicity. You could run multiple instances if you need to sent to multiple topics.

Add the concept of sequences in http mock

Currently there is no enforcement of specific sequences of requests. We want to be able to describe specific sequences of request, and it should exit 1 if a request is not made in the expected order.

Rate limit UDP output

The UDP output will dump all data as fast as possible. It will overload buffers of the recipient resulting in data loss. The output implementation should have some rate limit that's configurable (e.g. 500 KiB/sec).

There's a TODO here, but that's probably not the correct place for the logic as it would only apply to PCAP.

https://github.com/andrewkroh/stream/blob/7e151a6c712c398d433bc30cb9cd92c0d80959e6/command/pcap.go#L77

Add multiline input file support

Support reading multiple lines as a single event.

Add configurable framing for streams

Allow choosing newline delimited or octet counting.

Expand globs from paths

Rather than relying on a shell for path expansion use filepath.Glob to expand any path parameters received.

This will be helpful if we release an image that doesn't have a shell (just FROM scratch with the stream binary + libc + libpcap).

Add support for gzip

This issue here is to request support for sending the given log file in gzip format.

[ci] Produce multi-platform image (amd64, arm64)

The image produced by https://github.com/elastic/stream/blob/main/.buildkite/scripts/docker-build.sh is amd64. Many users of this image are running arm64 so they must run it in emulation. We should publish a multi-platform image so that it can run natively on arm64 too.

Add a REST API output

Add an "output" that hosts the log messages via a REST API. The format of the returned document could be configurable via a Go text/template

Things that should be configurable:

• path
• content-type
• paging style (timestamp based or http link headers)
• body format template
• tls settings

Add lumberjack output

Add an output that streams events over the lumberjack protocol. The github.com/elastic/go-lumber library has a client that can be used for streaming events to a server.

Relates: elastic/integrations#3796

Add syslog encoding options

Add the ability to add a syslog header (choice between RFC3164 and RFC5424).

The timestamp should be predictable. Start with 2020-01-01T00:00:00 and increment one second for every message.

'Accept' in `request_headers` doesn't match rule in certain cases when running http-server

Consider the rule:

rules:
  - path: /feed
    methods: ["GET"]
    request_headers:
      Authorization: "Token token=testing"
      Accept: "application/vnd.cif.v3+json"
    query_params:
      itype: "ipv4"

I couldn't get stream's http-server to match against the rule when a request came in that should have matched, e.g.:

`curl -H "Authorization: Token token=testing" -H "Accept: application/vnd.cif.v3+json" "http://localhost:8080/feed?itype=ipv4"

However, the moment I removed the "Accept" line from the request_headers key in the rule, the same request worked.

rules:
  - path: /feed
    methods: ["GET"]
    request_headers:
      Authorization: "Token token=testing"
    query_params:
      itype: "ipv4"

Either stream isn't picking up on it correctly or I've config'd something incorrectly in the rule or my clients. I tried from a few different HTTP clients including curl and the elastic-package system test (which I think uses Golang under the hood). Trying from either client exhibited the same failure.

Not sure if it has something to do with special chars or perhaps I have just missed something. Figured I'd file an issue in the event you're able to reproduce.

Move CI to buildkite

We have to migrate the CI pipeline for the repo to buildkite