ES backend should be able to pull more log messages than ES can provide on a single call about log-anomaly-detector

[zak-hassan]

Not trying to introduce too much changes to architecture but I have another alternative:

@vpavlin What if we skip ES completely and have the log messages stream to kafka then have a spark steaming job that will append the new messages to parquet storage which can be split. We could organize it into buckets and have it partitioned per day (24 hours).

[vpavlin]

Yeah, that is a valid point, we should decide whether to go directly with Kafka or base it on S3..we have issues for both:)

#3
#4

We surely will need S3 integration for you model storage proposal and for #35

All the data from Kafka should already go to S3 in our deployment (IIUC), so what you are saying is probably half way there from the infra point of view.

[goern]

is this still valid?

[sesheta]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[sesheta]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[sesheta]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

[sesheta]

@sesheta: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.