Elasticsearch output plugin - Errors on _bulk api post about grr

[micrictor]

Original dev of the Elasticsearch output plugin here.

I looked back at docs for old versions, I don't think there's a regression risk as all versions that still have live docs include that header value.

I submitted my changes via PR, so that seems like a decent route. If you want, I can incorporate your changes and add the requisite tests to ensure that my (broken ):) code would have failed.

[max-vogler]

Thanks for reporting and debugging this issue! Please send a pull request with your fix.

[fredx30]

The following code changes work in my environment. I belive the potential danger with this change is reverse and forward compatability with different versions of the elasticsearch api.

master...fredx30:grr:update-elasticsearch-output-plugin

Currently i am using these changes in a fairly volatile way where i have to reimplement them on container restart. Is there any way of getting these into a mainline release? Am i better off building my own releases to a private container registry of my choosing?

Edit: Is a PR here with these changes welcome?

[fredx30]

@micrictor Im not sure what these tests would entail. I see no apparent variable to test for in this change.

I could see a test for verifying the headers are in place, however if this worked in previous version without the headers then i see little reason for testing for this as the api could just as well change again.

If you have a good idea for a test i would gladly attempt to write it, this whole editing opensource code thing is new and exciting to me i could use the practice getting the PR across the finish line.

[micrictor]

I think you hit the nail on the head - we’d want to have a test for the Content-Type header value being one of the two valid ones for both 7.17 and 8.X, and for the presence of the trailing newline. You’re correct that the API could change again in the future, especially as Elasticsearch major versions change, but I think that having the test is still valuable as it prevents refactors or other changes from accidentally breaking it.

…

On Wed, Jul 13, 2022 at 6:30 AM Fredrik ***@***.***> wrote: @micrictor <https://github.com/micrictor> Im not sure what these tests would entail. I see no apparent variable to test for in this change. I could see a test for verifying the headers are in place, however if this worked in previous version without the headers then i see little reason for testing for this as the api could just as well change again. If you have a good idea for a test i would gladly attempt to write it, this whole editing opensource code thing is new and exciting to me i could use the practice getting the PR across the finish line. — Reply to this email directly, view it on GitHub <#990 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTCAGMMGFH7JXYDFJAWQZDVT3AGBANCNFSM53LU3JDA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[fredx30]

I ran into some issues while learning how to run the tests. After a bit of trying to get pdb to work on the wsl cli I wound up setting up a pycharm project for local debugging in windows. There were a few awkward moments but i think i got the basics working. I see the tests pass in your pipelines, my windows testing needed some patches to work. You can see them here if they intrest you. I think my changes to the elastic output plugin tests should work as its run in the ubuntu pipeline but cant be sure- working on a run of my own now.

Will cherry pick these commits into the pr linked to this issue.

[micrictor]

The changes to the plugin and the test look good so far. I think what's missing is that the test method _ParseEvents tries to parse an empty string - the last line, now that there's a trailing newline.

https://github.com/fredx30/grr/blob/adf66de6daa63855de31aee98c0846ba800de87a/grr/server/grr_response_server/output_plugins/elasticsearch_plugin_test.py#L72

As for running tests - I, personally, would just run the tests for the ES plugin locally, and rely on the CI jobs to run the full test suite.

[fredx30]

Well its passing the pipeline run tests now.
Edit: Dealing with PR comments now.

[micrictor]

@fredx30 do you plan on finalizing your PR, or would you mind if I created one to fix this?

[fredx30]

@micrictor I believe i finished the PR and am waiting on @max-vogler to review the requested changes that have been made.
Based on your answer im suspecting i may have missed something thats a todo on my end. Would you enlighten me?

I have a (i think signed, never tested that part) build going to my public github artifacts that i have been using since i got my the tests and the review finished. It uses a small change to the build pipeline to push to github instead of docker. If am am correct in waiting i hope it can serve as some inspiration for the time being. If not i would love to get cracking on finishing this PR up, such that i too can be on a mainstream release.

[micrictor]

That PR looks good, only thing I can think of is manually marking Max's requested changes as complete may be needed for it to request review again.

Thanks for following up on this, I know it's been a long time.

[fredx30]

What i have done to that end is

1. Request a re-review from max
2. Mark all the requested changes as resolved
3. Quoted/answered the main change request thread

I choose to believe the team is simply buzy or tasked with other things -as there have been no missed releases for this code. I trust max or someone else will get to this eventually. Latest activity is 6 months ago as far as releases are concerned ref: dockerhub.

With that said i dont think it would hurt to keep bumping this issue once every few months to keep it from getting stale on our end.