berglh / escroll Goto Github PK
View Code? Open in Web Editor NEWA command client for elasticsearch scroll queries in the spirit of curl. Shell tool for data extraction with URI filtering of fields.
License: MIT License
A command client for elasticsearch scroll queries in the spirit of curl. Shell tool for data extraction with URI filtering of fields.
License: MIT License
Time for completion upon success.
https://github.com/berglh/escroll/blob/master/escroll.go#L51
I am pretty sure this will fail in the case that the Size is a int, needs testing.
Add a progress indicator to show what escroll is doing. Scroll searches take a long time and if you're not receiving any stderr, you are bit unsure what's going on.
It would be cool to inspect the size of the search request, get the total number of matches and then create a visual progress of the number of scrolls completed.
Emulate curl by combining both the server and query flags into a single url flag that contains the resource type, server, path and query URI parameters.
Unexpected results from putting _source field filtering before filter_path. Maybe should reference elasticsearch documentation.
Adding INFO/ERROR logging to stderr to give an indication of what's happening.
Add version information flag to escroll
Add user flags and test the scroll functionality on a shield protected instance of ES.
Some stuff from a regular payload is omitted. At this point I can't see any benefit including data outside of Hits.Hits, but someone might have this requirement in the future.
Need to ensure that hits.total is in the URI query segment:
&filter_path=hits.total,hits.hits._source,_scroll_id
Check that the respJSON.Hits.Total not equal to zero, or throw error:
https://git.bytefuse.net/goes/escroll/src/master/escroll.go#L205
Create some documentation on this project.
Plain http is assumed, not useful for elasticsearch behind an SSL proxy or protected by TLS.
People need to know how to build it, it's pretty simple though.
After a successful scroll search, the final line prints out on the end of the previous line. This is probably because of the function that replaces the previous line to update the scroll stats.
Currently -h is used to specify the target host of escroll, but it's probably not the best flag. Choose a new flag and test the -h for help output.
Add add an option to direct search output to file
The scroll API has changed requiring a JSON payload with the scroll duration to be included in the request for the next scroll batch. The SearchScroll
function needs to be broken with the goal of being agnostic to the original query body and scroll body functions. Some version checking should occur on the target server test to determine whether the old or new scroll API body should be used.
POST /_search/scroll
{
"scroll" : "1m",
"scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ=="
}
ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html
These functions are almost identical, can reduce complexity substantially through refactoring these functions.
Create a test to parse the query file and data string. Should be JSON.
This has been resolve by L154 in commit: ee03e38
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.