Comments (6)
Yup, that's correct.
Nice, thanks for trying it out! We'll post here when full schema support is open-sourced.
from clp.
Hey @kavirajk,
Right, thereโs currently no easy way to specify them. Full schema support is in our open-sourcing pipeline. We are finishing up optimization & testing. Stay tuned!
from clp.
Thanks @kirkrodrigues for the information!.
So basically, now clp-core uses kinda static delimiter and variables check as done here and here respectively. Correct?
I also tested with your hadoop dataset. The compression ratio is very impressive!. Great work!. Looking forward for the full schema support!
from clp.
Hi @kirkrodrigues,
As mentioned in the paper, variable schema is pre-defined by the user. Are the regex as shown below currently manually written by the developer? I can see no part of the code checking for these schema. I understand it is not open-source yet, but can you please point out how is it done in currently?
dictionary_variables :
" task_ \ d + " # Task ID
" \ d {1 ,3}\.\ d {1 ,3}\.\ d {1 ,3}\.\ d {1 ,3} " # IP
" container_ \ d + " # Container ID
from clp.
Currently, we have a few schemas implemented implicitly (i.e., the logic is not implemented as a regular expression but as a set of conditions) in the code. Roughly, the logic works as follows:
EncodedVariableInterpreter::encode_and_add_to_dictionary
repeatedly callsLogTypeDictionaryEntry::parse_next_var
to parse every variable in a message.LogTypeDictionaryEntry::parse_next_var
callsget_bounds_of_next_var
to parse the next variable in the message.get_bounds_of_next_var
iterates the message and gets the bounds of the next token in the message (a token starts with a non-delimiter (!is_delim
) and ends before the next delimiter (is_delim
).get_bounds_of_next_var
then runs a few checks to determine if the token should be treated as a variable according to the implicit schemas.- Once the control-flow returns to
EncodedVariableInterpreter::encode_and_add_to_dictionary
, it will try to convert the given variable into a non-dictionary variable (e.g.,convert_string_to_representable_integer_var
). If it cannot be converted, it will be treated as a dictionary variable.
Encoding a query is a little bit more complicated since we need to handle wildcards (admittedly, the logic is a bit messy). To see how it works, I would start from Grep::process_raw_query
.
We are working hard to try and get an easy-to-use version of schema support open-sourced so that the logic won't be so complicated. I hoped we would be done by now, but development always takes longer than we expect.
from clp.
Thank you for your response @kirkrodrigues. Good to hear that development is in progress.
from clp.
Related Issues (20)
- clp-s: Implicit wildcard behaviour is unintuitive
- clp-s: Use standard syntax for conditions on values between ()
- clp-s: Support `?` to match single character in searches against variable value columns HOT 2
- CLP-package: search scheduler generating weird error logs
- clp-s: Incorrect search results with a wildcard character in the middle of a query
- clp-s: Incorrect search results with a "?" at the end of a query
- clp-s: Inifinite loop for queries with consecutive backslash and wildcard characters
- clp-s: Incorrect decompression results with array fields
- sbin/compress.sh keeps running without results HOT 4
- `clp::string_utils::wildcard_match_unsafe_case_sensitive` fails to match with certain queries
- CLP query parsing mistakenly treats static text with wildcards as int and float
- clp-s: Reported uncompressed size can be incorrect after archive-splitting
- start_clp.py start_webui() does not add a docker port binding HOT 3
- CLP WebUI bundle contains unnecessary development dependencies
- CLP package can't be started by root user
- On improving the efficiency of CLP compression and decompression HOT 2
- Building CLP package fails on clean repository setups
- Unhandled exceptions for some kinds of invalid JSON
- `network_reader_illegal_offset` unit test failure on macOS HOT 1
- cent OS image fails to build
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clp.