Tree Search Language (TSL) is a wonderful human readable filtering language.
The TSL language grammar is human readable and similar to SQL syntax.
- What I can do with it ?
- What does it do ?
- Cool logo
- Install
- Syntax examples
- Types
- Code examples
- CLI tools
- Grammar
You can use the TSL package to add uniform and powerful filtering to your RESTful API or GraphQL services, implement brewing-recipe searches on your smart tea brewer, or even make your own memory based "SQL like" server as we do in our tsl_mem CLI example.
kubesql uses TSL to select Kubernetes resources based on the value of one or more resource fields.
Here is our tsl_memCLI tool (code), it's an in-memory search engine, it is using the TSL package to filter through an in-memory array of books using "SQL like" tsl phrases:
$ ./tsl_mem -i "spec.rating is not null and author ~= 'Joe'" -o prettyjson[
{
"author": "Joe",
"spec.pages": 100,
"spec.rating": 4,
"title": "Book"
},
{
"author": "Joe",
"spec.pages": 150,
"spec.rating": 4,
"title": "Good Book"
},
{
"author": "Joe",
"spec.pages": 15,
"spec.rating": 5,
"title": "My Big Book"
}
] $ ./tsl_mem -i "spec.rating is null and spec.pages < 250" -o prettyjson[
{
"author": "Jane",
"spec.pages": 50,
"title": "Some Other Book"
}
]
The TSL package parses tsl phrases into tsl trees, it also include extra walkers that iterate (walk) over the tsl tree to perform exhilarating tasks, for example, convert a tsl tree into an SQL expression, create in-memory search engines, BSON object exporters and even more exciting stuff.
For example, this tsl phrase:
name like '%joe%' and (city = 'paris' or city = 'milan')Will be parsed into this tsl tree:
Awesome logo image by gophers....
$ go version
go version go1.11.2 linux/amd64Clone the TSL git repository, and run make:
git clone [email protected]:yaacov/tree-search-language.git
cd tree-search-language
makeOther make options include make lint for linting check and make test for tests.
Running make lint requires golangci-lint:
go get -u github.com/golangci/golangci-lint/cmd/golangci-lintRunning make test requires ginkgo:
go get -u github.com/onsi/ginkgo/ginkgo# Install the base package
go get "github.com/yaacov/tree-search-language/v5/pkg/tsl"
# Install all walkers
go get "github.com/yaacov/tree-search-language/v5/pkg/walkers/..."
# Or pick the walker needed
go get "github.com/yaacov/tree-search-language/v5/pkg/walkers/sql"
go get "github.com/yaacov/tree-search-language/v5/pkg/walkers/mongo"
go get "github.com/yaacov/tree-search-language/v5/pkg/walkers/ident"
go get "github.com/yaacov/tree-search-language/v5/pkg/walkers/graphviz"See CLI tools usage here.
go get -v "github.com/yaacov/tree-search-language/v5/cmd/tsl_parser"
go get -v "github.com/yaacov/tree-search-language/v5/cmd/tsl_mongo"
go get -v "github.com/yaacov/tree-search-language/v5/cmd/tsl_sqlite"
go get -v "github.com/yaacov/tree-search-language/v5/cmd/tsl_gorm"
go get -v "github.com/yaacov/tree-search-language/v5/cmd/tsl_graphql"This TSL phrase:
name like '%joe%' and (city = 'paris' or city = 'milan')Will be parsed into this TSL tree:
This TSL phrase:
name in ('joe', 'jane') and grade not between 0 and 50Will be parsed into this TSL tree:
This TSL phrase:
memory.total - memory.cache > 2000 and cpu.usage > 50Will be parsed into this TSL tree:
This TSL phrase:
(net.rx + net.tx) / 1000 > 3 or net.rx / 1000 > 6Will be parsed into this TSL tree:
This TSL phrase:
(net.rx + net.tx) < 1KiWill be parsed into this TSL tree:
This TSL phrase:
([net.rx] + `net.tx`) < 1024Will be parsed into this TSL tree:
Images created using the tsl_parser CLI example and Graphviz's dot utility:
$ ./tsl_parser -i "name like '%joe%' and (city = 'paris' or city = 'milan')" -o dot > file.dot
dot file.dot -Tpng > image.pngsupported = trueWill be parsed into this TSL tree:
date = 2020-01-01T20:00:00ZWill be parsed into this TSL tree:
For complete working code examples, see the CLI tools directory ( see more on TSL's CLI tools usage here ).
The tsl package include the ParseTSL code, doc method for parsing TSL into a search tree:
tree, err := tsl.ParseTSL("name in ('joe', 'jane') and grade not between 0 and 50")After parsing the TSL tree will look like this (image created using the tsl_parser cli utility using .dot output option):
The walkers sql package include a helper sql.Walk (code, doc) method that adds search to squirrel's SelectBuilder object:
import (
...
sq "github.com/Masterminds/squirrel"
"github.com/yaacov/tree-search-language/v5/pkg/walkers/sql"
...
)
// Parse a TSL phrase into a TSL tree.
tree, err := tsl.ParseTSL("name in ('joe', 'jane') and grade not between 0 and 50")
// Prepare squirrel filter.
filter, err := sql.Walk(tree)
// Create an SQL query.
sql, args, err := sq.Select("name", "city", "state").
From("users").
Where(filter).
ToSql()After SQL generation the sql and args vars will be:
SELECT name, city, state FROM users WHERE (name IN (?,?) AND grade NOT BETWEEN ? AND ?)["joe", "jane", 0, 50]
The walkers mongo package include a helper mongo.Walk (code, doc) method that adds search bson filter to mongo-go-driver:
import (
...
"github.com/yaacov/tree-search-language/v5/pkg/walkers/mongo"
...
)
// Parse a TSL phrase into a TSL tree.
tree, err := tsl.ParseTSL("name in ('joe', 'jane') and grade not between 0 and 50")
// Prepare a MongoDB BSON document as a filter.
filter, err = mongo.Walk(tree)
// Run query.
cur, err := collection.Find(ctx, filter)The walkers graphviz package include a helper graphviz.Walk (code, doc) method that exports .dot file nodes :
import (
...
"github.com/yaacov/tree-search-language/v5/pkg/walkers/graphviz"
...
)
// Parse a TSL phrase into a TSL tree.
tree, err := tsl.ParseTSL("name in ('joe', 'jane') and grade not between 0 and 50")
// Prepare .dot file nodes as a string.
s, err = graphviz.Walk("", tree, "")
// Wrap the nodes in a digraph wrapper.
s = fmt.Sprintf("digraph {\n%s\n}\n", s)The walkers ident package include a helper ident.Walk (code, doc) method that checks and mapps identifier names:
import (
...
"github.com/yaacov/tree-search-language/v5/pkg/walkers/ident"
...
)
...
// columnNamesMap mapps between user namespace and the SQL column names.
var columnNamesMap = map[string]string{
"title": "title",
"author": "author",
"spec.pages": "pages",
"spec.rating": "rating",
}
// checkColumnName checks if a column name is valid in user space replace it
// with the mapped column name and returns and error if not a valid name.
func checkColumnName(s string) (string, error) {
// Check for column name in map.
if v, ok := columnNamesMap[s]; ok {
return v, nil
}
// If not found return string as is, and an error.
return s, fmt.Errorf("column \"%s\" not found", s)
}
...
// Parse a TSL phrase into a TSL tree.
tree, err := tsl.ParseTSL("name in ('joe', 'jane') and grade not between 0 and 50")
// Check and replace user identifiers with the SQL table column names.
tree, err = ident.Walk(tree, checkColumnName)
...The walkers semantics package include a helper semantics.Walk (code, doc) method that helps filter a list of objects using a tsl tree, and a type semantics.EvalFunc (code, doc) that return a record's value for a record key:
import (
...
"github.com/yaacov/tree-search-language/v5/pkg/walkers/semantics"
...
)
...
// evalFactory creates an evaluation function for a data record.
//
// Returns:
// A function that gets a `key` for a record and returns the value.
// If no value can be found for this `key` in our record, it will return
// ok = false, if value is found it will return ok = true.
func evalFactory(r map[string]string) semantics.EvalFunc {
return func(k string) (interface{}, bool) {
v, ok := r[k]
return v, ok
}
}
// Check if a record complie with our tsl tree.
//
// For example:
// if our tsl tree represents the tsl phrase "author = 'Joe'"
// we will get the boolean value `true` for our record.
//
// if our tsl tree represents the tsl phrase "spec.pages > 50"
// we will get the boolean value `false` for our record.
record := map[string]string {
"title": "A good book",
"author": "Joe",
"spec.pages": 14,
"spec.rating": 5,
}
eval := evalFactory(record)
compliance, err = semantics.Walk(tree, eval)The example CLI tools showcase the TSL language and tsl golang package, see the cmd directory for code.
tsl_parser is a basic example, showing how to parse a tsl phrase into a tsl tree.
$ ./tsl_parser -h
Usage of ./tsl_parser:
-i string
the tsl string to parse (e.g. "animal = 'kitty'")
-o string
output format [json/yaml/prettyjson/sql/dot] (default "json")$ ./tsl_parser -i "(name = 'joe' or name = 'jane') and city = 'rome'" -o sqlsql: SELECT * FROM table_name WHERE ((name = ? OR name = ?) AND city = ?)
args: [joe jane rome]
$ ./tsl_parser -i "(name = 'joe' or name = 'jane') and city = 'rome'" -o prettyjson{
"func": "$and",
"left": {
"func": "$or",
"left": {
"func": "$eq",
"left": {
"func": "$ident",
"left": "name"
},
"right": {
"func": "$string",
"left": "joe"
}
},
"right": {
"func": "$eq",
"left": {
"func": "$ident",
"left": "name"
},
"right": {
"func": "$string",
"left": "jane"
}
}
},
"right": {
"func": "$eq",
"left": {
"func": "$ident",
"left": "city"
},
"right": {
"func": "$string",
"left": "rome"
}
}
}$ ./tsl_parser -i "city = 'rome'" -o dotdigraph {
root [shape=box color=black label="$eq"]
XVlB [shape=record color=red label="$ident | 'city'" ]
zgba [shape=record color=blue label="$string | 'rome'" ]
root -> { XVlB, zgba }
}tsl_mongo is an example showing tsl use with a mongodb.
$ ./tsl_mongo -h
Usage of ./tsl_mongo:
-c string
collection name to query on (default "books")
-d string
db name to connect to (default "tsl")
-i string
the tsl string to parse (e.g. "author = 'Jane'") (default "title is not null")
-p prepare a book collection for queries
...
-u string
url for mongo server (default "mongodb://localhost:27017")$ ./tsl_mongo -p -i "title is not null" | jq{
"title": "Book",
"author": "Joe",
"spec": {
"pages": 100,
"rating": 4
}
}$ ./tsl_mongo -i "title ~= 'Other' and spec.rating > 1" | jq{
"title": "Other Book",
"author": "Jane",
"spec": {
"pages": 200,
"rating": 3
}
}tsl_sqlite is an example showing tsl use with sqlite.
$ ./tsl_sqlite -h
Usage of ./tsl_sqlite:
-f string
the sqlite database file name (default "./sqlite.db")
-i string
the tsl string to parse (e.g. "Title = 'Book'")
-p prepare a book collection for queries$ SQL="title like '%Book%' and spec.pages > 100"
$ ./tsl_sqlite -i "$SQL" -p | jq{
"title": "Other Book",
"author": "Jane",
"spec": {
"pages": 200,
"rating": 3
}
}
{
"title": "Good Book",
"author": "Joe",
"spec": {
"pages": 150,
"rating": 4
}
}tsl_gorm is an example showing tsl use the gorm package.
$ ./tsl_gorm -h
Usage of ./tsl_gorm:
-f string
the sqlite database file name (default "./sqlite.db")
-i string
the tsl string to parse (e.g. "title = 'Book'") (default "title is not null")
-p prepare a book collection for queries$ SQL="title like '%Book%' and spec.pages > 100"
$ ./tsl_gorm -i "$SQL" -p | jq{
"title": "Other Book",
"author": "Jane",
"spec": {
"pages": 200,
"rating": 3
}
}
{
"title": "Good Book",
"author": "Joe",
"spec": {
"pages": 150,
"rating": 4
}
}tsl_mem is an advanced example showing a custom walker, implementing in-memory sql server.
$ ./tsl_mem -i "spec.rating > 4 and title ~= 'Big'" -o yaml- author: Joe
spec.pages: 15
spec.rating: 5
title: My Big Booktsl_graphql is an example showing a graphql serve using tsl.
$ ./tsl_graphql -h
Usage of ./tsl_graphql:
-f string
the sqlite database file name (default "./sqlite.db")
-p prepare a book collection for queries$ ./tsl_graphql -p
TSL GraphQL server listen on port: 8080
Query example:
curl -sG "http://localhost:8080/graphql" --data-urlencode \
"query={books(filter:\"title like '%Other%' and spec.pages>100\"){title,author,spec{pages}}}"$ curl -sG "http://localhost:8080/graphql" --data-urlencode \
"query={books(filter:\"title like '%Other%' and spec.pages>100\"){title,author,spec{pages}}}" | jq
{
"data": {
"books": [
{
"author": "Jane",
"spec": {
"pages": 200
},
"title": "Other Book"
},
{
"author": "Jane",
"spec": {
"pages": 250
},
"title": "Other Great Book"
}
]
}
}TSL parser is generated using Antlr4 tool, the antlr4 grammar file is TSL.g4.
and or not is null like between in
= <= >= != ~= ~! <> + - * / %
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent