mbuhot / eskotlin

Elasticsearch Query DSL for Kotlin

ES Kotlin

Elasticsearch Query DSL for Kotlin.

This library aims to minimize the gap between the Elasticsearch JSON query DSL, and the API used when writing kotlin applications. This integrates with the existing java API, only providing a nicer syntax to build the queries.

Getting Started

Gradle

repositories {
    mavenCentral()
    ...
    maven {
        url  "http://dl.bintray.com/mbuhot/maven"
   }
}
dependencies {
    compile 'mbuhot:eskotlin:0.7.0'
    ...
}

See CHANGELOG for older versions supporting previous elasticsearch client versions.

Maven

Full details on bintray

<dependency>
  <groupId>mbuhot</groupId>
  <artifactId>eskotlin</artifactId>
  <version>0.7.0</version>
  <type>pom</type>
</dependency>

Examples

Term Query

JSON:

{
    "term" : { "user" : "Kimchy" }
}

Kotlin:

val query = term {
    "user" to "Kimchy"
}

Bool Query

JSON:

{
    "bool" : {
        "must" : {
            "term" : { "user" : "kimchy" }
        },
        "filter": {
            "term" : { "tag" : "tech" }
        },
        "must_not" : {
            "range" : {
                "age" : { "from" : 10, "to" : 20 }
            }
        },
        "should" : [
            {
                "term" : { "tag" : "wow" }
            },
            {
                "term" : { "tag" : "elasticsearch" }
            }
        ],
        "minimum_should_match" : 1,
        "boost" : 1.0
    }
}

Kotlin:

val query = bool {
    must {
        term { "user" to "kimchy" }
    }
    filter {
        term { "tag" to "tech" }
    }
    must_not {
        range {
            "age" {
                from = 10
                to = 20
            }
        }
    }
    should = listOf(
        term { "tag" to "wow" },
        term { "tag" to "elasticsearch" })
    minimum_should_match = 1
    boost = 1.0f
}

Function Score Query

JSON:

{
    "function_score": {
        "query": {
            "match_all": {}
        },
        "functions": [
            {
                "filter": {
                    "term": {
                        "foo": "bar"
                    }
                },
                "gauss": {
                    "baz": {
                        "scale": 1.0
                    }
                }
            },
            {
                "filter": {
                    "match_all": {}
                },
                "random_score": {
                    "seed": 234
                }
            },
            {
                "exp": {
                    "qux": {
                        "scale": 2.3
                    }
                }
            }
        ],
        "score_mode": "max",
        "boost_mode": "multiply",
        "max_boost": 5.0,
        "boost": 1.2,
        "min_score": 0.001
    }
}

Kotlin:

val query = function_score {
    query = match_all { }
    functions = listOf(
        term { "foo" to "bar" } to gaussDecayFunction("baz", 1.0),
        match_all { } to randomFunction(234L),
        null to exponentialDecayFunction("qux", 2.3))

    boost = 1.2f
    boost_mode = "multiply"
    score_mode = "max"
    max_boost = 5.0f
    min_score = 0.001f
}

See the src/test directory for more examples.

API Coverage

Full Text Queries - Done

• Match
• Multi Match
• Common Terms

Term Queries - Done

• Term
• Terms
• Range
• Exists
• Prefix
• Wildcard
• Regex
• Fuzzy
• Type
• Ids

Compound Queries - Done

• Constant Score
• Bool
• Dis Max
• Function Score
• Boosting

Joining Queries - Done

• Nested
• Has Child
• Has Parent

Geo Queries - Contributions welcome!

• Geo Shape
• Geo Bounding Box
• Geo Distance Range
• Geo Polygon
• Geohash Cell

Specialized Queries - Contributions welcome!

• More Like This
• Template
• Script

Span Queries - Contributions welcome!

• Span Term
• Span Multi Term
• Span First
• Span Near
• Span Or
• Span Not
• Span Containing
• Span Within

Aggregations - Contributions welcome!

• Nested
• Filter
• Metrics Aggregations
• Bucket Aggregations
• Pipeline Aggregations
• Matrix Aggregations

License

MIT - See LICENSE file for full text.

eskotlin's Issues

development status?

hi the last contribution was month ago, i wonder what's the current status of this project.

I'd like to contribute, but not sure what's the plan.

JCenter shutdown impact on Gradle builds [February 2022]

On February 3 2021, JFrog announced that they will be shutting down Bintray and JCenter. This post tells you what you need to know and do to avoid disruptions to your build pipelines.

Use Kotlin range operator

Instead of constructing simple range queries like this:

range {
    "age" {
        from = 10
        to = 20
    }
}

we should leverage the Kotlin range operator (..)
and overload the in-operator operator to construct the same query like this:

"age" in 10..20

We then need to override the following function:

fun ClosedRange<Int>.contains(name: String): RangeQueryBuilder = 
        RangeQueryBuilder(name).from(start).to(endInclusive)

Similar methods could be added for other number types.

Additional consideration has to be done about non-inclusive floating point ranges, to support include_lower/include_upper. Same goes for the other optional parameters of RangeQueryBuilder.

ClassNotFoundException

I'm unable to use this library because of various ClassNotFoundExceptions. An example stack trace:

java.lang.ClassNotFoundException: org.elasticsearch.index.query.Operator
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_91]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_91]
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[na:1.8.0_91]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_91]
	at mbuhot.eskotlin.query.fulltext.MatchKt.match(Match.kt:50) ~[eskotlin-0.3.0.jar:na]
	at com.demo.search.services.QueryBuilder.termMatch(QueryBuilder.kt:18) ~[main/:na]
	at com.demo.search.services.QueryBuilder$build$1.invoke(QueryBuilder.kt:56) ~[main/:na]
	at com.demo.search.services.QueryBuilder$build$1.invoke(QueryBuilder.kt:8) ~[main/:na]
	at mbuhot.eskotlin.query.compound.BoolKt.bool(Bool.kt:37) ~[eskotlin-0.3.0.jar:na]
	at com.demo.search.services.QueryBuilder.build(QueryBuilder.kt:53) ~[main/:na]
	at com.demo.search.services.SearchService.search(SearchService.kt:42) ~[main/:na]
	at com.demo.search.controllers.SearchController.search(SearchController.kt:23) ~[main/:na]

DSL for bulk

I would love to use this for bulk indexing..

Support multiple [must, should, filter]

Hi there,

I just tried building a bool query like so:

fun makeQuery(terms: Map<String, String>) = mbuhot.eskotlin.query.compound.bool {
    terms.forEach { (key, value) ->
        must {
            match {
                key to {
                    query = value
                    fuzziness = Fuzziness.AUTO
                }
            }
        }
    }
}

However, it only renders the last match in the Map, instead of all.

I changed the BoolData.must() to the following to make it work:

fun must(f: () -> QueryBuilder) {
    must = must?.plus(f()) ?: listOf(f())
}

DSL for aggregations

Add support for some aggregations, eg sum/avg.
See #1 for some discussion.

field_value_factor

Elasticsearch allows you to boost/modify the score of documents using function_score and field_value_factor based on a property that exists on the documents. This is commonly used to implement a popularity system (likes, pageviews, etc...) to improve the perceived relevancy of results.

For example:

GET /_search
{
    "query": {
        "function_score": {
            "field_value_factor": {
                "field": "likes",
                "factor": 1.2,
                "modifier": "sqrt",
                "missing": 1
            }
        }
    }
}

Also discussed here: https://nocf-www.elastic.co/guide/en/elasticsearch/guide/master/boosting-by-popularity.html

Question #1: does ESKotlin support field_value_factor? A cursory search appears to show "no".

Question #2: If ESKotlin does not support field_value_factor how difficult would it be to add?

Use camelCase function names

The usage of snake_case function names is unidiomatic in Java/Kotlin.
We might consider to deprecate those functions in this DSL and instead use camelCase-named functions, e.g. matchAll instead of match_all.

yep, I'm interested

I noticed you post on reddit asking if anyone was interested 👍

Your project has a mention on elastic.co as a community Kotlin client. It's the only one on their list for Kotlin, so it's kinda visible, and it's looking really nice. Any plans to get it feature complete?

Cheers and thanks for what you've done so far!

Fix build script

The build script seems to be broken with recent versions of gradle and the bintray plugin.

Find a good reference build script that will build the package and publish to a Bintray maven repository including sources.

The error I get when trying to publish the package is:

Caused by: org.gradle.api.GradleException: Could not upload to 'https://api.bintray.com/content/mbuhot/maven/eskotlin/0.7.1/property(class java/lang/String, map(provider(?)))/property(class java.lang.String, fixed(class java.lang.String, es-kotlin))/property(class java.lang.String, map(provider(?)))/property(class java.lang.String, fixed(class java.lang.String, es-kotlin))-property(class java.lang.String, map(provider(?))).pom': 
HTTP/1.1 400 Bad Request [message:Unable to upload files: Maven group, artifact or version defined in the pom file do not match the file path 'property(class java/lang/String, map(provider(?)))/property(class java.lang.String, fixed(class java.lang.String, es-kotlin))/property(class java.lang.String, map(provider(?)))/property(class java.lang.String, fixed(class java.lang.String, es-kotlin))-property(class java.lang.String, map(provider(?))).pom']

which looks like it is trying to interpolate an object of some kind into a URL path?

Maintainers Wanted!

I haven't been keeping up with the developments in Kotlin or Elasticsearch over the last couple of years, so it's time to hand this project over to someone new.

If you're interested, please leave a comment on this issue.

Consider use invoke operator overloading instead of extension methods named 'to'?

For example, when I'am constructing a range query, I write codes like this:

val query = range {
    "age" to {
        gte = 10
        lte = 20
        boost = 2.0f
    }
}

This 'to' method is declared in RangeBlock class like:

infix fun String.to(init: RangeData.() -> Unit): RangeData {
     return RangeData(name = this).apply(init)
}

But I think it is better to use invoke operator overloading here, just replace the 'to' method with this:

operator inline fun String.invoke(init: RangeData.() -> Unit): RangeData {
    return RangeData(name = this).apply(init)
}

Then we can write a query like this:

val query = range {
    "age" {
        gte = 10
        lte = 20
        boost = 2.0f
    }
}

Now, we don't need to use the ugly key word anymore, the query looks clean.

Please consider this, or may I implement this myself and send a pull request to you?

Prefer extensions on existing types to new types storing only configuration

Currently for constructing queries, you're using classes like RangeData for storing parameters. Then later you apply those parameters to the framework RangeQueryBuilder.

I wonder why you don't configure the framework builder directly:

inline fun range(name: String, block: RangeQueryBuilder.() -> Unit): RangeQueryBuilder =
        RangeQueryBuilder(name).apply(block)

(Or as an extension function inRange on String, similar to #27.)

That way we would leverage existing APIs instead of creating additional overhead.

To make configuring more dogmatic in Kotlin we should would rather use extension vars on the framework types like this:

var RangeQueryBuilder.includeLower: Boolean
    get() = includeLower()
    set(value) {
        includeLower(value)
    }

DSL-like queries would then still be possible:

range("age") {
    from = 10
    to = 20
    includeLower = false
}

(Or "age" inRange { ... } as an extension function.)

Upgrade Kotlin, Gradle, Elastic

All dependencies should be updated.

Gradle -> 4.5.1
Kotlin -> 1.2.21
Elastic can be 5.6.5 for now and once it has been tested with 6.x, upped accordingly.