elastic / elasticsearch-java

Official Elasticsearch Java Client

Elasticsearch Java Client

The official Java client for Elasticsearch.

The Java client for Elasticsearch provides strongly typed requests and responses for all Elasticsearch APIs. It delegates protocol handling to an http client such as the Elasticsearch Low Level REST client that takes care of all transport-level concerns (http connection establishment and pooling, retries, etc).

The docs/design folder contains records of the major decisions in the design of the API. Most notably:

• Object construction is based on the builder pattern.
• Nested objects can be constructed with builder lambdas, allowing for clean and expressive DSL-like code.
• Optional values are represented as null with @Nullable annotations instead of the newer Optional, the Java ecosystem being still very null-based.

Installation

Download the latest version of Elasticsearch or sign-up for a free trial of Elastic Cloud.

Refer to the Installation section of the getting started documentation.

Connecting

Refer to the Connecting section of the getting started documentation.

Compatibility

The Elasticsearch client is compatible with currently maintained Java versions.

The Java client is forward compatible; meaning that the client supports communicating with greater or equal minor versions of Elasticsearch without breaking. It does not mean that the client automatically supports new features of newer Elasticsearch versions; it is only possible after a release of a new client version. For example, a 8.12 client version won't automatically support the new features of the 8.13 version of Elasticsearch, the 8.13 client version is required for that. Elasticsearch language clients are only backwards compatible with default distributions and without guarantees made.

Elasticsearch Version	Elasticsearch-Java Branch	Supported
main	main
8.x	8.x	8.x
7.x	7.x	7.17

Usage

• Creating an index
• Indexing a document
• Getting documents
• Searching documents
• Updating documents
• Deleting documents
• Deleting an index

Documentation

Please refer to the full documentation on elastic.co for comprehensive information.

Contributing

See CONTRIBUTING.md

Licence

This software is licensed under the Apache License 2.0.

elasticsearch-java's Issues

cannot read dependency from GitHub packages

Thanks for providing this, I just wanted to do a first test. Maybe I am missing something, but I don't get the dependency, I am getting an error about not being authorized:

$ ./gradlew classes
> Task :compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not resolve co.elastic.clients:elasticsearch-java:8.0.0-SNAPSHOT.
     Required by:
         project :
      > Could not resolve co.elastic.clients:elasticsearch-java:8.0.0-SNAPSHOT.
         > Unable to load Maven meta-data from https://maven.pkg.github.com/elastic/elasticsearch-java/co/elastic/clients/elasticsearch-java/8.0.0-SNAPSHOT/maven-metadata.xml.
            > Could not get resource 'https://maven.pkg.github.com/elastic/elasticsearch-java/co/elastic/clients/elasticsearch-java/8.0.0-SNAPSHOT/maven-metadata.xml'.
               > Could not GET 'https://maven.pkg.github.com/elastic/elasticsearch-java/co/elastic/clients/elasticsearch-java/8.0.0-SNAPSHOT/maven-metadata.xml'. Received status code 401 from server: Unauthorized

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 2s
1 actionable task: 1 executed

$  https maven.pkg.github.com/elastic/elasticsearch-java/co/elastic/clients/elasticsearch-java/8.0.0-SNAPSHOT/maven-metadata.xml                                                                             1 ↵
HTTP/1.1 401 Unauthorized
Access-Control-Allow-Origin: *
Content-Length: 0
Content-Security-Policy: default-src 'none';
Date: Thu, 03 Jun 2021 15:52:04 GMT
Server: GitHub Registry
Strict-Transport-Security: max-age=31536000;
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-GitHub-Request-Id: FC2A:7EF3:13680D:168AD5:60B8FAA4
X-XSS-Protection: 1; mode=block
access-control-allow-methods: GET, HEAD, OPTIONS
www-authenticate: Basic realm="GitHub Package Registry"

My setup for gradle is as in the documentation, when I try to open the link to the metada file, my browser wants me to login as well.

index request fails when using refresh=wait_for

final IndexResponse ir = client.index(builder -> builder.index("test").id("1") .value(new JsonMap("name", "alex1")) .refresh(Json.createValue("wait_for")));

fails with
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Unknown value for refresh: [\"wait_for\"]."}],"type":"illegal_argument_exception","reason":"Unknown value for refresh: [\"wait_for\"]."},"status":400}
I am not sure I am passing refresh parameter correctly but I could not find any examples

on another note. is this client ready for production use or it is in experimental state?
I really would like to get off low level REST client but not sure what this client's status is
would appreciate any info regarding this project status

[es/snapshot.create] Missing [X-Elastic-Product] header

Elasticsearch version (bin/elasticsearch --version): 7.16.2

Plugins installed: []

JVM version (java -version):

openjdk version "11.0.13" 2021-10-19
OpenJDK Runtime Environment (build 11.0.13+8-Ubuntu-0ubuntu1.21.10)
OpenJDK 64-Bit Server VM (build 11.0.13+8-Ubuntu-0ubuntu1.21.10, mixed mode, sharing)

OS version (uname -a if on a Unix-like system):

Linux pop-os 5.15.8-76051508-generic #202112141040163950527821.10~0ede46a SMP Tue Dec 14 22:38:29 U x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

When performing a snapshot using the new Java client, it fails due to missing X-Elastic-Product header sent by Elasticsearch.

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including (e.g.) index creation, mappings, settings, query etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

1. Create an FS repository,
2. Create a snapshot using the new java client:

final CreateSnapshotRequest request = new CreateSnapshotRequest
  .Builder()
  .repository(repository)
  .snapshot(name)
  .waitForCompletion(waitForCompletion)
  .build();
client.create(request);

Client being used (maven dep):

<elasticsearch.version>7.16.2</elasticsearch.version>

...

      <dependency>
        <groupId>co.elastic.clients</groupId>
        <artifactId>elasticsearch-java</artifactId>
        <version>${elasticsearch.version}</version>
      </dependency>

3. The code throws an exception.

Provide logs (if relevant):

co.elastic.clients.transport.TransportException: [es/snapshot.create] Missing [X-Elastic-Product] header. Please check that you are connecting to an Elasticsearch instance, and that any networking filters are preserving that header.

	at co.elastic.clients.transport.rest_client.RestClientTransport.checkProductHeader(RestClientTransport.java:340)
	at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:250)
	at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:144)
	at co.elastic.clients.elasticsearch.snapshot.ElasticsearchSnapshotClient.create(ElasticsearchSnapshotClient.java:142)

Duplicate `jakarta.json` classes on runtime classpath

When running some tests using this client we run into "jar hell". That is, when bringing in the elasticsearch-java dependency it's runtime classpath includes duplicate classes from the jakarta.json package. This seems to be due to us pulling in both jakarta.json-api and the glassfish imlementation jakarta.json dependencies.

There's a few things going on here:

1. The glassfish implementation includes the api classes, so that's what's causing the jar hell.
2. Since the dependency is marked implementation it's only added to the runtime classpath for consumers. Since the client exposes types from json-api through it's own API (for example) this means users will run into compile errors if trying to use those types. They can "fix" this by manually adding json-api as a compile dependency.

Two options to fix here:

1. Simple remove the superfluous json-api dependency and make the glassfish impl api instead of implementation. This means we also leak all the org.glassfish.json implementation classes onto the compile classpath of any consumers.
2. Make json-api a compileOnly dependency and the glassfish impl runtimeOnly. This should result in only the API classes being on the compile classpath, and both API and implementation classes available at runtime.

SourceConfig.Builder fluent API is broken

In version 7.16.2, unlike most of builders, the fluent API does not work for SourceConfig.Builder.

This code does not compile:

SourceConfig.Builder builder = new SourceConfig.Builder().filter(b -> b);

Error is "Type mismatch: cannot convert from ObjectBuilder to SourceConfig.Builder"

This code compiles:

SourceConfig.Builder builder = new SourceConfig.Builder();
builder.filter(b -> b);

Migration questions

Hi,

I've tried to replace deprecated high level elasticsearch client with this client and have couple questions.

1. Is this client stable? README says it's status is work in progress. Is it ready to be used on production?
2. Is there a replacement for BulkProcessor? It had many options I can't find where to set in this client like flush interval or backoff policy.
3. RestClient has runtime scope but it seems to be impossible to create ElasticsearchClient without it? Wouldn't it be better to make elasticsearch-rest-client a compile dependency?

Adding index block uses Enum name rather than value

If I'm understanding the syntax correctly, this should enable the write block on an index -

client.indices().addBlock(b -> b.index(sourceIndexName).block(IndicesBlockOptions.Write));

It results in a PUT operation with the enum name rather than the value.
From Apache http wire tracing -

http-outgoing-7 >> "PUT /tmp_public_id_and_name/_block/Write HTTP/1.1[\r][\n]"

This causes an exception -

"type":"illegal_argument_exception", "reason":"No block found with name Write"

It works like this, so maybe this is the correct way?

client.indices().putSettings(b -> b.index(sourceIndexName).settings(s -> s.blocks(w -> w.write(true))));

NullPointerException in LazyDeserializer due to error in double-checked locking code

The unwrap() method may return a null deserialiser if two threads access the method simultaneously.

elasticsearch-java/java-client/src/main/java/co/elastic/clients/json/LazyDeserializer.java

Lines 41 to 53 in 3e6f05c

	protected JsonpDeserializer<T> unwrap() {
	// See SEI CERT LCK10-J https://wiki.sei.cmu.edu/confluence/x/6zdGBQ
	JsonpDeserializer<T> d = deserializer;
	if (d == null) {
	synchronized (this) {
	if (deserializer == null) {
	d = ctor.get();
	deserializer = d;
	}
	}
	}
	return d;
	}

Stacktrace:

Caused by: java.lang.NullPointerException: null
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43) ~[elasticsearch-java-7.16.0.jar:na]
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72) ~[elasticsearch-java-7.16.0.jar:na]
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176) ~[elasticsearch-java-7.16.0.jar:na]
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137) ~[elasticsearch-java-7.16.0.jar:na]
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75) ~[elasticsearch-java-7.16.0.jar:na]
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79) ~[elasticsearch-java-7.16.0.jar:na]
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43) ~[elasticsearch-java-7.16.0.jar:na]
	at co.elastic.clients.transport.endpoints.EndpointWithResponseMapperAttr$1.deserialize(EndpointWithResponseMapperAttr.java:56) ~[elasticsearch-java-7.16.0.jar:na]
	at co.elastic.clients.transport.rest_client.RestClientTransport.decodeResponse(RestClientTransport.java:325) ~[elasticsearch-java-7.16.0.jar:na]
	at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:291) ~[elasticsearch-java-7.16.0.jar:na]
	at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:144) ~[elasticsearch-java-7.16.0.jar:na]
	at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1487) ~[elasticsearch-java-7.16.0.jar:na]

Debugging screenshot where you can see deserializer is set, but d is null:

Looking at the article linked to in the source code, it looks like the "Compliant Solution (Immutable)" is desired. In this case, there is one assignment (d = deserializer) missing just before the null check inside the synchonized block.

So the code below should work better, making sure d is getting assigned with deserializer, for the cases where it has been assigned by another thread after the current thread has entered the synchronized block:

    protected JsonpDeserializer<T> unwrap() {
        // See SEI CERT LCK10-J https://wiki.sei.cmu.edu/confluence/x/6zdGBQ
        JsonpDeserializer<T> d = deserializer;
        if (d == null) {
            synchronized (this) {
                d = deserializer;
                if (d == null) {
                    d = ctor.get();
                    deserializer = d;
                }
            }
        }
        return d;
    }

Version: 7.16.0

Using randomized testing in project

Elasticsearch is a big user of randomized testing. The library however features a lot more, like thread leak detection in test cases. So a test suite fails, if a thread is kept running - for more info check the annotation party in ESTestCase.

If you stick with junit4, this might make sense, if you upgrade to junit jupiter, I don't think there is a port just yet due to changes in the test infrastructure.

CombinedFieldQuery mimimumShouldMatch field misspelling causes error

Elasticsearch version (7.16.2):

Plugins installed: []

JVM version (16.0.2):

OS version (MacOS Montery 12.1):

Description of the problem including expected versus actual behavior:

The elastic java client CombinedFieldsQuery has field mimimumShouldMatch instead of minimumShouldMatch

Steps to reproduce:

Use the elastic java client co.elastic.clients.elasticsearch._types.query_dsl.CombinedFieldsQuery to build a query with a combined fields query that has any minimumShouldMatch
Confirm that CombinedFieldsQuery has mimimumShouldMatch and an error is returned when searching any index
Provide logs (if relevant):

{"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:1366] [combined_fields] unknown field [mimimum_should_match] did you mean [minimum_should_match]?"}],"type":"x_content_parse_exception","reason":"[1:1389] [bool] failed to parse field [must]","caused_by":{"type":"x_content_parse_exception","reason":"[1:1389] [bool] failed to parse field [should]","caused_by":{"type":"x_content_parse_exception","reason":"[1:1366] [combined_fields] unknown field [mimimum_should_match] did you mean [minimum_should_match]?"}}},"status":400}

https://github.com/elastic/elasticsearch-java/blob/main/java-client/src/main/java/co/elastic/clients/elasticsearch/_types/query_dsl/CombinedFieldsQuery.java

Could you supply a detail document about it like java high level client?

I think the api document of java high level client is easy to understand. But the java api client is simple and missing a lot of detail about how to use it to communicate with the elastic. Thanks.

SearchResponse.documents() return an empty list

• ES version: 7.16.2
• elasticsearch-java version: 7.16.2

Trying to use client.search().documents(), but I got an empty list:

List<Major> majorList = this.elasticsearchClient.search(s ->
                                                    s.size(RECOMMEND_SIZE)
                                                        .index("major")
                                                        .query(q ->
                                                            q.bool(bl ->
                                                                bl.must(mst ->
                                                                    mst.match(match ->
                                                                        match.field("name")
                                                                                .query(qr -> qr.stringValue(name))
                                                                                .fuzziness("AUTO"))))),
                                                Major.class).documents();

The majorList is empty. But when I use the client.search().hits().hits() the result contains what I need.

Am I using it the wrong way？

Map Builders should preserve order

Object Builders for maps use an InternalMap which is a sub-class of a hash map:

elasticsearch-java/java-client/src/main/java/co/elastic/clients/util/ObjectBuilderBase.java

Line 91 in 75feae5

private static final class InternalMap<K, V> extends HashMap<K, V> {

Hash maps are unordered, the order is defined by the internal hashing function and can be different for different runtime environments. Although the JSON standard does not define dictionaries to be ordered, it would be good to switch to a linked hash map with insertion order:

• for a user the order might have a meaning and it is easier for them if the order is preserved the way the user specified it, e.g. the output might be a ML job or transform configuration
• in some cases order can be important at runtime: elastic/elasticsearch#82350

Because this does not seems performance-critical, I suggest to switch to a linked hash map with insertion order

Filter aggregation is missing sub aggregations in result

It seems the client is not parsing/returning sub aggregations that appear under filter aggregation. Here is a sample query and response.

Request:

{ "_source": { "excludes": [ "attachmentContents.*" ], "includes": [ "*" ] }, "aggregations": { "count": { "value_count": { "field": "messageStorageId.messageId" } }, "login_filter": { "aggregations": { "date": { "terms": { "field": "year", "order": [ { "_key": "desc" } ] } }, "to_domain": { "terms": { "exclude": [ "unknown" ], "field": "to.domain" } }, "file_ext": { "terms": { "exclude": [ "" ], "field": "attachments.fileExt" } }, "has_att": { "terms": { "field": "hasAttachments" } }, "from": { "terms": { "exclude": [ "unknown" ], "field": "sender.name" } }, "from_domain": { "terms": { "exclude": [ "unknown" ], "field": "sender.domain" } }, "to": { "terms": { "exclude": [ "unknown" ], "field": "to.name" } } }, "filter": { "bool": { "must": [ { "bool": { "minimum_should_match": "1" } } ] } } } }, "from": 0, "highlight": { "fields": { "body": { "fragment_size": 0, "number_of_fragments": 0 }, "attachments.fileName": { "fragment_size": 0, "number_of_fragments": 0 }, "attachmentContents.*": { "fragment_size": 200, "number_of_fragments": 2 }, "subject": { "fragment_size": 0, "number_of_fragments": 0 } }, "post_tags": [ "</span>" ], "pre_tags": [ "<span class=\"blue-hilite\">" ] }, "query": { "bool": { "must": [ { "bool": { "must": [ { "query_string": { "default_operator": "and", "query": "Tsang" } } ] } }, { "bool": { "minimum_should_match": "1", "should": [ { "match_all": {} } ] } } ] } }, "size": 10, "sort": [ { "messageDate": { "order": "asc" } } ], "stored_fields": [ "_source" ] }

Response:
{ "took": 190, "timed_out": false, "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 }, "hits": { "total": { "value": 1, "relation": "eq" }, "max_score": null, "hits": [] }, "aggregations": { "count": { "value": 1 }, "login_filter": { "meta": { }, "doc_count": 1, "to_domain": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "cnn.com", "doc_count": 1 } ] }, "date": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "2008", "doc_count": 1 } ] }, "has_att": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": 1, "key_as_string": "true", "doc_count": 1 } ] }, "file_ext": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "doc", "doc_count": 1 } ] }, "from_domain": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "gmail.com", "doc_count": 1 } ] }, "from": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "John Tsang", "doc_count": 1 } ] }, "to": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "John Smith", "doc_count": 1 } ] } } } }

Specifically, sub aggregations above under "login_filter" section are not visible in the java client:

Better way to construct time/duration values?

I am trying to create a point-in-time (code is in Kotlin)

client.openPointInTime { builder ->
      builder.index(index)
          .keepAlive { timeBuilder -> timeBuilder.time("5m") }
  }

As can be seen, I have to specify the time and the unit as part of a string. This is neither very elegant nor type safe. Is this currently the only way to set these time values or am I missing something?

The old high-level client accepted a TimeValue, which was both a long and an associated TimeUnit. However, while the new client seems to also have TimeUnit class, it seems to be entirely disconnected from the Time class.
The only usage of TimeUnit I can find is in the DiskUsageRequest, but it seems to be used incorrectly there (TimeUnit does not contain a duration, but it tries to parse the timeout duration as a TimeUnit).

I also found a TimeBuilders class in the same package, but it is empty.

UpdateOperation is not able to use docAsUpsert or script options yet

I'm trying to build a Bulk request with some upsert operations, but it seems not supported in this version's Java Client yet.
I've seen the docAsUpsert and Script in the UpdateRequest. Will these options be supported at UpdateOperation in the next version?

Function Score weight returns the wrong reference

by the documentation a function score can contain only weight and no other inner functions
in the java api in order to build the FunctionScore we must "hold" a "ContainerBuilder" object, that is only provided as
a response to exp, gauss etc, the weight method (FunctionScore.java:306) returns builder instead of returning ContainerBuilder making this use case impossible to invoke within the java Api

Async client methods should not declare exceptions

For my example, let's take a look at ElasticsearchAsyncClient#index:

	public <TDocument> CompletableFuture<IndexResponse> index(IndexRequest<TDocument> request)
			throws IOException, ElasticsearchException {
		@SuppressWarnings("unchecked")
		JsonEndpoint<IndexRequest<?>, IndexResponse, ErrorResponse> endpoint = (JsonEndpoint<IndexRequest<?>, IndexResponse, ErrorResponse>) IndexRequest._ENDPOINT;

		return this.transport.performRequestAsync(request, endpoint, this.transportOptions);
	}

Is it true that this method can actually throw an IOException? I'd expect that the CompletableFuture might return a failure (essentially wrapping the exception), but it's not actually throwing said exception. Am I missing some other checked IOException further down in the call chain? In the above example, RestClientTransport#performRequestAsync doesn't throw an IOException.

Missing X-Elastic-Product header

Since upgrading from the 7.15.x client to the 7.16.0 client I am getting a TransportException:

co.elastic.clients.transport.TransportException: [es/search] Missing [X-Elastic-Product] header. Please check that you are connecting to an Elasticsearch instance, and that any networking filters are preserving that header.

Deserialization error when execute GET a non-existing alias

execute simple test case as follow:

import org.junit.jupiter.api.Test;

public class TestEsAliasCommands extends AbstractEsCallerTest {

    @Test
    void testGetNotExistingAlias() {
        try {
            getEsClient().indices().getAlias(g -> g.name("not-existing-alias")).toCompletableFuture().get();
        } catch (Throwable e) {
            e.printStackTrace();
        }
    }
}

where getEsClient() return an ElasticsearchAsyncClient instance. error as follow:

co.elastic.clients.json.UnexpectedJsonEventException: Unexpected JSON event 'VALUE_STRING' instead of '[START_OBJECT, KEY_NAME]'
	at co.elastic.clients.json.JsonpUtils.ensureAccepts(JsonpUtils.java:68)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:74)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:276)
	at co.elastic.clients.transport.rest_client.RestClientTransport.access$200(RestClientTransport.java:60)
	at co.elastic.clients.transport.rest_client.RestClientTransport$1.onSuccess(RestClientTransport.java:165)
	at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:649)
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:383)
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:377)
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.base/java.lang.Thread.run(Thread.java:833)

environment:

• server: docker image docker.elastic.co/elasticsearch/elasticsearch:7.16.2
• client: version 7.16.2

client failed to parse completion suggest response

with elasticsearch-java:7.16.0,
query:

var suggest = Suggester.of(builder -> {
    builder.text(request.prefix);
    for (String field : request.fields) {
        builder.suggesters(field, s -> s.completion(c -> c.field(field).skipDuplicates(true).size(request.limit)));
    }
    return builder;
});
var response = client.search(builder -> builder.index(index).suggest(suggest).source(s -> s.fetch(false)), documentClass);

response from es (successful and expected response):

{
  "took": 67,
  "timed_out": false,
  "_shards": {"total": 1, "successful": 1, "skipped": 0, "failed": 0},
  "hits": {"total": {"value": 0, "relation": "eq"}, "max_score": null, "hits": []},
  "suggest": {
    "completion#completion:completion1": [{"text": "hash", "offset": 0, "length": 4, "options": [{"text": "HashMap-Complete1", "_index": "document", "_type": "_doc", "_id": "2", "_score": 1.0}, {"text": "HashSet-Complete1", "_index": "document", "_type": "_doc", "_id": "1", "_score": 1.0}]}],
    "completion#completion:completion2": [{"text": "hash", "offset": 0, "length": 4, "options": [{"text": "HashMap-Complete2", "_index": "document", "_type": "_doc", "_id": "2", "_score": 1.0}, {"text": "HashSet-Complete2", "_index": "document", "_type": "_doc", "_id": "1", "_score": 1.0}]}]
  }
}

but the client failed to parse with this exception, it looks like it tried to parse it as TermSuggestOption, but it's actually CompletionSuggestOption

co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'TermSuggestOption.score'

	at co.elastic.clients.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:76)
	at co.elastic.clients.elasticsearch.core.search.TermSuggestOption.<init>(TermSuggestOption.java:66)
	at co.elastic.clients.elasticsearch.core.search.TermSuggestOption.<init>(TermSuggestOption.java:52)
	at co.elastic.clients.elasticsearch.core.search.TermSuggestOption$Builder.build(TermSuggestOption.java:167)
	at co.elastic.clients.elasticsearch.core.search.TermSuggestOption$Builder.build(TermSuggestOption.java:126)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:86)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:48)
	at co.elastic.clients.json.UnionDeserializer$SingleMemberHandler.deserialize(UnionDeserializer.java:75)
	at co.elastic.clients.json.UnionDeserializer.deserialize(UnionDeserializer.java:292)
	at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:316)
	at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:285)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:85)
	at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:316)
	at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:285)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:341)
	at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:327)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.transport.endpoints.EndpointWithResponseMapperAttr$1.deserialize(EndpointWithResponseMapperAttr.java:56)
	at co.elastic.clients.transport.rest_client.RestClientTransport.decodeResponse(RestClientTransport.java:325)
	at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:291)
	at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:144)
	at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1487)
	at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1504)

ElasticsearchIndicesClient#getMapping(GetMappingRequest) throws on reading response when mapping contains an object property

Using version 7.15.2.

When an index mapping contains a property that is an object,
co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient#getMapping(co.elastic.clients.elasticsearch.indices.GetMappingRequest) crashes on parsing the result.

This can be tested with the following code:

class Name {
	String first;
	String last;

	public Name(String first, String last) {
		this.first = first;
		this.last = last;
	}
       // getter+setter
}

class Person {
	String id;
	Name name;

	public Person(String id, Name name) {
		this.id = id;
		this.name = name;
	}
       // getter+setter
}

ElasticsearchClient client = ... // setup the ElasticsearchClient
String index = "testindex";
Person person = new Person("42", new Name("Ford", "Prefect"));
client.index(b -> b.index(index).id(person.id).document(person));
GetMappingResponse getMappingResponse = client.indices().getMapping(mrb -> mrb.index(index));

This will produce the following error:

jakarta.json.stream.JsonParsingException: Property 'type' not found

	at co.elastic.clients.json.JsonpUtils.lookAheadFieldValue(JsonpUtils.java:139)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:128)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:95)
	at co.elastic.clients.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:42)
	at co.elastic.clients.json.JsonpDeserializer$LazyDeserializer.deserialize(JsonpDeserializer.java:205)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:102)
	at co.elastic.clients.json.JsonpDeserializer$StringMapDeserializer.deserialize(JsonpDeserializer.java:461)
	at co.elastic.clients.json.JsonpDeserializer$StringMapDeserializer.deserialize(JsonpDeserializer.java:447)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:102)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:68)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:122)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:95)
	at co.elastic.clients.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:42)
	at co.elastic.clients.json.JsonpDeserializer$LazyDeserializer.deserialize(JsonpDeserializer.java:205)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:102)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:68)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:122)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:95)
	at co.elastic.clients.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:42)
	at co.elastic.clients.json.JsonpDeserializer$LazyDeserializer.deserialize(JsonpDeserializer.java:205)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:102)
	at co.elastic.clients.base.DictionaryResponse.lambda$setupDictionaryResponseDeserializer$0(DictionaryResponse.java:148)
	at co.elastic.clients.json.ObjectDeserializer.parseUnknownField(ObjectDeserializer.java:150)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:120)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:95)
	at co.elastic.clients.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:42)
	at co.elastic.clients.json.JsonpDeserializer$LazyDeserializer.deserialize(JsonpDeserializer.java:205)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:102)
	at co.elastic.clients.base.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:242)
	at co.elastic.clients.base.rest_client.RestClientTransport.performRequest(RestClientTransport.java:104)
	at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getMapping(ElasticsearchIndicesClient.java:1044)
	at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getMapping(ElasticsearchIndicesClient.java:1061)

The mapping returned from the server:

{
  "testindex" : {
    "mappings" : {
      "properties" : {
        "id" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "name" : {
          "properties" : {
            "first" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "last" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        }
      }
    }
  }
}

I debugged through the code; when reading the properties, the parser comes to the name entry and then on finding a START_OBJECT tries to get the type property of the object, basically expecting something like

{
    "type": "object",   <===  Elasticsearch does not send a type entry here
    "properties": {
      "first": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "last": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }

Even when the mapping is stored explicitly in Elasticsearch with the type object, it is not returned on getting the mapping.

significant terms aggregation missing regex include method

Hello,
by documentation of SignificantTermsAggregation should support regex aggregation
Documentation

The object new TermsAggregation.Builder() is missing the include method that supports it
it only has
public final Builder include(String value) when for exclude the currect implemtation exists (TermsExclude exclude)

it's expected that the object will have
private final TermsInclude include;
as in TermsAggregation.Builder

not having it causes the wrong behavior and elastic to not use regex for filtering

_msearch requires newline

Hi,

1. MsearchTemplateRequest
   I am trying to write tests with latest version of elasticsearch-java(7.15.1), elasticsearch(docker.elastic.co/elasticsearch/elasticsearch:7.15.0) for elastic-apm-agent. And got exception message below:

The msearch request must be terminated by a newline [\n]

As I understand it, the problem is in the serialization of the request - instead of json separated new line, the client sends an array of objects

Steps to reproduce:

 IndexResponse ir = doIndex(new IndexRequest(new IndexRequest.Builder()
            .index(INDEX)
            .type(DOC_TYPE)
            .id(DOC_ID)
            .refresh(JsonValue.TRUE)
            .document(Map.of(FOO, BAR))));
 PutScriptRequest putScriptRequest = new PutScriptRequest(builder -> builder
            .id("elastic-search-template")
            .script(new StoredScript(scriptBuilder ->
                scriptBuilder.lang(ScriptLanguage.Mustache)
                    .source("{" +
                        "  \"query\": { \"term\" : { \"{{field}}\" : \"{{value}}\" } }," +
                        "  \"size\" : \"{{size}}\"" +
                        "}")
            )));

        client.putScript(putScriptRequest);

   MsearchTemplateRequest multiRequest = new MsearchTemplateRequest(builder -> builder.searchTemplates(
            new TemplateItem(itemBuilder -> itemBuilder.index(INDEX)
                .id("elastic-search-template")
                .params(Map.of("field", JsonData.of(FOO), "value", JsonData.of(BAR), "size", JsonData.of(5))))
        ));

        try {
            MsearchTemplateResponse<Map> response = doMultiSearchTemplate(multiRequest, Map.class);
        } catch (ApiException e) {
            // reason -> {JsonStringImpl@7318} ""The msearch request must be terminated by a newline [\n]""
            LOGGER.error("Error", e);
        }

Addition logs:

2183 [main] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl  - http-outgoing-0 127.0.0.1:59098<->127.0.0.1:49297[ACTIVE][rw:r]: Event set [w]
2183 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalIODispatch  - http-outgoing-0 [ACTIVE] Request ready
2183 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.MainClientExec  - [exchange: 4] Attempt 1 to execute request
2183 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.MainClientExec  - [exchange: 4] Target auth state: UNCHALLENGED
2183 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.MainClientExec  - [exchange: 4] Proxy auth state: UNCHALLENGED
2183 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl  - http-outgoing-0 127.0.0.1:59098<->127.0.0.1:49297[ACTIVE][rw:w]: Set timeout 30000
2183 [I/O dispatcher 1] DEBUG org.apache.http.headers  - http-outgoing-0 >> POST /_msearch/template HTTP/1.1
2183 [I/O dispatcher 1] DEBUG org.apache.http.headers  - http-outgoing-0 >> Content-Length: 103
2183 [I/O dispatcher 1] DEBUG org.apache.http.headers  - http-outgoing-0 >> Content-Type: application/json; charset=UTF-8
2183 [I/O dispatcher 1] DEBUG org.apache.http.headers  - http-outgoing-0 >> Host: localhost:49297
2183 [I/O dispatcher 1] DEBUG org.apache.http.headers  - http-outgoing-0 >> Connection: Keep-Alive
2183 [I/O dispatcher 1] DEBUG org.apache.http.headers  - http-outgoing-0 >> User-Agent: elasticsearch-java/7.15.0-SNAPSHOT (Java/11.0.11)
2183 [I/O dispatcher 1] DEBUG org.apache.http.headers  - http-outgoing-0 >> X-Elastic-Client-Meta: es=7.15.0-SNAPSHOT,jv=11,t=7.15.0-SNAPSHOT,hc=4.1.4
2183 [I/O dispatcher 1] DEBUG org.apache.http.headers  - http-outgoing-0 >> Authorization: Basic ZWxhc3RpYy11c2VyOmVsYXN0aWMtcGFzcw==
2184 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl  - http-outgoing-0 127.0.0.1:59098<->127.0.0.1:49297[ACTIVE][rw:w]: Event set [w]
2184 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalIODispatch  - http-outgoing-0 [ACTIVE] Output ready
2184 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.MainClientExec  - [exchange: 4] produce content
2184 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.MainClientExec  - [exchange: 4] Request completed
2184 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalIODispatch  - http-outgoing-0 [ACTIVE] [content length: 103; pos: 103; completed: true]
2184 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl  - http-outgoing-0 127.0.0.1:59098<->127.0.0.1:49297[ACTIVE][rw:w]: 452 bytes written
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "POST /_msearch/template HTTP/1.1[\r][\n]"
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "Content-Length: 103[\r][\n]"
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "Content-Type: application/json; charset=UTF-8[\r][\n]"
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "Host: localhost:49297[\r][\n]"
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "User-Agent: elasticsearch-java/7.15.0-SNAPSHOT (Java/11.0.11)[\r][\n]"
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "X-Elastic-Client-Meta: es=7.15.0-SNAPSHOT,jv=11,t=7.15.0-SNAPSHOT,hc=4.1.4[\r][\n]"
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "Authorization: Basic ZWxhc3RpYy11c2VyOmVsYXN0aWMtcGFzcw==[\r][\n]"
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "[\r][\n]"
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "[{"id":"elastic-search-template","index":["my-index"],"params":{"size":5,"field":"foo","value":"bar"}}]"
2185 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalIODispatch  - http-outgoing-0 [ACTIVE] Request ready
2185 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl  - http-outgoing-0 127.0.0.1:59098<->127.0.0.1:49297[ACTIVE][r:w]: Event cleared [w]
2188 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl  - http-outgoing-0 127.0.0.1:59098<->127.0.0.1:49297[ACTIVE][r:r]: 696 bytes read
2188 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 << "HTTP/1.1 400 Bad Request[\r][\n]"
2188 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 << "X-elastic-product: Elasticsearch[\r][\n]"
2188 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 << "Warning: 299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."[\r][\n]"
2188 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 << "content-type: application/json; charset=UTF-8[\r][\n]"
2188 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 << "content-length: 249[\r][\n]"
2188 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 << "[\r][\n]"
2188 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 << "{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The msearch request must be terminated by a newline [\n]"}],"type":"illegal_argument_exception","reason":"The msearch request must be terminated by a newline [\n]"},"status":400}"
2188 [I/O dispatcher 1] DEBUG org.apache.http.headers  - http-outgoing-0 << HTTP/1.1 400 Bad Request
2188 [I/O dispatcher 1] DEBUG org.apache.http.headers  - http-outgoing-0 << X-elastic-product: Elasticsearch
2189 [I/O dispatcher 1] DEBUG org.apache.http.headers  - http-outgoing-0 << Warning: 299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."
2189 [I/O dispatcher 1] DEBUG org.apache.http.headers  - http-outgoing-0 << content-type: application/json; charset=UTF-8
2189 [I/O dispatcher 1] DEBUG org.apache.http.headers  - http-outgoing-0 << content-length: 249
2189 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalIODispatch  - http-outgoing-0 [ACTIVE(249)] Response received
2189 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.MainClientExec  - [exchange: 4] Response received HTTP/1.1 400 Bad Request
2189 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalIODispatch  - http-outgoing-0 [ACTIVE(249)] Input ready
2189 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.MainClientExec  - [exchange: 4] Consume content
2189 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalHttpAsyncClient  - [exchange: 4] Connection can be kept alive indefinitely
2189 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.MainClientExec  - [exchange: 4] Response processed
2189 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalHttpAsyncClient  - [exchange: 4] releasing connection
2189 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl  - http-outgoing-0 127.0.0.1:59098<->127.0.0.1:49297[ACTIVE][r:r]: Remove attribute http.nio.exchange-handler
2190 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager  - Releasing connection: [id: http-outgoing-0][route: {}->http://localhost:49297][total kept alive: 0; route allocated: 1 of 10; total allocated: 1 of 30]
2190 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager  - Connection [id: http-outgoing-0][route: {}->http://localhost:49297] can be kept alive indefinitely

What am I doing incorrectly?

2. Same thing for multi-search request.

 MsearchRequest multiSearchRequest = new MsearchRequest(builder -> builder
            .index(INDEX)
            .searches(
                Json.createObjectBuilder()
                    .add("index", Json.createArrayBuilder().add(INDEX))
                    .add("types", Json.createArrayBuilder())
                    .add("search_type", "query_then_fetch")
                    .add("ccs_minimize_roundtrips", true)
                    .build(),
                Json.createObjectBuilder()
                    .add("query", Json.createObjectBuilder()
                        .add("match", Json.createObjectBuilder()
                            .add(FOO, Json.createObjectBuilder()
                                .add("query", BAR)
                                .add("operator", "OR")))
                        .build()).build())
            .typedKeys(true));

        ApiException apiException = null;
        try {
            MsearchResponse response = doMultiSearch(multiSearchRequest, Map.class);
        } catch (ApiException e) {
            apiException = e;
        }

1836 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "POST /my-index/_msearch?typed_keys=true HTTP/1.1[\r][\n]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "Content-Length: 159[\r][\n]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "Content-Type: application/json; charset=UTF-8[\r][\n]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "Host: localhost:49173[\r][\n]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "User-Agent: elasticsearch-java/7.15.0-SNAPSHOT (Java/11.0.11)[\r][\n]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "X-Elastic-Client-Meta: es=7.15.0-SNAPSHOT,jv=11,t=7.15.0-SNAPSHOT,hc=4.1.4[\r][\n]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "Authorization: Basic ZWxhc3RpYy11c2VyOmVsYXN0aWMtcGFzcw==[\r][\n]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "[\r][\n]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 >> "[{"index":["my-index"],"types":[],"search_type":"query_then_fetch","ccs_minimize_roundtrips":true},{"query":{"match":{"foo":{"query":"bar","operator":"OR"}}}}]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalIODispatch  - http-outgoing-0 [ACTIVE] Request ready
1836 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl  - http-outgoing-0 127.0.0.1:60408<->127.0.0.1:49173[ACTIVE][r:w]: Event cleared [w]
1840 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl  - http-outgoing-0 127.0.0.1:60408<->127.0.0.1:49173[ACTIVE][r:r]: 696 bytes read
1840 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 << "HTTP/1.1 400 Bad Request[\r][\n]"
1840 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 << "X-elastic-product: Elasticsearch[\r][\n]"
1840 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 << "Warning: 299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."[\r][\n]"
1840 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 << "content-type: application/json; charset=UTF-8[\r][\n]"
1840 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 << "content-length: 249[\r][\n]"
1840 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 << "[\r][\n]"
1840 [I/O dispatcher 1] DEBUG org.apache.http.wire  - http-outgoing-0 << "{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The msearch request must be terminated by a newline [\n]"}],"type":"illegal_argument_exception","reason":"The msearch request must be terminated by a newline [\n]"},"status":400}"
1840 [I/O dispatcher 1] DEBUG org.apache.http.headers  - http-outgoing-0 << HTTP/1.1 400 Bad Request
1840 [I/O dispatcher 1] DEBUG org.apache.http.headers  - http-outgoing-0 << X-elastic-product: Elasticsearch
1840 [I/O dispatcher 1] DEBUG org.apache.http.headers  - http-outgoing-0 << Warning: 299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."

3. It would be great if you add api that allow convert Query to JsonValue.
   Does current api allow convert TermQuery or MatchQuery to JsonValue?

.searches(Json.createArrayBuilder().add(Json.createObjectBuilder()
                .add("query", Json.createObjectBuilder().add("match", Json.createObjectBuilder().add(FOO, BAR)))
                .build()).build()))

java.lang.NoClassDefFoundError: jakarta/json/JsonException

Hello, I'm trying to use the elasticsearch-java client in a new kotlin spring boot 2.7.0-SNAPSHOT project. As soon as I add the import for JacksonJsonpMapper() I get the following error:

	at com.codinghumans.framework.elasticsearch.ElasticsearchClient.<init>(ElasticsearchClient.kt:21) ~[main/:na]
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[na:na]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[na:na]
	at kotlin.reflect.jvm.internal.calls.CallerImpl$Constructor.call(CallerImpl.kt:41) ~[kotlin-reflect-1.6.0.jar:1.6.0-release-798(1.6.0)]
	at kotlin.reflect.jvm.internal.KCallableImpl.call(KCallableImpl.kt:108) ~[kotlin-reflect-1.6.0.jar:1.6.0-release-798(1.6.0)]
	at kotlin.reflect.jvm.internal.KCallableImpl.callDefaultMethod$kotlin_reflection(KCallableImpl.kt:159) ~[kotlin-reflect-1.6.0.jar:1.6.0-release-798(1.6.0)]
	at kotlin.reflect.jvm.internal.KCallableImpl.callBy(KCallableImpl.kt:112) ~[kotlin-reflect-1.6.0.jar:1.6.0-release-798(1.6.0)]
	at org.springframework.beans.BeanUtils$KotlinDelegate.instantiateClass(BeanUtils.java:867) ~[spring-beans-5.3.14.jar:5.3.14]
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:196) ~[spring-beans-5.3.14.jar:5.3.14]
	... 54 common frames omitted
Caused by: java.lang.ClassNotFoundException: jakarta.json.spi.JsonProvider
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) ~[na:na]
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) ~[na:na]
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520) ~[na:na]
	... 66 common frames omitted```

This is my build.gradle.kts

```import org.jetbrains.kotlin.gradle.tasks.KotlinCompile

plugins {
	id("org.springframework.boot") version "2.7.0-SNAPSHOT"
	id("io.spring.dependency-management") version "1.0.11.RELEASE"
	id("org.flywaydb.flyway") version "8.2.0"
	kotlin("jvm") version "1.6.0"
	kotlin("plugin.spring") version "1.6.0"
}

group = "com.codinghumans"
version = "0.0.1-SNAPSHOT"
java.sourceCompatibility = JavaVersion.VERSION_17

repositories {
	mavenCentral()
	maven { url = uri("https://repo.spring.io/milestone") }
	maven { url = uri("https://repo.spring.io/snapshot") }
}

dependencies {
	implementation("co.elastic.clients:elasticsearch-java:7.16.2")
	implementation("com.fasterxml.jackson.core:jackson-databind:2.13.1")
	implementation("com.ibm.icu:icu4j:70.1")
	implementation("mysql:mysql-connector-java:8.0.27")
	implementation("org.flywaydb:flyway-mysql:8.3.0")

	implementation("org.springframework.boot:spring-boot-starter-web")
	implementation("org.springframework.boot:spring-boot-starter-validation")
	implementation("org.springframework.boot:spring-boot-starter-data-jpa")
	implementation("org.springframework.boot:spring-boot-starter-security")
	implementation("com.fasterxml.jackson.module:jackson-module-kotlin")
	implementation("org.jetbrains.kotlin:kotlin-reflect")
	implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8")

	testImplementation("org.springframework.boot:spring-boot-starter-test")
	developmentOnly("org.springframework.boot:spring-boot-devtools")
}

tasks.withType<KotlinCompile> {
	kotlinOptions {
		freeCompilerArgs = listOf("-Xjsr305=strict")
		jvmTarget = "17"
	}
}

tasks.withType<Test> {
	useJUnitPlatform()
}

I'm quite new to java, so I'm a bit lost...Any idea what I might be doing wrong?

How can I disable api-version-compatibility headers?

Since some 7.x service do not support these headers, how can I disable this in Java client?

Converting a request/response back to its builder

I am trying to implement a generic deep pagination function that takes a SearchRequest, creates a point-in-time and then paginates through all the values using search_after.

This requires modifying the search request I am given to include the PIT and set the search_after field correctly. However, using the new client, I am unable to modify an existing request due to all data being immutable and all mutability being contained within the corresponding builder classes.

A solution to this problem could be adding a generated method that creates a builder from the values contained within the object. Alternatively, a new constructor could be added to the builder classes that take a constructed object and initialize their values from the object.

ElasticsearchIndicesClient getSettings does not parse default values returned from server

When calling the method ElasticsearchIndicesClient.getSettings(...) method, it is possible to add the includeDefaults parameter and set that to true:

GetSettingsResponse getSettingsResponse = client.indices()
		.getSettings(srb -> srb.index(index).includeDefaults(true));

But the GetSettingsResponse that is returned does not contain this information, although it is returned from the server (the following json was extracted from an intercepting proxy):

{
  "testindex" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "testindex",
        "creation_date" : "1636661459560",
        "number_of_replicas" : "1",
        "uuid" : "EYazsV63SXmlmFcNhRFvRw",
        "version" : {
          "created" : "7150299"
        }
      }
    },
    "defaults" : {
      "index" : {
        "flush_after_merge" : "512mb",
        "final_pipeline" : "_none",
        "max_inner_result_window" : "100",
        "unassigned" : {
          "node_left" : {
            "delayed_timeout" : "1m"
          }
        },
        "max_terms_count" : "65536",
        "rollup" : {
          "source" : {
            "name" : "",
            "uuid" : ""
          }
        },
        "lifecycle" : {
          "name" : "",
          "parse_origination_date" : "false",
          "step" : {
            "wait_time_threshold" : "12h"
          },
          "indexing_complete" : "false",
          "rollover_alias" : "",
          "origination_date" : "-1"
        },
        "routing_partition_size" : "1",
        "force_memory_term_dictionary" : "false",
        "max_docvalue_fields_search" : "100",
        "merge" : {
          "scheduler" : {
            "max_thread_count" : "1",
            "auto_throttle" : "true",
            "max_merge_count" : "6"
          },
          "policy" : {
            "floor_segment" : "2mb",
            "max_merge_at_once_explicit" : "30",
            "max_merge_at_once" : "10",
            "max_merged_segment" : "5gb",
            "expunge_deletes_allowed" : "10.0",
            "segments_per_tier" : "10.0",
            "deletes_pct_allowed" : "33.0"
          }
        },
        "max_refresh_listeners" : "1000",
        "max_regex_length" : "1000",
        "load_fixed_bitset_filters_eagerly" : "true",
        "number_of_routing_shards" : "1",
        "write" : {
          "wait_for_active_shards" : "1"
        },
        "verified_before_close" : "false",
        "mapping" : {
          "coerce" : "false",
          "nested_fields" : {
            "limit" : "50"
          },
          "depth" : {
            "limit" : "20"
          },
          "field_name_length" : {
            "limit" : "9223372036854775807"
          },
          "total_fields" : {
            "limit" : "1000"
          },
          "nested_objects" : {
            "limit" : "10000"
          },
          "ignore_malformed" : "false"
        },
        "source_only" : "false",
        "soft_deletes" : {
          "enabled" : "true",
          "retention" : {
            "operations" : "0"
          },
          "retention_lease" : {
            "period" : "12h"
          }
        },
        "max_script_fields" : "32",
        "query" : {
          "default_field" : [
            "*"
          ],
          "parse" : {
            "allow_unmapped_fields" : "true"
          }
        },
        "format" : "0",
        "frozen" : "false",
        "sort" : {
          "missing" : [ ],
          "mode" : [ ],
          "field" : [ ],
          "order" : [ ]
        },
        "priority" : "1",
        "codec" : "default",
        "max_rescore_window" : "10000",
        "max_adjacency_matrix_filters" : "100",
        "analyze" : {
          "max_token_count" : "10000"
        },
        "gc_deletes" : "60s",
        "top_metrics_max_size" : "10",
        "optimize_auto_generated_id" : "true",
        "max_ngram_diff" : "1",
        "hidden" : "false",
        "translog" : {
          "generation_threshold_size" : "64mb",
          "flush_threshold_size" : "512mb",
          "sync_interval" : "5s",
          "retention" : {
            "size" : "-1",
            "age" : "-1"
          },
          "durability" : "REQUEST"
        },
        "auto_expand_replicas" : "false",
        "mapper" : {
          "dynamic" : "true"
        },
        "recovery" : {
          "type" : ""
        },
        "requests" : {
          "cache" : {
            "enable" : "true"
          }
        },
        "data_path" : "",
        "highlight" : {
          "max_analyzed_offset" : "1000000"
        },
        "routing" : {
          "rebalance" : {
            "enable" : "all"
          },
          "allocation" : {
            "include" : {
              "_tier" : ""
            },
            "disk" : {
              "watermark" : {
                "ignore" : "false"
              }
            },
            "exclude" : {
              "_tier" : ""
            },
            "require" : {
              "_tier" : ""
            },
            "enable" : "all",
            "total_shards_per_node" : "-1"
          }
        },
        "search" : {
          "slowlog" : {
            "level" : "TRACE",
            "threshold" : {
              "fetch" : {
                "warn" : "-1",
                "trace" : "-1",
                "debug" : "-1",
                "info" : "-1"
              },
              "query" : {
                "warn" : "-1",
                "trace" : "-1",
                "debug" : "-1",
                "info" : "-1"
              }
            }
          },
          "idle" : {
            "after" : "30s"
          },
          "throttled" : "false"
        },
        "fielddata" : {
          "cache" : "node"
        },
        "default_pipeline" : "_none",
        "max_slices_per_scroll" : "1024",
        "shard" : {
          "check_on_startup" : "false"
        },
        "xpack" : {
          "watcher" : {
            "template" : {
              "version" : ""
            }
          },
          "version" : "",
          "ccr" : {
            "following_index" : "false"
          }
        },
        "percolator" : {
          "map_unmapped_fields_as_text" : "false"
        },
        "allocation" : {
          "max_retries" : "5",
          "existing_shards_allocator" : "gateway_allocator"
        },
        "refresh_interval" : "1s",
        "indexing" : {
          "slowlog" : {
            "reformat" : "true",
            "threshold" : {
              "index" : {
                "warn" : "-1",
                "trace" : "-1",
                "debug" : "-1",
                "info" : "-1"
              }
            },
            "source" : "1000",
            "level" : "TRACE"
          }
        },
        "compound_format" : "0.1",
        "blocks" : {
          "metadata" : "false",
          "read" : "false",
          "read_only_allow_delete" : "false",
          "read_only" : "false",
          "write" : "false"
        },
        "max_result_window" : "10000",
        "store" : {
          "stats_refresh_interval" : "10s",
          "type" : "",
          "fs" : {
            "fs_lock" : "native"
          },
          "preload" : [ ],
          "snapshot" : {
            "snapshot_name" : "",
            "index_uuid" : "",
            "cache" : {
              "prewarm" : {
                "enabled" : "true"
              },
              "enabled" : "true",
              "excluded_file_types" : [ ]
            },
            "repository_uuid" : "",
            "uncached_chunk_size" : "-1b",
            "index_name" : "",
            "partial" : "false",
            "blob_cache" : {
              "metadata_files" : {
                "max_length" : "64kb"
              }
            },
            "repository_name" : "",
            "snapshot_uuid" : ""
          }
        },
        "queries" : {
          "cache" : {
            "enabled" : "true"
          }
        },
        "shard_limit" : {
          "group" : "normal"
        },
        "warmer" : {
          "enabled" : "true"
        },
        "max_shingle_diff" : "3",
        "query_string" : {
          "lenient" : "false"
        }
      }
    }
  }
}

With the old RestHighLevelClient the org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse contained two entries: indexToSettings for the custom settings made by the user and indexToDefaultSettings for the default settings.

Client fails on parsing search response when `track_total_hits` is set to `false`

When the track_total_hits property of a query is set to false like this:

client.search(srb -> srb
		.index(index)
		.trackTotalHits(thb -> thb.enabled(false)), Entity.class);

The client fails on parsing the response because it expects the response to contain the total element which is not returned in this case:

co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'HitsMetadata.total'

	at co.elastic.clients.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:76)
	at co.elastic.clients.elasticsearch.core.search.HitsMetadata.<init>(HitsMetadata.java:68)
	at co.elastic.clients.elasticsearch.core.search.HitsMetadata.<init>(HitsMetadata.java:53)
	at co.elastic.clients.elasticsearch.core.search.HitsMetadata$Builder.build(HitsMetadata.java:220)
	at co.elastic.clients.elasticsearch.core.search.HitsMetadata$Builder.build(HitsMetadata.java:139)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:80)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.transport.endpoints.EndpointWithResponseMapperAttr$1.deserialize(EndpointWithResponseMapperAttr.java:56)
	at co.elastic.clients.transport.rest_client.RestClientTransport.decodeResponse(RestClientTransport.java:325)
	at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:291)
	at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:144)
	at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1487)
	at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1504)
	at org.springframework.data.elasticsearch.backend.elasticsearch.DevTests.someTest(DevTests.java:96)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)

UpdateOperation don't support script

in the old RestHighLevelClient, I can execute a bulk request with an update-using-script operation. It doesn't seem to work
in this new client.

Link to Javadoc doesn't contain version

Not sure if this is intentional, but the PR to add links to the Javadoc and Github repo added a link to the javadoc that doesn't work. Not sure how link templating works, but it isn't properly formatting the links. Happy to open a PR if it makes sense that this should be fixed

SearchMvtResponse is not Json serializable

Trying a SearchMvtRequest gives an error as we are trying to serialise the response to Json:

RestClient restClient = RestClient.builder(new HttpHost("localhost", 9200)).build());
Transport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());
ElasticsearchClient client = new ElasticsearchClient(transport);
SearchMvtResponse response = client.searchMvt(
        (builder) -> builder.zoom(0).x(0).y(0).addIndex("points").field("location")
 );

Gives the following error:

Caused by: com.fasterxml.jackson.core.JsonParseException: Illegal character ((CTRL-CHAR, code 26)): only regular white space (\r, \n, \t) is allowed between tokens

How can I create a simple match query?

Going on with my tests I wanted to create a simple match query for a call like this:

GET /test-index/_search
{
  "query": {
    "match": {
      "message": {
        "query": "this is a test"
      }
    }
  }
}

I try to use this code:

		SearchRequest searchRequest = new SearchRequest.Builder()
				.index("test-index")
				.query(b -> b
						.match(
								// what to use here????
						))
				.build();

The QueryContainer.Builder has these 2 match methods:

        public ObjectBuilder<QueryContainer> match(NamedQuery<JsonValue> v) {
            this.$variant = v;
            this.$tag = QueryContainer.Tag.match;
            return new Constant(this.build());
        }

        public ObjectBuilder<QueryContainer> match(Function<co.elastic.clients.elasticsearch._types.query_dsl.NamedQuery.Builder<JsonValue>, ObjectBuilder<NamedQuery<JsonValue>>> f) {
            return this.match((NamedQuery)((ObjectBuilder)f.apply(new co.elastic.clients.elasticsearch._types.query_dsl.NamedQuery.Builder())).build());
        }

what would I need to put there? I am missing a MatchQuery class here. Or something like

.match(m -> m.field("message").query("this is a test").build()

The same problem comes up when trying to build a simple terms query.

Cluster stats fails due to Jackson error

jakarta.json.stream.JsonParsingException: Jackson exception: Numeric value (-9223372036854775808) out of range of int (-2147483648 - 2147483647)

ClusterStatsResponse stats = cluster.getClient().cluster().stats();

Request:

curl http://localhost:60512/_cluster/stats\?pretty

Response:

{ "_nodes" : { "total" : 2, "successful" : 2, "failed" : 0 }, "cluster_name" : "testCluster", "cluster_uuid" : "mFsnsmy6R4mIEyAcxOdRGg", "timestamp" : 1640886234666, "status" : "green", "indices" : { "count" : 0, "shards" : { }, "docs" : { "count" : 0, "deleted" : 0 }, "store" : { "size_in_bytes" : 0, "total_data_set_size_in_bytes" : 0, "reserved_in_bytes" : 0 }, "fielddata" : { "memory_size_in_bytes" : 0, "evictions" : 0 }, "query_cache" : { "memory_size_in_bytes" : 0, "total_count" : 0, "hit_count" : 0, "miss_count" : 0, "cache_size" : 0, "cache_count" : 0, "evictions" : 0 }, "completion" : { "size_in_bytes" : 0 }, "segments" : { "count" : 0, "memory_in_bytes" : 0, "terms_memory_in_bytes" : 0, "stored_fields_memory_in_bytes" : 0, "term_vectors_memory_in_bytes" : 0, "norms_memory_in_bytes" : 0, "points_memory_in_bytes" : 0, "doc_values_memory_in_bytes" : 0, "index_writer_memory_in_bytes" : 0, "version_map_memory_in_bytes" : 0, "fixed_bit_set_memory_in_bytes" : 0, "max_unsafe_auto_id_timestamp" : -9223372036854775808, "file_sizes" : { } }, "mappings" : { "field_types" : [ ], "runtime_field_types" : [ ] }, "analysis" : { "char_filter_types" : [ ], "tokenizer_types" : [ ], "filter_types" : [ ], "analyzer_types" : [ ], "built_in_char_filters" : [ ], "built_in_tokenizers" : [ ], "built_in_filters" : [ ], "built_in_analyzers" : [ ] }, "versions" : [ ] }, "nodes" : { "count" : { "total" : 2, "coordinating_only" : 0, "data" : 2, "data_cold" : 2, "data_content" : 2, "data_frozen" : 2, "data_hot" : 2, "data_warm" : 2, "ingest" : 2, "master" : 2, "ml" : 0, "remote_cluster_client" : 2, "transform" : 2, "voting_only" : 0 }, "versions" : [ "7.16.1" ], "os" : { "available_processors" : 16, "allocated_processors" : 16, "names" : [ { "name" : "Mac OS X", "count" : 2 } ], "pretty_names" : [ { "pretty_name" : "Mac OS X", "count" : 2 } ], "architectures" : [ { "arch" : "aarch64", "count" : 2 } ], "mem" : { "total_in_bytes" : 34359738368, "free_in_bytes" : 118784000, "used_in_bytes" : 34240954368, "free_percent" : 0, "used_percent" : 100 } }, "process" : { "cpu" : { "percent" : 0 }, "open_file_descriptors" : { "min" : 337, "max" : 338, "avg" : 337 } }, "jvm" : { "max_uptime_in_millis" : 51439, "versions" : [ { "version" : "17.0.1", "vm_name" : "OpenJDK 64-Bit Server VM", "vm_version" : "17.0.1+12", "vm_vendor" : "Eclipse Adoptium", "bundled_jdk" : true, "using_bundled_jdk" : true, "count" : 2 } ], "mem" : { "heap_used_in_bytes" : 824238880, "heap_max_in_bytes" : 17179869184 }, "threads" : 72 }, "fs" : { "total_in_bytes" : 994662584320, "free_in_bytes" : 722047512576, "available_in_bytes" : 722047512576 }, "plugins" : [ ], "network_types" : { "transport_types" : { "netty4" : 2 }, "http_types" : { "netty4" : 2 } }, "discovery_types" : { "zen" : 2 }, "packaging_types" : [ { "flavor" : "default", "type" : "tar", "count" : 2 } ], "ingest" : { "number_of_pipelines" : 2, "processor_stats" : { "gsub" : { "count" : 0, "failed" : 0, "current" : 0, "time_in_millis" : 0 }, "script" : { "count" : 0, "failed" : 0, "current" : 0, "time_in_millis" : 0 } } } } }

null value in SearchResponse.hits.hits.sort causes parsing failure

From https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511/8

When a request is sorted on fields that may have null values, the responses's hits.hits.sort array will contain null values that cause a parsing failure as arrays generally don't accept null values in the API spec.

This is related to elastic/elasticsearch-specification#1109 and will be solved once sort is considered as an opaque JSON blob.

/cc @ilgrosso

Invalid enum values in DataStreamHealthStatus

The enum in the DataStreamHealthStatus contains the wrong statuses, elastic return the status of the data stream in capitals. Currently the parsing of the object return a exception.

package co.elastic.clients.elasticsearch.indices.DataStreamHealthStatus

Green("green"),
Yellow("yellow"),
Red("red");

jakarta.json.stream.JsonParsingException: Invalid enum [GREEN] at co.elastic.clients.util.StringEnum$Deserializer.deserialize(StringEnum.java:75) at co.elastic.clients.util.StringEnum$Deserializer.deserialize(StringEnum.java:61) at co.elastic.clients.util.StringEnum$Deserializer.deserialize(StringEnum.java:45) at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:102) at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:68) at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:122) at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:95) at co.elastic.clients.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:42) at co.elastic.clients.json.JsonpDeserializer$LazyDeserializer.deserialize(JsonpDeserializer.java:205) at co.elastic.clients.json.JsonpDeserializer$ArrayDeserializer.deserialize(JsonpDeserializer.java:430) at co.elastic.clients.json.JsonpDeserializer$ArrayDeserializer.deserialize(JsonpDeserializer.java:409) at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:102) at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:68) at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:122) at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:95) at co.elastic.clients.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:42) at co.elastic.clients.json.JsonpDeserializer$LazyDeserializer.deserialize(JsonpDeserializer.java:205) at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:102) at co.elastic.clients.base.RestClientTransport.getHighLevelResponse(RestClientTransport.java:231) at co.elastic.clients.base.RestClientTransport.performRequest(RestClientTransport.java:93) at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getDataStream(ElasticsearchIndicesClient.java:838)

TermVectorOption enum is missing with_positions_payloads value

tested with 7.16.0-SNAPSHOT (20.11.2021, 05:44)

In the TermVectorOption enum
https://github.com/elastic/elasticsearch-java/blob/main/java-client/src/main/java/co/elastic/clients/elasticsearch/_types/mapping/TermVectorOption.java the value with_positions_payloads is missing (https://www.elastic.co/guide/en/elasticsearch/reference/current/term-vector.html)

Example from the readme doc fails with Jackson, works with Yasson

I have set up a minimal Java application to test the client (https://github.com/sothawo/elastic-java-test) It's basically the code from the readme showing how to use the Elastisearch Client.

I have Elasticsearch 7.12.1 running in docker on port 9002 and am using an intercepting proxy on port 8080 to check what's going over the wire.

So this simple code

public class App {

    public static void main(String[] args) throws IOException {

        // Create the low-level client
        RestClient restClient = RestClient.builder(new HttpHost("localhost", 9200))
            .setHttpClientConfigCallback(builder -> {
                builder.setProxy(new HttpHost("localhost", 8080));
                return builder;
            })
            .build();

        // Create the transport that provides JSON and http services to API clients
        Transport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());

        // And create our API client
        ElasticsearchClient client = new ElasticsearchClient(transport);

        // Search all items in an index that contains documents of type AppData
        SearchResponse<AppData> search = client.search(s -> s
                .index("appdata-index"),
            AppData.class
        );

        if (search.hits().hits().isEmpty()) {
            System.out.println("No match");
        } else {
            for (Hit<AppData> hit : search.hits().hits()) {
                processAppData(hit._source());
            }
        }
    }

    private static void processAppData(AppData appData) {
        if (appData != null) {
            System.out.println(appData.toString());
        }
    }
}

leads to this error:

20:06:06: Executing task 'App.main()'...

> Task :compileJava UP-TO-DATE
> Task :processResources NO-SOURCE
> Task :classes UP-TO-DATE

> Task :App.main()
Exception in thread "main" java.lang.NullPointerException: took
	at java.base/java.util.Objects.requireNonNull(Objects.java:233)
	at co.elastic.clients.elasticsearch._global.SearchResponse.<init>(SearchResponse.java:105)
	at co.elastic.clients.elasticsearch._global.SearchResponse$Builder.build(SearchResponse.java:381)
	at co.elastic.clients.elasticsearch._global.SearchResponse$Builder.build(SearchResponse.java:365)
	at co.elastic.clients.json.JsonpObjectBuilderParser.parse(JsonpObjectBuilderParser.java:43)
	at co.elastic.clients.json.JsonpValueParser.parse(JsonpValueParser.java:91)
	at co.elastic.clients.base.RestClientTransport.getHighLevelResponse(RestClientTransport.java:175)
	at co.elastic.clients.base.RestClientTransport.performRequest(RestClientTransport.java:65)
	at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1227)
	at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1246)
	at com.sothawo.elasticjavatest.App.main(App.java:39)

That's when parsing the response, not even on parsing the returned data.

In the proxy I see the following request

POST http://localhost:9200/appdata-index/_search HTTP/1.1
Content-Length: 2
Content-Type: application/json; charset=UTF-8
Proxy-Connection: Keep-Alive
User-Agent: elasticsearch-java/7.12.1-SNAPSHOT (Java/16.0.1)
X-Elastic-Client-Meta: es=7.12.1-SNAPSHOT,jv=16,t=7.12.1-SNAPSHOT,hc=4.1.4
Host: localhost:9200

{}

and this response:

HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 408

{
    "took": 2,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 2,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": "appdata-index",
                "_type": "_doc",
                "_id": "5sko2HkB1yiMr4eS5Dmo",
                "_score": 1.0,
                "_source": {
                    "foo": "foo1",
                    "bar": "bar1"
                }
            },
            {
                "_index": "appdata-index",
                "_type": "_doc",
                "_id": "58ko2HkB1yiMr4eS-jlu",
                "_score": 1.0,
                "_source": {
                    "foo": "foo2",
                    "bar": "bar2"
                }
            }
        ]
    }
}

So the request and response over the wire are ok. Taking the proxy out yields the same result.

What could be the reason for this error?

Scroll API support

How do you implement the equivalent of

POST /_search/scroll 
{
    "scroll" : "1m", 
    "scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ==" 
}

with this library?

Then the following code:

client.clearScroll(new ClearScrollRequest.Builder().scrollId(scrollId).build());

where scrollId is not null, raises an ApiException with root cause error:

Validation Failed: 1: no scroll ids specified;"

Why?

How to add documents in bulk operation?

I only found a method for adding single doc: co.elastic.clients.elasticsearch.core.BulkRequest.Builder::addDocument

No java docs ?

DO we have documentation for this?

Directory structure of elasticsearch-java-8.0.0-SNAPSHOT.jar contains `/./`

Not sure if this is an issue with the generated code, my IntelliJ setup or perhaps a bug in IntelliJ I am using the current version 2021.1.1).

I just want to write the code to create a SearchRequest.Builder and add some values.

When I have entered

SearchRequest.Builder builder = new SearchRequest.Builder();
builder.i

just after entering the i after the period, when IntelliJ IDEA is starting to autocomplete, the UI freezes and in the IntelliJ logs I can see, that the JVM is running out of heap space. Happens on two machines, the first has IDEA configured with 8GB heap, the second with 16GB.

Is this working for other people using IntelliJ IDEA?

How to get aggregation result in searchTemplate [7.16.2]

7.16，Elasticsearch Java API Client

SearchTemplateResponse<Object> response = ElasticSearchClient.searchTemplate(request, Object.class);

The response just have the hits property to get the search result, but how to get aggregation result ?

Nested aggregation is missing sub aggregations in result

I have created a Search Request with an Aggregation on a nested object. ES server returns the correct response (as seen from the logs / trace). However, the client does not parse the response completely, and the sub aggregations are missing

Request

http://localhost:9200/my_providers/_search?typed_keys=true' -d '{"aggregations":{"agg_services":{"aggregations":{"grp_by_service_slug":{"aggregations":{"nested_agg_attributes":{"aggregations":{"grp_by_service_attr_name":{"terms":{"field":"providerServiceDTOList.providerServiceAttrValDTOSet.attrName"}},"grp_by_service_attr_value":{"terms":{"field":"providerServiceDTOList.providerServiceAttrValDTOSet.attrValValue"}}},"nested":{"path":"providerServiceDTOList.providerServiceAttrValDTOSet"}}},"terms":{"field":"providerServiceDTOList.serviceSlug"}}},"nested":{"path":"providerServiceDTOList"}}},"query":{"term":{"marketplaceId":{"value":1}}}}

Response from Server

I have removed the data in the hits[] for brevity

{"took":3,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":1.0,"hits":[]},"aggregations":{"nested#agg_services":{"doc_count":2,"sterms#grp_by_service_slug":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"appliances-repair","doc_count":1,"nested#nested_agg_attributes":{"doc_count":2,"sterms#grp_by_service_attr_name":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"Product Type","doc_count":2}]},"sterms#grp_by_service_attr_value":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"Refrigerator","doc_count":1},{"key":"Washing Machine","doc_count":1}]}}},{"key":"carpenter","doc_count":1,"nested#nested_agg_attributes":{"doc_count":2,"sterms#grp_by_service_attr_name":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"Service Type","doc_count":2}]},"sterms#grp_by_service_attr_value":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"Doors and Windows","doc_count":1},{"key":"Modular Kitchen","doc_count":1}]}}}]}}}}

Response as parsed by the client, and made available for use in API

• Only docCount is available

How to print search query?

hi,

before used "elasticsearch-rest-high-level-client.jar", in SearchSourceBuilder.java can print the DSL query

SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(boolQuery);
...
logger.info("=====DSL====="+searchSourceBuilder.toString());

I‘m trying "elasticsearch-java.jar" , please tell me how to print the DSL SearchRequest query?

SearchRequest searchRequest = SearchRequest.of(s -> s
                .index(indexName)
                .query(q -> q
                        .term(t -> t
                                .field("goodsId")
                                .value(v -> v.stringValue("1"))
                        )));

i',m debugging. no found the record。 please help！

Wrapper query implementation missing

Client version 7.16.1:

It seems that the co.elastic.clients.elasticsearch._types.query_dsl.Query is missing the wrapper query type (https://www.elastic.co/guide/en/elasticsearch/reference/7.16/query-dsl-wrapper-query.html).

SortOptions not implement function deserialize(parser, mapper, event)

When deserialize the json to SearchRequest

{
"size": 9999,
"query": {
"match_all": {}
},
"sort": [
{
"modify_time": {
"order": "desc"
}
}
],
"track_total_hits": 2147483647
}
Array deserializer use function deserialize(parser, mapper, event) to deserialize each item in sort block, it will be failed because SortOptions not implement this function

NPE while getting doc with GET

Hi gyus!

I know that project is still in progress, but got some problems in simple usages.
Gets NPE in ElasticsearchError.class constructor when requesting nonexistent doc through get. Fails on Objects.requireNonNull(builder.status, "status") validation

client.get(s -> s.index(index).id("asd"), Document.class)

How to create a range query

I create a client to perform a search and most of the builder types build a type of query except for range:

client.search({ s ->
   s.index("myindex")
     .query { q -> q.range(someJsonValue) }

Most of the functions on the "query builder" allow something to be built, where range wants a JsonValue, how do we build that JsonValue?

Unable to use in a Java EE 8 environment (javax.json vs. jakarta.json)

The co.elastic.clients.json.JsonpMapper class already uses the "new" jakarta.json.* classes (which are implemented in org.glassfish:jakarta.json). This causes an issue when trying to use the elasticsearch client in an environment which still has the "old" implementation org.glassfish:javax.json on the classpath (which is the case for most if not any Java EE 8 application server).

This is because there is a class "org.glassfish.json.JsonProviderImpl" in both org.glassfish:javax.json and org.glassfish:jakarta.json. One uses javax.json classes and the other one uses jakarta.json classes.

The elastic client now (in class JsonValueParser) simply calls jakarta.json.spi.JsonProvider.provider() which basically calls:

Class.forName("org.glassfish.json.JsonProviderImpl");

However, in an Java EE 8 environment, this will return the JsonProviderImpl from org.glassfish:javax.json because the classes unfortunately share the same name and package.

During runtime, this will lead to an exception:

jakarta.json.JsonException: Provider org.glassfish.json.JsonProviderImpl could not be instantiated: java.lang.ClassCastException: org.glassfish.json.JsonProviderImpl cannot be cast to jakarta.json.spi.JsonProvider

I have created a post on StackOverflow to ask if there is any possibility to use both in the same application. However, I do not expect there to be a solution unless the implementation of the "new" "org.glassfish.json.JsonProviderImpl" (in org.glassfish:jakarta.json) is moved to a different package or renamed. (I am also not sure why the authors of org.glassfish:jakarta.json chose to reuse the same package and class name for the new version).

However, in order to use the Elastic Client in Java EE 8 environments, there should be some sort of workaround for the time being: Maybe there could be two builds - one using the jakarta.json.* classes and one using the javax.json.* classes (or potentially using the maven shade plugin).

Any ideas?