Elastic client issue about iudx-resource-server

[abhi4578]

I can trace back the origins of failure of requests from 08 Feb . Initially it was a connection timeout issue - , that would have triggered. Attached detailed log of origins elastic-execptions-origins.txt.

time=2022-02-08T09:55:41+0000 level="�[1;31mERROR�[m" loggerName=iudx.resource.server.database.archives.ElasticClient message="Timeout connecting to [elasticsearch-mcd-headless.elastic.svc.cluster.local/10.42.153.136:9200]" source=rs sourceUrl=https://rs.iudx.org.in exception=""
022-02-08 15:25:41	
ERROR [InternalHttpAsyncClient:66] 2022-02-08 09:55:41,032 - I/O reactor terminated abnormally
	
2022-02-08 15:25:41	
java.lang.StringIndexOutOfBoundsException: begin -1, end 0, length 26
	
2022-02-08 15:25:41	
	at java.base/java.lang.String.checkBoundsBeginEnd(Unknown Source)
	
2022-02-08 15:25:41	
	at java.base/java.lang.String.substring(Unknown Source)
	
2022-02-08 15:25:41	
	at iudx.resource.server.database.archives.ElasticClient$2.onFailure(ElasticClient.java:161)
	
2022-02-08 15:25:41	
	at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:598)
	
2022-02-08 15:25:41	
	at org.elasticsearch.client.RestClient$1.failed(RestClient.java:359)
	
2022-02-08 15:25:41	
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137)
	
2022-02-08 15:25:41	
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:101)
	
2022-02-08 15:25:41	
	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426)
	
2022-02-08 15:25:41	
	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.connectionRequestFailed(AbstractClientExchangeHandler.java:348)
	
2022-02-08 15:25:41	
	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.access$100(AbstractClientExchangeHandler.java:62)
	
2022-02-08 15:25:41	
	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler$1.failed(AbstractClientExchangeHandler.java:392)
	
2022-02-08 15:25:41	
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137)
	
2022-02-08 15:25:41	
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$1.failed(PoolingNHttpClientConnectionManager.java:316)
	
2022-02-08 15:25:41	
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137)
	
2022-02-08 15:25:41	
	at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:169)
	
2022-02-08 15:25:41	
	at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:628)
	
2022-02-08 15:25:41	
	at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:894)
	
2022-02-08 15:25:41	
	at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:184)
	
2022-02-08 15:25:41	
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:214)
	
2022-02-08 15:25:41	
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:158)
	
2022-02-08 15:25:41	
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
	
2022-02-08 15:25:41	
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	
2022-02-08 15:25:41	
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	
2022-02-08 15:25:41	
	at java.base/java.lang.Thread.run(Unknown Source)
	
2022-02-08 15:25:41	
time=2022-02-08T09:55:41+0000 level="�[1;31mERROR�[m" loggerName=iudx.resource.server.database.archives.ElasticClient message="Timeout connecting to [elasticsearch-mcd-headless.elastic.svc.cluster.local/10.42.153.136:9200]" source=rs sourceUrl=https://rs.iudx.org.in exception=""

• The above timeout issue -> made the request fail and goto OnFailure method -> which in turn triggered string exception -> which triggered reactor stop --> resulting in failure in subsequent requests --> OnFailure method --> ...

[swaminathanvasanth]

It was for all the requests made to the Elastic client @abhi4578
Looping @rraks

[abhi4578]

The reason like for alternate query suceeding is, in a pod two verticle instances of a service/verticle is deployed by default. So this issue trigger would have happened in one of verticle instance of database service/verticle but not the other.

[abhi4578]

Handling the string out of bounds exception should solve the problem. at In current-deployment branch, at the code

iudx-resource-server/src/main/java/iudx/resource/server/database/archives/ElasticClient.java

Line 161 in 03915b1

String error = e.getMessage().substring(e.getMessage().indexOf("{"),

and at current master branch at

iudx-resource-server/src/main/java/iudx/resource/server/database/archives/elastic/ElasticClient.java

Line 172 in b239da2

String error = e.getMessage().substring(e.getMessage().indexOf("{"),

[abhi4578]

but can any other exceptions happen in that method? if so can have a generalized exception catacher inside failure method, so that no exception leave onFailure method .

[abhi4578]

This also should be fixed in other parts of elastic client.

iudx-resource-server/src/main/java/iudx/resource/server/database/archives/ElasticClient.java

Line 92 in 03915b1

public void onFailure(Exception e) {

iudx-resource-server/src/main/java/iudx/resource/server/database/archives/elastic/ElasticClient.java

Line 103 in b239da2

public void onFailure(Exception e) {

This issue might happen in other IUDX servers which uses elastic client - file-server, catalogue. It needs to be checked the against the possible methods and fixed appropriately.

[kailash]

fixed and resolved