Solandra: Issue with shards about solandra

[compfix]

I was able to get past the original issue by changing the String shard = addr.getHostAddress() + ":8983/solr/" + indexName + "" + i; to String shard = addr.getHostAddress() + ":8983/solandra/" + indexName + "" + i;

However we are now getting a null pointer exception:

15:18:58,292 ERROR SolrCore:139 - java.lang.NullPointerException
at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:462)
at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:298)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:290)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at solandra.SolandraDispatchFilter.execute(SolandraDispatchFilter.java:169)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at solandra.SolandraDispatchFilter.doFilter(SolandraDispatchFilter.java:133)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:722)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:404)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:206)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:828)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:514)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)

when we run a query.

[tjake]

Did you update recently? I may have broken this while renaming things, I'll look at this tomorrow on the plane

[tjake]

Well I added your fix for now. Have your upgraded all nodes to the latest?

[compfix]

I'm planning on pulling your latest tomorrow. I think my issue right now is that some of the documents come back with null column families which cause the returned document to be null. Perhaps an issue with cassandra? I'll let you know if my results are better after I pull the latest.

Thanks,

Cliff

[compfix]

Jake,

It appears as those Solandra is based on the assumption that there will be multiple nodes in the cassandra cluster that will store the "shards". Is this the case? What if we only have one node? Can we prevent Solandra from thisnking there are multiple shards once our doc count goes beyond the 2^17 limit on maxDocsPerShard ?

Thanks,

Cliff

[tjake]

Yes, just set SHARDS_AT_ONCE="1" in start-solandra.sh

[compfix]

I did see that setting and it has been set that way. Unfortunately we still experience a problem. Based on the log information, it seems to pull a result from the database, however it fails to merge the results to present to the user because it reports "Missing document in multiget_slice for" ... I'm not sure if this is because the doc isn't in the database or because it thinks it actually resides in a different shard the cassandra isn't aware of.

I'm able to reproduce this fairly easily but haven't made much headway in determining root cause. Any guidance on this would be appreciated.

Thanks,

Cliff

[tjake]

Can you post the log?

[compfix]

Here's the excerpt from the log when I query: (There are two Solr cores one called config and one called log) The one called log is the one with issues.

16:17:57,306 INFO SolandraCoreContainer:72 - Loading Solandra core: log0
16:17:57,306 INFO SolandraCoreContainer:72 - Loading Solandra core: log1
16:17:57,338 INFO SolandraCoreContainer:72 - Loading Solandra core: config
16:17:57,340 INFO UpdateRequestProcessor:171 - {add=[3e7d197b-2582-4847-a6fa-6bea1b13aecf:current]} 0 2
16:17:57,360 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b88
16:17:57,342 INFO SolandraCoreContainer:72 - Loading Solandra core: log
16:17:57,362 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b89
16:17:57,362 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b82
16:17:57,362 INFO SolrCore:1324 - [config] webapp=/solandra path=/update params={} status=0 QTime=2
16:17:57,363 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b83
16:17:57,364 INFO UpdateRequestProcessor:171 - {add=[bc28ab8a-c92d-40b5-b164-d6cd028b92f1]} 0 2
16:17:57,364 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b80
16:17:57,364 INFO SolrCore:1324 - [log] webapp=/solandra path=/update params={} status=0 QTime=2
16:17:57,365 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b81
16:17:57,365 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b86
16:17:57,365 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b87
16:17:57,365 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b84
16:17:57,365 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b85
16:17:57,366 INFO SolrCore:1324 - [log] webapp=/solandra path=/select params={fl=id,score&start=0&q=uuid:&isShard=true&wt=javabin&fsv=true&rows=10&version=1} hits=547 status=0 QTime=60
16:17:57,389 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿189
16:17:57,393 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿188
16:17:57,393 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿183
16:17:57,393 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿182
16:17:57,393 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿181
16:17:57,394 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿180
16:17:57,394 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿187
16:17:57,394 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿186
16:17:57,394 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿185
16:17:57,394 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿184
16:17:57,394 INFO SolrCore:1324 - [log] webapp=/solandra path=/select params={fl=id,score&start=0&q=uuid:&isShard=true&wt=javabin&fsv=true&rows=10&version=1} hits=33536 status=0 QTime=88
16:17:57,396 INFO SolandraCoreContainer:72 - Loading Solandra core: config
16:17:57,401 INFO SolandraCoreContainer:72 - Loading Solandra core: log
16:17:57,402 INFO UpdateRequestProcessor:171 - {add=[29595742-3036-4dab-bd38-c7fecec28da0:current]} 0 5
16:17:57,402 INFO SolrCore:1324 - [config] webapp=/solandra path=/update params={} status=0 QTime=5
16:17:57,408 INFO UpdateRequestProcessor:171 - {add=[a0c7f456-5a98-4ed7-bbac-3ef8cbc4f146]} 0 6
16:17:57,408 INFO SolrCore:1324 - [log] webapp=/solandra path=/update params={} status=0 QTime=6
16:17:57,408 ERROR SolrCore:139 - java.lang.NullPointerException
at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:462)
at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:298)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:290)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at solandra.SolandraDispatchFilter.execute(SolandraDispatchFilter.java:169)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at solandra.SolandraDispatchFilter.doFilter(SolandraDispatchFilter.java:133)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:722)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:404)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:206)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:828)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:514)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)

16:17:57,408 INFO SolrCore:1324 - [log] webapp=/solandra path=/select params={q=uuid:*} status=500 QTime=568
16:17:57,409 ERROR SolrDispatchFilter:139 - java.lang.NullPointerException
at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:462)
at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:298)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:290)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at solandra.SolandraDispatchFilter.execute(SolandraDispatchFilter.java:169)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at solandra.SolandraDispatchFilter.doFilter(SolandraDispatchFilter.java:133)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:722)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:404)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:206)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:828)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:514)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)

16:17:57,438 INFO SolandraCoreContainer:72 - Loading Solandra core: config

You can get the full log here:
https://docs.google.com/leaf?id=0BxyBcfQyaiLCMzM1NTA1NmMtZjc3My00NGY1LWJlZWMtMDFkOGQ4YzU2MDk1&hl=en&authkey=CNriscMO

[tjake]

Hmm, have you been deleting documents? could be a bug in the delete logic leaving behind old references

[compfix]

No. Everything is an add at this point. We aren't deleting anything.

[tjake]

What changes have you made to the code and/or settings?

BTW, you can chat me directly on irc freenode #solandra

[tjake]

Could you also post the schema.xml you are using?

[tjake]

This ended up being the in schema.xml wasn't marked stored="true"