after ingesting for about 20 minutes solandra throws protocol violation error about solandra

[ceocoder]

little more background,

4 machine cluster with replicatin_level =1,

jvm set to 6G (I've tried with jvm at 12 and 18 to same effect)
on boxes swappiness is set to 0

for disks I have 3PAR iSCSI mounts of 500GB each

[tjake]

Looks like you had a big GC that caused the reads to back up and the JVM was running out of memory.

What kind of writes per second were you seeing? How many solr cores?

[ceocoder]

UPDATE

Turning off row_cache completely and disabling swap I was able to ingest using one of the 4 nodes for about 15 hours and counting, and just started ingesting on a second node, and keep getting this error frequently -

ERROR

10:09:26,663 INFO SolrCore:1324 - [htmlnew] webapp=/solandra path=/update params={commit=true} status=500 QTime=1112
10:09:26,664 ERROR SolrDispatchFilter:139 - java.lang.IllegalStateException: Unable to reserve an id
at lucandra.cluster.CassandraIndexManager.getNextId(CassandraIndexManager.java:341)
at solandra.SolandraIndexWriter.addDoc(SolandraIndexWriter.java:194)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61)
at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at solandra.SolandraDispatchFilter.execute(SolandraDispatchFilter.java:170)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at solandra.SolandraDispatchFilter.doFilter(SolandraDispatchFilter.java:134)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:536)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:930)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:747)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)

[tjake]

This should be fixed. Please retest

[tjake]

Verified fix