collective / collective.solr

Solr search engine integration for Plone

collective.solr - Solr integration for the Plone CMS

collective.solr integrates the Solr search engine with Plone.

Apache Solr is based on Lucene and is the enterprise open source search engine. It powers the search of sites like Twitter, the Apple and iTunes Stores, Wikipedia, Netflix and many more.

Solr does not only scale to any level of content, but provides rich search functionality, like faceting, geospatial search, suggestions, spelling corrections, indexing of binary formats and a whole variety of powerful tools to configure custom search solutions. It has integrated clustering and load-balancing to provide a high level of robustness.

collective.solr comes with a default configuration and setup of Solr that makes it extremely easy to get started, yet provides a vastly superior search quality compared to Plone's integrated text search based on ZCTextIndex.

Features

Solr Features

• Schema and Schemaless Configuration
• Information Retrieval System
• Speed (in comparission to ZCTextIndex)

Features of Solr Integration into Plone

Search Enhancements

• Facets
• Indexing of binary documents
• Spellchecking / suggestions
• Wildcard searches
• Exclude from search
• Elevation

Detailed Documentation

A full Documentation of the Solr integration of Plone could be found on collectivesolr.readthedocs.org.

Installation & Configuration

Download the latest default Solr configuration from github:

$ wget https://github.com/collective/collective.solr/raw/master/solr.cfg
$ wget https://raw.githubusercontent.com/collective/collective.solr/master/solr-4.10.x.cfg

Extend your buildout to use those files and make sure collective.solr is added to the eggs in your instance section. Your full buildout file should look something like this:

[buildout]
parts += instance
extends =
    https://dist.plone.org/release/4.3.8/versions.cfg
    solr.cfg
    solr-4.10.x.cfg

[instance]
recipe = plone.recipe.zope2instance
http-address = 8080
user = admin:admin
eggs =
    Plone
    collective.solr

[versions]
collective.recipe.solrinstance = 5.3.2

After saving this to let's say buildout.cfg, the buildout can be run and the Solr server and Plone instance started:

$ python bootstrap-buildout.py
$ bin/buildout
...
$ bin/solr-instance start
$ bin/instance start

Next you should activate the collective.solr (site search) add-on in the add-on control panel of Plone. After activation you should review the settings in the new Solr Settings control panel. To index all your content in Solr you can call the provided maintenance view:

http://localhost:8080/plone/@@solr-maintenance/reindex

Solr connection configuration in ZCML

The connections settings for Solr can be configured in ZCML and thus in buildout. This makes it easier when copying databases between multiple Zope instances with different Solr servers.

Example:

zcml-additional =
    <configure xmlns:solr="http://namespaces.plone.org/solr">
        <solr:connection host="localhost" port="8983" base="/solr/plone"/>
   </configure>

Current Project Status

The code is used in production in many sites and considered stable. This add-on can be installed in a Plone 4.3 (or later) site to enable indexing operations as well as searching (site and live search) using Solr. Doing so will not only significantly improve search quality and performance -especially for a large number of indexed objects, but also reduce the memory footprint of your Plone instance by allowing you to remove the SearchableText, Description and Title indexes from the catalog as well as the lexicons if no other indexes are using them.

In large sites with 100000 content objects and more, searches using ZCTextIndex often taken 10 seconds or more and require a good deal of memory from ZODB caches. Solr will typically answer these requests in 10ms to 50ms at which point network latency and the rendering speed of Plone's page templates are a more dominant factor.

Plone Compatibility

collective.solr works with Plone 5.2 and Plone 6.

Python Compatibility

collective.solr works with Python 3.8 and 3.9. Older versions might still work but we do not test them.

Solr Compatibility

collective.solr works with Solr 7,8, and 9. Older versions might work as well but we do not test them.

Bug Reporting & Development

Releases can be found on the Python Package Index at https://pypi.org/project/collective.solr. The code and issue trackers can be found on GitHub at https://github.com/collective/collective.solr.

For outstanding issues and features remaining to be implemented please see the issue tracker__.

collective.solr's Issues

Two SolrException classes

Seems that I rushed to create a src/collective/solr/exceptions.py SolrException base class but there exists one on src/collective/solr/solr.py as well.

Geospatial search support.

I implemented support for geospatial search in c.solr:

https://github.com/collective/collective.solr/tree/spatial-filters

Though, there are a few pieces missing and I currently don't have time to continue to work on it. If anybody is interested in this, let me know.

commit with parameter "waitFlush" raises exception

Using solr v4.5.1 and collective.solr v4.0.2, the commit to solr after page creation fails, because the commit function (solr.py:236) uses the removed parameter "waitFlush".

Add Solr Test Fixtures

There should be one test fixture that starts/stops solr and another fixture that configures collective.solr.

The test fixture should use a "test" Solr core, otherwise tests will clear the real solr data (on your local machine).

We also have to make sure that solr is properly stopped, even if we just run a single test.

The fixture should be usable for both "unit" tests against the solr configuration and for integration, functional and robot-framework-based acceptance tests.

collective.solr should use that test fixture to replace the alltests script.

Todo:

• Add SolrFixture (for Unit Tests)
• Add SolrPloneFixture (for Integration, Functional, Acceptance Tests)
• Document test fixtures
• Make sure test teardown works when running single tests
• Make tests use a test solr core
• Make collective.solr use the new fixtures and replace the alltests script.

Get rid of activateAndReindex method in testing.py

https://github.com/collective/collective.solr/blob/master/src/collective/solr/testing.py#L171

Almost all tests in c.solr require Solr activated in the control panel. Therefore the activation should happen in the test setup, to simplify things. We can specifically deactivate Solr in a test if we need to.

I guess this method is just legacy code from the pre-plone.app.testing era. We could first rename the method to just "reindex" and then either keep it, or automatically reindex on test setup/teardown.

@do3cc @witsch opinions?

Docs need review: not following best practices and style guide yet

Hi,

before we can include them into docs.plone.org, we have to review the, a 'quick' review showed that currently the docs are not following best practices and our style guide in terms of rst syntax, semantic linefeeds and some more stuff :)

I say this is a perfect topic to sprint :)

Move @@solr-controlpanel to use plone.app.registry.

Disable option to set solr hostname, port and core TTW if this has been set on ZCML.

Solr is aimed at larger deployments. Allowing to set server configuration TTW is superfluous. ZCML configuration should be the only way to configure solr.

If people care much about TTW configuration and are willing to work on that, an alternative would be to display the zcml settings on the control panel. Though, personally I don't think this is worth the effort.

support multiple instances

I have multiple plone instances and tried to use a separate solr core for each.
But apparently this does not work.
Although I can configure different values for host and solr base this has no effect.
The reason seems to be that SolrConnectionManager.getConnection reuses old connections even if they are create by a different plone instance.

File uploading fails after installing collective.solr on Plone 4.1.3

After installing collective.solr 4.1.0 on Plone 4.1.3, I am unable to upload new Files or Images;
The traces is as follows:

Traceback (innermost last):

Module ZPublisher.Publish, line 126, in publish
Module ZPublisher.mapply, line 77, in mapply
Module ZPublisher.Publish, line 46, in call_object
Module Products.CMFPlone.FactoryTool, line 453, in call
Module ZPublisher.mapply, line 77, in mapply
Module ZPublisher.Publish, line 46, in call_object
Module Products.CMFFormController.FSControllerPageTemplate, line 91, in call
Module Products.CMFFormController.BaseControllerPageTemplate, line 28, in _call
Module Products.CMFFormController.ControllerBase, line 231, in getNext
Module Products.CMFFormController.Actions.TraverseTo, line 38, in call
Module ZPublisher.mapply, line 77, in mapply
Module ZPublisher.Publish, line 46, in call_object
Module Products.CMFFormController.FSControllerPythonScript, line 107, in call
Module Products.CMFFormController.ControllerBase, line 231, in getNext
Module Products.CMFFormController.Actions.TraverseTo, line 38, in call
Module ZPublisher.mapply, line 77, in mapply
Module ZPublisher.Publish, line 46, in call_object
Module Products.CMFFormController.FSControllerPythonScript, line 105, in call
Module Products.CMFFormController.Script, line 145, in call
Module Products.CMFCore.FSPythonScript, line 130, in call
Module Shared.DC.Scripts.Bindings, line 322, in call
Module Products.PloneHotfix20121106.python_scripts, line 63, in _patched_bindAndExec
Module Shared.DC.Scripts.Bindings, line 359, in _bindAndExec
Module Products.PythonScripts.PythonScript, line 344, in _exec
Module script, line 1, in content_edit
<FSControllerPythonScript at ..../portal_factory/File/file.2015-09-29.7961599938>
Line 1
Module Products.CMFCore.FSPythonScript, line 130, in call
Module Shared.DC.Scripts.Bindings, line 322, in call
Module Products.PloneHotfix20121106.python_scripts, line 63, in _patched_bindAndExec
Module Shared.DC.Scripts.Bindings, line 359, in _bindAndExec
Module Products.PythonScripts.PythonScript, line 344, in _exec
Module script, line 13, in content_edit_impl
<FSPythonScript at ..../content_edit_impl used for..../portal_factory/File/file.2015-09-29.7961599938>
Line 13
Module Products.LinguaPlone.I18NBaseObject, line 420, in processForm
Module Products.Archetypes.BaseObject, line 661, in processForm
Module Products.Archetypes.BaseObject, line 769, in _renameAfterCreation
Module transaction._manager, line 101, in savepoint
Module transaction._transaction, line 260, in savepoint
Module transaction._transaction, line 257, in savepoint
Module transaction._transaction, line 690, in init
Module ZODB.Connection, line 1123, in savepoint
Module ZODB.Connection, line 623, in _commit
Module ZODB.Connection, line 658, in _store_objects
Module ZODB.serialize, line 422, in serialize
TypeError: Can't pickle objects in acquisition wrappers.

I have tried to do a clear and rebuild of the catalog, but I get the following error:

Traceback (innermost last):

Module ZPublisher.Publish, line 134, in publish
Module Zope2.App.startup, line 301, in commit
Module transaction._manager, line 89, in commit
Module transaction._transaction, line 323, in commit
Module transaction._transaction, line 393, in _callBeforeCommitHooks
Module collective.indexing.transactions, line 52, in before_commit
Module collective.indexing.queue, line 163, in process
Module collective.solr.indexer, line 212, in index
Module collective.solr.indexer, line 138, in call
Module collective.solr.indexer, line 116, in getpath
AttributeError: blob

BinaryAdder write on Solr before the end (commit) of Zope's transaction

The case is:

create a File object
do a catalog search during same Zope transaction, this will fire the emptying indexing queue by collective.indexing and a call to BinaryAdder that do a doPost (update/extract method) on Solr
abort the Zope’s transaction

Now the problem is that the file is indexed on Solr, but not exists on Zope DB.

Another case:

modify an existing File
do a search catalog (again BinaryAdder sends updated data to Solr)
abort the Zope’s transaction

In this case Solr and Zope DB will have different data for same object

Utility registered with wrong name

In collective/solr/configure.zcml, the utlity for the titles vocabulary is registered with the wrong name ("portal_types") - it should be "portal_type".

Language specific solr configurations.

c.solr or c.r.solrinstance could be shipped with language specific Solr configurations or examples. English stemming algorithms for instance do not work well in other languages (e.g. German).

Those configurations should be tested (on travis) and included directly in the docs to make sure we have working and up-to-date configurations in our docs.

Ability to clear/cleanup/reindex a single core.

When working with multiple Solr cores, it would be nice to be able to clear/cleanup/reindex just a single core.

This is somehow related to #21.

Default buildout fails

If I bootstrap/buildout the buildout.cfg it fails with:

%> ./bin/buildout                                                          ~/pkgs/bern.web/src/collective.solr
Upgraded:
  zc.buildout version 2.3.1;
restarting.
Generated script '/home/userpkgs/bern.web/src/collective.solr/bin/buildout'.
Develop: '/home/userpkgs/bern.web/src/collective.solr/.'
warning: no previously-included files found matching 'better-defaults.diff'
warning: no previously-included files found matching 'bootstrap-buildout.py'
warning: no previously-included files found matching '*.cfg'
warning: no previously-included files found matching '.coveragerc'
warning: no previously-included files found matching 'docs/Makefile'
warning: no previously-included files found matching '.gitattributes'
warning: no previously-included files found matching 'tox.ini'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '.*py.un~' found anywhere in distribution
warning: no files found matching '*.txt'
no previously-included directories found matching 'buildout'
no previously-included directories found matching 'docs/source'
no previously-included directories found matching 'etc'
Getting distribution for 'plone.recipe.codeanalysis==2.0b1'.
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '**/.DS_Store' found anywhere in distribution
warning: no previously-included files matching '**/__pycache__' found anywhere in distribution
Got plone.recipe.codeanalysis 2.0b1.
Getting distribution for 'flake8==2.4.1'.
zip_safe flag not set; analyzing archive contents...
flake8.reporter: module references __file__
flake8.tests._test_warnings: module references __file__
flake8.tests.test_integration: module references __file__
Got flake8 2.4.1.
Getting distribution for 'check-manifest'.
Got check-manifest 0.25.
Getting distribution for 'collective.recipe.sphinxbuilder'.
warning: no files found matching '*.txt'
warning: no previously-included files matching '*pyc' found anywhere in distribution
Got collective.recipe.sphinxbuilder 0.8.2.
Installing _mr.developer.
Generated script '/home/userpkgs/bern.web/src/collective.solr/bin/develop'.
Installing solr-download.
solr-download: Extracting package to /home/userpkgs/bern.web/src/collective.solr/parts/solr-download
Installing solr-instance.
solr-instance: Generated file 'jetty.xml'.
solr-instance: Generated file 'log4j.properties'.
solr-instance: Generated file 'logging.properties'.
While:
  Installing solr-instance.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/home/user.buildout/eggs-cache/zc.buildout-2.3.1-py2.7.egg/zc/buildout/buildout.py", line 1946, in main
    getattr(buildout, command)(args)
  File "/home/user.buildout/eggs-cache/zc.buildout-2.3.1-py2.7.egg/zc/buildout/buildout.py", line 626, in install
    installed_files = self[part]._call(recipe.install)
  File "/home/user.buildout/eggs-cache/zc.buildout-2.3.1-py2.7.egg/zc/buildout/buildout.py", line 1370, in _call
    return f()
  File "/home/user.buildout/eggs-cache/collective.recipe.solrinstance-5.3.2-py2.7.egg/collective/recipe/solrinstance/__init__.py", line 625, in install
    directoryFactory=self.solropts['directoryFactory'],
  File "/home/user.buildout/eggs-cache/collective.recipe.solrinstance-5.3.2-py2.7.egg/collective/recipe/solrinstance/__init__.py", line 471, in generate_solr_conf
    name='solrconfig.xml', **kwargs)
  File "/home/user.buildout/eggs-cache/collective.recipe.solrinstance-5.3.2-py2.7.egg/collective/recipe/solrinstance/__init__.py", line 439, in _generate_from_template
    with open(output_file, 'wb') as outfile:
IOError: [Errno 2] No such file or directory: '/home/userpkgs/bern.web/src/collective.solr/parts/solr-instance/solr/conf/solrconfig.xml'

Well, the solrconfig.xml is in ..solr-instance/solr/collection1/conf....
It seems like the receipe does not recognise the right solr version.

Any suggestions how to solve this?

I got it running with collective.receipe.solrinstance 6.0.0b3, but I need this test setup to make some contributions.

Travis times out occasionally

e.g. https://travis-ci.org/collective/collective.solr/jobs/3493803

Installing solr-download.


Executing your install (bin/buildout -N) took longer than 15 minutes and was terminated.

For more information about the test timeouts please checkout the section Build Timeouts at http://about.travis-ci.org/docs/user/build-configuration/.

That seems to be a hard limit:: http://about.travis-ci.org/docs/user/build-configuration/#Build-Timeouts

searching for "-something" gives a Bad Request Error

searching for anything with a - (minus) in front gives a Bad Request Error:

     Module zope.tales.pythonexpr, line 59, in __call__                                                                                                                                                                                  
    __traceback_info__: ( view.results(b_start=b_start))                        
    Module <string>, line 1, in <module>                                        
    Module plone.app.search.browser, line 54, in results                        
    Module collective.indexing.monkey, line 84, in searchResults                
    Module collective.solr.monkey, line 31, in searchResults                    
    Module collective.solr.dispatcher, line 45, in __call__                     
    Module collective.solr.dispatcher, line 94, in solrSearchResults            
    __traceback_info__: ({'allowedRolesAndUsers': '+allowedRolesAndUsers:(Manager OR Authenticated OR Anonymous OR user$admin)', 'path_parents': '+path_parents:"\\/plone"', 'SearchableText': '+SearchableText:-1B3'}, {'start': 0, 'row
    Module collective.solr.search, line 72, in search                           
    Module collective.solr.solr, line 246, in search                            
    Module collective.solr.solr, line 119, in doPost                            
    Module collective.solr.solr, line 107, in __errcheck                        

SolrException: HTTP code=400, reason=Bad Request

custom patterns with base_query unreliable, can create invalid syntax

In the configuration panel of solr, I can define a custom pattern for simple search queries. The description offers me to insert the search terms via value or base_value. base_value is stripped of '*'
A query that looks like this: 'movu *' will create an illegal query like this: '(movu) ()'
The PR I am going to link has a fialing test already, and it should be trivial to fix it. But I don't feel 100% confident that removing the star here is the right solution.
@saily

searching with collective.solr does not work if it happens within an async request from plone.app.async

An async request by plone.app.async gets executed without a request.
Without a real request, the search returns SolrFlares instead of PloneFlares.
Without PloneFlares basic things like getObject() don't work.
this, for example, prevents me from using uuidToObject.

use tika for indexing files

tomgross already implemented this.
https://github.com/tomgross/collective.solr

Index comments?

I'm trying collective.solr and our website (www.freitag.de) is heavily used for commenting articles (being a newspaper with an open community as our website is).

So I installed collective.solr, configure it and then hit /@@solr-maintenance/reindex and I don't see any comment being indexed.

Is there any reason not to do so?

Getting started howto.

Write a document about how to get started with c.solr and c.r.solrinstance.

New release 5.0.0

I would really appreciate a new release of collective.solr!

1. We could go ahead with several open PR's
2. I'm able to port more features from ftw.solr to collective.solr.
3. No deployments from master branch anymore 😨 😜

Or someone may giving me access on pypi to the repo and I'm gonna make a new release (4.2.0).

\cc @do3cc @tisto

Elevation support

c.solr or c.r.solrinstance could be shipped with elevation support configuration example.

Those configurations should be tested (on travis) and included directly in the docs to make sure we have working and up-to-date configurations in our docs.

https://cwiki.apache.org/confluence/display/solr/The+Query+Elevation+Component

solr index not being updated

The Plone site's catalog shows new pages (Documents) being created but those new pages do not appear in the Solr index, which I query at localhost:8983/solr

The files in the zeocluster/var/solr/data/index directory are also not being updated (their last mod date time is several weeks old) even after I run @@solr-maintenance/resync and @@solr/maintenance/sync

At some point in the past I must have stumbled on a way to make it work, some combination of the above views and/or stopping and restarting the ZEO client and the solr-instance (via "sudo -u plone_buildout bin/solr-instance start")

The solr.pid file is being updated correctly. The files in the var/solr directory tree are owned by plone_buildout:plone_group (it seemed to be a problem before when they were owned by root).

This is with Plone 4.3 and collective.solr 3.1.

Purpose of this package: API or I-feel-lucky Plone Solr integration?

While polishing this package for a Plone 5 compatible release I found myself asking the purpose of this package. Should it be

1. an API for plugging in other (UI) products or
2. a full fledged solution for all relevant Plone-Solr integration tasks?

I'm tending to go for 1. because of the following reasons:

With an interface to Solr and a complex (react based) UI we have many dependencies which are hard

• to track. We need to make sure newer versions of Solr and UI components are supported and tested.
• to test. We have low level network tests side-by-side with UI tests.
• for new people to start with. You have to understand Plone, Solr and React based UIs
• Searches are mostly custom tailerd. It will we hard to maintain a one-size-fits-all-approach.

I suggest to focus on a stable API and do the following for the next major release:

1. Remove the complex React based search UI and put it in a seperate product. I myself don't use it because I work with eea.facetednavigation and there might be other scenarios.
2. (Re)add a simple BrowserView based search form using the product with some (faceting) enhancements. This is far more easier to test across diferent Plone versions!
   1. Document how to override Plones search UI with a custom one in Plone 4.3 and Plone 5.

Any feedback? @tisto @gforcada

Add link for reindexing (and purging?) to control panel

Plone 5 compatibility

The current master is compatible with Plone 5 and comes with an new React-based search page.

Roadmap:

• Create 5.x branch from current master
• Merge this pr and make it the new master branch
• Release plone.restapi 1.0a1
• Make this pr use plone.restapi 1.0a1
• Fix remaining test isolation issues
• Make sure we do not skip any tests
• Make sure test coverage is > 90 % (as it was before the refactoring)
• Code cleanup
• Remove support for old Plone versions (< 4)
• Remove support for old Solr versions (< 4)
• Remove skins folder
• Remove browser views for Plone 4
• Remove search profile
• Release collective.solr 6.0 with Plone 5 support and a new react-based search UI
• Upgrade step to remove utilities and move settings to registry

Update search view to work with Plone > 4.0.

The search view that c.solr overrides is outdated. It still uses the old "search" view instead of the newer "@@search". We should create a new view that supports facets, autocomplete and suggest.

ISearch utility

When querying Solr a utility ISearch is used for building the search query and actually quering Solr. It was introduced with df834b9 as a global utility
Meanwhile it changed to a local utility because there were issues with AJAX calls to search (together with eea.facetednavigation). Occassionally the utility can not be found and the dispatcher falls back to the catalog implementation.
Here is my question: Does the ISearch instance really have to be a utility? I guess it was introduced for memory saving purpose and/or performance? Or is there a usecase to override it?
Since it is the most stable solution I found I want to replace the utility getter of ISearch in the dispatcher with a standard instance. Any objections, ideas?

Query parameters need to be quoted

While working on #96 (pull request about to be pushed) I noticed that once I have comments being indexed I don't get any result back from Solr.

The problem is that the catalog query I make filters by portal_type and comments' portal type is Discussion Item (note the space).

Putting quotes around it "Discussion Item" solves the problem.

Pull request to come.

UnicodeDecodeError with python 2.7

I get the following traceback with python 2.7

Traceback (innermost last):
  Module ZPublisher.Publish, line 126, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 46, in call_object
  Module collective.solr.browser.maintenance, line 150, in reindex
  Module collective.solr.browser.maintenance, line 49, in checkpointIterator
  Module collective.solr.browser.maintenance, line 122, in checkPoint
  Module collective.solr.indexer, line 125, in __call__
  Module collective.solr.solr, line 144, in flush
  Module collective.solr.solr, line 156, in doSendXML
  Module collective.solr.solr, line 117, in doPost
  Module httplib, line 958, in request
  Module httplib, line 992, in _send_request
  Module httplib, line 954, in endheaders
  Module httplib, line 812, in _send_output
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2274: ordinal not in range(128)

with python 2.6 it works though

Problem with solr-instance - _version_ field must exist in schema

I tried to setup c.solr in existing plone instances, but I stuck on starting sorl-instance. After start it, I see in logs (and when I try to reindex solr):

3011 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.core.CoreContainer  – Error creating core [collection1]: Unable to use updateLog: _version_ field must exist in schema, using indexed="true" or docValues="true", stored="true" and multiValued="false" (_version_ does not exist)
org.apache.solr.common.SolrException: Unable to use updateLog: _version_ field must exist in schema, using indexed="true" or docValues="true", stored="true" and multiValued="false" (_version_ does not exist)
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:881)
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:654)
    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
    at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
    at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unable to use updateLog: _version_ field must exist in schema, using indexed="true" or docValues="true", stored="true" and multiValued="false" (_version_ does not exist)
    at org.apache.solr.update.UpdateLog.init(UpdateLog.java:280)
    at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:134)
    at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:94)
    at org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:100)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:558)
    at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:628)
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
    ... 8 more
Caused by: org.apache.solr.common.SolrException: _version_ field must exist in schema, using indexed="true" or docValues="true", stored="true" and multiValued="false" (_version_ does not exist)
    at org.apache.solr.update.VersionInfo.getAndCheckVersionField(VersionInfo.java:56)
    at org.apache.solr.update.VersionInfo.<init>(VersionInfo.java:82)
    at org.apache.solr.update.UpdateLog.init(UpdateLog.java:277)
    ... 18 more
3013 [main] INFO  org.apache.solr.servlet.SolrDispatchFilter  – user.dir=/home/mstaniszczak/Projekty/portal/parts/solr-instance
3013 [main] INFO  org.apache.solr.servlet.SolrDispatchFilter  – SolrDispatchFilter.init() done
3052 [main] INFO  org.eclipse.jetty.server.AbstractConnector  – Started [email protected]:8983

I thought, that is problem with our instance, so I also tested it on vanilla Plone. But with exactly the same result. Here you are buildout which I used for vanilla instances (buildoud.cfg default, just from paster) :

solr.cfg

[buildout]
extends =
  buildout.cfg
  https://github.com/collective/collective.solr/raw/master/solr.cfg
  https://raw.githubusercontent.com/collective/collective.solr/master/solr-4.10.x.cfg

[instance]
eggs += collective.solr

Any idea what can be reason of that? Maybe I missed something?

ImportError: IIndexableObjectWrapper in Plone5

It doesn't work on Plone5 as there is no plone.app.content.interfaces.IIndexableObjectWrapper.

GS profile which does not set connection settings

Hi,
We're using integration and deployment servers with different solr servers. When transporting a project to our integration system let's say: 'A', we have to use Solr 'A', when transporting to deployment server 'B' we should use Solr 'B'.

Unfortunately collective.solr always sets the connection properties. When setting the connection properties in a custom profile i have to maintain 2 different solr profiles for integration/deployment.

I'd like to see a more comfortable solution here, any ideas?

indexer.py line 121 throws 'Uncommitted changes' when trying to index a new file

I am using the latest version of collective.solr. My SOLR server and Plone server are located on different linux systems. I tried to index a file and received an error at line 121 of indexer.py. It seems like calling the function blob.committed() is causing the problem. How can this be fixed?

what if Solr is on another machine?

Indexing binary files does not work.

But I found that in collective.solr.indexer
line 121

--        postdata['stream.file'] = self.getpath()
++        postdata['stream.url'] = self.context.absolute_url()

it works.
The server must "see" each other on the networks of course.

reindex command and dexteritytextindexer don't play well together

While trying to migrate a plone site to Solr (900k documents) with ./bin/instance solr_reindex all documents get indexed at Solr but SearchableText is empty in my dexterity content types that use collective.dexteritytextindexer.

Probably related to collective/collective.dexteritytextindexer#12 ?

Run tests against multiple solr versions on travis.

Solr docs gone.

The solr docs are gone: http://docs.plone.org/4/en/search.html?q=solr#

@loechel @svx any idea what happened to the solr docs? I would like to make a c.solr release but I don't want to make a release with broken docs...

TypeError: Could not adapt

When I do a search I get the following error.
LiveSearch seems to work though.

I am using the current default branch.

Traceback (innermost last):

Module ZPublisher.Publish, line 126, in publish
Module ZPublisher.mapply, line 77, in mapply
Module ZPublisher.Publish, line 46, in call_object
Module Products.Five.browser.metaconfigure, line 476, in __call__
Module Products.Five.browser.pagetemplatefile, line 125, in __call__
Module Products.Five.browser.pagetemplatefile, line 59, in __call__
Module zope.pagetemplate.pagetemplate, line 132, in pt_render
Module five.pt.engine, line 93, in __call__
Module z3c.pt.pagetemplate, line 149, in render
Module chameleon.zpt.template, line 257, in render
Module chameleon.template, line 190, in render
Module chameleon.template, line 172, in render
Module 97a9139883530dc2d43861f390bf8dd9b09ca074.py, line 3445, in render
Module 3a13850b926d726ad6d44822f6b99c8d45184ca8.py, line 1398, in render_master
Module 3a13850b926d726ad6d44822f6b99c8d45184ca8.py, line 614, in render_content
Module 97a9139883530dc2d43861f390bf8dd9b09ca074.py, line 2238, in __fill_main
Module plone.app.search.browser, line 58, in results
TypeError: ('Could not adapt', <collective.solr.parser.SolrResponse object at 0xb2974c0c>, <InterfaceClass plone.app.contentlisting.interfaces.IContentListing>) - Expression: "python: view.results(b_start=b_start)" - Filename: ... -testing/src/plone.app.search/plone/app/search/search.pt - Location: (95:33) - Source: batch python: view.results(b_start=b_start); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Arguments: repeat: {...} (0) template: <ViewPageTemplateFile - at 0xb2e27eccL> views: <ViewMapper - at 0xf864e8c> modules: <instance - at 0x919fe0c> args: <tuple - at 0xb756102cL> here: <ImplicitAcquisitionWrapper testing at 0xde0f75c> user: <ImplicitAcquisitionWrapper - at 0xde1c2fc> nothing: <NoneType - at 0x816fdb8> container: <ImplicitAcquisitionWrapper testing at 0xde0f75c> request: <instance - at 0xb2d6b04cL> wrapped_repeat: <SafeMapping - at 0xde0f70c> traverse_subpath: <list - at 0xb29759ecL> default: <object - at 0xb75807d0L> loop: {...} (1) context: <ImplicitAcquisitionWrapper testing at 0xde0f75c> view: <SimpleViewClass from /usr/local/Plone/zeocluster-testing/src/plone.app.search/plone/app/search/search.pt search at 0xf8648cc> translate: <function translate at 0xf56a764> root: <ImplicitAcquisitionWrapper Zope at 0xde0f784> options: {...} (0) target_language: <NoneType - at 0x816fdb8>

leading * are removed

Currently it is hard coded that leading * are removed before querying.
But my understanding is that with solr.ReversedWildcardFilterFactory solr actually supports leading *.
So I think this should be at least an optional behavior.

Synonyms

c.solr or c.r.solrinstance could be shipped with a synonym configuration example. The synonym data can be provided as json file or modified via a REST api.

Those configurations should be tested (on travis) and included directly in the docs to make sure we have working and up-to-date configurations in our docs.

https://cwiki.apache.org/confluence/display/solr/Managed+Resources

Solr 5 Support

Create a new Plone instance, enable solr, go to reindex view.

2015-06-25 09:20:18 ERROR Zope.SiteErrorLog 1435216818.110.045122177306 http://localhost:8080/Plone/@@solr-maintenance/reindex
Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module collective.solr.browser.maintenance, line 119, in reindex
  Module collective.solr.manager, line 138, in getSchema
  Module collective.solr.solr, line 300, in getSchema
  Module collective.solr.solr, line 116, in __errcheck
SolrException: HTTP code=500, reason=JSP support not configured

Additions to the maintenance view

Solr does support partial reindexing.

It would be great to have a maintenance GUI for "reindex" and "sync" methods. Ideally, it would be great if partial reindexing goes as far as (a) select content type(s), (b) select fields to be reindexed for each content type chosen.

That would speed up things so much!

Allow unmarshallers for dynamic fields

Currently unmarshallers are not usable for dyamicFields.

Evaluate Python bindings for solr.

We should use one of the existing Python Solr binding packages, e.g.:

• pysolr
• solrpy
• mysolr

Module Products.CMFPlone.interfaces has no global IReorderedEvent

Current master raises an error:

ConfigurationError: ('Invalid value for', 'for', 'ImportError: Module Products.CMFPlone.interfaces has no global IReorderedEvent in Products.CMFPlone.interfaces.IReorderedEvent')

Make facets available through collective.solr

Hi,
Accessing the result is easy through SolrResult wrapper. Accessing the returned facets is manual work. We should implement a SolrFacets wrapper as well.

New Documentation Structure

As dicussed during Plone Strategic Summit 2015 (Section: Enterprise Search and ZCatalog) Collective.solr should become more visibile and documentation should be included in docs.plone.org

Therefore a move away from a monolitic README file to a Sphinx Documentation is necessary.
I started a structuring of the documentation and created a branch docs-restruturing.

Todos:

• Discuss what to include (collective.solr and setup of solr for Plone, maybe by including collective.recipe.solrinstance documentation in the docs) (at least @tisto, @saily, @reinhardt, @pilz, @loechel needed for discussion)
• Base Structure and Transfer of existing documentation in new Structure (started in branch docs-restruturing)
• Setup external for docs.plone.org to reference from in README's (@svx needed)
• Merge of branch docs-restruturing into master.