swirlai / swirl-search

Swirl is open-source software that uses AI to simultaneously search multiple content and data sources, finds the best results using a reader LLM, then prompts Generative AI, enabling you to get answers from your own data.

Swirl

Swirl is open source software that simultaneously searches multiple content sources and returns AI ranked results.

Start Searching · Slack · Key Features · Contribute · Documentation · Connectors

---

Swirl is open source software that simultaneously searches multiple content sources and returns AI ranked results. Prompt your choice of Generative AI using the top N results to get answers incorporating your own data.

Swirl can connect to:

• Databases (SQL, NoSQL, Google BigQuery)
• Public data services (Google Programmable Search Engines, ArXiv.org, etc.)
• Enterprise sources (Microsoft 365, Jira, Miro, etc.)

And generate insights with AI and LLMs like ChatGPT. Start discovering and generating the answers you need based on your data.

Swirl is as simple as ABC: (a) Download YML, (b) Start in Docker, (c) Search with Swirl. From there, add credentials to preloaded SearchProviders to get results from more sources.

🚀 Try Swirl with ChatGPT

Swirl with ChatGPT as a configured AI Model.

Note We need your help 🙏. Help us create more examples of things you can or want to do with Swirl. Join our Slack Community to discuss and learn more. We'd be very happy to help you contribute 🤗!

🔎 How Swirl Works

Swirl adapts and distributes user queries to anything with a search API - search engines, databases, noSQL engines, cloud/SaaS services, data siloes, etc. and uses Large Language Models to re-rank the unified results without extracting or indexing anything.

🔌 List of Connectors

➕ For Enterprise Support on Connectors Contact the Swirl Team at: [email protected]

🚀 Help Us Expand! Want to see a new connector? Contribute by adding a connector and join our growing community of contributors.

🔥 Try Swirl Now In Docker

Prerequisites

• To run Swirl in Docker, you must have the latest Docker app for MacOS, Linux, or Windows installed and running locally.

• Windows users must also install and configure either the WSL 2 or the Hyper-V backend, as outlined in the System Requirements for installing Docker Desktop on Windows.

Start Swirl in Docker

Warning Make sure the Docker app is running before proceeding!

• Download the YML file: https://raw.githubusercontent.com/swirlai/swirl-search/main/docker-compose.yaml

curl https://raw.githubusercontent.com/swirlai/swirl-search/main/docker-compose.yaml -o docker-compose.yaml

• Optional: To enable Swirl's Real-Time Retrieval Augmented Generation (RAG) in Docker, run the following commands from the Console using a valid OpenAI API key:

export MSAL_CB_PORT=8000
export MSAL_HOST=localhost
export OPENAI_API_KEY=‘<your-OpenAI-API-key>’

🔑 Check out OpenAI's YouTube video if you don't have an OpenAI API Key.

• In MacOS or Linux, run the following command from the Console:

docker-compose pull && docker-compose up

• In Windows, run the following command from PowerShell:

docker compose up

After a few minutes the following or similar should appear:

• Open this URL with a browser: http://localhost:8000 (or http://localhost:8000/galaxy)

• If the search page appears, click Log Out at the top, right. The Swirl login page will appear.

• Enter the username admin and password password, then click Login.

• Enter a search in the search box and press the Search button. Ranked results appear in just a few seconds:

• To view the raw JSON, open http://localhost:8000/swirl/search/

The most recent Search object will be displayed at the top. Click on the result_url link to view the full JSON Response.

Notes 📝

Warning The Docker version of Swirl does not retain any data or configuration when shut down!

🔑 Swirl includes five (5) Google Programmable Search Engines (PSEs) to get you up and running right away. The credentials for these are shared with the Swirl Community.

🔑 Using Swirl with Microsoft 365 requires installation and approval by an authorized company Administrator. For more information, please review the M365 Guide or contact us.

Next Steps 👇

• Check out the details of our latest release!

• Head over to the Quick Start Guide and install Swirl locally!

🌟 Key Features

✦	Feature
📌	Microsoft 365 integration and OAUTH2 support
🔍	SearchProvider configurations for all included Connectors. They can be organized with the active, default and tags properties.
✏️	Adaptation of the query for each provider such as rewriting NOT term to -term, removing NOTted terms from providers that don't support NOT, and passing down the AND, + and OR operators.
⏳	Synchronous or asynchronous search federation via APIs
🛎️	Optional subscribe feature to continuously monitor any search for new results
🛠️	Pipelining of Processor stages for real-time adaptation and transformation of queries, responses and results
🗄️	Results stored in SQLite3 or PostgreSQL for post-processing, consumption and/or analytics
➡️	Built-in Query Transformation support, including re-writing and replacement
📖	Matching on word stems and handling of stopwords via NLTK
🚫	Duplicate detection on field or by configurable Cosine Similarity threshold
🔄	Re-ranking of unified results using Cosine Vector Similarity based on spaCy's large language model and NLTK
🎚️	Result mixers order results by relevancy, date or round-robin (stack) format, with optional filtering of just new items in subscribe mode
📄	Page through all results requested, re-run, re-score and update searches using URLs provided with each result set
📁	Sample data sets for use with SQLite3 and PostgreSQL
✒️	Optional spell correction using TextBlob
⌛	Optional search/result expiration service to limit storage use
🔌	Easily extensible Connector and Mixer objects

👩‍💻 Contributing to Swirl

Do you have a brilliant idea or improvement for Swirl? We're all ears, and thrilled you're here to help!

🔗 Get Started in 3 Easy Steps:

1. Connect with Fellow Enthusiasts - Jump into the Swirl Slack Community and share your ideas. You'll find a welcoming group of Swirl enthusiasts and team members eager to assist and collaborate.
2. Branch It Out - Always branch off from the develop branch with a descriptive name that encapsulates your idea or fix.
3. Start Your Contribution - Ready to get your hands dirty? Make sure all contributions come through a GitHub pull request. We roughly follow the Gitflow branching model, so all changes destined for the next release should be made to the develop branch.

📚 First time contributing on GitHub? No worries, the GitHub documentation has you covered with a great guide on contributing to projects.

💡 Every contribution, big or small, makes a difference. Join us in shaping the future of Swirl!

☁ Use the Swirl Cloud

For information about Swirl as a managed service, please contact us!

📖 Documentation

Overview | Quick Start | User Guide | Admin Guide | M365 Guide | Developer Guide | Developer Reference | AI Guide

👷‍♂️ Need Help? We're Here for You!

At Swirl, every user matters to us. Whether you're a beginner finding your way or an expert with feedback, we're here to support, listen, and help. Don't hesitate to reach out to us.

• 🎉 Join the Conversation: Dive into our vibrant Swirl Community on Slack - it's where all the magic happens!

• 📧 Direct Support: For any questions, suggestions, or even a simple hello, drop us an email at [email protected]. We cherish every message and promise to get back to you promptly!

• 💼 Request A Connector (Enterprise Support) Want to see a new connector quickly and fast. Contact the Swirl Team at: [email protected]

Remember, you're part of our family now. 🌍💙

swirl-search's Issues

Task: Create file(s) and code snippets for SWIRL favicon

We would like to have a nice favicon setup for SWIRL that will work on multiple devices.

The initial logo image files are located in: swirl-search/swirl/static

Favicon creation has become rather more complex than the old skool favicon.ico file and one entry in the HTML <head> tag alone. Some information and potentially useful resources for helping with this task:

• https://evilmartians.com/chronicles/how-to-favicon-in-2021-six-files-that-fit-most-needs
• Shorter summary of the above: https://css-tricks.com/how-to-favicon-in-2021
• Favicon generator and checker tool: https://realfavicongenerator.net

I'll take this one for now, but if someone else with experience doing this wants to grab it, please do! :)

PostgreSQL driver Psycopg2 issues

The psycopg2 driver for PostgreSQL has been removed from requirements.txt. This is because PostgreSQL must be installed, and the pg_config executable must be in the path.

If that is the case, simply do:

pip install psycopg2
python swirl.py restart core

Note that there is error handling to take care of the case where the PostgreSQL connector is requested, but the package is not installed. An error will appear in the message section of the search object.

Add metadata to Result List indicating whether query was run using q= or qs=

At the moment, the Result List page does not indicate whether a search was run using the q= parameter or the qs= parameter. It would be nice to see that, perhaps in the search block.

key error: 'searchprovider_rank' when processing results with GenericResultProcessor

This will be fixed in 1.6.2

Workaround: use MappingResultProcessor

During installation, python swirl.py setup sometimes hangs after message "Collecting Statics:"

Pressing ENTER produces an error message.

swirl.py not working on some ubuntu configs

If rabbit becomes 'lost', e.g.

sid@swirl1:/usr/local/swirl$ python swirl.py status
##S#W#I#R#L##1#.#5##############################################################

Service: rabbitmq...UNKNOWN, pid:108968 not found
Service: django...RUNNING, pid:109050
Service: celery-worker...RUNNING, pid:109051
Service: celery-beats...RUNNING, pid:109056

    PID TTY          TIME CMD
 109050 pts/4    00:00:00 daphne
 109051 pts/4    00:00:02 celery
 109056 pts/4    00:00:02 celery

Shutdown swirl:

python swirl.py stop

An error will occur since rabbit can't be found at the original pid. Remove the .pid file. Find rabbit and terminate it, e.g.

ps -ef | grep rabbit
...
kill rabbits-new-pid

As a workaround: start and stop processes manually

Highlighting across apostrophe not working

Example, query term "musks" not found in "Since the start of Elon Musk’s announcement"

The date_published field is not reported by most connectors

It appears that date_published is not being reported by most connectors.

More details to follow

Docker PostgreSQL driver psycopg2 error on startup

This error seems to appear in the Docker Terminal window immediately after the Container comes up:

swirl-161f-app-1  | 2022-11-26 13:08:35,922 ERROR    postgresql.py: Error: can't load psycopg2: No module named 'psycopg2', see https://github.com/sidprobstein/swirl-search/wiki/4.-Object-Reference#postgresql

Screen grab from a 1.6.1 final Docker install:

Logging the issue because I do have psycopg2 installed already, yet Docker seems unable to find it, perhaps? I installed it earlier today as part of a fresh MacOS installation of 1.6.1 final and had the funding DB example working just fine with PostgreSQL; so the driver is there and functioning (or at least, it was).

This results in an error when attempting to query the funding DB example in PostgreSQL from the docs, for example:
"PostgreSQL_8_4: Error: name 'psycopg2' is not defined",

SQLite funding example SearchProvider returns errors on federation

Follow the steps here for setting up the SQLite funding example SearchProvider: https://github.com/sidprobstein/swirl-search/wiki/4.-Object-Reference#loading-into-sqlite3

Swirl indicates that the new SP is created correctly:

Performing a search that should return something from this source reveals the following errors on the Results List page, messages clause:

        "Sqlite3_4_2: Error: db_path does not exist",
        "Sqlite3_4_2: Error: db_path does not exist",
        "Sqlite3_4_2: Error: 'NoneType' object is not subscriptable",

The Results List info clause returns with this (note the "-1" values):

Restricting the search to only this SearchProvider also displays the issue:

Some URLs create two search objects

This is browser prefetch

The CLI help for swirl.py isn't working correctly (MacOS)

Doing either of the following in the swirl-directory results in the same output to the terminal window.
./swirl.py -h
./swirl.py -help

Output to the Terminal window:

No way to tell which version is running

@trouserless reports this and it is true

SWIRL won't highlight terms that have preceding or trailing quotes in the query

From the v. 1.7 Release Notes:

For example 'hello or 'hello'. These may be quite acceptable to search engines as phrase searches. This will be fixed in a future release.

This is also true of double quotes; searching returns expected results but SWIRL doesn't highlight the matching word if the query is passed with quotation marks in front, for example.

Result List page explanations have the incorrect parameter symbol

On the SWIRL Results List page (http://localhost:8000/swirl/results/?search_id=*), there is a block of test at the top of the page that outlines additional parameters that can be added the Results List URL to reveal additional information. Three of these explanations indicate that the question mark (?) is required before adding the additional parameter:

• ?result_mixer=
• ?explain=
• ?provider=

As seen here:

This is incorrect for these 3 options; instead, they require the ampersand (&) before the parameter. For example, &explain=true is correct and activates query explanation within the Result Lis, whereas ?explain=true gives a 500 error.

The code responsible for generating the text at the top of the Result List page should be update such that the ampersand is presented to users for these 3 options.

Need workflow in logs under INFO and query/result details under DEBUG

To be resolved in 1.8

HTTP Post Connector

create a connector that can use POST

Swirl 1.8RC: Swagger API link is 404

From the Swirl homepage in v. 1.8RC, the "Swagger" link is 404.

Link URL: http://localhost:8000/swagger-ui/

Re-checked in v. 1.7, working there. The issue is new to 1.8RC

Django Admin page > View Site link returns "Not Found"

To see this issue:

• Click the "Admin" link from the main SWIRL page to access the Django Admin area.
• On the top right of that page, click the "View Site" link.

This returns a page "Not Found" message at this URL: http://localhost:8000/

Perhaps this link should take the user back to the main SWIRL homepage? http://localhost:8000/swirl/

Docker warnings for unresolved variable in compose file

When running the docker compose up with Swirl v. 1.8, we see these Warnings, courtesy of Docker:

swirl1-8rc % docker compose up
WARN[0000] The "jjf" variable is not set. Defaulting to a blank string. 
WARN[0000] The "jjf" variable is not set. Defaulting to a blank string. 
WARN[0000] The "jjf" variable is not set. Defaulting to a blank string. 

They are just warnings and don't disrupt bringing up Swirl.

This behavior seems to be similar to this issue filed in the compose-cli project: docker/compose-cli#1907

Specifically this comment: docker/compose-cli#1907 (comment)

Filing this issue as tech-debt so that we can track this and potentially sort it out in future.

Salesforce Connector

Ability to use swirl search to connect to salesforce

Server Error 500 when Patch-ing a SearchProvider

To reproduce:

• Go to an individual Search Provider Instance page (e.g. http://localhost:8000/swirl/searchproviders/1/)
• Select the "Raw data" tab
• Select the "Patch" option (you don't need to make a config change)
• You should get the Server Error 500

From the django.log file:

2022-12-10 16:48:59,996 ERROR    Internal Server Error: /swirl/searchproviders/1/
Traceback (most recent call last):
  File "/Users/erikspears/.pyenv/versions/3.9.9/lib/python3.9/site-packages/asgiref/sync.py", line 472, in thread_handler
    raise exc_info[1]
  File "/Users/erikspears/.pyenv/versions/3.9.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 42, in inner
    response = await get_response(request)
  File "/Users/erikspears/.pyenv/versions/3.9.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 253, in _get_response_async
    response = await wrapped_callback(
  File "/Users/erikspears/.pyenv/versions/3.9.9/lib/python3.9/site-packages/asgiref/sync.py", line 435, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "/Users/erikspears/.pyenv/versions/3.9.9/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
    return await fut
  File "/Users/erikspears/.pyenv/versions/3.9.9/lib/python3.9/site-packages/asgiref/current_thread_executor.py", line 22, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/erikspears/.pyenv/versions/3.9.9/lib/python3.9/site-packages/asgiref/sync.py", line 476, in thread_handler
    return func(*args, **kwargs)
  File "/Users/erikspears/.pyenv/versions/3.9.9/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/Users/erikspears/.pyenv/versions/3.9.9/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/Users/erikspears/.pyenv/versions/3.9.9/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/Users/erikspears/.pyenv/versions/3.9.9/lib/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/Users/erikspears/.pyenv/versions/3.9.9/lib/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/Users/erikspears/.pyenv/versions/3.9.9/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/Users/erikspears/.pyenv/versions/3.9.9/lib/python3.9/site-packages/rest_framework/mixins.py", line 82, in partial_update
    return self.update(request, *args, **kwargs)
TypeError: update() got an unexpected keyword argument 'partial'

Newsdata.io Search Provider setup throws an error

Attempting to add the Newsdata.io source as a Search Provider per the User Guide > Copy/Paste Install instructions using the JSON here (1.3 Final): https://github.com/sidprobstein/swirl-search/blob/main/SearchProviders/newsdata_api.json

After the copy/paste and clicking POST, as documented, the Search Provider List page (http://localhost:8000/swirl/searchproviders/) returns an error:

{
    "non_field_errors": [
        "Invalid data. Expected a dictionary, but got list."
    ]
}

See here:

v. 1.8 - Cannot add PostgreSQL SearchProvider: "\"PostgreSQL\" is not a valid choice."

Attempting to validate the Postgres funding example from the Developer's Guide, encountered an error attempting to add the corresponding SearchProvider: HTTP 400 Bad Request

Most direct way to reproduce is try adding the Postgres funding db example SearchProvider: funding_db_postgres.json
...available here in 1.8: https://github.com/sidprobstein/swirl-search/blob/main/SearchProviders/funding_db_postgres.json

Copy/paste this json into the Raw Data tab of the Search Provider List page: http://localhost:8000/swirl/searchproviders/

Both with and without updates to the "url:" line of the SP, SWIRL returns the 404 above:

Behavior confirmed in:

• MacOS installation -- SWIRL 1.8
• Docker installation -- SWIRL 1.8

Can't save date_mixer in a search object due to model validation (MINOR)

Description

Using SWIRL Search 1.1, if you try to create a search object with "result_mixer": "date mixer", it will fail due to this option not being in the models.py choice list.

Workaround

Specify the default when creating the search object, then change the mixer when viewing results. For example, if the default URL is:

http://localhost:8000/swirl/results/?search_id=2

And this is not date mixed, manually add the following to the URL:

http://localhost:8000/swirl/results/?search_id=2&result_mixer=date_mixer

Notes

The result_mixer stored in the search_object is just the default, but it doesn't impact the way data is stored, and isn't validated once results are available.

This will be fixed in 1.1.1.

Clean-up: Remove celery beat?

Technical clean-up discussion / task: remove celery beat?

CC: @sidprobstein and @dnicodemus

Hit highlighting: no highlighting of title terms with adjacent punctuation

A search that included the term tesla yielded the title match below; however hit highlighting was not applied to the title match next to the question mark.

Hit title: Where is shipping's Elon Musk and Tesla?

Note missing highlighting in title below:

Error: PostgreSQL_7_6: Error: name 'psycopg2' is not defined

Running v. 1.5, I got the Postgresql funding example from docs all setup and working. It was returning results as expected.

I then pulled the 1.6rc branch down to a separate directory. I stopped SWIRL 1.5 and installed 1.6rc. Postgres itself was still running the whole time. No other SWIRL dependencies were uninstalled.

I setup the SP in 1.6rc for the Postgres funding example, then ran a search and got this error on the Results List:
PostgreSQL_7_6: Error: name 'psycopg2' is not defined

This is odd because pip knows all about psycopg2 already:
% pip show psycopg2

Name: psycopg2

Version: 2.9.5
S
ummary: psycopg2 - Python-PostgreSQL Database Adapter

Home-page: https://psycopg.org/

Author: Federico Di Gregorio

Author-email: [email protected]
L
icense: LGPL with exceptions

Location: /Users/erikspears/.pyenv/versions/3.9.9/lib/python3.9/site-packages�
Requires: 

Required-by: 


Restarting SWIRL did not get rid of the error. Instead, I had to uninstall and reinstall psycopg2 (using PIP). That error is no longer returned; and I am getting results from the Postgres funding example, as expected.

While there is a work-around, it would be good to know why that error was thrown in the first place. And if it will happen with a proper upgrade from 1.5 -> 1.6 vs. pulling the 1.6rc branch separately.

Add Bearer token Authentication to the "Requests Get" connector

Swirl can't talk to rabbitmq

celery cannot talk to rabbitmq and is throwing this error

Here is the error:
amqp.exceptions.InternalError: Connection.open: (541) INTERNAL_ERROR - access to vhost '/' refused for user 'guest': vhost '/' is down

Docker shutdown / restart issue with v. 1.8 Swirl?

It appears that doing a ctrl-c to gracefully stop Docker (MacOS, Terminal) doesn't result in rabbitmq being properly terminated ... or, in the shutdown process not fully clearing or deleting the .swirl file.

To see this in action:

• Get the Docker version of Swirl 1.8 up and running (do a search to make sure, etc.)
• Issue CTRL-C in the Terminal window where you ran docker compose up
• Give it a moment or few to finish shutdown, then issue a fresh docker compose up
• You'll see this warning about rabbitmq running already:

swirl1-8rc-app-1  | Error: rabbitmq appears to be running, please check .swirl or remove it; exiting...
swirl1-8rc-app-1  | Command start reported an error

• Docker will continue to bring up Swirl
• Once it's up, do a search (e.g. ?q=tesla)
• You will get this rather gnarly error in the results window along with a mighty stacktrace:

OperationalError at /swirl/search/
[Errno 111] Connection refused

To remediate this issue:

• While Docker is still running
• Go the the Docker GUI, Containers > open the container in question > three-dot menu > Open in Terminal
• Remove the .swirl file
• Stop Docker
• Restart Docker

Screen grab of the rabbitmq error when starting Docker:

Screen grab of Operational Error in the UI:

The might stack from the Terminal window where Docker was running:

swirl1-8rc-app-1  | 2023-01-15 14:10:41,744 INFO     views.py: Search.create() from ?q
swirl1-8rc-app-1  | Internal Server Error: /swirl/search/
swirl1-8rc-app-1  | Traceback (most recent call last):
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/utils/functional.py", line 30, in __call__
swirl1-8rc-app-1  |     return self.__value__
swirl1-8rc-app-1  | AttributeError: 'ChannelPromise' object has no attribute '__value__'
swirl1-8rc-app-1  | 
swirl1-8rc-app-1  | During handling of the above exception, another exception occurred:
swirl1-8rc-app-1  | 
swirl1-8rc-app-1  | Traceback (most recent call last):
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 446, in _reraise_as_library_errors
swirl1-8rc-app-1  |     yield
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 433, in _ensure_connection
swirl1-8rc-app-1  |     return retry_over_time(
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
swirl1-8rc-app-1  |     return fun(*args, **kwargs)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 877, in _connection_factory
swirl1-8rc-app-1  |     self._connection = self._establish_connection()
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 812, in _establish_connection
swirl1-8rc-app-1  |     conn = self.transport.establish_connection()
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/transport/pyamqp.py", line 201, in establish_connection
swirl1-8rc-app-1  |     conn.connect()
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/amqp/connection.py", line 323, in connect
swirl1-8rc-app-1  |     self.transport.connect()
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/amqp/transport.py", line 129, in connect
swirl1-8rc-app-1  |     self._connect(self.host, self.port, self.connect_timeout)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/amqp/transport.py", line 184, in _connect
swirl1-8rc-app-1  |     self.sock.connect(sa)
swirl1-8rc-app-1  | ConnectionRefusedError: [Errno 111] Connection refused
swirl1-8rc-app-1  | 
swirl1-8rc-app-1  | The above exception was the direct cause of the following exception:
swirl1-8rc-app-1  | 
swirl1-8rc-app-1  | Traceback (most recent call last):
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/asgiref/sync.py", line 486, in thread_handler
swirl1-8rc-app-1  |     raise exc_info[1]
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 42, in inner
swirl1-8rc-app-1  |     response = await get_response(request)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 253, in _get_response_async
swirl1-8rc-app-1  |     response = await wrapped_callback(
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/asgiref/sync.py", line 448, in __call__
swirl1-8rc-app-1  |     ret = await asyncio.wait_for(future, timeout=None)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/asyncio/tasks.py", line 455, in wait_for
swirl1-8rc-app-1  |     return await fut
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/asgiref/current_thread_executor.py", line 22, in run
swirl1-8rc-app-1  |     result = self.fn(*self.args, **self.kwargs)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/asgiref/sync.py", line 490, in thread_handler
swirl1-8rc-app-1  |     return func(*args, **kwargs)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
swirl1-8rc-app-1  |     return view_func(*args, **kwargs)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
swirl1-8rc-app-1  |     return self.dispatch(request, *args, **kwargs)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
swirl1-8rc-app-1  |     response = self.handle_exception(exc)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
swirl1-8rc-app-1  |     self.raise_uncaught_exception(exc)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
swirl1-8rc-app-1  |     raise exc
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
swirl1-8rc-app-1  |     response = handler(request, *args, **kwargs)
swirl1-8rc-app-1  |   File "/app/swirl/views.py", line 191, in list
swirl1-8rc-app-1  |     search_task.delay(new_search.id)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/celery/app/task.py", line 425, in delay
swirl1-8rc-app-1  |     return self.apply_async(args, kwargs)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/celery/app/task.py", line 575, in apply_async
swirl1-8rc-app-1  |     return app.send_task(
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/celery/app/base.py", line 788, in send_task
swirl1-8rc-app-1  |     amqp.send_task_message(P, name, message, **options)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/celery/app/amqp.py", line 510, in send_task_message
swirl1-8rc-app-1  |     ret = producer.publish(
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/messaging.py", line 177, in publish
swirl1-8rc-app-1  |     return _publish(
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 523, in _ensured
swirl1-8rc-app-1  |     return fun(*args, **kwargs)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/messaging.py", line 186, in _publish
swirl1-8rc-app-1  |     channel = self.channel
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/messaging.py", line 209, in _get_channel
swirl1-8rc-app-1  |     channel = self._channel = channel()
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/utils/functional.py", line 32, in __call__
swirl1-8rc-app-1  |     value = self.__value__ = self.__contract__()
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/messaging.py", line 225, in <lambda>
swirl1-8rc-app-1  |     channel = ChannelPromise(lambda: connection.default_channel)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 895, in default_channel
swirl1-8rc-app-1  |     self._ensure_connection(**conn_opts)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 433, in _ensure_connection
swirl1-8rc-app-1  |     return retry_over_time(
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
swirl1-8rc-app-1  |     self.gen.throw(type, value, traceback)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 450, in _reraise_as_library_errors
swirl1-8rc-app-1  |     raise ConnectionError(str(exc)) from exc
swirl1-8rc-app-1  | kombu.exceptions.OperationalError: [Errno 111] Connection refused
swirl1-8rc-app-1  | 2023-01-15 14:10:48,327 ERROR    Internal Server Error: /swirl/search/
swirl1-8rc-app-1  | Traceback (most recent call last):
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/utils/functional.py", line 30, in __call__
swirl1-8rc-app-1  |     return self.__value__
swirl1-8rc-app-1  | AttributeError: 'ChannelPromise' object has no attribute '__value__'
swirl1-8rc-app-1  | 
swirl1-8rc-app-1  | During handling of the above exception, another exception occurred:
swirl1-8rc-app-1  | 
swirl1-8rc-app-1  | Traceback (most recent call last):
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 446, in _reraise_as_library_errors
swirl1-8rc-app-1  |     yield
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 433, in _ensure_connection
swirl1-8rc-app-1  |     return retry_over_time(
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
swirl1-8rc-app-1  |     return fun(*args, **kwargs)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 877, in _connection_factory
swirl1-8rc-app-1  |     self._connection = self._establish_connection()
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 812, in _establish_connection
swirl1-8rc-app-1  |     conn = self.transport.establish_connection()
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/transport/pyamqp.py", line 201, in establish_connection
swirl1-8rc-app-1  |     conn.connect()
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/amqp/connection.py", line 323, in connect
swirl1-8rc-app-1  |     self.transport.connect()
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/amqp/transport.py", line 129, in connect
swirl1-8rc-app-1  |     self._connect(self.host, self.port, self.connect_timeout)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/amqp/transport.py", line 184, in _connect
swirl1-8rc-app-1  |     self.sock.connect(sa)
swirl1-8rc-app-1  | ConnectionRefusedError: [Errno 111] Connection refused
swirl1-8rc-app-1  | 
swirl1-8rc-app-1  | The above exception was the direct cause of the following exception:
swirl1-8rc-app-1  | 
swirl1-8rc-app-1  | Traceback (most recent call last):
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/asgiref/sync.py", line 486, in thread_handler
swirl1-8rc-app-1  |     raise exc_info[1]
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 42, in inner
swirl1-8rc-app-1  |     response = await get_response(request)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 253, in _get_response_async
swirl1-8rc-app-1  |     response = await wrapped_callback(
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/asgiref/sync.py", line 448, in __call__
swirl1-8rc-app-1  |     ret = await asyncio.wait_for(future, timeout=None)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/asyncio/tasks.py", line 455, in wait_for
swirl1-8rc-app-1  |     return await fut
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/asgiref/current_thread_executor.py", line 22, in run
swirl1-8rc-app-1  |     result = self.fn(*self.args, **self.kwargs)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/asgiref/sync.py", line 490, in thread_handler
swirl1-8rc-app-1  |     return func(*args, **kwargs)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
swirl1-8rc-app-1  |     return view_func(*args, **kwargs)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
swirl1-8rc-app-1  |     return self.dispatch(request, *args, **kwargs)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
swirl1-8rc-app-1  |     response = self.handle_exception(exc)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
swirl1-8rc-app-1  |     self.raise_uncaught_exception(exc)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
swirl1-8rc-app-1  |     raise exc
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
swirl1-8rc-app-1  |     response = handler(request, *args, **kwargs)
swirl1-8rc-app-1  |   File "/app/swirl/views.py", line 191, in list
swirl1-8rc-app-1  |     search_task.delay(new_search.id)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/celery/app/task.py", line 425, in delay
swirl1-8rc-app-1  |     return self.apply_async(args, kwargs)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/celery/app/task.py", line 575, in apply_async
swirl1-8rc-app-1  |     return app.send_task(
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/celery/app/base.py", line 788, in send_task
swirl1-8rc-app-1  |     amqp.send_task_message(P, name, message, **options)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/celery/app/amqp.py", line 510, in send_task_message
swirl1-8rc-app-1  |     ret = producer.publish(
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/messaging.py", line 177, in publish
swirl1-8rc-app-1  |     return _publish(
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 523, in _ensured
swirl1-8rc-app-1  |     return fun(*args, **kwargs)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/messaging.py", line 186, in _publish
swirl1-8rc-app-1  |     channel = self.channel
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/messaging.py", line 209, in _get_channel
swirl1-8rc-app-1  |     channel = self._channel = channel()
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/utils/functional.py", line 32, in __call__
swirl1-8rc-app-1  |     value = self.__value__ = self.__contract__()
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/messaging.py", line 225, in <lambda>
swirl1-8rc-app-1  |     channel = ChannelPromise(lambda: connection.default_channel)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 895, in default_channel
swirl1-8rc-app-1  |     self._ensure_connection(**conn_opts)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 433, in _ensure_connection
swirl1-8rc-app-1  |     return retry_over_time(
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
swirl1-8rc-app-1  |     self.gen.throw(type, value, traceback)
swirl1-8rc-app-1  |   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 450, in _reraise_as_library_errors
swirl1-8rc-app-1  |     raise ConnectionError(str(exc)) from exc
swirl1-8rc-app-1  | kombu.exceptions.OperationalError: [Errno 111] Connection refused

Highlighting interacts with tags

For example foo bar<b>

Ubuntu installation doesn't work

This needs a total overhaul.

1. Don't use

python install.py

Install, do:

sudo pip install -r requirements.txt

This is not ideal, instructions with venv will be forthcoming.

2. Manually start services instead of using swirl.py, which does not work reliable after start.

https://github.com/sidprobstein/swirl-search/wiki/5.-Admin-Guide#manually-starting-services

3. Manually kill services

Replace Django REST logo/link with SWIRL logo/link

Somewhat described here: https://stackoverflow.com/questions/38799315/change-default-django-rest-framework-home-page-title

Email support at swirl dot today and we will happily share a template that attempts to implement this article... but doesn't work.

Duplicate error messages generated when SOLR SearchProvider is borked

When federating against a non-functional SOLR SearchProvider, SWIRL is generating multiple copies of the same error message. It should produce only one error message in this case.

Example of 2 duplicate error messages:

"RequestsGet_5_128: Error: request.get returned: 500 Server Error from: techproducts (local/solr) for: http://localhost:8983/solr/techproducts/select?wt=json&start=0&q=federated+search",

 "RequestsGet_5_128: Error: request.get returned: 500 Server Error from: techproducts (local/solr) for: http://localhost:8983/solr/techproducts/select?wt=json&start=0&q=federated+search",
        "[search.py](http://search.py/): No response from provider: ['techproducts (local/solr)']"

Release Notes for v. 1.3 - updates needed

There are 2 items on the v. 1.3 Release Notes that might need attention.

1. Under "Known Issues" > first issue > third line: the "rerun function" link returns a 404; the href should be corrected.
2. Unser "Known Issues" > second issue: the reference page in Django Admin (for managing results objects) does NOT appear to be returning a 404. Perhaps this can be removed as a known issue, if confirmed?
   Link in the release notes: http://localhost:8000/admin/swirl/result/
   ...opens this page:
   

Need to highlight alternative word forms

For example knowledge management should match
knowledge management
managing knowledge
etc

Relevancy never scores the first hit in a response

Appears to have the last term instead?

Evaluate default celery execution pool

What should SWIRL's default celery execution pool be (if we're sticking with celery). Per @dnicodemus , Eventlet might be a better option.

See this doc: https://docs.celeryq.dev/en/master/userguide/concurrency/eventlet.html#enabling-eventlet

Users with no searchprovider* Django permissions

In the Django Admin page, create a User group with all result* and search* permissions but no searchprovider* permissions. Or, remove the "Can view search provider" from an existing User group.

Verify that the SearchProviders list is 403 Forbidden, as it should be.

Run a new search, and you will get results.

The bug is that you should be able to create/send a search but not get search results back since SearchProviders should completely be unavailable to the User due to the Django permissions setup.

Incorrect links: Readme and Quick Start pages

On the Homepage / Readme

• “Result mixers” link in the 5th bullet under "Features" goes to the top of the User Guide, not to the Result Mixers section: https://github.com/sidprobstein/swirl-search/wiki/2.-User-Guide#mixers
• “to-do list” link under "Contributing" returns a 404: https://github.com/sidprobstein/swirl-search/blob/main/docs/TO_DO.md
• "Documentation" section, those links are the only one that display horizontally; were they supposed to be a vertical, bulleted list?

On the Quick Start page

• "Next Steps" section, the “Relevancy Explanation” link leads to raw content version of the page, here:
https://raw.githubusercontent.com/wiki/sidprobstein/swirl-search/2.-User-Guide.md#relevancy-explanation

Replace the federation process in search.py with a chord()

Enhancement to replace the existing federation process in search.py with a chord()

Accept Google spelling suggestion

Accept Google's spelling suggestions when returned from Google sources.

Google Chrome renamed prefect to predictive page loading

See update here: https://www.ghacks.net/2019/04/23/missing-chromes-use-a-prediction-service-setting/

/swirl/search/?rerun=search_id doesn't remove message, needs timestamp

"##S#W#I#R#L##1#.#5##############################################################",
"Retrieved 1 of 1 results from: Microsoft M365 (Outlook/Messages)",
"Post processing of results by CosineRelevancyProcessor updated 0 results",
"Post processing of results by CosineRelevancyProcessor updated 1 results",
"Post processing of results by CosineRelevancyProcessor updated 1 results"

Deprecation warnings from PIP to keep an eye on

Not a barn-burner in terms of priority, but perhaps we should keep an eye on these deprecation warnings from PIP? Verified all three still exist in Swirl 1.6.1 final.

These two are seen when running ./install.sh and reference the django-restframework and bs4 packages respectively.

And a similar warning is seen when installing psycopg2 with PIP.

docker error appears during startup

swirl-15-f-app-1 | Error: .swirl not found, exiting
swirl-15-f-app-1 | Command stop reported an error

Daphne doesn't work on ubuntu

$ daphne
Traceback (most recent call last):
File "/home/sid/.local/bin/daphne", line 5, in
from daphne.cli import CommandLineInterface
File "/home/sid/.local/lib/python3.10/site-packages/daphne/cli.py", line 10, in
from .server import Server
File "/home/sid/.local/lib/python3.10/site-packages/daphne/server.py", line 36, in
from twisted.internet.endpoints import serverFromString
File "/home/sid/.local/lib/python3.10/site-packages/twisted/internet/endpoints.py", line 63, in
from twisted.python.systemd import ListenFDs
File "/home/sid/.local/lib/python3.10/site-packages/twisted/python/systemd.py", line 18, in
from attrs import Factory, define
ModuleNotFoundError: No module named 'attrs'

Django UI returns only the search providers list when asking for pages beyond the last result

At present, SWIRL has a limit of 10 results per search provider. So, when querying the 3 pre-set GSE providers, users can get a max of 30 results. In the Django UI, users can ask for additional pages beyond the max hits returned (e.g. by adding &page=5 to the results URL), and the UI will return a HTTP 200 OK page with only the search providers listed. This occurs anytime the user asked for a page beyond the one containing the last result. This scenario is potentially confusing for users.

Is it possible for the Django UI to return a more user-friendly message indicating that the maximum number of results has been exceeded? Or, perhaps even just what the max number of hits for the given query was? It's also unclear if having the search providers returned in this case is useful.

Labeling as a possible enhancement, @sidprobstein

Highlighting collisions

[2022-10-30 19:10:49,529: ERROR/ForkPoolWorker-8] highlight_list: failed to find match: ranked, Nov 1, 2019 ... Learn how the machine learning method , learning to rank , helps you serve up results that are not only relevant but that are ranked by relevancy .