dx-prog / distributedindexing

2DB Distributed Indexing Sprocket System: Loads files of various types through extractors into a distributed indexing system

2DB Distributed Indexing Sprocket System

2DB Distributed Indexing Sprocket System (the "Project") is a research project by David Garcia into the areas of distributed search, indexing, and complex integrated learning systems. The contributor(s), author(s), and/or sponsor(s), or other Contributor of the Project make the Project available under the Apache 2.0 License.

FEATURES SO FAR

LEXER

Project File: Sprockets.Lexer The project (the "Lexer") is designed to help parse content, queries, and scripts for consumption by indexing systems or script engines.

Lucene Query Sanitizer

Project File: Sprockets.DocumentIndexer.Lucene Lucene Query Sanitizer is now functional. This tool allows developers to expose search engines that can support boolean and fuzzy searches in a manner that is flexible and resistant to malformed queries. This is ideal for enabling advance search without punishing users who are not familiar with strict boolean logic. Examples:

(house AND garden)) OR land becomes (house AND garden) OR land

---

FEATURE SET TODO

The following is a set of ideal features for my research project.

SEARCH DATABASE FEATURES

The intent is to support saving and searching objects into searchable databases where in the search is handled by services, and the queries and scripting interface are abstracted away from database concerns. The user of the system is intended only to have to worry about their specific programming language, and not the underlying connectivity to any particular database, or set of databases in the case of distributed search.

SEARCHABLE FORMATS

The intent is to be able to store documents of various formats into a database for search wherein the extract is handled by services. The formats intended to be supported are as follows:

• XML
• JSON
• HTML
• PLAIN

SEARCH METHODS

My desire is to make the project scriptable and usable in multiple languages. The current set of languages targeted are as follows:

• CSHARP
• Python (IronPython)
• Ruby (IronRuby)
• F#

ANALYSIS FEATURES

This project is intended to make exploring data and the relationships there-in (data-mining) easier.

---

License

Copyright 2017 David Garcia

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

---

THIRD PARTY FORMATS

The Project may contain extractors for various formats. The extractors are not intended to support the full scope, scale, structure, intent, design, purpose, philosophy or other quality of the format or problem space for the format. Furthermore, descriptions, information, or other details about the formats are neither stated to be correct nor complete. All information hereafter is provided "AS-IS" in accordance to the license for the Project.

Possible Inclusions

• (Legal XML) [http://www.legalxml.org/]
• PDF
• Open Office XML
• OpenDocument Format (ODF)
• Markdown

DISCLAIMER

An extractor for any particular format is:

1. NOT AN ENDORSEMENT OF ANY PARTICULAR FORMAT
2. NOT AN ENDORSEMENT OF ANY THIRD-PARTY
3. NEITHER IMPLIES NOR CONSTITUTES AN ENDORSEMENT BY ANY SUCH THIRD-PARTY OF THIS PROJECT
4. NEITHER IMPLIES NOR CONSTITUTES AN ENDORSEMENT BY ANY SUCH THIRD-PARTY OF OR FOR ANY PROJECT CONTRIBUTOR