Scientist is a self-learning search engine library. It offers the possibility to enter search elements, which it then evaluates. It has a self-learning algorithm to determine results on their revelance to improve them afterwards.
Currently it is still a bit slow, but improvements are already planned. At the beginning, the results might not be correct, but they will improve over time. Suitable results are also delivered unlearned. These may not always correspond to the best
Features
- Simple handling & clear Desgin
- Possible with all storage methods, for example MySQL, Json, Sqlite
- Easy self learning
- The learning process can be shared
Example found in ./examples
Currently it is not possible to install it via PIP
Add the folder scienntist
to your project folder and import it
In the examples the database SQLite is used, but it is not necessary.
- Create a database connection with
databaseConnector
or use an existing one. More details can be found further down in the text. - Set up the database with the correct tables, described in
databaseTables.md
.
Before starting the script, the database must already have been created.
import scientist
from scientist import DCSqlite # example database connector
dbConnector = DCSqlite("./mydb.db")
sc = scientist.DataScientist(dbConnector)
The whole system has a logger, by default it logs to a file s2.log
in overwrite mode.
To create your own logger, use LogSettings
.
In this you can configure the logger as you want it.
A detailed description will follow. It uses the Python module logging
.
Only one logger per Python script is possible in the library, if you change it, all will be changed.
Example
from scientist import LogSettings
import logging
logSet = LogSettings(
loggerName="logger scientist",
filepath="./s2.log",
filemode="a",
textFmt="[%(asctime)s] [%(levelname)s]: %(message)s >> %(pathname)s:%(lineno)d",
dateFmt="%Y-%m-%d %H:%M:%S",
level=logging.DEBUG
)
Adding the logger at init
sc = scientist.DataScientist(dbConnector, logSet)
Modify the logger later
sc.updateLogger(logSet)
The library loads all data once at startup, currently it is not possible to force a reload.
Currently there is no automatic saving, use for it
sc.save()
The system works with an index, in which it stores important things for searching. This index must always be kept up to date. Since otherwise new realizations are not incorporated.
The library can do this automatically. Initialize it then with the addition autoRecreateIndex = True
.
Example
sc = scientist.DataScientist(dbConnector, autoRecreateIndex = True)
or do it manually with sc.recreateIndex()
. Since this function is thread-safe you can also use it with autoReCreateIndex
Results are stored in collection
's. But these are not created directly.
A collection is a class for better handling of the data.
There are two possibilities
The best way to add something to the system. Use .addElement()
to create an entry, the following parameters can be passed.
Parameters
name: str (Importand)
give the entry a name.extraSearchs: str
add extra text to the entry, which will be considered in the search. Like a description.count: int
add how often the entry has been called. (Currently not connected)- relevance: int`` add how relevant the entry is. (Currently not connected)
_category: list[str]
categorize the entry. The input is a list containing the categories.identifikator: int
give your entry a recognizable ID to be able to assign the entry to your data, this is not important for the functions or the search.ignore: bool
should the entry be invisible?
Example
sc.addElement(
name="Cars",
extraSearchs="Cars is a 2006 American computer-animated sports comedy film produced by Pixar Animation Studios and released by Walt Disney Pictures. (Wikipedia)",
_category=["cars", "disney", "pixar", "route 66", "Lightning McQueen", "Matter"],
identifikator=1,
)
In this example all important entries are present. Even if not all parameters have been used.
Use .insert()
to insert a whole text into collections, but this is very inaccurate and will be
will surely be removed soon, because this system only separates words and inserts them, without any
Parameters
text: str
Text to read.startAsThread: bool
Should the function start as a thread? If enabled a thread will be returned.replacer: list[list[str, str]]
characters to replace on read, example structure[["a", "b"], [".", " "], ["\n", " "]]
.
Return
thread
returns the running thread if used.
Now to use the search function, use .match()
. This returns the result based on what was learned.
Parameters
search: str
What is searched for.user: [User, int]
Pass the searching user or its ID to personalize the result (Not implemented).
Return
Record
After a search a record is returned which contains the results and is important for the learning progress, so don't lose it.
Example
rec = sc.match("Cars")
A record contains all found results sorted by importance, this is also important for the learning progress
Important & Interesting Functions
Function | Description | Optional parameters |
---|---|---|
setResult(index) |
Set which result was selected | |
clearResult() |
Set the result back to 0 | |
getResult() |
Get the result as Collection |
|
getUser() |
Get the user of the record | |
setSearchText() |
Overwrite the text searched with match > search |
|
getAsDRec() |
Get the record as DRec to display it user friendly |
maxShows = int |
DRec is a way of displaying the record data nicely, it helps to create a user friendly display. You can think of it like a book.
It displays 25 elements per page by default.
INIT
It is possible to initialize using one of the 4 options.
rec.getAsDRec()
get it directly from the recordrec.getAsDRec(maxShows)
specify the elements per pageDRec(rec)
initialize it with the recordDRec(rec, maxShows)
specify the elements per page
Functions
function | description | optional parameters | return |
---|---|---|---|
get() |
Get the current page elements | list with collections | |
nextPage() ` Go to the next page |
amount = 1 specify how many pages to skip |
bool |
|
previousPage() |
Go to the page before | amount = 1 specify how many pages to skip |
bool |
addIndex() |
- | amount = 1 how much to add |
bool |
removeIndex() |
- | amount = 1 how much to remove |
bool |
The index is the number of the first element from which the displayed elements will be counted, it is possible to move it.
After something has been selected for the result, it must be inserted into the system.
example for setting the result rec.setResult(1)
The insertion is very simple, this works by simply giving the record back to the library.
sc.insertRecord(rec)
The insert can sometimes take a little longer, because it works thread-safe.
Used translator for the english version
Translated with www.DeepL.com/Translator (free version)