mc-cat-tty / placerank Goto Github PK

• Enable/disable query expansion
• Expanded query preview

An ad-hoc folder has been created for this task: presskit/assets

• Check installation process (requirements, setup.py)
• Check TUI
• Check benchmarks

Add reviews generation to setup

• EDA - Exploratory Dataset Analysis. Eg. unbalanced reviews sentiment
• Objectives and functional requirements. Assumptions and user's degrees of freedom (TUI).
• System architecture
• Basic retrieval models
• Weighting models
• Query expansion: global with BERT vs WordNet, local with Whoosh (?)
• Sentiment analysis
• Show results
• Future improvements: Clustering tree 4 w2v (embeddings from BIG LLM, not BERT)

An exception is raised by the sentiment analyzer when the input text dimension is larger than the maximum allowed (which is 512 characters). Should we define a light preprocessing for reviews to remove markup and other characters that don't influence the sentiment?

The raised exception is in detail:

Token indices sequence length is longer than the specified maximum sequence length for this model (647 > 512). Running this sequence through the model will result in indexing errors

Cache listings when first downloading it, since different inverted index buildings require to download the dataset each time and on slow connections can take up to a minute of total processing time.

• Sentiment-aware scorer
• Call scorer from TUI

Ref.:

• https://gate.ac.uk/sale/olp-book/main.pdf
• https://aclanthology.org/P19-1328.pdf

Proposals:

• BERT fine-tuning on custom corpus
• Online (ie. run BERT for each query in order to extract semantically alternative tokens for highly-informative words) vs offline (ie. pre-define a set of highly-informative tokens and generate an offline "lexicon/ontology/thesaurus" for them) assessment

To speed up things, we should provide a convenient method to change the corpus analyzer on the fly.

Right now you have to redefine the getDefaultAnalyzer, which is referenced in placerank.logic_views.DocumentLogicView but this makes it impossible to change the default analyzer at runtime.

We could change the function code for getDefaultAnalyzer by making its func_code field referencing the func_code of another ad-hoc function, but I think this is somewhat inelegant and convoluted.

Maybe let's make a factory that returns the appropriate analyzer. Another problem arises: the call to getDefaultAnalyzer, or this virtual factory, is inside the constructor of a class field. I think we should move the inverted index schema outside and create a separate class.