demetrius-mp / sesg

SeSG (Search String Generator) python package repository.

Hi 👋, I'm Demetrius

Pursuing a Msc. in Computer Science

• 🔭 I’m currently working on my graduation

• 🌱 I’m currently learning text mining and topic extraction

• 👯 I’m looking to collaborate on open source development

• 📫 How to reach me [email protected]

• ⚡ Fun fact I like listening to city pop while programming

Connect with me:

Languages and Tools:

 

sesg's Issues

test: add scopus api tests

useful links:

• respx

fix: inconsistent citation graph architecture

in sesg.snowballing.fuzzy_backward_snowballing the graph is returned as an adjacency list, however in sesg.graph we provide a function named edges_to_adjacency_list.

we should provide a way to turn make a directed adjacency list into an undirected one (actually, just need to put it back where it was, see this commit

docs: ideas

"Experiment workflow" page, describing how the experiment is executed.

"Proposed usage" page, describing how a researcher could use the package.

feat: mypy

check the viability of using mypy, and add a task to run mypy against the codebase

discussion: incluse british and american versions for the words

words such as behavio(u)r, analy(z/s)ed, could be included in the search string with both of their versions in order to increase the recall

feat: remove bert OOV words

remove out-of-vocabulary words found by bert when enriching a string

this is done by removing every token that starts with "##"

refactor: remove unused db queries

CLI code now makes heavy use of sqlalchemy sessions, so theres probably some queries that are no longer used

lint: replace docformatter for pydocstyle

this would remove the docformatter dependency, and we could just use ruff with the pydocstyle rules

refactor: rename `generate_search_string` parameter name

rename n_similar_words to n_similar_words_per_word just to be more explicit

bug: installing with poetry wont install nvidia drivers

happens also when installing bertopic with poetry, so probably something either in bertopic's end, or poetry's end.

feat: cache BERT model on CI

have no idea how to do it, but it would definetly make CI faster

fix: weird `KeyError` on ScopusClient

another weid issue (like #42).

seems like scopus sometimes will return a JSON response without the search-results key. this happend non-deterministically which means that if you simply retry the request, you might get a correct JSON response.

fix: transformers vulnerability

refer to dependabot

refactor: simplify the API to the user

for example, it shouldnt require much work to generate a search string.

maybe provide a simple function with "sensible defaults"

refactor: rename `generate_enrichment_text` function name

rename to create_enrichment_text to stay consistent with create_docs

also allow passing the studies list as positional argument

feat: add caching for similar words

not sure if it would make it noticeably faster, but worth a shot

im thinking of using a class SearchStringBuilder that receives the enrichment text, and stores a cache dict, in which the key is the word, and the value is a previously computed list of similar words.

docs: add a jupyter notebook or google colab example

probably better to add a jupyter notebook to ensure usage of latest version of the package

see if theres a way to display the notebook on the docs page

fix: weird `JSONDecodeError` on `ScopusClient`

seems like sometimes the Scopus API does not return a valid JSON response for a string. However, if we simply redo the request, the API responds correctly.

i see no other option but to add a retry on exception mechanism to avoid this problem, (tenacity seems like the perfect solution)

docs: development/contribution guide

add a page to the docs giving some instructions on how to contribute, and some development recommendations

tests: multi client scopus search

figure out how to test the abstract base class

tests: make codecov not fail when code coverage is reduced

tests have the lower priority in the roadmap of development, due to the short time available for development

refactor: word enrichment

maybe create a differente function to concatenate the enriched words

main goal is to make testing easier

docs: deploy

figure out how to deploy, and where to deploy.

feat: enhance scopus search

add a recipe on how to use sesg.scopus.create_clients_with_disjoint_api_keys to execute more performatic searchs

the idea is to use either multiprocessing or multithreading and initialize each client at the same time

this would hugely increase the performance, theoretically it could be n_clients times faster

refactor: rename `extract_topics_with_lda` parameter name

rename number_of_topics to n_topics

docs: add link to leo's SeSG repository

https://github.com/LeoFuchs/SeSG

refactor: rename `on_search_finalization` to `on_completed`

on sesg.scopus.BaseScopusClientConsumer rename the on_search_finalization hook name to on_completed

tests/discussion: should we mock BERT, LDA, and BERTopic?

not sure if we should mock them, and if yes, what should we mock?

feat: script to export requirements.txt file

in a first moment, we can just run poetry export with --only flag to separate each dependency group, and add the full command to a taskipy task

refactor: no need to create a generic cache protocol

on this line, the cache is generic, but the only place where a cache is used is in this module

better to keep it simple, just create a SimilarWordsFinderCache.

tests: figure out how to test the package

set a minimum coverage and study testing strategies

refactor: install only the needed dependency groups

since i changed the groups quite a bit, still need to verify which groups are required.

refactor: decouple search string formulation from similar word generation

this would allow for a higher modularity

right now, we pass a list of topics (where a topic is a list of words) to a function that returns a formulated search string with the similar words already injected

this approach makes it hard to try out other similar word generation strategies

i thought about providing a new function:

def generate_similar_words(
  topics: list[list[str]],
) -> dict[str, list[str]]:
  ...

which would be used as follows:

topics = extract_topics_with_bertopic(
    docs,
    kmeans_n_clusters=2,
    umap_n_neighbors=5,
)

similar_words = generate_similar_words(topics)

search_string = generate_search_string(
    topics,
    similar_words,
    n_words_per_topic=5,
    n_similar_words_per_word=1,
)

notice that the generate_similar_words function might receive additional parameters, according to the strategy used.

docs: add "sesg process" page to docs

add a page to explain how sesg works

feat: logging

add some sort of logging to show time taken by functions and other informations

ci: lint and format

add a ci task to lint and format the code

docs: add "secondary studies" page to docs

add a page with some theoretical background on secondary studies. the target audience is potential researchers that want to continue the work on the tool

refactor: move evaluation functions to a new module `evaluation`

placing everything on the root directory seems a little confusing, since the main modules are search_string and topic_extraction

fix: remove export-dependencies script from CI

it has no effects whatsoever, the precommit already forces to export them before pushing

refactor: rename backward_snowballing function

rename backward_snowballing to fuzzy_backward_snowballing in order to be explicit about the strategy used to perform the backward snowballing

might be good to improve the docstring mentioning the fact that it will use fuzzy matching to detect references.