thanatoz-1 / text-technology Goto Github PK

Augment the keywords with all-cap words and cluster representatives. Other post-process like filter out too-shot and too-long keywords are needed.

todo: clean and update KeywordCleaner class

Prepare the final files

• collect part outputs:
  • papers.xml,
• process part outputs: augmented_papers.xml, all_cap.dict, cluster.dict
  • augmented_papers.xml and json files
  • all_cap.dict
  • cluster.dict
• access
  • sql data examples

• views.py, year range
• fetch_display_given_keywords, alert, reminds users that some keywords don't exist

Issues with the old methods

• the most frequent word can't represent the cluster well
• the cluster itself could be problematic
  • e.g. Very large l2 distance stddev, up to 60.
  • lacking domain knowledge: it categorize "student teacher methods" as "human"

todo: clean and upload the following code

• weighted Levenshtein distance based kmeans jupyter notebook
• code for building the index-term-to-cluster-representative-word mapper

A baseline API is required having the following necessary requirements:

• flask rest API that simply emits data to the console for now
• setting manageable from the config.ini file
• Modularity of choice of database

link

New feature
loc: Keywords Page:
In the form "keywords", if the input is "LSTM;ASR", then the return only shows the change curve w.r.t to "LSTM" and "ASR"

• A naive method, loop over each keyword at Django, query the database how many related papers are published during the given range

Structure:

Inputs:
---
Argument name: a description of that argument

Outputs:
---
i.e. return

Example:
---
inputs:...
outputs:...```

collect: main.py, xml_loader.py, info_extractor.py, converter.py
process: augment and to_json.py

Redirect the home url to ver0

• review the views.py code comments
• briefly comment the other codes
• bug fix: How PDF parser work.md incorrect pdf parser description

• Keyword extraction: change the workflow figure
• Access Readme: update the screen shots
• add database schema to the main readme
• directly show links to example outputs on the main readme
• extension, update the table and the main readme

Other checks, tbc

A prototype to test the basic features, e.g. list keywords over time, list the institute names and their papers

Issues with the old one

• failed to load a lot of pdfs(7k4 out of 9989 were successfully processed)
• incorrectly categorized some abstract content as authors or index terms

todo: clean and update code

Create a new README describing the project ideas. Please stick to the Markdown format.

• remove unused functions
• remove useless todos
• remove debug code
• better default values
• format the code

convert query results to xml files

create a basic schema for conf papers, will be written in xsd