kevinbretonnelcohen / cord-19 Goto Github PK
View Code? Open in Web Editor NEWNatural language processing code for coronavirus corpus CORD-19
Natural language processing code for coronavirus corpus CORD-19
Currently has text that marks where the abstract is in the debugging output. Needs similar markers for the title and for the article body.
The code for frequency analysis needs to be refactored. The "pipeline" should be broken down into functions so that multiple datasets can be compared.
The input and output directory paths should be put into easy-to-find variables in order for this to be more easily adaptable by others.
Currently pulls text from titles, or abstracts, or the article body, or the full paper---but, not title + abstract, which seems like a reasonable unit of analysis.
When I read in the article body, I'm not preserving the divisions between paragraphs. It would be nice to keep those, although I don't need them right at this moment, since I'm doing purely lexical stuff.
Last action of the script should be to output a count of the files that have been processed....which it is, but clearly that count is not getting incremented, as it is being output as 0, despite the fact that a ton of output is being produced.
Currently has text that marks where the abstract is in the debugging output. Needs similar markers for the title and for the article body.
All ratios are coming out at 1, despite the fact that we are passing both the structured and the metamorphic tests...
I have some files for validation, but this still needs actual test cases.
I have a logic error: I need to go through all words in one corpus (and then, optionally, the other), but I'm going through the union of the words in both the corpus of interest and the reference corpus.
Currently I get the words that are over-represented in the corpus of interest as compared to the reference corpus, but not the ones that are over-represented in the reference corpus as compared to the corpus of interest. That's fine from the point of view of the research questions about the corpus of interest, but knowing which words are "over-represented" in the reference corpus as compared to the corpus of interest is a good sanity check on the functioning of the script, so let's add that in, too. It's a simple return to the previous approach of calculating the ratios for the union of the words in the vocabularies, rather than the ratios for only those words that are in the corpus of interest.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.