bloominstituteoftechnology / ds-unit-4-sprint-1-nlp

DS-Unit-4-Sprint-1-NLP

ds-unit-4-sprint-1-nlp's Issues

Table Comparing SKlearn API to Gensim Comparison

Link in module 1 assignment still points to "Master" instead of "main"

data link in https://github.com/LambdaSchool/DS-Unit-4-Sprint-1-NLP/blob/main/module1-text-data/LS_DS_411_Text_Data_Assignment.ipynb should point to main instead of master.

module 1 lecture notebook regex

re.sub(r'[^a-zA-Z ^0-9]', '', sample)

that regex pattern will not replace any '^' in the string.

re.sub(r'[^a-zA-Z ^0-9]', '', 'Hi, Joe, look up ^ there.')

'Hi Joe look up ^ there'

should be
re.sub(r'[^a-zA-Z 0-9]', '', sample)

Downgrade Python Version

Thinc and Spacy require Python 3.7 - the conda environment instructions say to use Python 3.8.

Training the model with gridsearch outside of the pipeline.

Due to the computational expense of vectorizing raw text data, if the model is trained via the full pipeline, then for every param combo in the gridsearch, the training data is revectorizing unnecessarily. Also the .fit method for pipeline does not allow for passing parameters to do more robust things such as early stopping.

Time Difference:
full pipeline - several minutes to train one param combo
outside pipeline - 2 seconds to train one param combo all the way to early stopping limit

Proposed Solution:
create preprocess pipeline -> prepocess training data -> train model -> add best_estimator_ to pipeline via nesting

Unused variables in compute_coherence_values

In your function for computing coherence values in LS_DS_414_Topic_Modeling_Lecture.ipynb, both tokens and stream are unused, which means that the parameter path is also unused. Recommend removing them.

Typo in Module1 Objective 1

This doesn't work.

Sketch out NLP project for students

Requirements - update

update numpy to numpy==1.16 to avoid gensim breaking

---

add:
import warnings
warnings.filterwarnings('ignore')

to avoid this:

DeprecationWarning: Calling np.sum(generator) is deprecated, and in the future will give a different result. Use np.sum(np.from_iter(generator)) or the python sum builtin instead.
score += np.sum(cnt * logsumexp(Elogthetad + Elogbeta[:, int(id)]) for id, cnt in doc)

happening about 100 times.

Litterary Typo

Litterary == literary

Update Pandas code to 1.0

From Maxie:

pointer:
df['brand'] = df['brand'].apply(lambda x: x.lower())
can be simplified to
df['brand'] = df['brand'].str.lower()

Pip installation issues

pip install -r requirements.txt

Results in...

ERROR: Could not find a version that satisfies the requirement mkl-fft (from -r requirements.txt (line 45)) (from versions: none)
ERROR: No matching distribution found for mkl-fft

And then also...

ERROR: Could not find a version that satisfies the requirement mkl-random (from -r requirements.txt (line 52)) (from versions: none)
ERROR: No matching distribution found for mkl-random (from -r requirements.txt (line 52))

Gensim + Generators

Remind students that gensim can use more than generators

Fix Submissions No in M3

Max submissions should be 20 instead of 2 in nb copy.