jasonkessler / scattertext-pydata

Notebooks for the Seattle PyData 2017 talk on Scattertext

Scattertext-PyData

Notebooks for the Seattle PyData 2017 talk on Scattertext

A guide to using the python package Scattertext. If you feel so moved, please star it, fork it, or even contribute!

Check out the introductory presentation here.

Video

Using the notebooks

The notebooks look best in Chrome.

Slow but interactive way

In order to use these notebooks, please execute the following commands, please clone this repo and run (in Python 3):

$ git clone https://github.com/JasonKessler/Scattertext-PyData
$ pip3 install scattertext agefromname
$ cd Scattertext-PyData
$ jupyter notebook

Fast and non-interative way

• First Notebook how to use Scattertext to visualize differences in document types.
• Second Notebook how to use Scattertext and AgeFromName to understand how lanugage, gender and political party intersect.
• Third Notebook how to use Scattertext to visualize how the same word or semantic type is discussed different between document categories. In this case, we explore how "jobs" is discussed differently by Republicans and Democrats.

scattertext-pydata's Issues

Cannot import scattertext

I am trying to import scattertext on AWS SageMaker with a conda_python3 kernel. I downgraded numpy to 1.20.3 and installed a certain daal version based on the error messages.
!pip install --upgrade numpy==1.20.3
!pip install daal==2021.2.3

Now I get the error message "ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject"

This still seems like a numpy package collision. Is there a Requirements.txt for scattertext?

PyData-Scattertext-Part-1.ipynb needs to be updated

Hi Jason, just want to say thanks for your codes! I have been looking for a long time for something like this and was almost going to give up. Now, I just hope it will work on my dataset. By the way, your corpus code does not work and needs to be updated. It should be:

corpus = st.CorpusFromPandas(convention_df, category_col='party', text_col='text',nlp=nlp).build()

instead of

corpus = st.CorpusFromParsedDocuments(convention_df, category_col='party', parsed_col='parsed').build()

Harmonic mean error

Hello,
I use the Jupyter Notebook with my own data. When running the following part of the code

def normcdf(x):
    return norm.cdf(x, x.mean(), x.std())
term_freq_df['eight_precision_normcdf'] = normcdf(term_freq_df['eight_precision'])
term_freq_df['eight_freq_pct_normcdf'] = normcdf(term_freq_df['eight_freq_pct'])
term_freq_df['eight_scaled_f_score'] = hmean([term_freq_df['eight_precision_normcdf'], term_freq_df['eight_freq_pct_normcdf']])
term_freq_df.sort_values(by='eight_scaled_f_score', ascending=False).iloc[:10]

I get the following error

The column categories run from 'zero' to 'eight'. Any suggestion to overcome this problem is welcome

Dimensionality reduction

This is great! How can one incorporate dimensionality reduction into the pipeline? For substantive and speed reasons, I'd like to exclude the most and least common words:

corpus = st.CorpusFromPandas(df,
category_col='country',
text_col='text',
nlp=nlp,
# can we discard 1st and 99th percentile of words here?
).build()