hundredblocks / concrete_nlp_tutorial Goto Github PK

import keras
import nltk
import pandas as pd
import numpy as np
import re
import codecs

input_file = codecs.open("/home/vikash/gen/social_media-disaster-tweets-DFE.csv", 'r'
, encoding='utf-8', errors='replace')
print input_file
output_file = open("social_media_relevant_cols_clean.csv", "w")

def sanitize_characters(raw, clean):
print("enter the function")
for line in input_file:
output_file.write(line)

sanitize_characters(input_file, output_file)
questions = pd.read_csv("social_media_relevant_cols_clean.csv")
questions.columns = ('text', 'choose_one', 'choose_label')
questions.head()
questions.tail()

The following is the error

Traceback (most recent call last):
File "/home/vikash/gen/concept.py", line 22, in
questions.columns = ('text', 'choose_one', 'choose_label')
File "/home/vikash/.local/lib/python2.7/site-packages/pandas/core/generic.py", line 4385, in setattr
return object.setattr(self, name, value)
File "pandas/_libs/properties.pyx", line 69, in pandas._libs.properties.AxisProperty.set
File "/home/vikash/.local/lib/python2.7/site-packages/pandas/core/generic.py", line 645, in _set_axis
self._data.set_axis(axis, labels)
File "/home/vikash/.local/lib/python2.7/site-packages/pandas/core/internals.py", line 3323, in set_axis
'values have {new} elements'.format(old=old_len, new=new_len))
ValueError: Length mismatch: Expected axis has 13 elements, new values have 3 elements

Hi,

Can you please provide the requirements.txt for the versions of python and all libraries?

Thanks.

@hundredblocks

ValueError: Classification metrics can't handle a mix of continuous and multiclass targets

def plot_important_words(top_scores, top_words, bottom_scores, bottom_words, name):
y_pos = np.arange(len(top_words))
top_pairs = [(a,b) for a,b in zip(top_words, top_scores)]
top_pairs = sorted(top_pairs, key=lambda x: x[1])

bottom_pairs = [(a,b) for a,b in zip(bottom_words, bottom_scores)]
bottom_pairs = sorted(bottom_pairs, key=lambda x: x[1], reverse=True)

top_words = [a[0] for a in top_pairs]
top_scores = [a[1] for a in top_pairs]

bottom_words = [a[0] for a in bottom_pairs]
bottom_scores = [a[1] for a in bottom_pairs]

fig = plt.figure(figsize=(10, 10))  

plt.subplot(121)
plt.barh(y_pos,bottom_scores, align='center', alpha=0.5)
plt.title('Irrelevant', fontsize=20)
plt.yticks(y_pos, bottom_words, fontsize=14)
plt.suptitle('Key words', fontsize=16)
plt.xlabel('Importance', fontsize=20)

plt.subplot(122)
plt.barh(y_pos,top_scores, align='center', alpha=0.5)
plt.title('Disaster', fontsize=20)
plt.yticks(y_pos, top_words, fontsize=14)
plt.suptitle(name, fontsize=16)
plt.xlabel('Importance', fontsize=20)

plt.subplots_adjust(wspace=0.8)
plt.show()

top_scores = [a[0] for a in importance[1]['tops']]
top_words = [a[1] for a in importance[1]['tops']]
bottom_scores = [a[0] for a in importance[1]['bottom']]
bottom_words = [a[1] for a in importance[1]['bottom']]

plot_important_words(top_scores, top_words, bottom_scores, bottom_words, "Most important words for relevance")

Following is the erro
KeyError Traceback (most recent call last)
in ()
33
34
---> 35 bottom_scores =[a[1] for a in importance[1]['bottom']]
36 bottom_words = [a[0] for a in importance[1]['bottom']]
37 top_scores =[a[1] for a in importance[1]['tops']]

KeyError: 1

Hi there,

This is there error that comes up when trying to run the code. Any suggestions?
I'm currently running python 3.7.4.

Thanks,
Stef

When running questions = pd.read_csv("socialmedia_relevant_cols.csv")

I am having trouble setting up the environment and i think many are facing similar problems like #2 .If we can have a requirements.txt or environment.yml file,please add in the repo.It would be very helpful

I am new to ML and NLP do we need to add the Column class_label.
I have downloaded from this link https://www.figure-eight.com/data-for-everyone/ pertaining to Disasters on social media as in your blog do help. Thanks in advance.

On writing the cleaned file -
output_file = open("socialmedia_relevant_cols_clean.csv", "w")

It gave error -
UnicodeEncodeError: 'charmap' codec can't encode characters in position 64-65: character maps to

It has been replaced by Concatenate, so the code will not run as is.