lstm value error of different shape about tflearn

[aymericdamien]

You have to change last fully connected layer to:

net = tflearn.fully_connected(net, 10, activation='softmax')

In your example you seems to have 10 classes and not 2, so you need your softmax layer to have a 10 output dimension.

[aymericdamien]

in that line you have to change input_dim by your dictionary size (total number of different ids):
net = tflearn.embedding(net, input_dim=20000, output_dim=128)

[aymericdamien]

I guess it should be np.max(trainX)+1

[aymericdamien]

Can you tell what are these numbers? I thought they were ids? I think your main issue here is parsing your data.

[aymericdamien]

Please can you tell what does these data mean? It would be easier to understand what you are actually trying to do. I mean, these integers are an id (that represent words or whatever)? or are they real values?

[aymericdamien]

I see, first you can normalize your data by assigning an id (from 0 to your total number of event) for every event. After you can apply the embedding. But note that if you total number of event is too large, it will be very slow, so you can try to find some ways to reduce your data dimension (keep only events occurring more than X times, or apply a PCA transformation, etc...)

[aymericdamien]

oh, your problem is not about number of rows, but about your embedding layer dimensions, you have to first normalize your data and give them ids (0 to total number of event). What is your total number of distinct events?

[vinayakumarr]

I I modified but it is generating the following error

[vinayakumarr]

that i understood by looking at the error itself. But what exactly that dictionary size (total number different ids) with respect to my data set.

[vinayakumarr]

generating the following error

Segmentation fault (core dumped)

[aymericdamien]

What np.max(trainX)+1 is returning? Maybe your dictionary size is too large..

[vinayakumarr]

np.max(trainX)+1 returning 1930563585. How to solve this..

[vinayakumarr]

when i use print(np.max(trainX)+1), it returns 1930563585.

reading and parsing code is given below

//for reading csv file
data = pd.read_csv('Train.csv')

X = data.iloc[:,1:1805] //all columns
y = data.iloc[:,0] // only first clumn - class label

X_train, X_test, y_train, y_test = train_test_split(X, y,test_size=0.2,random_state=42) // using scikit learn train_test_split

// for converting into list of values
print("Preprocessing")
X_train1 = X_train.values.T.tolist()
X_test1 = X_test.values.tolist()
y_train1 = y_train.values.T.tolist()
y_test1 = y_test.values.tolist()

Sequence padding

trainX = pad_sequences(X_train1, maxlen=200, value=0.)
testX = pad_sequences(X_test1, maxlen=200, value=0.)

Converting labels to binary vectors

trainY = to_categorical(y_train, nb_classes=0)
testY = to_categorical(y_test, nb_classes=0)

Then finally model creation

[vinayakumarr]

The above code is for reading and parsing my dataset. Is there any problem in that?

[vinayakumarr]

I reduced my feature set now. I have 2000 row and 20 24 columns. Then also it is showing same error