andpol5 / credit-classifier Goto Github PK

Hi,

I reproduced your code but found some problems with the cross validation step.

Since the 10 cv folds are run in the same tensorflow session, the data used for validation in later folds are already seen by the model when they were used as training data in earlier folds. In my opinions, this produces overly optimistic estimations.

In fact, when I tried leaving some data out just for testing, the test results give a precision at about 0.7, not very far from the results of the linear regression.

I reproduced your code getting similar results.

I have some doubts on the validation strategy.

The following line made me think that the code was selecting different sets of rows for train and validation:

for train_indices, val_indices in kf:

On the other hand, looking at the complete snippet:

for train_indices, val_indices in kf:
    # split the data into train and validation
    train_dataset = dataset[train_indices,:]
    val_dataset = dataset[train_indices,:]

Actually val_indices is not used in any other part of the code, and in reality looking at the above code it appears that we are using the same set for training and validation.

Am I missing something?