aapatel09 / handson-unsupervised-learning

Code for Hands-on Unsupervised Learning Using Python (O'Reilly Media)

Hands-on Unsupervised Learning Using Python

This repo contains the code for the O'Reilly Media, Inc. book "Hands-on Unsupervised Learning Using Python: How to Build Applied Machine Learning Solutions from Unlabeled Data" by Ankur A. Patel.

Official Book Website: https://www.unsupervisedlearningbook.com/thebook

Available on Amazon: https://www.amazon.com/Hands-Unsupervised-Learning-Using-Python/dp/1492035645

Available on O'Reilly Safari: https://www.oreilly.com/library/view/hands-on-unsupervised-learning/9781492035633/

More on the Author: https://www.ankurapatel.io

Release Updates

May 2021: Added support for TensorFlow 2.x, Fashion MNIST examples, and Tensorboard for Dimensionality Reduction.

Book Description

Many industry experts consider unsupervised learning the next frontier in artificial intelligence, one that may hold the key to the holy grail in AI research, the so-called general artificial intelligence. Since the majority of the world's data is unlabeled, conventional supervised learning cannot be applied; this is where unsupervised learning comes in. Unsupervised learning can be applied to unlabeled datasets to discover meaningful patterns buried deep in the data, patterns that may be near impossible for humans to uncover.

Author Ankur Patel provides practical knowledge on how to apply unsupervised learning using two simple, production-ready Python frameworks - scikit-learn and TensorFlow. With the hands-on examples and code provided, you will identify difficult-to-find patterns in data and gain deeper business insight, detect anomalies, perform automatic feature engineering and selection, and generate synthetic datasets. All you need is programming and some machine learning experience to get started.

• Compare the strengths and weaknesses of the different machine learning approaches: supervised, unsupervised, and reinforcement learning
• Set up and manage a machine learning project end-to-end - everything from data acquisition to building a model and implementing a solution in production
• Use dimensionality reduction algorithms to uncover the most relevant information in data and build an anomaly detection system to catch credit card fraud
• Apply clustering algorithms to segment users - such as loan borrowers - into distinct and homogeneous groups
• Use autoencoders to perform automatic feature engineering and selection
• Combine supervised and unsupervised learning algorithms to develop semi-supervised solutions
• Build movie recommender systems using restricted Boltzmann machines
• Generate synthetic images using deep belief networks and generative adversarial networks
• Perform clustering on time series data such as electrocardiograms
• Explore the successes of unsupervised learning to date and its promising future

Google Colaboratory

If you wish use Google Colab (instead of your local machine), please follow these instructions to run the code on Google Colab.

Setup Main Conda Environment

If you wish to run this repo on your local machine, please follow these instructions below.

1. If you are on macOS, install Xcode Command Line Tools using xcode-select --install in Terminal.

2. Install the Miniforge distribution of Python 3.8 based on your OS. If you are on Windows, you can choose the Anaconda distribution of Python 3.8 instead of the Miniforge distribution, if you wish to.

3. For NVIDIA GPU support, install CUDA 11.0. This is only available on select NVIDIA GPUs.

4. Set up new Anaconda environment and follow these instructions based on your OS.

For Windows:

```
conda env create -f environment_windows.yml
conda activate unsupervisedLearning
pip install -r requirements_windows.txt
```

For macOS:

```
conda env create -f environment_mac.yml
conda activate unsupervisedLearning
pip install -r requirements_mac.txt
```

5. Download data from Google Drive (files are too large to store and access on Github).

   https://drive.google.com/drive/folders/1TQVOPUU4tVOYZvdpbxUo6uOCh0jvWNhv?usp=sharing
   

6. Run the notebooks using Jupyter.

   jupyter notebook
   

7. If you encounter any issues or errors with the setup or the code or anything else, please email the author at [email protected].

Set up TensorFlow for macOS Conda Environment

Please follow these instructions to set up TensorFlow for macOS.

For macOS:

```
conda env create -f environment_tensorflow_mac.yml
conda activate tensorflow_mac
pip install -r requirements_tensorflow_mac.txt

For Apple Silicon Mac (M1):
	pip install --upgrade --force --no-dependencies https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_addons_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl

For Intel Mac:
	pip install --upgrade --force --no-dependencies https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_x86_64.whl https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_addons_macos-0.1a3-cp38-cp38-macosx_11_0_x86_64.whl 
```

Please refer to this TensorFlow for macOS guide if you run into issues or contact us.

handson-unsupervised-learning's Issues

credit_card.csv is missing

08 Autoencoders possible error

On line 36, you are passing two arguments to anomalyScores; X_test and predictions. Is that right? Shouldn't the arguments be y_test and predictions?

about reproducibility

In[50] of 08_autoencoders.ipynb cannot be reproduced.
I'm using Python 3.7 tensorflow 1.14.

Unnecessary complexity (I think)

I do not know if this is an issue of the code, or just the consequence of my limited understanding of what is going on in a piece of code. In the notebook 02_end_to_end_machine_learning_project.ipynb, why the for loop is like this in the section about logistic regression:

for train_index, cv_index in k_fold.split(np.zeros(len(X_train)), y_train.ravel()):

Instead of just

for train_index, cv_index in k_fold.split(X_train, y_train):

What is behind that np.zeros and the call to the method ravel()?

If there is no reason for doing it the way it is done, I think the simpler version would be better, given that this is code to teach people how to use scikitlearn library.

Group segmentation

NameError Traceback (most recent call last)
in
----> 1 im = impute.SimpleImputer(strategy='mean')
2 data.loc[:,fillWithMean] = im.fit_transform(data[fillWithMean])
3
4 data.loc[:,fillWithZero] = data.loc[:,fillWithZero].fillna(value=0,axis=1)

NameError: name 'impute' is not defined

How to use this repository with the Hands-on Unsupervised Learning Book? / Missing files

Missing

https://github.com/aapatel09/handson-unsupervised-learning/tree/master/datasets/credit_card_data/credit_card.csv

file.

Otherwise very nice book. 🙂

Anomaly score for anomaly detection

I have to configure an anomaly detection system. I found very useful what you wrote but I have a doubt.
I train PCA on the the train set. Then I get anomalyScores with the following function

def anomalyScores(originalDF, reducedDF):
    loss = np.sum((np.array(originalDF)-np.array(reducedDF))**2, axis=1)
    loss = pd.Series(data=loss,index=originalDF.index)
    loss = (loss-np.min(loss)) / (np.max(loss)-np.min(loss))
    return loss

At this point I try to use this trained algorithm for the test set.

The point is that I'am not sure that anomalyScores so defined is suitble for this task.
Maybe using a MinMaxScaler would be better (or using this function but annotating the minimum and maximum obtained during the first evaluation) ?
This because consider the following example at inference time:

#during training I got the following losses
rec_error = np.array([0, 0.1, 1])
loss = (loss-np.min(loss)) / (np.max(loss)-np.min(loss)) #returns [0, 0.1, 1]

#now during testing suppose I got these reconstruction errors
rec_error = np.array([0, 0.1, 0.2])
loss = (loss-np.min(loss)) / (np.max(loss)-np.min(loss)) #returns [0, 0.5, 1]

So a reconstruction error of 0.1 wuold be considered much larger in the second example even if the trained model is the same.
While with a MinMaxScaler I would get:

from sklearn.preprocessing import MinMaxScaler
from sklearn.utils.validation import check_is_fitted

rec_error_train = np.array([ [0.0], [0.1], [1]])
rec_error_test = np.array([ [0.0], [0.1], [0.2]])
scaler = MinMaxScaler((0,1)).fit(rec_error_train)

loss_train = scaler.transform(rec_error_train) #returns [0, 0.1, 1]
loss_test = scaler.transform(rec_error_test) #returns [0, 0.1, 0.2]

What do you think?

02_end_to_end_machine_learning_project.ipynb Error

In [32]:
preds = pd.concat([y_train,predictionsBasedOnKFolds.loc[:,1]], axis=1)

the above is wrong... it would always grab prediction for class 1

so if we had predict_proba give us something like
index prob0 prob1
1 0.9 0.1

and label is
index label
1 0

so above would produce preds as
index trueLabel prection
1 0 0.1

should be 0.9, not 0.1

Train set for anomaly detection with PCA

I am trying to do anomaly detection with Normal PCA and 27 principal components as presented in the book but if I run the model on the entire dataset (without splitting into train and set) I get a completely different result:

data = pd.read_csv('credit_card.zip')
X = data.copy().drop(["Class"],axis=1)
y = data['Class'].copy()

featuresToScale = X.columns
scaler = StandardScaler(copy=True)
X.loc[:,featuresToScale] = scaler.fit_transform(X[featuresToScale])

pca = PCA(n_components=n_components, random_state=42)

X_train_PCA = pca.fit_transform(X)
X_train_PCA = pd.DataFrame(data=X_train_PCA, index=X.index)

X_train_PCA_inverse = pca.inverse_transform(X_train_PCA)
X_train_PCA_inverse = pd.DataFrame(data=X_train_PCA_inverse, index=X.index)

anomalyScoresPCA = anomalyScores(X, X_train_PCA_inverse)
plotResults(y, anomalyScoresPCA, False) 

The score training PCA only on the train set (after splitting into train and test) is 0.69

Do you have any ideas on why this appens ?

xgboost on mac

the easiest way for me to install xgboost was through conda:
conda install -c conda-forge xgboost

09_semisupervised - LightGBMError: Do not support special JSON characters in feature name.

Just trying to execute the code as is and I'm getting the following error:
trainingScores = []
cvScores = []
predictionsBasedOnKFolds = pd.DataFrame(data=[], index=y_train.index,
columns=['prediction'])

for train_index, cv_index in k_fold.split(np.zeros(len(X_train)),
y_train.ravel()):
X_train_fold, X_cv_fold = X_train.iloc[train_index,:],
X_train.iloc[cv_index,:]
y_train_fold, y_cv_fold = y_train.iloc[train_index],
y_train.iloc[cv_index]

lgb_train = lgb.Dataset(X_train_fold, y_train_fold)
lgb_eval = lgb.Dataset(X_cv_fold, y_cv_fold, reference=lgb_train)
gbm = lgb.train(params_lightGB, lgb_train, num_boost_round=2000,
               valid_sets=lgb_eval, early_stopping_rounds=200)

loglossTraining = log_loss(y_train_fold, gbm.predict(X_train_fold, \
                            num_iteration=gbm.best_iteration))
trainingScores.append(loglossTraining)

predictionsBasedOnKFolds.loc[X_cv_fold.index,'prediction'] = \
    gbm.predict(X_cv_fold, num_iteration=gbm.best_iteration) 
loglossCV = log_loss(y_cv_fold, \
    predictionsBasedOnKFolds.loc[X_cv_fold.index,'prediction'])
cvScores.append(loglossCV)

print('Training Log Loss: ', loglossTraining)
print('CV Log Loss: ', loglossCV)

loglossLightGBMGradientBoosting = log_loss(y_train,
predictionsBasedOnKFolds.loc[:,'prediction'])
print('LightGBM Gradient Boosting Log Loss: ',
loglossLightGBMGradientBoosting)

---

LightGBMError Traceback (most recent call last)
in
14 lgb_eval = lgb.Dataset(X_cv_fold, y_cv_fold, reference=lgb_train)
15 gbm = lgb.train(params_lightGB, lgb_train, num_boost_round=2000,
---> 16 valid_sets=lgb_eval, early_stopping_rounds=200)
17
18 loglossTraining = log_loss(y_train_fold, gbm.predict(X_train_fold, \

~/.local/lib/python3.6/site-packages/lightgbm/engine.py in train(params, train_set, num_boost_round, valid_sets, valid_names, fobj, feval, init_model, feature_name, categorical_feature, early_stopping_rounds, evals_result, verbose_eval, learning_rates, keep_training_booster, callbacks)
226 # construct booster
227 try:
--> 228 booster = Booster(params=params, train_set=train_set)
229 if is_valid_contain_train:
230 booster.set_train_data_name(train_data_name)

~/.local/lib/python3.6/site-packages/lightgbm/basic.py in init(self, params, train_set, model_file, model_str, silent)
1712 self.handle = ctypes.c_void_p()
1713 _safe_call(_LIB.LGBM_BoosterCreate(
-> 1714 train_set.construct().handle,
1715 c_str(params_str),
1716 ctypes.byref(self.handle)))

~/.local/lib/python3.6/site-packages/lightgbm/basic.py in construct(self)
1083 init_score=self.init_score, predictor=self._predictor,
1084 silent=self.silent, feature_name=self.feature_name,
-> 1085 categorical_feature=self.categorical_feature, params=self.params)
1086 if self.free_raw_data:
1087 self.data = None

~/.local/lib/python3.6/site-packages/lightgbm/basic.py in _lazy_init(self, data, label, reference, weight, group, init_score, predictor, silent, feature_name, categorical_feature, params)
913 raise TypeError('Wrong predictor type {}'.format(type(predictor).name))
914 # set feature names
--> 915 return self.set_feature_name(feature_name)
916
917 def __init_from_np2d(self, mat, params_str, ref_dataset):

~/.local/lib/python3.6/site-packages/lightgbm/basic.py in set_feature_name(self, feature_name)
1366 self.handle,
1367 c_array(ctypes.c_char_p, c_feature_name),
-> 1368 ctypes.c_int(len(feature_name))))
1369 return self
1370

~/.local/lib/python3.6/site-packages/lightgbm/basic.py in _safe_call(ret)
43 """
44 if ret != 0:
---> 45 raise LightGBMError(decode_string(_LIB.LGBM_GetLastError()))
46
47

LightGBMError: Do not support special JSON characters in feature name.

09_semisupervised - ValueError: Error when checking target: expected dense_3 to have shape (29,) but got array with shape (30,)

X_train has 30 columns but the input layer to the autoencoder expects 29.
np.shape(X_train)
(193823, 30)

model = Sequential()
model.add(Dense(units=40, activation='linear',
activity_regularizer=regularizers.l1(10e-5),
input_dim=29,name='hidden_layer'))

Were you supposed to drop the Amount column in addition to Class and Time?
dataX.columns
Index(['Unnamed: 0', 'V1', 'V2', 'V3', 'V4', 'V5', 'V6', 'V7', 'V8', 'V9',
'V10', 'V11', 'V12', 'V13', 'V14', 'V15', 'V16', 'V17', 'V18', 'V19',
'V20', 'V21', 'V22', 'V23', 'V24', 'V25', 'V26', 'V27', 'V28',
'Amount'],
dtype='object')

function view_digit (chapter 3) has hard-coded value

def view_digit(example):
label = y_train.loc[example] #compared to old label = y_train.loc[0]

Denoising Autoencoders target label

In model 9-10 of chapter 8 I don't understand why the target label of the fitting phase is the noisy input. Doing this the autoencoders learns to reconstruct the noisy input, isn't it ?
Should instead be written in this way ?

history = model.fit(x=X_train_AE_noisy, y=X_train_AE,  #using the original input as target
                            epochs=10,
                            batch_size=32,
                            shuffle=True,
                            validation_data=(X_val_AE_noisy, X_val),
                            verbose=1)

Original code is:

history = model.fit(x=X_train_AE_noisy, y=X_train_AE_noisy,
                        epochs=num_epochs,
                        batch_size=batch_size,
                        shuffle=True,
                        validation_data=(X_train_AE, X_train_AE),
                        verbose=1)

I have also replaced
validation_data=(X_train_AE, X_train_AE)
with
validation_data=(X_val_AE_noisy, X_val)
Where X_val is obtained by splitting the train set into 2 subsets. What do you think ?

Moreover why predictions are made on the noised test set ? I tried to predict the original test set.
Executing the code with these updates I got:

Mean average precision over 10 runs: 0.6327761266275229
Coefficient of variation over 10 runs: 0.018682160539079985
[0.6243557484686052,
0.6346261618115446,
0.6111520781820431,
0.6294291611519081,
0.6387461341507094,
0.6258058349652793,
0.6550653293459515,
0.6473704778074738,
0.6260815948098096,
0.6351287455819044]

Where is the code for the optimal number of principal components (Chapter 4)?

In Chapter 4 (Anomaly Detection), in the PCA part, it is mentioned the the code for finding the optimal number of principal components to be used (reduced from 30 to 27 in the book) is in this repository, but I could not find it.

Where is it?

why not splitting them before scaling datasets

In your book, chapter2, if you scaling datasets first, it will leak the information of testing dataset.

02_end_to_end_machine_learning_project - LightGBM model not working

I had trouble installing lightgbm v3.2.1 (as suggested in the environment .yml) so I installed the latest I could, version 4.3.0. When I got to the lgb.train function in the book, the "early_stopping_rounds=200" parameter is used, but this parameter no longer (directly) exists in that function.

Omitting that parameter yields terrible results, but I found a compatibility fix on the LightGBM github repo. They removed this parameter and put it in something called callbacks.

I replaced "early_stopping_rounds=200" parameter with "callbacks=[lgb.early_stopping(stopping_rounds=200)]" and it worked just fine with results very similar to the book.

First time using github, so apologies if this doesn't go here. Hopefully this helps someone encountering the same issue!