• Lab 1 - On the Gateway host, run a simulator to send IoT sensors data to the MQTT broker.

• Lab 2 - On the Gateway host, configure and start MiNiFi, which will read from the MQTT broker, filter and forward to the NiFi cluster.

• Lab3 - Create the MiNiFi flow on the Edge Flow Manager and publish it for the MiNiFi agent to start sending data to the NiFi cluster.

• Lab 4 - On Schema Registry, register the schema describing the data generated by the IoT sensors.

• Lab 5 - On the NiFi cluster, prepare the data and send it to the Kafka cluster.

• Lab 6 - On the Streams Messaging Manager (SMM) Web UI, monitor the Kafka cluster and confirm data is being ingested correctly.

• Lab 7 - On the CDH cluster, process each record using Spark Streaming, calling the Model endpoint and save results to Kudu.

• Lab 8 - On the CDH cluster, pull reports on upcoming predicted machine failures using Impala and Hue.

• Lab 9 - On the CDSW cluster, train your model with the Experiment feature.

• Lab 10 - On the CDSW cluster, deploy the model into production with the Model feature.

• Laptop with a supported OS (Windows 7 not supported).

• Ability to SSH into remote hosts from Windows or Mac. For Windows machines, install Putty or even better install OpenSSH for PowerShell.

• A modern browser like Google Chrome (IE not supported).

You should have 2 addresses for you one-node cluster: the public DNS name and the public IP address. With those addresses you can test the following connectivity to your cluster:

1. Ensure you can SSH into the cluster (using either the DNS name or IP address)

2. Ensure you can connect to the following service using your browser:

   Service	URL	Credentials
   Cloudera Manager	http://<public_dns>:7180/	admin/admin
   Edge Flow Manager	http://<public_dns>:10080/efm/ui/
   NiFi	http://<public_dns>:8080/nifi/
   NiFi Registry	http://<public_dns>:18080/nifi-registry/
   Schema Registry	http://<public_dns>:7788/
   SMM	http://<public_dns>:9991/
   Hue	http://<public_dns>:8888/
   CDSW	http://cdsw.<public_IP>.nip.io/	admin/supersecret1

3. Login into Cloudera Manager and familiarize yourself with the services installed

4. Login into Hue. As you are the first user to login into Hue, you are granted admin privileges. At this point, you won’t need to do anything on Hue, but by logging in, CDH has created your HDFS user and folder, which you will need for the next lab.

5. When you log into CDSW for the first time, click on "Sign Up for a New Account" and create your own account.

Ensure you remember the username and password, as you will use these throughout this workshop.

Below a screenshot of Chrome open with 8 tabs, one for each service.

Before we start building our flow, let’s create a Process Group to help organizing the flows in the NiFi canvas and also to enable flow version control.

1. Open the NiFi Web UI, create a new Process Group and name it something like Process Sensor Data.

   

2. We want to be able to version control the flows we will add to the Process Group. In order to do that, we first need to connect NiFi to the NiFi Registry. On the NiFi global menu, click on "Controller Services", navigate to the "Registry Clients" tab and add a Registry client with the following URL:

   Name: NiFi Registry
   URL: http://edge2ai-1.dim.local:18080

   
   

3. On the NiFi Registry Web UI, add another bucket for storing the Sensor flow we’re about to build'. Call it SensorFlows:

   

4. Back on the NiFi Web UI, to enable version control for the Process Group, right-click on it and select Version > Start version control and enter the details below. Once you complete, a will appear on the Process Group, indicating that version control is now enabled for it.

   Registry: NiFi Registry
   Bucket: SensorFlows
   Flow Name: SensorProcessGroup

5. Let’s also enable processors in this Process Group to use schemas stored in Schema Registry. Right-click on the Process Group, select Configure and navigate to the Controller Services tab. Click the + icon, add a HortonworksSchemaRegistry service and configure it with the following URL:

   URL: http://edge2ai-1.dim.local:7788/api/v1

   

6. Still on the Controller Services screen, let’s add two additional services to handle the reading and writing of JSON records. Add the following two services:

   • JsonTreeReader, with the following properties:

     Schema Access Strategy: Use 'Schema Name' Property
     Schema Registry: HortonworksSchemaRegistry
     Schema Name: SensorReading
     Schema Version: 1

   • JsonRecordSetWriter, with the following properties:

     Schema Write Strategy: HWX Content-Encoded Schema Reference
     Schema Access Strategy: Inherit Record Schema
     Schema Registry: HortonworksSchemaRegistry

7. Start all the 3 services you just created (click on the lightning bolt icons)

   

1. Double-click on the newly created process group to expand it.

2. Inside the process group, add a new Input Port and name it "Sensor Data"

3. Add a PublishKafkaRecord_2.0 processor and configure it as follows:

   Kafka Brokers: edge2ai-1.dim.local:9092
   Topic Name: iot
   Record Reader: JsonTreeReader
   Record Writer: JsonRecordSetWriter
   Use Transactions: false

4. Connect the Input Port to the PublishKafkaRecord processor.

5. Add a new Funnel to the canvas and connect the PublishKafkaRecord processor to it. When the "Create connection" dialog appears, select "success" and click "Add".

   

6. Double-click the PublishKafkaRecord processor, navigate to the Settings tab and check "failure" under the Automatically Terminate Relationships section to terminate here records that were not successfully sent to Kafka.

   

7. Start the PublishKafkaRecord processor and the Input Port. Your canvas should now look like the one below:

   

8. The only thing that remains to be configured now is to finally connect the "from Gateway" Input Port to the flow in the "Processor Sensor Data" group. To do that, first go back to the root canvas by clicking on the NiFi Flow link on the status bar.

   

9. Connect the Input Port to the Process Sensor Data Process Group by dragging the destination of the current connection from the funnel to the Process Group. When prompted, ensure the "To input" fields is set to the Sensor data Input Port.

   
   

10. Refresh the screen (Ctrl+R on Linux/Windows; Cmd+R on Mac) and you should see that the records that were queued on the "from Gateway" Input Port disappeared. They flowed into the Process Sensor Data flow. If you expand the Process Group you will see those records on the "success" output queue after they were processed by the KafkaPublishRecord processor.

    

    At this point, the messages are already in the Kafka topic. You can add more processors as needed to process, split, duplicate or re-route your FlowFiles to all other destinations and processors.

11. To complete this Lab, let’s commit and version the work we’ve just done. Go back to the NiFi root canvas, clicking on the "Nifi Flow" breadcrumb. Right-click on the Process Sensor Data Process Group and select Version > Commit local changes. Enter a descriptive comment and save.

1. Open CDSW Web UI and click on sign up for a new account. As you’re the first user to login into CDSW, you are granted admin privileges.

   

2. Navigate to the CDSW Admin page to fine tune the environment:

   1. in the Engines tab, add in Engines Profiles a new engine (docker image) with 2 vCPUs and 4 GB RAM, while deleting the default engine.

   2. add the following in Environmental Variables:

      HADOOP_CONF_DIR=/etc/hadoop/conf/

      

1. Return to the main page and click on New Project, using this GitHub project as the source: https://github.com/asdaraujo/edge2ai-workshop

   

2. Now that your project has been created, click on Open Workbench and start a Python3 Session

   

3. Once the Engine is ready, run the following command to install some required libraries:

   !pip3 install --upgrade pip scikit-learn

4. The project comes with a historical dataset. Copy this dataset into HDFS:

   !hdfs dfs -put data/historical_iot.txt /user/$HADOOP_USER_NAME

   

5. You’re now ready to run the Experiment to train the model on your historical data.

6. You can stop the Engine at this point.

Open the file cdsw.iot_exp.py. This is a python program that builds a model to predict machine failure (the likelihood that this machine is going to fail). There is a dataset available on hdfs with customer data, including a failure indicator field.

The program is going to build a failure prediction model using the Random Forest algorithm. Random forests are ensembles of decision trees. Random forests are one of the most successful machine learning models for classification and regression. They combine many decision trees in order to reduce the risk of overfitting. Like decision trees, random forests handle categorical features, extend to the multiclass classification setting, do not require feature scaling, and are able to capture non-linearities and feature interactions.

spark.mllib supports random forests for binary and multiclass classification and for regression, using both continuous and categorical features. spark.mllib implements random forests using the existing decision tree implementation. Please see the decision tree guide for more information on trees.

The Random Forest algorithm expects a couple of parameters:

• numTrees: Number of trees in the forest.

  Increasing the number of trees will decrease the variance in predictions, improving the model’s test-time accuracy. Training time increases roughly linearly in the number of trees.

• maxDepth: Maximum depth of each tree in the forest.

  Increasing the depth makes the model more expressive and powerful. However, deep trees take longer to train and are also more prone to overfitting. In general, it is acceptable to train deeper trees when using random forests than when using a single decision tree. One tree is more likely to overfit than a random forest (because of the variance reduction from averaging multiple trees in the forest).

In the cdsw.iot_exp.py program, these parameters can be passed to the program at runtime, to these python variables:

Also note the quality indicator for the Random Forest model, are written back to the Data Science Workbench repository:

These indicators will show up later in the Experiments dashboard.

1. Now, run the experiment using the following parameters:

   numTrees = 20 numDepth = 20

2. From the menu, select Run → Run Experiments…​. Now, in the background, the Data Science Workbench environment will spin up a new docker container, where this program will run.

   

3. Go back to the Projects page in CDSW, and hit the Experiments button.

4. If the Status indicates ‘Running’, you have to wait till the run is completed. In case the status is ‘Build Failed’ or ‘Failed’, check the log information. This is accessible by clicking on the run number of your experiments. There you can find the session log, as well as the build information.

   

5. In case your status indicates ‘Success’, you should be able to see the auroc (Area Under the Curve) model quality indicator. It might be that this value is hidden by the CDSW user interface. In that case, click on the ‘3 metrics’ links, and select the auroc field. It might be needed to de-select some other fields, since the interface can only show 3 metrics at the same time.

   

6. In this example, ~0.8478. Not bad, but maybe there are better hyper parameter values available.

1. Go back to the Workbench and run the experiment 2 more times and try different values for NumTrees and NumDepth. Try the following values:

1. When all runs have completed successfully, check which parameters had the best quality (best predictive value). This is represented by the highest ‘area under the curve’, auroc metric.

   

1. Select the run number with the best predictive value, in this case, experiment 2. In the Overview screen of the experiment, you can see that the model in spark format, is captured in the file iot_model.pkl. Select this file and hit the Add to Project button. This will copy the model to your project directory.

   
   

1. Open the project you created in the previous lab, and examine the file in the Workbench. This PySpark program uses the pickle.load mechanism to deploy models. The model it refers to the iot_modelf.pkl file, was saved in the previous lab from the experiment with the best predictive model.

   There is a predict definition which is the function that calls the model, using features, and will return a result variable.

2. Before deploying the model, try it out in the Workbench: launch a Python3 engine and run the code in file cdsw.iot_model.py. Then call the predict() method from the prompt:

   predict({"feature": "0, 65, 0, 137, 21.95, 83, 19.42, 111, 9.4, 6, 3.43, 4"})

   

3. The functions returns successfully, so we know we can now deploy the model. You can now stop the engine.

1. From the projects page of your project, select the Models button. Select New Model and populate specify the following configuration:

   Name:          IoT Prediction Model
   Description:   IoT Prediction Model
   File:          cdsw.iot_model.py
   Function:      predict
   Example Input: {"feature": "0, 65, 0, 137, 21.95, 83, 19.42, 111, 9.4, 6, 3.43, 4"}
   Kernel:        Python 3
   Engine:        2 vCPU / 4 GB Memory
   Replicas:      1

   

2. If all parameters are set, you can hit the Deploy Model button. Wait till the model is deployed. This will take several minutes.

1. After several minutes, your model should get to the Deployed state. Now, click on the Model Name link, to go to the Model Overview page. From the that page, hit the Test button to check if the model is working.

2. The green color with success is telling that our REST call to the model is technically working. And if you examine the response: {"result": 1}, it returns a 1, which mean that machine with these features is likely to stay healthy.

   

3. Now, lets change the input parameters and call the predict function again. Put the following values in the Input field:

   {
     "feature": "0, 95, 0, 88, 26.62, 75, 21.05, 115, 8.65, 5, 3.32, 3"
   }

4. With these input parameters, the model returns 0, which means that the machine is likely to break. Take a note of the AccessKey as you will need this for lab 7.