microsoft / mcw-internet-of-things

MCW Internet of Things

Internet of Things

This workshop is archived and no longer being maintained. Content is read-only.

Fabrikam provides services and smart meters for enterprise energy (electrical power) management. Their You-Left-The-Light-On service enables the enterprise to understand their energy consumption. Fabrikam would like to become an authorized energy management solution provider. According to their Director of Analytics, Sam George, "We are investigating a move to the cloud to help our customers not only to meet data collection and reporting requirements, but also become the number one energy management solution provider." They are intending to enable their enterprise customers with a web-based dashboard where they can see historical trends of power consumption.

October 2021

Target audience

• Application Developer
• IoT Developer

Abstracts

Workshop

This workshop will guide you through an implementation of an end-to-end IoT solution simulating high velocity data emitted from smart meters and analyzed in Azure. You will design a lambda architecture, filtering a subset of the telemetry data for real-time visualization on the hot path, and storing all the data in long-term storage for the cold path.

At the end of this workshop, you will be better able to construct an IoT solution implementing device registration with the IoT Hub Device Provisioning Service and visualizing hot data with Power BI.

Whiteboard design session

In this whiteboard design session, you will work with a group to design an implementation of an end-to-end IoT solution simulating high velocity data emitted from smart meters and analyzed in Azure. You will design a lambda architecture, filtering a subset of the telemetry data for real-time visualization on the hot path, and storing all the data in long-term storage for the cold path.

At the end of this whiteboard design session, you will be better able to design an IoT solution implementing device registration with the IoT Hub Device Provisioning Service and visualizing hot data with Power BI.

Hands-on lab

In this hands-on lab, you will construct an end-to-end IoT solution simulating high velocity data emitted from smart meters and analyzed in Azure. You will design a lambda architecture, filtering a subset of the telemetry data for real-time visualization on the hot path, and storing all the data in long-term storage for the cold path.

At the end of this hands-on lab, you will be better able to build an IoT solution implementing device registration with the IoT Hub Device Provisioning Service and visualizing hot data with Power BI.

Azure services and related products

• Azure App Services
• Azure Blob Storage
• Azure Data Factory
• Azure Databricks
• Azure SQL Database
• Azure Stream Analytics
• Device Provisioning Service
• IoT Hub
• Power BI Desktop
• Visual Studio

Azure solution

Internet of Things

Related references

MCW

Help & Support

We welcome feedback and comments from Microsoft SMEs & learning partners who deliver MCWs.

Having trouble?

• First, verify you have followed all written lab instructions (including the Before the Hands-on lab document).
• Next, submit an issue with a detailed description of the problem.
• Do not submit pull requests. Our content authors will make all changes and submit pull requests for approval.

If you are planning to present a workshop, review and test the materials early! We recommend at least two weeks prior.

Please allow 5 - 10 business days for review and resolution of issues

mcw-internet-of-things's Issues

WDS trainer guide - Internet of Things: IoT Central

https://github.com/microsoft/MCW-Internet-of-Things/blob/main/Whiteboard%20design%20session/WDS%20trainer%20guide%20-%20Internet%20of%20Things.md#checklist-of-preferred-objection-handling

IoT central has been re-positioned as aPaaS (application platform as a service) See here as a reference
https://docs.microsoft.com/en-us/azure/architecture/example-scenario/iot/iot-central-iot-hub-cheat-sheet

As such, will suggest to update the paragraph to reflect.

June 2020 Update Suggestions

March 2019 - Scheduled content update

@ZoinerTejada @joelhulen @kylebunting
Solliance team -
This workshop is scheduled for an update in March. Please update this issue with suggested content changes. Once done, we'll assign to our SME team for review and additional feedback.

Next round of content update

For Solliance-
In the next round of content update, we need to:

1. Create content around how this scenario would be addressed via Delta architecture, as opposed to lambda architecture. Let me know if you need help
2. Include Edge Gateway Management
3. Include how to monitor Telemetry for debugging

Potential Security Alert

Hey Joel,

Can you please look at this and mark it accordingly?
https://github.com/Microsoft/MCW-Internet-of-Things/network/alert/Hands-on%20lab/lab-files/starter-project/SmartMeterSimulator/packages.config/Microsoft.Data.OData/open

Thanks,
Dawnmarie

October 2021 suggestions

Small remarks after guided HOL

Exercise 4.11
SELECT AVG(temp) AS Average**,** id
INTO powerbi
FROM temps
GROUP BY TumblingWindow**(minute,** 5**),** id

with the asterix (*), the query will not work.

Exercise 5 --> There are no steps that inform about the storage account, Add steps to add storage account

Thanks for the lab!

Work email is required

You can't complete this lab if you don't have work email where PowerBI is enabled. You can't open trial with personal accounts, and some organizations have disabled PowerBI from general use. This should be mentioned in the prerequisites.

March Update - WDS Trainer guide

In WDS trainer guide - Internet of Things.md, this is not correct:
"IoT Hub Message Routing might also be used, but this has the limitation that messages follow a single path, so with the current implementation, it would not be possible to send all messages to the cold path, while simultaneously sending some of the same messages into the hot path."

IoT Message Routing supports multiple routes. This, along with consumer groups, allows for data to go to multiple paths / applications.

I would also suggest replacing mention of "pre-configured solutions" with "Solution Accelerators"

Issue while running Command 3 in notebooks - Exercise 4 Task 4

In Excercise 4 Task 4 the command 3 in notebook smartmeters giving an error.

@DawnmarieDesJardins - please look into this as we have a workshop scheduled for 11th next week.

Will this lab be updated to VS2019?

Lab uses VS2017.

Issue in Exercise - 6

In Exercise - 6> Task - 2> step - 26, The edgeAgent status is in Failed state state. According to the lab guide it should be in running state.

We tried checking the logs and we found a permission denied error. We ran chmod 777 command and gave all permission to /etc/aziot/config.toml file. But still we are facing the same issue. Please look into this.

June 2021 Update Suggestions

Exercise 3: Task 2:

Exercise 3: Task 2 Step 2: Dataset avgtemps is not showing up in PowerBI.

Can you please look into this?

'DeviceState' could not be found

Not able to run the simulator following the instructions. Getting below error

Severity Code Description Project File Line Suppression State
Error CS0246 The type or namespace name 'DeviceState' could not be found (are you missing a using directive or an assembly reference?) SmartMeterSimulator C:\MCW-Internet-of-Things-master\MCW-Internet-of-Things-master\Hands-on lab\lab-files\starter-project\SmartMeterSimulator\Sensor.cs 18 Active

Getting Path error

While running the following command on Databriks (Exercise 4 > Task 4 > Step 17) I am getting a path error.

#Create a Dataframe containing data from all the files in blob storage, regardless of the folder they are located within.
df = spark.read.options(header='true', inferSchema='true').csv("dbfs:/mnt/smartmeters/*/*/*.csv",header=True)
print(df.dtypes)

I am getting an error that the path "dbfs:/mnt/smartmeters///*.csv" is not correct.

Instead of that, I tried the following command and it solved the error.

#Create a Dataframe containing data from all the files in blob storage, regardless of the folder they are located within.
df = spark.read.options(header='true', inferSchema='true').csv("dbfs:/mnt/smartmeters/*/*/*/*/*.csv",header=True)
print(df.dtypes)

September 2019 - Content update

This MCW is scheduled for content update.
Solliance - please review current content, open issue #20 and open PR #22 and recommended updates for SME review.

Broken link on page "https://github.com/snowstormlife/MCW-Internet-of-Things/blob/master/HOL/Hands-on%20lab%20step-by%20step%20-%20Internet%20of%20Things.md"

Link in the instructions that points to "Before the hands-on lab" is broken.

March updates to Trainer presentation

For the Trainer presentation:

• Slide 6: I don't think we refer to IoT Suite anymore. May need marketing to confirm.
• Slide 17: HTTP/1 Polling or AMQP is mentioned. Why isn't MQTT listed? MQTT is the preferred communication protocol. In terms of 'recommendations': 1. MQTT, 2. AMQP, 3. HTTP
• Slide 18: In the notes section we mention pre-configured solutions and IoT Suite. I think the new terminology is IoT Solution Accelerators.

Review June update and September test/fix

All changes are in the june-2018-update branch. Please review and merge to master.

MainForm.cs not opening

In Hands-on-lab, In Exercise 1 Task 2 the MainForm.cs in SmartMeterSimulator.sln is not opening up. Seems like there is some error in the MainForm.Designer.cs. Please look into this

Also the Virtual Machine Image Visual Studio Community 2019 (latest) on Windows Server 2016 (x64) mentioned in Before Hands-on-Lab is currently not available in the Azure Marketplace

@DawnmarieDesJardins Please check these issues

Change Step 8 of task 3

In this section

Under Step 8, change

sudo -HE /usr/bin/anaconda/bin/conda install pandas

to

sudo -HE /usr/bin/anaconda/bin/conda install pandas=0.20.3

The former causes an error when you are running the Scala code in Task 4, step 5

This MCW is tied to only MSDN subscription

This MCW requires that the user is using MSDN subscription, as Visual Studio Community with Windows Server is not available to another subscription types, such as Pay As You Go or Azure Passes. There is no mention about this requirement.

November test/fix

Please submit a PR for the November test/fix.