esri / event-hub-sender

C#.Net console application to send data from a delimited text file to an Azure Event Hub.

event-hub-sender

This is a C# .Net console application to send data from a delimited text file to an Azure Event Hub. Publish this console app to your Azure portal to create a continuously running source of messages to your Event Hub to support demos of Velocity and GeoEvent Server.

Features

• Use event-hub-sender to send records from your delimited data file as event messages to an Azure Event Hub.
• Provides a source of messages for your Velocity and GeoEvent Server demos.

Instructions

1. Clone the repo in Visual Studio Code.

2. Update the app.config file as follows:

   • connectionString – The complete connection string to your Event Hub.
   • fileUrl – Enter the URL to the simulation delimited file containing the data to be sent between the empty quotes for the value of fileUrl. If using our sample file, set this value to “https://a4iot-test-data.s3.us-west-2.amazonaws.com/point/Charlotte_Simulations/57Buses_in_CharlotteNC.csv”.
   • hasHeaderRow – Enter true if the simulation file has a header row of field names, false if not. If using our sample csv file, set this value to “true”.
   • fieldDelimiter – the delimiter between fields in the simulation file. If using our sample csv file, set this value to “,”.
   • convertToJson - Set to true to convert the delimited records to JSON, false to leave them in delimited format.
   • numLinesPerBatch – Enter the number of lines to send with each batch. The app will read this number of lines from the simulation file, bundle them into a batch of events and send them to the Event Hub all at once. Then it will read the next set of lines into a batch, send them and repeat until the end of the simulation csv file is reached and all lines have been sent. You might set this value to be equal to the number of unique track ids in your data or use it in conjunction with the sendInterval to simply control the rate of events into your Event Hub. If using our sample csv file, there are 57 unique track ids.
   • sendInterval – Enter the number of milliseconds between batches sent to the Event Hub. This time includes the time required to send a batch. Thus, if this value is set to 1000ms, and it takes 700ms to send a batch, the app will wait 300ms before sending the next batch. If it takes longer than this value to send a batch, it will not wait before sending the next batch.
   • timeField – The zero-based index of the field in the simulation file containing date values. If using our sample csv file, set this value to “0”.
   • setToCurrentTime – Enter true to update the values in the date field to the date and time the event is sent to the Event Hub. If using our sample csv file, set this value to “true”.
   • dateFormat – Optional, only used if setToCurrentTime is true. In that case the date values will be formatted as strings according to this formatter. If this value is empty, date values will be epochs. Formatting string can be standard or custom. See https://docs.microsoft.com/en-us/dotnet/standard/base-types/standard-date-and-time-format-strings and https://docs.microsoft.com/en-us/dotnet/standard/base-types/custom-date-and-time-format-strings
   • dateCulture - Optional, examples: "en-US", "es-ES", "fr-FR"; only used if setToCurrentTime is true and dateFormat is not empty. In that case date strings will be formatted according to the culture specified in this setting or the default culture if empty.
   • repeatSimulation - Enter true to indicate if the app, upon reaching the end of the simulation csv file, should return to the top of the file and repeat the simulation. or false to send the file once and stop.

3. Commit the changes in the app.config file

4. Deploy to Azure App Service to your Azure portal.

5. In the resulting App Service, configure the deployment source to be LocalGit.

6. Deploy to Web App.

7. Set up a feed in ArcGIS Velocity or an input in ArcGIS GeoEvent Server to ingest your Event Hub messages.

Detailed instructions are in the "Deploy EventHubSender to Azure App Service using Visual Studio Code .docx" file in this repo.

Requirements

• Microsoft Azure account with and active subscription. (Create one for free: https://azure.microsoft.com/free/?utm_source=campaign&utm_campaign=vscode-tutorial-appservice-extension&mktingSource=vscode-tutorial-appservice-extension)
• An Azure Event Hub. (Create one this way: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-create)
• Visual Studio Code (VS Code) installed on your local machine. (Get it here: https://code.visualstudio.com/)
• The Azure App Service extension for VS Code (install from within VS Code or get it here: https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-azureappservice)
• Git installed on your local machine. (Get it here: https://git-scm.com/)
• A delimited text file of events you wish to simulate. It must be hosted in a place where it is accessible by URL such as an Amazon S3 bucket. To get started you may use the sample file hosted in this repo (https://a4iot-test-data.s3.us-west-2.amazonaws.com/point/Charlotte_Simulations/57Buses_in_CharlotteNC.csv).

Resources

Issues

Find a bug or want to request a new feature? Please let us know by submitting an issue.

Contributing

Esri welcomes contributions from anyone and everyone. Please see our guidelines for contributing.

Licensing

Copyright 2021 Esri

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

A copy of the license is available in the repository's license.txt file.

ArcGIS Velocity, ArcGIS GeoEvent Server, Microsoft Azure, Azure Event Hub, C#, .Net