azure / integration-services-landing-zone-accelerator

Introduction to Azure Integration Services Cloud Adoption Framework

This article series outlines the process for establishing an Azure Integration Services platform into your organization’s cloud adoption efforts.

Executive summary of Azure Integration Services

These articles describe how Azure Integration Services workloads impact your overall strategy, cloud adoption plan, and environmental readiness efforts, with detailed guidance on common drift for each effort. This document highlights important best practices to adopt when provisioning Azure Integration Services and also provides automation that allows you to provision Azure Integration Services based upon the best practices that we outline in this document. To support your cloud adoption needs, the series also outlines considerations and best practices for managing governance and operations throughout an Azure Integration Services implementation.

Azure Integration Services (AIS) is the collective name given to a suite of related, but separate, resource offerings within Azure, which enable you to build effective integration solutions.

Those resources include:

• API Management

• Data Factory

• Event Grid

• Event Hubs

• Function Apps

• Logic Apps

• Service Bus

• Storage Accounts

To accelerate these efforts, the articles also include detailed technical resources that describe how to build an enterprise-scale landing zone that can support your mission-critical Azure Integration Services needs.

The enterprise architecture is broken down into six different design areas. You can find the links to each here:

Design Area	Considerations	Recommendations
Identity and Access Management	Design Considerations	Design Recommendations
Network Topology and Connectivity	Design Considerations	Design Recommendations
Security	Design Considerations	Design Recommendations
Management	Design Considerations	Design Recommendations
Governance	Design Considerations	Design Recommendations
Platform Automation and DevOps	Design Considerations	Design Recommendations
Reference Implementation

Next Steps to implement the Integration Services Landing Zone Accelerator

Pick one of the scenarios below to get started on a reference implementation.

▶️ Scenario 1: Enterprise Deployment of AIS

▶️ Scenario 2: Integrating ServiceNow with AIS

---

Got a feedback

Please leverage issues if you have any feedback or request on how we can improve on this repository.

---

Data Collection

The software may collect information about you and your use of the software and send it to Microsoft. Microsoft may use this information to provide services and improve our products and services. You may turn off the telemetry as described in the repository. There are also some features in the software that may enable you and Microsoft to collect data from users of your applications. If you use these features, you must comply with applicable law, including providing appropriate notices to users of your applications together with a copy of Microsoft's privacy statement. Our privacy statement is located at https://go.microsoft.com/fwlink/?LinkId=521839. You can learn more about data collection and use in the help documentation and our privacy statement. Your use of the software operates as your consent to these practices.

Telemetry Configuration

Telemetry collection is on by default.

To opt-out, set the variable enableTelemetry to false in Bicep/ARM file and disable_terraform_partner_id to false on Terraform files.

---

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

integration-services-landing-zone-accelerator's Issues

Apply identity based and network based security methods to the inter-service communication

Image links now broken in doc pages

The recent fix to the broken links ( PR #32 ) has indeed fixed the broken links on the landing page. However, now the image links are broken in those pages. Example would be the Network Topology page which has lots of embedded images.

Hope we can fix this before my INTEGRATE presentation in London on 7th June - as I was hoping to show this page! :-)

Pattern: Correlated Parallel Receive

Deployable Demo that shows how to receive two different, related messages and continue processing only when both messages have been received.

• Self-contained demo
• independently deployable
• independent documentation
• simple, intended to be built upon
• include message tracking and property promotion

Links to the design articles are broken

The recent reorganisation (commit # 8cdd84d) has broken all the links in the table on the main ReadMe page. The docs were moved from the root of the /docs folder into the /scenario1 folder, but the links were not updated.

Personally, I'm not sure why they were moved? The concepts discussed within these excellent pages would apply to all scenarios. Burying them under a specific scenario page makes them harder to find (especially when the links on the landing page don't take you there. ;-) )

Unable to deploy the landing zone into Australia East

Hi,
The deployment fails at times when resource name (like key vault) exceeds the allowed number of characters. For example, if ais is used as name and landing zone needs to be deployed in Australia East, it exceeds the limit i.e. between 3-24 characters. Would be good if character limit can be enforced at the time of creation.

Thanks

Scenario proposal: Orchestrate HTTP-triggered Logic Apps through the Durable Functions

Add Bicep Templates for Scenario 1

Functions Extension Version

The Logic Apps deployment script (logicApp.json) references V3 of Functions. Now that Logic Apps Supports V4 this can be updated to reflect latest version of runtime

"appSettings": [
{
"name": "FUNCTIONS_EXTENSION_VERSION",
"value": "~3"
},

Guidance: Service Bus Session Management

Adding in examples and guidance for session management for functions and logic apps

Guidance: LogicApp Standard Connector Extension Development

Guidance, tips and tricks regarding LogicApp Standard Connector development.

• Simplified documentation and examples
• Visual Studio Scaffolding to get you started, much like asp.net scaffolding, ask some questions, generat3e the code
• Detailed guidance of how to setup a local development environment where you can debug and step through code when running the designer and when running an instance
• Discuss how to deploy the custom connector with the logicApp when it is being deployed through Azure Arc

Enhancement: Reference implementation using Service Endpoints instead of Private Endpoints

Would be great to see a reference implementation that uses Service Endpoints instead of Private Endpoints; as not all enterprise grade implementations mandate the need for using Private Endpoints.

Enhancement: Adding Cost Optimization and Guidance

Would be great to see a section about typical costs associated with this landing zone accelerator and how to best manage cost within Azure Integration Services.

Enhancement: LogicApps can leverage SQL for storage, can we leverage CosmosDB

In the documentation, it is discussed that LogicApps can leverage SQL Server for storage and persistence instead of an Azure Storage account. How difficult would it be to also have the ability to leverage CosmosDB. My thoughts here are, if we can use CosmosDB, that could lead to a logicApp that runs in multiple regions. Outage in region A, logicApp keeps running without issue since it was running in regions A and B simultaneously.

I think this would be a great talking point during compete scenarios.

Pattern: Non-Uniform Correlated Receive

Show how to receive n related messages for processing. Correlate based on message contents.

• independently deployable
• self-contained demo
• independent documentation
• intended to be built upon
• message tracking and reporting with promoted properties
• Show completion based on control message, number of messages, duration
• Show how to raise alerts if messages no subsequent messages received if expected

Add Azure Pricing estimates for Scenario 1

Scenario proposal: Load levelling with low capacity 3rd party back-end

Pattern: ESB

Demonstrate an ESB solution using iPaaS similar to the BizTalk ESB toolkit.

• itinerary processing
• onramp implementation
• offramp implementation
• property promotion
• message tracking and reporting

Guidance: Multi Region Topologies and Failover

Help a customer understand how to increase the resilience of an integration scenario by leveraging multiple azure regions. Need to focus on scenarios that utilize multiple azure service. Need to handle scenarios where messages have only been partially processed when an outage occurs.

Pattern: Claim Check

Deployable demo that illustrates large message handling with a claim check pattern.

• demonstrate how to read message contents in order to promote properties
• demonstrate how to store the large message
• self contained demo
• independently deployable demo
• independent documentation

Azure App Configuration is missing

Azure App Configuration should be a key part of this landing zone; it helps organise all the different environments.

Pattern: Message Envelopes

Show how to implement a message envelope where important message information is promoted and contained within a common envelope structure allowing for flexible message contents.

• find data within a message
• promote important data into envelope
• How to process across LogicApp, Functions, ServiceBus, etc.
• Envelope tracking and reporting

AppGW, Firewall and NATGW are missing

if the selling point of scenario 1 is to make networking easier in AIS, then you're definitely missing AppGW, Firewall and NATGW for ingress and egress. those are the ones that are difficult to setup. Make them optional please.

I'd also suggest to include a sample hello world app in scenario 1

Missing A records for Log Analytics with PE and AMPLS

Hello,

With the current logging.bicep, no A records are created for the Log Analytics Workspace with private endpoint and AMPLS. This code is needed.

// For DNS A records for Azure Monitor with PE and AMPLS.
resource existingStorageAccountPrivateDnsZone 'Microsoft.Network/privateDnsZones@2020-06-01' existing = {
  name: storageAccountPrivateDnsZoneName
}
resource existingMonitorPrivateDnsZone 'Microsoft.Network/privateDnsZones@2020-06-01' existing = {
  name: monitorPrivateDnsZoneName
}
resource existingOmsPrivateDnsZone 'Microsoft.Network/privateDnsZones@2020-06-01' existing = {
  name: omsPrivateDnsZoneName
}
resource existingOdsPrivateDnsZone 'Microsoft.Network/privateDnsZones@2020-06-01' existing = {
  name: odsPrivateDnsZoneName
}
resource existingAgentSvcPrivateDnsZone 'Microsoft.Network/privateDnsZones@2020-06-01' existing = {
  name: agentSvcPrivateDnsZoneName
}

param storageAccountPrivateDnsZoneName string
param monitorPrivateDnsZoneName string
param omsPrivateDnsZoneName string
param odsPrivateDnsZoneName string
param agentSvcPrivateDnsZoneName string

resource privateEndpointDnsGroup 'Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2022-09-01' = if(usePrivateEndpoint){
  parent: privateLinkScopePrivateEndpoint
  name: 'privateDnsZoneGroup'
  properties: {
    privateDnsZoneConfigs: [
      {
        name: replace(monitorPrivateDnsZoneName,'.','-')
        properties: {
          privateDnsZoneId: existingMonitorPrivateDnsZone.id
        }
      }
      {
        name: replace(omsPrivateDnsZoneName,'.','-')
        properties: {
          privateDnsZoneId: existingOmsPrivateDnsZone.id
        }
      }
      {
        name: replace(odsPrivateDnsZoneName,'.','-')
        properties: {
          privateDnsZoneId: existingOdsPrivateDnsZone.id
        }
      }
      {
        name: replace(agentSvcPrivateDnsZoneName,'.','-')
        properties: {
          privateDnsZoneId: existingAgentSvcPrivateDnsZone.id
        }
      }
      {
        name: replace(storageAccountPrivateDnsZoneName,'.','-')
        properties: {
          privateDnsZoneId: existingStorageAccountPrivateDnsZone.id
        }
      }                        
    ]
  }
}

Guidance: Load and Chaos Testing

Integration scenarios are spread across multiple services. A failure or degradation within a service could potentially cause a ripple effect through the system. Show how to leverage Azure Load Test service to drive load against the integration scenario to ensure it is able to handle the expected load. At the same time, show how to build a Chaos Studio experiment which will simulate a degraded service or service outage during the load test so app resilience is understood.

Show how to incorporate load testing and chaos testing within AzDo and GitHub pipelines.

Add TF Templates for Scenario 1

Guidance: End to End testing

Show a customer the best way to setup functional testing of a integration scenario which leverages multiple different azure services. Want to show how to deploy the solution via a pipeline, run some functional tests from the pipeline, and based on the results, make a deploy don't deploy decision.

Pattern: Request/Response Aggregator

Deployable demo that shows a customer a couple of implementations of a request/response aggregator

• Show how messages are fanned out and processed in parallel
• Show how to implement in LogicApps
• Show how to implement in Durable Functions
• Discuss When to use what (logicapps vs functions)
• demonstrate message tracking with property promotion, i.e. show all instances where po xyz was processed
• Self-contained demo
• independently deployable
• independent documentation
• Design as a starting point for customers, simple, easy to build upon

Guidance: Mainframe integration

How can we leverage HIS capabilities with logicApps? Mainframes are still around and integration needs persist. What are the best practices, what technologies do you need.

Pattern: Generic Batching and DeBatching

While we have EDI batching... so of, there are other scenarios where we need to batch and debatch messages. Need to have a pattern than makes this simple to implement, track, and manage for our customers.

Logic apps with Private Storage Endpoints

Keen to understand it the logic apps implements the pattern mentioned at https://learn.microsoft.com/en-us/azure/logic-apps/deploy-single-tenant-logic-apps-private-storage-account and associated code https://github.com/VeeraMS/LogicApp-deployment-with-Secure-Storage

looking at the code at https://github.com/Azure/Integration-Services-Landing-Zone-Accelerator/blob/main/src/infra/scenario1/la/logicApp.json it seems like is enabled
"publicNetworkAccess": "Enabled", // Need to disable after deployment of all artefacts.