azure / azure-functions-dotnet-worker

Azure Functions out-of-process .NET language worker

Branch	Status
main
release/1.x

Azure Functions .NET Worker

Welcome to the Azure Functions .NET Worker Repository. Introduced in 2020, Azure Functions' Isolated Worker Model moves function execution into a separate language worker process. Isolating function execution from the Azure Functions runtime allows you to have full control over your application's dependencies and easily incorporate advanced .NET features such as middleware and dependency injection.

A .NET Isolated worker process function works differently from a .NET in-process function. For .NET Isolated, you build an executable that imports the .NET Isolated language worker as a NuGet package. Your app includes a Program.cs that starts the worker.

Binding Model

.NET Isolated introduces a new binding model, slightly different from the binding model exposed in .NET in-process Azure Functions. More information can be found here. Please review our samples for usage information.

Middleware

The Azure Functions .NET Isolated supports middleware registration, following a model similar to what exists in ASP.NET and giving you the ability to inject logic into the invocation pipeline, pre and post function executions.

Samples

You can find samples on how to use different features of the .NET Worker under samples (link).

Create and run .NET Isolated Worker functions

Please see our Guide for running C# Azure Functions in an isolated worker process for information on how to develop, debug and deploy using the Isolated Worker model.

Running E2E Tests

Requirements

• Powershell 7
• CosmosDb Emulator
• Azurite (the set up script will download this automatically)

Instructions

1. Run setup-e2e-tests.ps1. Once the build succeeds and the emulators are started correctly, you are done with the setup.
2. Run run-e2e-tests.ps1 to run the tests or use the Test Explorer in VS.

Note: Do not add the switch to skip the core-tools download when running set-up-e2e-tests.ps1 as it will lead to an incomplete setup.

Deploying to Azure

Create the Azure resources

1. To deploy the app, first ensure that you've installed the Azure CLI.

2. Login to the CLI.

   az login

3. If necessary, use az account set to select the subscription you want to use.

4. Create a resource group, Storage account, and Azure Functions app. If you would like to use an existing Windows .NET Core 3 function app, please skip this step.

   az group create --name AzureFunctionsQuickstart-rg --location westeurope
   az storage account create --name <STORAGE_NAME> --location westeurope --resource-group AzureFunctionsQuickstart-rg --sku Standard_LRS
   az functionapp create --resource-group AzureFunctionsQuickstart-rg --consumption-plan-location westeurope --runtime dotnet-isolated --functions-version 3 --name <APP_NAME> --storage-account <STORAGE_NAME>

Deploy the app

1. Ensure you are in your functions project folder.

2. Deploy the app.

   func azure functionapp publish <APP_NAME>

Known issues

• Optimizations are not all in place in the consumption plan and you may experience longer cold starts.

Feedback

Please create issues in this repo. Thanks!

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

azure-functions-dotnet-worker's Issues

Add assembly strongname signing

@fabiocav -- should we be signing these assemblies? I think we can just use our own key now, right?

[Core tools] Add support for .NET 5 worker

• Add templates
• Configure local.settings.json to start worker
• func azure functionapp publish to configure worker (build the worker first)

Fail build when source generation fails

It seems that any error today with source generation still makes the build be successful and leaves a small info message saying that source wasn't generated. We should force the build to fail in such cases with some helpful message.

We can catch all exceptions and use SourceGeneratorContext.ReportDiagnostic to create a diagnostic with a severity of error.

Change JSON serialization to use ObjectSerializer

See example here -- we should use ObjectSerializer from the Azure SDK to abstract away serialization technology. This likely should happen for GA because it could impact public API and slightly change behavior.

See here for an example: https://github.com/Azure/azure-sdk-for-net/blob/ef4430baed/sdk/core/Microsoft.Azure.Core.NewtonsoftJson/README.md. We'd likely add a Serializer property to our Options.

Update to drive Portal UX

• Plan update to stacks that allows us to specify the language, version, and "mode" (embedded or isolated) for .NET
• Talk with Elliot/Allison about whether we can update the UX to look more like Web Apps:

Functions:

Web Apps:

• This would remove the need for a third dropdown in the Function create flow and move everything to one selection. For example, we'd have ".NET Core 3.1 embedded", ".NET Core 3.1 isolated", and ".NET 5 isolated"

Publish package(s) to NuGet

ACTION REQUIRED: Microsoft needs this private repository to complete compliance info

There are open compliance tasks that need to be reviewed for your azure-functions-dotnet-worker repo.

Action required: 4 compliance tasks

To bring this repository to the standard required for 2021, we require administrators of this and all Microsoft GitHub repositories to complete a small set of tasks within the next 60 days. This is critical work to ensure the compliance and security of your Azure GitHub organization.

Please take a few minutes to complete the tasks at: https://repos.opensource.microsoft.com/orgs/Azure/repos/azure-functions-dotnet-worker/compliance

• The GitHub AE (GitHub inside Microsoft) migration survey has not been completed for this private repository
• No Service Tree mapping has been set for this repo. If this team does not use Service Tree, they can also opt-out of providing Service Tree data in the Compliance tab.
• No repository maintainers are set. The Open Source Maintainers are the decision-makers and actionable owners of the repository, irrespective of administrator permission grants on GitHub.
• Classification of the repository as production/non-production is missing in the Compliance tab.

You can close this work item once you have completed the compliance tasks, or it will automatically close within a day of taking action.

If you no longer need this repository, it might be quickest to delete the repo, too.

GitHub inside Microsoft program information

More information about GitHub inside Microsoft and the new GitHub AE product can be found at https://aka.ms/gim or by contacting [email protected]

FYI: current admins at Microsoft include @jeffhollan, @pragnagopa, @mhoeger, @fabiocav, @shimedh, @ahmelsayed, @mathewc, @cgillum, @brettsam, @anirudhgarg, @paulbatum, @davidmrdavid, @ehamai

Logging is not translating LogLevel to Rpc LogLevel

Add tests for source generation

Use source generators to create the definition factory

The DefaultFunctionDefinitionFactory is a type that should be generated at build time

Come up with better worker name than "dotnet5"

We need something like dotnetworker to differentiate from dotnet -- but needs to support future versions of .NET

Move function invocation middleware into an explicit middleware registration

Right now we're auto-wiring up the invoker as the last piece of middleware. We should make this explicit and expose a UseFunctionExecutionMiddleware() that we wire-up in Program.cs.

Support for .NET Core 3.1

Currently the preview of Microsoft.Azure.Functions.Worker only targets net5.0.
Is there a plan to support netcoreapp3.1 or netstandard2.0/2.1?

We'd like to be able to update our functions to this new out-of-process mode, but we won't be able to use .NET 5 as it is not an LTS version (EOL is too soon). It would be great if we wouldn't have to wait until .NET 6 is released.

DI enable function class activation

In order to support constructor injection, we need to activate the function class using the container to fulfill constructor parameters.

E2E tests: Http

Track invocations dispatched in the process call

We need to to ensure the dispatch path is not just a "fire and forget". Messages need to be tracked.

.NET worker detectors

Issue to track the following:

• Review existing detectors to identify whether any of them would need to be updated
• Identify new detectors needed

Review entire public surface

Minimize surface area for v1. Make sure that anything referencing GRPC or that couples us to System.Text.Json isn't exposed. Things like Converters can be all internal and exposed later to give us more time for API design.

Runtimes folder containing gRPC binaries not copied for non-Windows

After building an app, we end up with 2 runtimes folders in the output:

• bin/Debug/net5.0/runtimes
• bin/Debug/net5.0/bin/runtimes

The second one only contains win and win7-x64 folders. Needs Linux and OSX ones too.

Build out mechanism for system logs

In the functions host, everything uses ILogger, which makes it hard to differentiate between System and User logs. We should take a different approach in the worker and make it explicit that logs written by system components do not use the "vanilla" ILogger, but maybe some internal wrapper class that allows us to differentiate.

Handle async Functions

Today we're just assuming that functions are synchronous and we're not awaiting the call.

Wire up function ILogger to logging system

Today we explicitly pass an ILogger to the ExecutionContext.

• We need to wire this up as an ILoggerProvider, which allows ILoggers used in customer DI services to log back to the host.
• Investigate how we handle categories, etc, from these loggers as they pass back through the host. For example, does the host respect the category on the RpcLog type and use it for its own logs?

Linux work needs to be done to enable "linuxfxversion" for .NET

Functions Integration Tests—Onboard DotNet worker

Please follow the instructions in the OneNote to onboard your language worker.

Review FunctionExecutionContext API

Make sure that it doesn't expose anything GRPC-related and that it is extensible for future usage.

Throw error when using Extension Bundles

If host.json has extension bundle configured, everything works fine except no functions show up. It's really hard to tell the root cause. And this can happen if your app is missing host.json as even the default one has extension bundles.

We likely need to fix this in Host, but tracking here for .NET worker work.

Worker config needs to generate assembly name rather than being hardcoded

Add nuget package signing to devops build

Flow scoped DI services through pipeline

This includes creating the function class with the DI services (if not static).

Validate worker exception handling and logging

I ran into an issue while testing -- we were throwing a NullRef in here: https://github.com/Azure/azure-functions-dotnet-worker/blob/main/src/DotNetWorker/Definition/DefaultFunctionDefinitionFactory.cs#L20, but I wasn't seeing any log in the host. We need to ensure that exceptions in code like this make it back to the host for logging/debugging.

Add SourceGenerator that generates an implementation of IFunctionProvider

We're going to try to skip over the need for function.json files in dotnet worker projects. In order to do that, we are hoping that we can auto-generate a class that implements IFunctionProvider, which will provide the host with all the function metadata it needs without having an actual json file artifact as part of the deployment.

Is TimeTrigger supported?

Getting this error:

The 'XXX' function is in error: Can't figure out which ctor to call.

Fix Functions Host's "/function" APIs and "sync triggers"

All ARM calls related to "/function(s)" API are missing binding information for .NET 5 worker. These API uses the payload created by the Functions Host during a sync triggers call.

When a sync triggers call is made, Functions Host somewhat relies on the physical function.json files to create the config portion of the request. This causes binding information to be missing in Functions that come outside of function.json. Here's the code that relies on function.json.

We need to decouple the call from the physical file and use the FunctionMetadata. All this work needs to happen in the Functions Host, but tracking here because they are crucial for .NET 5 support.

Update Core Tools to set site and app settings on deployment

Update sample app to use new package(s)

FunctionName with hyphen (-) breaks build

FunctionName with hyphen (-) breaks build

To repro, take a working example and just change the function name to something like this:

[FunctionName("CosmosData-Get")]

This is the results that show up in the failed build:

11>C:\Dev\ServiceExtensions\src\Example.Function.Net5\Microsoft.Azure.Functions.Worker.Sdk\SourceGenerator.FunctionProvider\_GeneratedFunctionProvider.cs(42,59,42,60): error CS1002: ; expected
11>C:\Dev\ServiceExtensions\src\Example.Function.Net5\Microsoft.Azure.Functions.Worker.Sdk\SourceGenerator.FunctionProvider\_GeneratedFunctionProvider.cs(42,49,42,59): error CS0818: Implicitly-typed variables must be initialized
11>C:\Dev\ServiceExtensions\src\Example.Function.Net5\Microsoft.Azure.Functions.Worker.Sdk\SourceGenerator.FunctionProvider\_GeneratedFunctionProvider.cs(42,60,42,63): error CS0103: The name 'Get' does not exist in the current context
11>C:\Dev\ServiceExtensions\src\Example.Function.Net5\Microsoft.Azure.Functions.Worker.Sdk\SourceGenerator.FunctionProvider\_GeneratedFunctionProvider.cs(49,12,49,15): error CS0103: The name 'Get' does not exist in the current context
11>C:\Dev\ServiceExtensions\src\Example.Function.Net5\Microsoft.Azure.Functions.Worker.Sdk\SourceGenerator.FunctionProvider\_GeneratedFunctionProvider.cs(56,66,56,69): error CS0103: The name 'Get' does not exist in the current context
11>C:\Dev\ServiceExtensions\src\Example.Function.Net5\Microsoft.Azure.Functions.Worker.Sdk\SourceGenerator.FunctionProvider\_GeneratedFunctionProvider.cs(56,55,56,111): error CS0201: Only assignment, call, increment, decrement, await, and new object expressions can be used as a statement
11>C:\Dev\ServiceExtensions\src\Example.Function.Net5\Microsoft.Azure.Functions.Worker.Sdk\SourceGenerator.FunctionProvider\_GeneratedFunctionProvider.cs(59,57,59,60): error CS0103: The name 'Get' does not exist in the current context
11>C:\Dev\ServiceExtensions\src\Example.Function.Net5\Microsoft.Azure.Functions.Worker.Sdk\SourceGenerator.FunctionProvider\_GeneratedFunctionProvider.cs(59,46,59,102): error CS0201: Only assignment, call, increment, decrement, await, and new object expressions can be used as a statement
11>C:\Dev\ServiceExtensions\src\Example.Function.Net5\Microsoft.Azure.Functions.Worker.Sdk\SourceGenerator.FunctionProvider\_GeneratedFunctionProvider.cs(60,29,60,32): error CS0103: The name 'Get' does not exist in the current context
11>Done building project "Example.Function.Net5.csproj" -- FAILED.

Simply removing the hyphen in the name will allow this to build properly

Allow simpler testing with ConfigureFunctionsWorker

Currently, Functions worker is configured with UseFunctionExecutionMiddleware() to provide the necessary infrastructure for triggers and output bindings. I'd like to be able to test my functions w/o running the full middleware when using the container DI based Functions. This will allow simpler testing scenarios that do not require Functions runtime to be installed and bootstrapped.

Placeholder investigations

None is not supported yet

When any binding is unavailable such that the datatype that is received is null, we get a NotSupportedException. We need to also handle the case for TypedData.DataOneofCase.None.

We probably need to handle a bunch of other types as well in future. But None, I think, is more crucial than others.

Fix core tools to prevent auto-updating of FUNCTIONS_WORKER_RUNTIME on deploy

right now if you use "dotnet5" (or anything unknown) -- the core tools will overwrite the app setting when publishing

Improve function metadata generation

Today we use Source Generators... but this won't work for non-C# or if the assembly has been AOT compiled. Need to come up with alternative or fallback...

Move TargetFramework to NetStandard2.0

E2E tests: Cosmos

Define behavior when using rich types in a function

Rich types like CloudBlockBlob, etc, don't work when used in the worker. We need to define how to handle this so it's not a runtime exception.

Will binding to http route elements from the HTTP trigger be supported?

Before, parts of a url could be bound to, which could then be used in other bindings.

E.g. you could take a part of the HTTP trigger url, and use that for a binding to a storage item.

Or see this sample.

Will this still be possible?

Update readme instructions in preview repo

Some limitations to note:

• There may be breaking changes in the future
• Only core tools workflow supported
• Rich bindings not supported (give examples) - durable, etc

Release updated Functions CLI with dotnet-isolated support

E2E tests: Timer

Http[Request/Response]Data contract classes design discussion

Preface: As the title notes, this is probably more of a discussion than an issue, but it doesn't look like discussions are enabled on this repo yet... which is understandable since it's only just recently (i.e. 3 days ago) gone into public beta. 😊

In looking at the samples and what immediately jumped out at me as someone who utilizes HTTP triggered functions the majority of the time is the introduction of two new data contract classes to represent the request and response: HttpRequestData and HttpResponseData.

First, I would like to ask that we could have some kind of short explanation of why these classes were introduced. I believe it's because the SDK does not want to pin itself to a specific version of System.Net.Http, is that correct? If so, would you care to expand on why you think it's a bad idea to tie a specific extension to another dependency? Frankly, I think it would be a big mistake to force new types on .NET developers in the context of an Azure Function when in so many other contexts System.Net.Http is what we all use and which itself has an extension ecosystem built around it.

Second, if it comes down to decoupling, well then I have concerns about the design of these contracts. For example, the Body being a string: this is dangerous for performance on multiple levels. First and most basic, even if I actually do want to return a string... I shouldn't have to build a whole contiguous string like that. Second, if I don't want to return a string and want to return binary data... what option do I have now? I would have to return base-64 encoded bodies and that's just, well, not a good idea from an HTTP perspective and could completely box out some standard HTTP scenarios like file downloads or serving images. I would also suggest that the Headers property being defined as the concrete type Dictionary<string, string> and always allocated even if not used is a terrible idea from a performance perspective. Anyway, we can split off and go into deeper on into this in separate issues once we have the core discussion established, but the point is that so much of this is already well thought out, designed and implemented into the existing System.Net.Http type system and I think it's a mistake to eschew that for an extension where I'm very much opting in to using HTTP in the first place.

Decouple initialization hosted service from communication channel (gRPC)

We need to bring back some of the abstractions we previously had in the worker POC.