azure / azure-functions-openai-extension

An extension that adds support for Azure OpenAI/ OpenAI bindings in Azure Functions for LLM (GPT-3.5-Turbo, GPT-4, etc)

Azure Functions bindings for OpenAI's GPT engine

This project adds support for OpenAI LLM (GPT-3.5-turbo, GPT-4) bindings in Azure Functions.

This extension depends on the Azure AI OpenAI SDK.

NuGet Packages

The following NuGet packages are available as part of this project.

Preview Bundle

Add following section to host.json of the function app for non dotnet languages to utilise the preview bundle and consume extension packages:

"extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle.Preview",
    "version": "[4.*, 5.0.0)"
  }

Requirements

• .NET 6 SDK or greater (Visual Studio 2022 recommended)
• Azure Functions Core Tools v4.x
• Update settings in Azure Function or the local.settings.json file for local development with the following keys:
  1. For Azure, AZURE_OPENAI_ENDPOINT - Azure OpenAI resource (e.g. https://***.openai.azure.com/) set.
  2. For Azure, assign the user or function app managed identity Cognitive Services OpenAI User role on the Azure OpenAI resource. It is strongly recommended to use managed identity to avoid overhead of secrets maintenance, however if there is a need for key based authentication add the setting AZURE_OPENAI_KEY and its value in the settings.
  3. For non- Azure, OPENAI_API_KEY - An OpenAI account and an API key saved into a setting.
     If using environment variables, Learn more in .env readme.
  4. Update CHAT_MODEL_DEPLOYMENT_NAME and EMBEDDING_MODEL_DEPLOYMENT_NAME keys to Azure Deployment names or override default OpenAI model names.
  5. If using user assigned managed identity, add AZURE_CLIENT_ID to environment variable settings with value as client id of the managed identity.
  6. Visit binding specific samples README for additional settings that might be required for each binding.
• Azure Storage emulator such as Azurite running in the background
• The target language runtime (e.g. dotnet, nodejs, powershell, python, java) installed on your machine. Refer the official supported versions.

Features

The following features are currently available. More features will be slowly added over time.

• Text completions
• Chat completion
• Assistants
• Embeddings generators
• Semantic search

Text completion input binding

The textCompletion input binding can be used to invoke the OpenAI Chat Completions API and return the results to the function.

The examples below define "who is" HTTP-triggered functions with a hardcoded "who is {name}?" prompt, where {name} is the substituted with the value in the HTTP request path. The OpenAI input binding invokes the OpenAI GPT endpoint to surface the answer to the prompt to the function, which then returns the result text as the response content.

C# example

Setting a model is optional for non-Azure OpenAI, see here for default model values for OpenAI.

[Function(nameof(WhoIs))]
public static HttpResponseData WhoIs(
    [HttpTrigger(AuthorizationLevel.Function, Route = "whois/{name}")] HttpRequestData req,
    [TextCompletionInput("Who is {name}?")] TextCompletionResponse response)
{
    HttpResponseData responseData = req.CreateResponse(HttpStatusCode.OK);
    responseData.WriteString(response.Content);
    return responseData;
}

TypeScript example

import { app, input } from "@azure/functions";

// This OpenAI completion input requires a {name} binding value.
const openAICompletionInput = input.generic({
    prompt: 'Who is {name}?',
    maxTokens: '100',
    type: 'textCompletion',
    model: 'gpt-35-turbo'
})

app.http('whois', {
    methods: ['GET'],
    route: 'whois/{name}',
    authLevel: 'function',
    extraInputs: [openAICompletionInput],
    handler: async (_request, context) => {
        var response: any = context.extraInputs.get(openAICompletionInput)
        return { body: response.content.trim() }
    }
});

PowerShell example

using namespace System.Net

param($Request, $TriggerMetadata, $TextCompletionResponse)

Push-OutputBinding -Name Response -Value ([HttpResponseContext]@{
        StatusCode = [HttpStatusCode]::OK
        Body       = $TextCompletionResponse.Content
    })

In the same directory as the PowerShell function, define the bindings in a function.json file.

If using Azure OpenAI, update CHAT_MODEL_DEPLOYMENT_NAME key in local.settings.json with the deployment name or update model property directly in function.json for textCompletion input binding or use it to override the default model value for OpenAI.

{
    "type": "textCompletion",
    "direction": "in",
    "name": "TextCompletionResponse",
    "prompt": "Who is {name}?",
    "maxTokens": "100",
    "model": "gpt-3.5-turbo"
}

Python example

Setting a model is optional for non-Azure OpenAI, see here for default model values for OpenAI.

@app.route(route="whois/{name}", methods=["GET"])
@app.generic_input_binding(arg_name="response", type="textCompletion", data_type=func.DataType.STRING, prompt="Who is {name}?", maxTokens="100", model = "gpt-3.5-turbo")
def whois(req: func.HttpRequest, response: str) -> func.HttpResponse:
    response_json = json.loads(response)
    return func.HttpResponse(response_json["content"], status_code=200)

Running locally

You can run the above function locally using the Azure Functions Core Tools and sending an HTTP request, similar to the following:

GET http://localhost:7127/api/whois/pikachu

The result that comes back will include the response from the GPT language model:

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Date: Tue, 28 Mar 2023 18:25:40 GMT
Server: Kestrel
Transfer-Encoding: chunked

Pikachu is a fictional creature from the Pokemon franchise. It is a yellow
mouse-like creature with powerful electrical abilities and a mischievous
personality. Pikachu is one of the most iconic and recognizable characters
from the franchise, and is featured in numerous video games, anime series,
movies, and other media.

You can find more instructions for running the samples in the corresponding project directories. The goal is to have samples for all languages supported by Azure Functions.

Chat completion

Chat completions are useful for building AI-powered assistants.

There are three bindings you can use to interact with the assistant:

1. The assistantCreate output binding creates a new assistant with a specified system prompt.
2. The assistantPost output binding sends a message to the assistant and saves the response in its internal state.
3. The assistantQuery input binding fetches the assistant history and passes it to the function.

You can find samples in multiple languages with instructions in the chat samples directory.

Assistants

Assistants build on top of the chat functionality to provide assistants with custom skills defined as functions. This internally uses the function calling feature of OpenAIs GPT models to select which functions to invoke and when.

You can define functions that can be triggered by assistants by using the assistantSkillTrigger trigger binding. These functions are invoked by the extension when a assistant signals that it would like to invoke a function in response to a user prompt.

The name of the function, the description provided by the trigger, and the parameter name are all hints that the underlying language model use to determine when and how to invoke an assistant function.

C# example

public class AssistantSkills
{
    readonly ITodoManager todoManager;
    readonly ILogger<AssistantSkills> logger;

    // This constructor is called by the Azure Functions runtime's dependency injection container.
    public AssistantSkills(ITodoManager todoManager, ILogger<AssistantSkills> logger)
    {
        this.todoManager = todoManager ?? throw new ArgumentNullException(nameof(todoManager));
        this.logger = logger ?? throw new ArgumentNullException(nameof(logger));
    }

    // Called by the assistant to create new todo tasks.
    [Function(nameof(AddTodo))]
    public Task AddTodo([AssistantSkillTrigger("Create a new todo task")] string taskDescription)
    {
        if (string.IsNullOrEmpty(taskDescription))
        {
            throw new ArgumentException("Task description cannot be empty");
        }

        this.logger.LogInformation("Adding todo: {task}", taskDescription);

        string todoId = Guid.NewGuid().ToString()[..6];
        return this.todoManager.AddTodoAsync(new TodoItem(todoId, taskDescription));
    }

    // Called by the assistant to fetch the list of previously created todo tasks.
    [Function(nameof(GetTodos))]
    public Task<IReadOnlyList<TodoItem>> GetTodos(
        [AssistantSkillTrigger("Fetch the list of previously created todo tasks")] object inputIgnored)
    {
        this.logger.LogInformation("Fetching list of todos");

        return this.todoManager.GetTodosAsync();
    }
}

You can find samples with instructions in the assistant samples directory.

Embeddings Generator

OpenAI's text embeddings measure the relatedness of text strings. Embeddings are commonly used for:

• Search (where results are ranked by relevance to a query string)
• Clustering (where text strings are grouped by similarity)
• Recommendations (where items with related text strings are recommended)
• Anomaly detection (where outliers with little relatedness are identified)
• Diversity measurement (where similarity distributions are analyzed)
• Classification (where text strings are classified by their most similar label)

Processing of the source text files typically involves chunking the text into smaller pieces, such as sentences or paragraphs, and then making an OpenAI call to produce embeddings for each chunk independently. Finally, the embeddings need to be stored in a database or other data store for later use.

C# embeddings generator example

[Function(nameof(GenerateEmbeddings_Http_RequestAsync))]
public async Task GenerateEmbeddings_Http_RequestAsync(
    [HttpTrigger(AuthorizationLevel.Function, "post", Route = "embeddings")] HttpRequestData req,
    [EmbeddingsInput("{RawText}", InputType.RawText)] EmbeddingsContext embeddings)
{
    using StreamReader reader = new(req.Body);
    string request = await reader.ReadToEndAsync();

    EmbeddingsRequest? requestBody = JsonSerializer.Deserialize<EmbeddingsRequest>(request);

    this.logger.LogInformation(
        "Received {count} embedding(s) for input text containing {length} characters.",
        embeddings.Count,
        requestBody.RawText.Length);

    // TODO: Store the embeddings into a database or other storage.
}

Semantic Search

The semantic search feature allows you to import documents into a vector database using an output binding and query the documents in that database using an input binding. For example, you can have a function that imports documents into a vector database and another function that issues queries to OpenAI using content stored in the vector database as context (also known as the Retrieval Augmented Generation, or RAG technique).

Only raw text files are supported at present for data ingestion.

The supported list of vector databases is extensible, and more can be added by authoring a specially crafted NuGet package. Visit the currently supported vector specific folder for specific usage information:

• Azure AI Search - See source code
• Azure Data Explorer - See source code
• Azure Cosmos DB using MongoDB (vCore) - See source code

More may be added over time.

C# document storage example

This HTTP trigger function takes a URL of a file as input, generates embeddings for the file, and stores the result into an Azure AI Search Index.

public class EmbeddingsRequest
{
    [JsonPropertyName("Url")]
    public string? Url { get; set; }
}

[Function("IngestFile")]
public static async Task<EmbeddingsStoreOutputResponse> IngestFile(
    [HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req)
{
    using StreamReader reader = new(req.Body);
    string request = await reader.ReadToEndAsync();

    EmbeddingsRequest? requestBody = JsonSerializer.Deserialize<EmbeddingsRequest>(request);

    if (requestBody == null || requestBody.Url == null)
    {
        throw new ArgumentException("Invalid request body. Make sure that you pass in {\"Url\": value } as the request body.");
    }

    Uri uri = new(requestBody.Url);
    string filename = Path.GetFileName(uri.AbsolutePath);

    HttpResponseData response = req.CreateResponse(HttpStatusCode.Created);

    return new EmbeddingsStoreOutputResponse
    {
        HttpResponse = response,
        SearchableDocument = new SearchableDocument(filename)
    };
}

Tip - To improve context preservation between chunks in case of large documents, specify the max overlap between chunks and also the chunk size. The default values for MaxChunkSize and MaxOverlap are 8 * 1024 and 128 characters respectively.

C# document query example

This HTTP trigger function takes a query prompt as input, pulls in semantically similar document chunks into a prompt, and then sends the combined prompt to OpenAI. The results are then made available to the function, which simply returns that chat response to the caller. Tip - To improve the knowledge for OpenAI model, the number of result sets being sent to the model with system prompt can be increased with binding property - MaxKnowledgeCount which has default value as 1. Also, the SystemPrompt in SemanticSearchRequest can be tweaked as per user instructions on how to process the knowledge sets being appended to it.

public class SemanticSearchRequest
{
    [JsonPropertyName("Prompt")]
    public string? Prompt { get; set; }
}

[Function("PromptFile")]
public static IActionResult PromptFile(
    [HttpTrigger(AuthorizationLevel.Function, "post")] SemanticSearchRequest unused,
    [SemanticSearchInput("AISearchEndpoint", "openai-index", Query = "{Prompt}", ChatModel = "%CHAT_MODEL_DEPLOYMENT_NAME%", EmbeddingsModel = "%EMBEDDING_MODEL_DEPLOYMENT_NAME%")] SemanticSearchContext result)
{
    return new ContentResult { Content = result.Response, ContentType = "text/plain" };
}

The responses from the above function will be based on relevant document snippets which were previously uploaded to the vector database. For example, assuming you uploaded internal emails discussing a new feature of Azure Functions that supports OpenAI, you could issue a query similar to the following:

POST http://localhost:7127/api/PromptFile
Content-Type: application/json

{
    "Prompt": "Was a decision made to officially release an OpenAI binding for Azure Functions?"
}

And you might get a response that looks like the following (actual results may vary):

HTTP/1.1 200 OK
Content-Length: 454
Content-Type: text/plain

There is no clear decision made on whether to officially release an OpenAI binding for Azure Functions as per the email "Thoughts on Functions+AI conversation" sent by Bilal. However, he suggests that the team should figure out if they are able to free developers from needing to know the details of AI/LLM APIs by sufficiently/elegantly designing bindings to let them do the "main work" they need to do. Reference: Thoughts on Functions+AI conversation.

IMPORTANT: Azure OpenAI requires you to specify a deployment when making API calls instead of a model. The deployment is a specific instance of a model that you have deployed to your Azure OpenAI resource. In order to make code more portable across OpenAI and Azure OpenAI, the bindings in this extension use the Model, ChatModel and EmbeddingsModel to refer to either the OpenAI model or the Azure OpenAI deployment ID, depending on whether you're using OpenAI or Azure OpenAI.

All samples in this project rely on default model selection, which assumes the models are named after the OpenAI models. If you want to use an Azure OpenAI deployment, you'll want to configure the Model, ChatModel and EmbeddingsModel properties explicitly in your binding configuration. Here are a couple examples:

// "gpt-35-turbo" is the name of an Azure OpenAI deployment
[Function(nameof(WhoIs))]
public static string WhoIs(
    [HttpTrigger(AuthorizationLevel.Function, Route = "whois/{name}")] HttpRequest req,
    [TextCompletion("Who is {name}?", Model = "gpt-35-turbo")] TextCompletionResponse response)
{
    return response.Content;
}

public record SemanticSearchRequest(string Prompt);

// "my-gpt-4" and "my-ada-2" are the names of Azure OpenAI deployments corresponding to gpt-4 and text-embedding-ada-002 models, respectively
[Function("PromptEmail")]
public static IActionResult PromptEmail(
    [HttpTrigger(AuthorizationLevel.Function, "post")] SemanticSearchRequest unused,
    [SemanticSearch("KustoConnectionString", "Documents", Query = "{Prompt}", ChatModel = "my-gpt-4", EmbeddingsModel = "my-ada-2")] SemanticSearchContext result)
{
    return new ContentResult { Content = result.Response, ContentType = "text/plain" };
}

Default OpenAI models

1. Chat Completion - gpt-3.5-turbo
2. Embeddings - text-embedding-ada-002
3. Text Completion - gpt-3.5-turbo

While using non-Azure OpenAI, you can omit the Model specification in attributes to use the default models.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

azure-functions-openai-extension's Issues

dotnet isolated sample creation with http trigger template doesn't return any response and runs until time out.

Function specification from template which should be changed according to samples/textcompletion/csharp-inproc:

[Function("Function1")]
public IActionResult Run([HttpTrigger(AuthorizationLevel.Function, "get", "post")] HttpRequest req)
{
    _logger.LogInformation("C# HTTP trigger function processed a request.");
    return new OkObjectResult("Welcome to Azure Functions!");
}

Program.cs

using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

var host = new HostBuilder()
    .ConfigureFunctionsWebApplication()
    .ConfigureServices(services =>
    {
        services.AddApplicationInsightsTelemetryWorkerService();
        services.ConfigureFunctionsApplicationInsights();
    })
    .Build();

host.Run();

References

    <FrameworkReference Include="Microsoft.AspNetCore.App" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.20.1" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.1.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.16.4" />
    <PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.21.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.1.0" />

Sample expects the Program.cs to be:

using Microsoft.Extensions.Hosting;


var host = new HostBuilder()
    .ConfigureFunctionsWorkerDefaults()
    .Build();

host.Run();

Action required: migrate or opt-out of migration to GitHub inside Microsoft

Migrate non-Open Source or non-External Collaboration repositories to GitHub inside Microsoft

In order to protect and secure Microsoft, private or internal repositories in GitHub for Open Source which are not related to open source projects or require collaboration with 3rd parties (customer, partners, etc.) must be migrated to GitHub inside Microsoft a.k.a GitHub Enterprise Cloud with Enterprise Managed User (GHEC EMU).

Action

✍️ Please RSVP to opt-in or opt-out of the migration to GitHub inside Microsoft.

❗Only users with admin permission in the repository are allowed to respond. Failure to provide a response will result to your repository getting automatically archived.🔒

Instructions

Reply with a comment on this issue containing one of the following optin or optout command options below.

✅ Opt-in to migrate

@gimsvc optin --date <target_migration_date in mm-dd-yyyy format>

Example: @gimsvc optin --date 03-15-2023

OR

❌ Opt-out of migration

@gimsvc optout --reason <staging|collaboration|delete|other>

Example: @gimsvc optout --reason staging

Options:

• staging : This repository will ship as Open Source or go public
• collaboration : Used for external or 3rd party collaboration with customers, partners, suppliers, etc.
• delete : This repository will be deleted because it is no longer needed.
• other : Other reasons not specified

Need more help? 🖐️

• Email [email protected]. ✉️
• Post your questions in GitHub inside Microsoft Team in Microsoft Teams. 🗨️

TotalTokens Count after initialization should count instructions tokens.

In the current implementation for TotalTokens in ChatCompletion.

1. Initialise the chat state with PUT method

   PUT http://localhost:7071/api/chats/test123
   Content-Type: application/json
   
   {
     "instructions": "You are a helpful chatbot. In all your English responses, speak as if you are Shakespeare."
   }
   

2. Retrieve the chat state with GET method (TotalTokens is 0) because nothing has been posted to OpenAI yet and it has not returned consumed tokens.
   GET http://localhost:7071/api/chats/test123?timestampUTC=2023-08-10T07:51:10Z

Behavior:
Actual: The response returns totalTokens to be 0.
Expected: it should count the instructions tokens.

Cannot use semantic search bindings in .NET 8

When I change my standard sample demo project to target .NET 8 in the csproj, the project builds clean, but then I get this runtime error when I test a post to IngestData (uses Embeddings binding):

This branch exactly repros it as you use test.http and post to the local IngestData function:.

Functions:

        IngestData: [POST] http://localhost:7071/api/IngestData

        PromptData: [POST] http://localhost:7071/api/PromptData

        WhoIs:  http://localhost:7071/api/whois/{name}

        OpenAI::ChatBotEntity: entityTrigger

For detailed output, run func with --verbose flag.
[2024-02-15T18:38:50.834Z] Host lock lease acquired by instance ID '000000000000000000000000731A3487'.
[2024-02-15T18:38:52.563Z] Executing 'IngestData' (Reason='This function was programmatically called via the host APIs.', Id=e43173fd-3292-44f4-b59c-c519224a4bd8)
[2024-02-15T18:38:52.586Z] Executed 'IngestData' (Failed, Id=e43173fd-3292-44f4-b59c-c519224a4bd8, Duration=39ms)
[2024-02-15T18:38:52.586Z] System.Private.CoreLib: Exception while executing function: IngestData. Microsoft.Azure.WebJobs.Host: One or more errors occurred. (Exception binding parameter 'req') (Exception binding parameter 'embeddings'). Exception binding parameter 'req'. System.Private.CoreLib: Could not load type 'System.Runtime.CompilerServices.NullableContextAttribute' from assembly 'System.Runtime, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.

It is throwing on this line of code that uses the EmbeddingsRequest:embeddings record object, and note I have two fields, with RawText being null here:

        public record EmbeddingsRequest(string RawText, string FilePath);

//...
            await output.AddAsync(new SearchableDocument(title, embeddings));

If i changed EmbeddingsRequest definition to match the sample having only a single field in the record, I go back to getting Build errors all the time (maybe this was bound to happen and I forced a rebuild?):

        public record EmbeddingsRequest(string FilePath);
        public record SemanticSearchRequest(string Prompt);

        [FunctionName("IngestData")]
        public static async Task<IActionResult> IngestData(
            [HttpTrigger(AuthorizationLevel.Function, "post")] EmbeddingsRequest req,
            [Embeddings("{FilePath}", inputType: InputType.FilePath, 
            Model = "%AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT%")] EmbeddingsContext embeddings,
            [SemanticSearch("KustoConnectionString", "Documents")] IAsyncCollector<SearchableDocument> output)
        {
            string title = Path.GetFileNameWithoutExtension(req.FilePath);
            await output.AddAsync(new SearchableDocument(title, embeddings));
            return new OkObjectResult(new { status = "success", title, chunks = embeddings.Count });
        }

[2024-02-15T18:34:32.589Z] Found /Users/paulyuk/src/paulyuk/functions/ai/ai-bindings/ai-bindings.csproj. Using for user secrets file configuration.
[2024-02-15T18:34:33.488Z] No job functions found. Try making your job classes and methods public. If you're using binding extensions (e.g. Azure Storage, ServiceBus, Timers, etc.) make sure you've called the registration method for the extension(s) in your startup code (e.g. builder.AddAzureStorage(), builder.AddServiceBus(), builder.AddTimers(), etc.).
[2024-02-15T18:34:33.507Z] The 'IngestData' function is in error: The binding type(s) 'embeddings, semanticSearch' are not registered. Please ensure the type is correct and the binding extension is installed.
[2024-02-15T18:34:33.507Z] The 'PromptData' function is in error: The binding type(s) 'semanticSearch' are not registered. Please ensure the type is correct and the binding extension is installed.
[2024-02-15T18:34:33.507Z] The 'WhoIs' function is in error: The binding type(s) 'textCompletion' are not registered. Please ensure the type is correct and the binding extension is installed.
For detailed output, run func with --verbose flag.

The net of this (no pun intended :) ) is I cannot use .NET 8, or figure out a workaround.

Remove ability to use Azure OpenAI API keys

Expected Behavior

As part of a product review with executive leadership, an ask was made to remove the ability to use Azure OpenAI API keys. Instead, we should support only managed identity authentication with Azure OpenAI and our Azure-hosted search providers. This ensures that customers are always using the most secure configurations.

Additional Information

Naturally, we'll want to support both system and user-assigned identities. New APIs may be required specifically for configuring user-assigned identities.

Making small changes to a client project using extension leads to errors

When I make changes or come back to my sample that uses our new Nugets, I see these types of errors a lot when I func start:

Example 1

[2024-02-15T18:11:35.534Z] Found /Users/paulyuk/src/paulyuk/functions/ai/ai-bindings/ai-bindings.csproj. Using for user secrets file configuration.
[2024-02-15T18:11:42.982Z] A host error has occurred during startup operation '9cf9f0e1-32ad-4175-8390-d95c53563625'.
[2024-02-15T18:11:42.982Z] Azure.Identity: Method not found: 'Void Azure.Core.ClientOptions..ctor(Azure.Core.DiagnosticsOptions)'.
Value cannot be null. (Parameter 'provider')
[2024-02-15T18:11:43.072Z] Host startup operation has been canceled

Example 2

[2024-02-15T18:12:29.734Z] Found /Users/paulyuk/src/paulyuk/functions/ai/ai-bindings/ai-bindings.csproj. Using for user secrets file configuration.
[2024-02-15T18:12:30.737Z] No job functions found. Try making your job classes and methods public. If you're using binding extensions (e.g. Azure Storage, ServiceBus, Timers, etc.) make sure you've called the registration method for the extension(s) in your startup code (e.g. builder.AddAzureStorage(), builder.AddServiceBus(), builder.AddTimers(), etc.).
[2024-02-15T18:12:30.757Z] The 'InngestData' function is in error: The binding type(s) 'embeddings, semanticSearch' are not registered. Please ensure the type is correct and the binding extension is installed.
[2024-02-15T18:12:30.757Z] The 'PromptData' function is in error: The binding type(s) 'semanticSearch' are not registered. Please ensure the type is correct and the binding extension is installed.
[2024-02-15T18:12:30.757Z] The 'WhoIs' function is in error: The binding type(s) 'textCompletion' are not registered. Please ensure the type is correct and the binding extension is installed.
For detailed output, run func with --verbose flag.
[2024-02-15T18:12:36.449Z] Host lock lease acquired by instance ID '000000000000000000000000731A3487'.

Closing my terminal and restarting usually mitigates it and errors go away.

Steps to reproduce:

1. clone this repo and change to this folder: https://github.com/paulyuk/functions/tree/main/ai/ai-bindings
2. func start <-- should be ok
3. change any value in ai-bindings.csproj
4. func start again <-- error often starts

[Bug] Semantic Search with Kusto returns unexpected match

What language are you using?

Dotnet (OOP)

Expected Behavior

Index couple of documents in Kusto, I indexed two small documents with chunk size 3 and 8 respectively.

I am asking a very simple question based on document 2.
Kusto should return an embedding match - chunk or snippet which has closest match to my query.

Actual Behavior

It doesn't matter what question I ask, Kusto query always returns me the same chunk.
It goes to OpenAI, since the Kusto snippet doesn't make sense, OpenAI tries to answer from its own knowledge pool and gives a random document name as reference since SystemPrompt asks to give references.

Host.json

{
"version": "2.0",
"logging": {
"logLevel": {
"Microsoft.Azure.WebJobs.Extensions.OpenAI": "Information"
}
},
"extensions": {
"openai": {
"searchProvider": {
"type": "kusto"
}
}
}
}

Steps to Reproduce

1. Set up local.settings.json for SemanticSearchEmbeddings sample for Kusto
2. Run the sample
3. In demo.http, provide two file paths for ingestion API, ingest the documents into Kusto
4. In demo.http, give a prompt based on document which was ingested later
5. Have breakpoints to check Kusto response
6. Also, observe the final response from the API.

Relevant code being tried

No response

Relevant log output

No response

Where are you facing this problem?

Local - Core Tools

Additional Information

No response

Readme for assistants should include creation of testdb database, my-todos container, and id for partition key

In the Readme.md for assistants, https://github.com/Azure/azure-functions-openai-extension/blob/main/samples/assistant/README.md#prerequisites, we should add instructions to create the testdb database, my-todos container, and id for partition key.

[BUG] Assistant Functions - GET assistants call has too many function calls in 1106 model version of GPT

When testing, we sporadically see multiple calls for "The function call succeeded. Let the user know you completed the action", when we should only be seeing one. This is likely caused by the model's response and needs to be investigated further. Here is an example JSON output of what we're seeing:

{
    "id": "assistant678",
    "exists": true,
    "status": null,
    "createdAt": "2024-03-01T17:31:26.849899Z",
    "lastUpdatedAt": "2024-03-01T17:32:22.5504361Z",
    "totalMessages": 6,
    "totalTokens": 173,
    "recentMessages": [
        {
            "content": "Don't make assumptions about what values to plug into functions.\r\nAsk for clarification if a user request is ambiguous.",
            "role": "system"
        },
        {
            "content": "Remind me to call my dad",
            "role": "user"
        },
        {
            "content": "The function call succeeded. Let the user know that you completed the action.",
            "role": "function"
        },
        {
            "content": "The function call succeeded. Let the user know that you completed the action.",
            "role": "function"
        },
        {
            "content": "The function call succeeded. Let the user know that you completed the action.",
            "role": "function"
        },
        {
            "content": "I have added a reminder for you to call your dad.",
            "role": "assistant"
        }
    ]
}

ASP.NET WhoIs example should use IActionResult

Looking at this example, the code is perfectly fine, just it's not modern ASP.NET like.

[FunctionName(nameof(WhoIs))]
public static string WhoIs(
    [HttpTrigger(AuthorizationLevel.Function, Route = "whois/{name}")] HttpRequest req,
    [TextCompletion("Who is {name}?", Model = "gpt-35-turbo")] TextCompletionResponse response)
{
    return response.Content;
}

Suggest changing this in all places to be this instead:

        [FunctionName(nameof(WhoIs))]
        public static IActionResult WhoIsFunction(
            [HttpTrigger(AuthorizationLevel.Function, Route = "whois/{name}")] HttpRequest req,
            [TextCompletion("Who is {name}?")] TextCompletionResponse response)
        {
            return new OkObjectResult(response.Content);
        }

[Bug] Assistant Functions API not triggerring skills with 0301 GPT 35 model version

What language are you using?

Dotnet (OOP)

Expected Behavior

Assistant Functions should work as demonstrated in the sample, add todo tasks and recognise the skills

Actual Behavior

No task being added to ToDo, errors in the logs (shared)
Not retrieving the tasks.

Host.json

No response

Steps to Reproduce

No response

Relevant code being tried

https://github.com/Azure/azure-functions-openai-extension/tree/main/samples/assistant/csharp-ooproc/AssistantSample

Relevant log output

[2024-03-20T18:49:47.419Z] Executing 'Functions.PostUserQuery' (Reason='This function was programmatically called via the host APIs.', Id=8144aae9-3bb5-40fa-b79f-6200344c742e)
[2024-03-20T18:49:47.434Z] Posting message to assistant 'assistant128': Remind me to call my dad
[2024-03-20T18:49:47.441Z] Posting message to assistant entity 'assistant128'
[2024-03-20T18:49:47.455Z] [assistant128] Received message: Remind me to call my dad
[2024-03-20T18:49:55.468Z] [assistant128] Found 1 function call(s) in response
[2024-03-20T18:49:55.470Z] [assistant128] Calling function 'functions' with arguments: // Assuming this is a request to create a new todo task
// We need clarification on the task details, like when should it be completed, or if there's any additional notes for the task
type AddTodo = (_: {
dueDate?: Date,
notes?: string,
}) => any;
 
// We can also add an optional reminder if necessary, so the assistant can remind the user about the task before the due date
type AddReminder = (_: {
reminderDate: Date,
}) => any;
 
[2024-03-20T18:49:55.476Z] [assistant128] Function 'functions' failed with an unhandled exception
[2024-03-20T18:49:55.476Z] WebJobs.Extensions.OpenAI: No skill registered with name 'functions'.
[2024-03-20T18:49:56.130Z] [assistant128] Got LLM response consisting of 23 tokens: I'm sorry, I didn't understand what action you are requesting. Can you please give me more context?
[2024-03-20T18:49:56.132Z] [assistant128] Chat length is now 4 messages
[2024-03-20T18:49:56.139Z] Executed 'Functions.PostUserQuery' (Succeeded, Id=8144aae9-3bb5-40fa-b79f-6200344c742e, Duration=8722ms)
 
 
[2024-03-20T19:01:24.923Z] Created assistant 'assistant134'
[2024-03-20T19:01:24.924Z] Executed 'Functions.CreateAssistant' (Succeeded, Id=0d978090-5e4c-497f-9d27-06615272d746, Duration=42076ms)
[2024-03-20T19:01:27.261Z] Executing 'Functions.PostUserQuery' (Reason='This function was programmatically called via the host APIs.', Id=b132c6c6-bfb5-42a8-90f1-482d42b3e152)
[2024-03-20T19:01:27.285Z] Posting message to assistant 'assistant134': Remind me to call my dad
[2024-03-20T19:01:27.285Z] Posting message to assistant entity 'assistant134'
[2024-03-20T19:01:27.289Z] [assistant134] Received message: Remind me to call my dad
[2024-03-20T19:01:28.624Z] [assistant134] Got LLM response consisting of 35 tokens: Sure thing! Should I add this as a todo task or just remind you right now? Also, do you want a specific date and time or just a general reminder?
[2024-03-20T19:01:28.625Z] [assistant134] Chat length is now 3 messages
[2024-03-20T19:01:28.628Z] Executed 'Functions.PostUserQuery' (Succeeded, Id=b132c6c6-bfb5-42a8-90f1-482d42b3e152, Duration=1367ms)
[2024-03-20T19:02:13.462Z] Executing 'Functions.PostUserQuery' (Reason='This function was programmatically called via the host APIs.', Id=427db8ae-d455-4fbe-afdc-4db086d590cf)
[2024-03-20T19:02:13.480Z] Posting message to assistant 'assistant134': add as a todo task with a general reminder
[2024-03-20T19:02:13.481Z] Posting message to assistant entity 'assistant134'
[2024-03-20T19:02:13.485Z] [assistant134] Received message: add as a todo task with a general reminder
[2024-03-20T19:02:15.111Z] [assistant134] Found 1 function call(s) in response
[2024-03-20T19:02:15.113Z] [assistant134] Calling function 'python' with arguments: // Assuming there's a function to add a new todo task
functions.addTodo({
  taskDescription: "Call my dad",
});
 
// You can set a reminder for yourself to check your todo list at a specific time, or just set an alarm on your phone to remind you to check your list periodically.
[2024-03-20T19:02:15.186Z] [assistant134] Function 'python' failed with an unhandled exception
[2024-03-20T19:02:15.187Z] WebJobs.Extensions.OpenAI: No skill registered with name 'python'.
[2024-03-20T19:02:15.882Z] [assistant134] Got LLM response consisting of 39 tokens: Oops, it looks like I don't have access to a todo list to add this task to. Do you want me to create a new list for you or do you already have one?
[2024-03-20T19:02:15.885Z] [assistant134] Chat length is now 6 messages
[2024-03-20T19:02:15.892Z] Executed 'Functions.PostUserQuery' (Succeeded, Id=427db8ae-d455-4fbe-afdc-4db086d590cf, Duration=2429ms)

Where are you facing this problem?

Local - Core Tools

Additional Information

Model specs, it works with

Model specs, it doesn't work with

Add support and samples for URL based input types for Embeddings Binding

I noticed in the recent Nuget samples we have the user write their own record for embeddings requests, and it only handles this one input type:

public record EmbeddingsRequest(string FilePath);

In prior samples we handled 2 of the 3 cases ; - )

public record EmbeddingsRequest(string RawText, string FilePath);

My opinion is we should probably give the user a standard type for embedding request and handle all 3 of the cases: RawText, FilePath, and Url (cool new one!). But at least have sample code that is aware of each case like i have above pls.

Rename "chat bots" to "assistants"

This issue tracks rebranding chat bots as assistants, which is consistent with OpenAI's naming conventions.

As part of this, we should also rename the "Agents" namespace to Assistants ("agent" is a misnomer, and we might decide to use it for something else later).