aaronpowell / fsharp.cosmosdb

An F# wrapper around Cosmos DB's .NET SDK to make it more friendly for F# developers

FSharp.CosmosDb 🌍

This project is a wrapper around the Cosmos DB v4 .NET SDK to make it a bit more friendly to the F# language.

Install

Install via NuGet:

dotnet add package FSharp.CosmosDb

Or using Paket:

dotnet paket add FSharp.CosmosDb

Usage

All operations will return an AsyncSeq via FSharp.Control.AsyncSeq that contains the data fetched, data inserted or data updated.

Insert

open FSharp.CosmosDb

let connStr = "..."

let insertUsers data =
    connStr
    |> Cosmos.fromConnectionString
    |> Cosmos.database "UserDb"
    |> Cosmos.container "UserContainer"
    |> Cosmos.insertMany<User> data
    |> Cosmos.execAsync

Upsert

open FSharp.CosmosDb

let connStr = "..."

let insertUsers data =
    connStr
    |> Cosmos.fromConnectionString
    |> Cosmos.database "UserDb"
    |> Cosmos.container "UserContainer"
    |> Cosmos.upsertMany<User> data
    |> Cosmos.execAsync

Update

open FSharp.CosmosDb

let connStr = "..."

let updateUser id partitionKey =
    connStr
    |> Cosmos.fromConnectionString
    |> Cosmos.database "UserDb"
    |> Cosmos.container "UserContainer"
    |> Cosmos.update<User> id partitionKey (fun user -> { user with IsRegistered = true })
    |> Cosmos.execAsync

Query

open FSharp.CosmosDb

let host = "https://..."
let key = "..."
let findUsers() =
    host
    |> Cosmos.host
    |> Cosmos.connect key
    |> Cosmos.database "UserDb"
    |> Cosmos.container "UserContainer"
    |> Cosmos.query "SELECT u.FirstName, u.LastName FROM u WHERE u.LastName = @name"
    |> Cosmos.parameters [ "@name", box "Powell" ]
    |> Cosmos.execAsync<User>

[<EntryPoint>]
let main argv =
    async {
        let users = findUsers()
        do! users
        |> AsyncSeq.iter (fun u -> printfn "%s %s" u.FirstName u.LastName)

        return 0
    } |> Async.RunSynchronously

DeleteItem

open FSharp.CosmosDb

let connStr = "..."

let updateUser id partitionKey =
    connStr
    |> Cosmos.fromConnectionString
    |> Cosmos.database "UserDb"
    |> Cosmos.container "UserContainer"
    |> Cosmos.deleteItem id partitionKey
    |> Cosmos.execAsync

DeleteContainer

open FSharp.CosmosDb

let connStr = "..."

connStr
|> Cosmos.container "ContainerName"
|> Cosmos.deleteContainer
|> Cosmos.execAsync
|> Async.Ignore

FSharp.CosmosDb.Analyzer 💡

Also part of this repo is a F# Analyzer for use from the CLI or in Ionide.

Features

• Validation of database name against databases in Cosmos
  • Quick fix provided with list of possible db names
• Validation of container name against containers in the database
  • Quick fix provided with list of possible container names
• Detection of unused parameters in the query
  • Quick fix provided with list of defined parameters (if any)
• Detection of supplied but unused parameters
  • Quick fix provided with list of declared parameters
• Detection of missing @ for parameter name
  • Quick fix provided to add it in

Analyzer Usage

1. Provide connection information

Connection information can be provided as either environment variables or using an appsettings.json/appsettings.Development.json file.

Environment Variables

The analyzer will look for the following environment variables:

• FSHARP_COSMOS_CONNSTR -> A full connection string to Cosmos DB
• FSHARP_COSMOS_HOST & FSHARP_COSMOS_KEY -> The URI endpoint and access key

The FSHARP_COSMOS_CONNSTR will take precedence if both sets of environment variables are provided

App Settings

The analyzer will look for a file matching appsettings.json or appsettings.Development.json in either the workspace root of the VS Code instance or relative to the file being parsed. The file is expected to have the following JSON structure in it:

{
  "CosmosConnection": {
    "ConnectionString": "",
    "Host": "",
    "Key": ""
  }
}

If CosmosConnection.ConnectionString exists, it will be used, otherwise it will use the CosmosConnection.Host and CosmosConnection.Key to connect.

2. Install the Analyzer from paket

paket add FSharp.CosmosDb.Analyzer --group Analyzers

3. Enable Analyzers in Ionide

Add the following settings (globally or in the workspace):

{
  "FSharp.enableAnalyzers": true,
  "FSharp.analyzersPath": ["./packages/analyzers"]
}

License

MIT

Thank You

• Zaid Ajaj for the Npgsql Analyzer. Without this I wouldn't have been able to work out how to do it (and there's some code lifted from there)
• Krzysztof Cieślak for the amazing Ionide plugin
• Isaac Abraham for helping fix the parser

fsharp.cosmosdb's Issues

Release 0.1.1 ready for review

🚀 Release 0.1.1 ready for review

Changelog

---

[0.1.1] - 2020-03-13

Changed

• CI/CD pipeline working

Release 0.2.0 ready for review

🚀 Release 0.2.0 ready for review

Changelog

---

[0.2.0]

Added

• Ability to create a connection from a connection string with Cosmos.fromConnectionString
• Insert API
• Update API
• Delete API

Changed

• Introduced a maybe computational expression to simplify option types
• Major refactor of the internals
• Change analyzer to support using appsettings not just environment variables to find connection info
• Bumped dependency for FSAC to 35.0.0

Any chance of UPSERT support?

Hey!
This is a really handy package, I am enjoying playing with it.
Any chance of supporting the UPSERT command from Cosmos?

My current method is to query for matching IDs and Insert the difference, but I feel like UPSERT would be more idiomatic.

Thanks!

Replace doesn't resolve correct id value

When doing a replace of an item the value "id" is resolved instead of the value of the property "id" if you haven't added an IdAttribute on the id property.

I think this is where the issue lies.

let getIdFieldName<'T> (item: 'T) =
    let idAttr = IdAttributeTools.findId<'T> ()

    match idAttr with
    | Some attr -> attr.GetValue(item).ToString()
    | None -> "id"

Does it support Stored Procedure?

Nice work with the library!

Is it possible to execute stored procedure with it somehow? Like with using the query or something?

Release 0.2.0 ready for review

🚀 Release 0.2.0 ready for review

Changelog

---

[0.2.0]

Added

• Ability to create a connection from a connection string with Cosmos.fromConnectionString
• Insert API
• Update API
• Delete API

Changed

• Introduced a maybe computational expression to simplify option types
• Major refactor of the internals
• Change analyzer to support using appsettings not just environment variables to find connection info
• Bumped dependency for FSAC to 35.0.0

Is Paket required for FSharp.CosmosDb.Analyzer?

I am using dotnet add package, but I don't have the packages directory anywhere.

Release 0.1.0 ready for review

🚀 Release 0.1.0 ready for review

Changelog

---

[0.1.0] - 2020-03-12

Initial Release 🎉

Added

• Basic query API
• Basic Analyzer support
• Tests

Release 0.2.0 ready for review

🚀 Release 0.2.0 ready for review

Changelog

---

[0.2.0]

Added

• Ability to create a connection from a connection string with Cosmos.fromConnectionString
• Insert API
• Update API
• Delete API

Changed

• Introduced a maybe computational expression to simplify option types
• Major refactor of the internals
• Change analyzer to support using appsettings not just environment variables to find connection info
• Bumped dependency for FSAC to 35.0.0

Analyzer not loading

I'm trying to use the CosmosDB analyzer but it doesn't load one of the dependencies

[19:04:39.861 ERR] [Analyzers] Analyzer Analyzer errored while processing api\src\FooHandler.fs: Exception has been thrown by the target of an invocation.
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.IO.FileNotFoundException: Could not load file or assembly 'Azure.Cosmos, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified. ---> System.IO.FileNotFoundException: Could not load the specified file.
   at System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(AssemblyName assemblyName)
[....]
   at [e-mail](Context ctx)
   at [email protected](Tuple`2 tupledArg)

.vscode\settings.json

{
    "FSharp.enableAnalyzers": true,
    "FSharp.analyzersPath": [
        "packages/Analyzers",
        "analyzers",
        "./analyzers"
    ]
}

Folder structure:

api
├───.ionide
├───.vscode
│        └───settings.json
├───analyzers
│   ├───azure.core
│   │   └───1.0.0
│   ├───azure.cosmos
│   │   └───4.0.0-preview3
│   ├───fsharp.analyzers.sdk
│   │   └───0.4.1
│   ├───fsharp.compiler.service
│   │   └───35.0.0
│   ├───fsharp.control.asyncseq
│   │   └───2.0.23
│   ├───fsharp.core
│   │   ├───4.7.0
│   │   └───4.7.1
│   ├───fsharp.cosmosdb
│   │   └───0.2.0
│   ├───fsharp.cosmosdb.analyzer
│   │   └───0.2.0

[...]

├───src [*.fsproject]
│   └───applicationSettings.Development.json
├───tests
└───applicationSettings.Development.json

Linq Query Support

It would be nice to be able to query a database using linq without directly writing sql.

The following syntax would be possible:

open FSharp.CosmosDb
open Newtonsoft.Json

type User = 
  { [<JsonProperty(PropertyName="id")>]
    Id: string
    Name: string }

let users =
    Cosmos.fromConnectionString "..."
    |> Cosmos.database "UserDb"
    |> Cosmos.container "UserContainer"
    |> Cosmos.linq (fun users ->
        query {
            for user in users do
            where (user.Name = "Aaron")
        })
    |> Cosmos.execAsync<User>

I've been experimenting with extended query builders that can replace F# functions with .Net BCL methods for the purposes of querying cosmos db

open Microsoft.Azure.Cosmos
open Microsoft.Azure.Cosmos.Linq
open Newtonsoft.Json
open System
open System.Linq
open FSharp.Linq
open FSharp.Quotations
open FSharp.Quotations.ExprShape
open FSharp.Quotations.DerivedPatterns

let rec replace expr =
    // replaces 'abs' with 'System.Math.Abs' and 'acos' with 'System.Math.Acos'
    match expr with 
    | SpecificCall <@@ abs @@> (_, [typ], args) ->
        let e = replace args.Head
        if typ = typeof<int8> then
            <@@ (Math.Abs : int8 -> int8) %%e @@>
        elif typ = typeof<int16> then
            <@@ (Math.Abs : int16 -> int16) %%e @@>
        elif typ = typeof<int32> then
            <@@ (Math.Abs : int32 -> int32) %%e @@>
        elif typ = typeof<int64> then
            <@@ (Math.Abs : int64 -> int64) %%e @@>
        elif typ = typeof<float32> then
            <@@ (Math.Abs : float32 -> float32) %%e @@>
        elif typ = typeof<float> then
            <@@ (Math.Abs : float -> float) %%e @@>
        elif typ = typeof<decimal> then
            <@@ (Math.Abs : decimal -> decimal) %%e @@>
        else 
            failwith $"Invalid argument type for translation of 'abs': {typ.FullName}"
    | SpecificCall <@@ acos @@> (_, [typ], args) ->
        let e = replace args.Head
        <@@ (Math.Acos : float -> float) %%e @@>
    | ShapeVar v -> Expr.Var v
    | ShapeLambda(v, expr) -> Expr.Lambda(v, replace expr)
    | ShapeCombination(o, args) -> 
        RebuildShapeCombination(o, List.map replace args)

type CosmosQueryBuilder() =
    inherit QueryBuilder()

    member _.Run(e: Expr<QuerySource<'a, IQueryable>>) =
        let r = Expr.Cast<Linq.QuerySource<'a, System.Linq.IQueryable>>(replace e)
        base.Run r

let cosmosQuery = CosmosQueryBuilder()

(task {
    use cosmosClient = new CosmosClient("...")
    let! resp = cosmosClient.CreateDatabaseIfNotExistsAsync "TestDB"
    let database = resp.Database
    let! resp = database.CreateContainerIfNotExistsAsync("TestContainer", "/id")
    let container = resp.Container

    let q = 
        // Utilization, this doesn't work with the normal 'query' ce
        cosmosQuery {
            for p in container.GetItemLinqQueryable<Person>() do
            let o = abs p.Number
            where (o > 2)
            select ((float o) + 1.2)
        }
    use setIterator = q.ToFeedIterator<float>()

    while setIterator.HasMoreResults do
        let! items = setIterator.ReadNextAsync()
        for item in items do
            printfn $"%A{item}"
}).Wait()

📣 Announcement - Road to v1

Hey friends 👋 ,

It was around 18 months ago that I created this project, and it was originally designed to tackle some challenges I was having myself more than anything.

Since then, I've plugged away here and there, made a few tweaks to it, fixed some bugs, but it just trickled along.

After some recent activity around it, off the back of some conference talks I gave, I figured it was time to actively look at getting as major release out.

Today I've created PR #53, which is the work going in to making a v1 release, and here I want to mention some of the major changes, as there are breaking changes.

Changing the Cosmos SDK dependency

When I created the library I decided to use the v4 SDK, which was in preview, working on the assumption that it was coming along soon. Well, 18 months later and it's still in preview, with no indication as to when it's likely to be released (and hasn't had any updates in months).

This represented a fork in the road, either I stuck with the v4 SDK and hope it does come out one day, but miss all the improvements in v3, or roll back to using the v3 SDK.

I'm made the decision that v1 will roll back to the v3 SDK, and this does represent a breaking change.

The main change that happens between v3 and v4 (for the context of this library) is that v4 uses IAsyncEnumerable as a way to iterate over the result set, which with F# we map using AsyncSeq.

To try and reduce the impact on existing code, I've changed the internals to continue pushing out an AsyncSeq as results, by creating that myself, rather than relying on a type conversion.

Changes to batch requests

With #23, pagination support was added, and it was represented by Page<'T>, which was returned from the SDK. With the downgrade of the SDK it no longer natively uses Page<'T>, meaning that when doing execBatchAsync, the return type is different.

On the plus side, I actually figured out paging properly, so now you can pass in a page size to the batch request and it'll return you an AsyncSeq containing in batches of the required size, so I call it a win!

New features

There's three new features being added for v1, so it's not all breaking changes!

Client caching

#42 has been open for ages and since we now use the v3 SDK I figured I should follow the advice and cache the CosmosClient. Caching is done by connection string (or host/access key) and will be reused for all operations that use that value.

I've also added a Cosmos.dispose method do dispose of the client. I'll also look to making the record that contains the connection info implement IDisposable, but I need to work out how to best make it access the cache.

Change feed processor support

One of the nifty features of Cosmos is change feed processors, but they feel C#-ish to work with, so I've created a F# wrapper around it:

        let onChange changes _ =
            printfn "Changes: %A" changes
            System.Threading.Tasks.Task.CompletedTask

        let processor =
            conn
            |> Cosmos.ChangeFeed.create<Family> "changeFeedSample" onChange
            |> Cosmos.ChangeFeed.withInstanceName "consoleHost"
            |> Cosmos.ChangeFeed.leaseContainer (conn |> Cosmos.container "leaseContainer")
            |> Cosmos.ChangeFeed.build

        printfn "Starting change feed"
        do! processor.StartAsync() |> Async.AwaitTask

        printfn "Change feed started. Press any key to exit"

        Console.Read() |> ignore

Raw SDK access

FSharp.CosmosDb doesn't do everything that the Cosmos SDK does, so for those scenarios, rather than having to create your own CosmosClient, it'd be nice to access the one that's in cache. Well, now you can, using the Cosmos.Raw module, you can get the client, database and container SDK objects.

Looking for help

So this is what it's looking like for v1. I don't have a hard and fast date on when v1 will ship, the main hurdle I have is testing. While I can test against my own apps, I'd really appreciate anyone who can test it in their app, does so. You can get the pre-release package from GitHub packages - https://github.com/aaronpowell/FSharp.CosmosDb/packages/530463 and install it via NuGet.

Thanks, and let's go for a major release!

Paging support?

This looks like a great new library that I'd love to use in place of my own hacked together wrapper. Are you expecting the caller to do the paging, or should that be included when returning a query result? I'm not sure if ofAsyncEnum does that or not, but the code doesn't seem to check .HasMoreResults.

Performance benefits in sharing the connection?

I'm wondering whether there are any performance benefits to sharing the connection? Something like this:

let host = "https://..."
let key = "..."

// Create my container once and hold it as a value.
let container =
    host
    |> Cosmos.host
    |> Cosmos.connect key
    |> Cosmos.database "UserDb"
    |> Cosmos.container |> "UserContainer"

let userByName name =
    container
    |> Cosmos.query "SELECT u.FirstName, u.LastName FROM u WHERE u.LastName = @name"
    |> Cosmos.parameters [ "name", box "Powell" ]
    |> Cosmos.execAsync<User>

let userByEmail email =
    // etc.

Cannot open in dev container when on Windows

Hey 🙋‍♀️

Thanks for the repo, which I wanted to give a try after your latest post on Cosmos DB in a dev container.

I'm on a Windows machine:

C:\Users\me\dev\FSharp.CosmosDb [main ≡]
λ  git config --get core.autocrlf
true

Steps I took

1. Clone the repo
2. Open in VS Code
3. Click the "Reopen in container" button in the pop-up notification

Result

Building the container fails.

Toggle me to see the errors

#7 3.064 /tmp/library-scripts/common-debian.sh: line 11: $'\r': command not found: invalid optionbrary-scripts/common-debian.sh: line 12: set: -
#7 3.064 set: usage: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]
#7 3.065 /tmp/library-scripts/common-debian.sh: line 13: $'\r': command not found
#7 3.068 /tmp/library-scripts/common-debian.sh: line 23: $'\r': command not found
#7 3.068 /tmp/library-scripts/common-debian.sh: line 38: syntax error near unexpected token `$'do\r''
#7 3.068 /tmp/library-scripts/common-debian.sh: line 38: `    for CURRENT_USER i' ${POSSIBLE_USERS[@]}; do
------
executor failed running [/bin/sh -c apt-get update     && /bin/bash /tmp/library-scripts/common-debian.sh "${INSTALL_ZSH}" "${USERNAME}" "${USER_UID}" "${USER_GID}" "${UPGRADE_PACKAGES}" "true" "true"     && /bin/bash /tmp/library-scripts/docker-debian.sh "${ENABLE_NONROOT_DOCKER}" "/var/run/docker-host.sock" "/var/run/docker.sock" "${USERNAME}"     && apt-get autoremove -y && apt-get clean -y &&rm -rf /var/lib/apt/lists/* /tmp/library-scripts/]: exit code: 2
ERROR: Service 'app' failed to build : Build failed
[10452 ms] Error: Command failed: docker-compose --project-name fsharpcosmosdb_devcontainer -f c:\Users\me\dev\FSharp.CosmosDb\.devcontainer\docker-compose.yml build
[10453 ms]     at gu (c:\Users\me\.vscode\extensions\ms-vscode-remote.remote-containers-0.245.2\dist\spec-node\devContainersSpecCLI.js:220:419)
[10453 ms]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
[10453 ms]     at async _S (c:\Users\me\.vscode\extensions\ms-vscode-remote.remote-containers-0.245.2\dist\spec-node\devContainersSpecCLI.js:220:2358)
[10454 ms]     at async AS (c:\Users\me\.vscode\extensions\ms-vscode-remote.remote-containers-0.245.2\dist\spec-node\devContainersSpecCLI.js:204:2361)
[10454 ms]     at async KS (c:\Users\me\.vscode\extensions\ms-vscode-remote.remote-containers-0.245.2\dist\spec-node\devContainersSpecCLI.js:261:2177)
[10455 ms]     at async co (c:\Users\me\.vscode\extensions\ms-vscode-remote.remote-containers-0.245.2\dist\spec-node\devContainersSpecCLI.js:261:3110)
[10456 ms]     at async LP (c:\Users\me\.vscode\extensions\ms-vscode-remote.remote-containers-0.245.2\dist\spec-node\devContainersSpecCLI.js:360:9352)
[10456 ms]     at async DP (c:\Users\me\.vscode\extensions\ms-vscode-remote.remote-containers-0.245.2\dist\spec-node\devContainersSpecCLI.js:360:9108)
[10470 ms] Exit code 1
[10476 ms] Command failed: C:\Users\me\AppData\Local\Programs\Microsoft VS Code\Code.exe c:\Users\me\.vscode\extensions\ms-vscode-remote.remote-containers-0.245.2\dist\spec-node\devContainersSpecCLI.js up --user-data-folder c:\Users\me\AppData\Roaming\Code\User\globalStorage\ms-vscode-remote.remote-containers\data --workspace-folder c:\Users\me\dev\FSharp.CosmosDb --workspace-mount-consistency cached --id-label devcontainer.local_folder=c:\Users\me\dev\FSharp.CosmosDb --log-level debug --log-format json --config c:\Users\me\dev\FSharp.CosmosDb\.devcontainer\devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true

Quick analysis

All the Bash files in the .devcontainer folder were checked out CRLF-style.
Manually changing them to LF-style made the error disappear.

Should we enforce LF line endings on these files through EditorConfig and .gitattributes or am I holding part of it wrong?

Getting a warning about package downgrade

Functions.fsproj : warning NU1605: Detected package downgrade: FSharp.Core from 5.0.1 to 5.0.0. Reference the package directly from the project to select a different version. 
Functions.fsproj : warning NU1605:  Functions -> FSharp.CosmosDb 0.5.2 -> FSharp.Core (>= 5.0.1) 
Functions.fsproj : warning NU1605:  Functions -> FSharp.Core (>= 5.0.0)

Is there something in this package that requires FSharp.Core 5.0.0 strictly?

Use a singleton Azure Cosmos DB client for the lifetime of your application

The official performance guidance (at least for v3 SDK) is to keep a single client for the lifetime of the app.

I think this library should either implement the singleton pattern, or provide an argument to execAsync that takes in the client.

Unless, of course, this advice is no longer relevant for v4. I could not find a similar page for v4.

Issue deleting an item from a container

Is the code in this repository for deleting items correct?

Taken from samples directory:

let deletePowell = deleteFamily conn "Powell.1" "Powell"

do!
    deletePowell
    |> AsyncSeq.iter (fun f -> printfn "Deleted: %A" f)

It seems to do nothing. iter does not seem to be called.

Release 0.2.0 ready for review

🚀 Release 0.2.0 ready for review

Changelog

---

[0.2.0]

Added

• Ability to create a connection from a connection string with Cosmos.fromConnectionString
• Insert API
• Update API
• Delete API

Changed

• Introduced a maybe computational expression to simplify option types
• Major refactor of the internals
• Change analyzer to support using appsettings not just environment variables to find connection info
• Bumped dependency for FSAC to 35.0.0

Release 0.2.0 ready for review

🚀 Release 0.2.0 ready for review

Changelog

---

[0.2.0]

Added

• Ability to create a connection from a connection string with Cosmos.fromConnectionString
• Insert API
• Update API
• Delete API

Changed

• Introduced a maybe computational expression to simplify option types
• Major refactor of the internals
• Change analyzer to support using appsettings not just environment variables to find connection info
• Bumped dependency for FSAC to 35.0.0

Release 0.2.0 ready for review

🚀 Release 0.2.0 ready for review

Changelog

---

[0.2.0]

Added

• Ability to create a connection from a connection string with Cosmos.fromConnectionString
• Insert API
• Update API
• Delete API

Changed

• Introduced a maybe computational expression to simplify option types
• Major refactor of the internals
• Change analyzer to support using appsettings not just environment variables to find connection info
• Bumped dependency for FSAC to 35.0.0

Feature req: identity-based connection

Do you only support Connection-String connections?

Please add support for identity-based connection.

In the upstream, this is done using the DefaultAzureCredential class provided by the Azure.Identity client library.

This would be a function called Cosmos.fromCredential which works like

open Microsoft.Azure.Cosmos
open Azure.Identity
open FSharp.CosmosDb

let credential = new DefaultAzureCredential()

let insertUsers data =
    credential 
    |> Cosmos.fromCredential
    |> Cosmos.database "UserDb"
    |> Cosmos.container "UserContainer"
    |> Cosmos.insertMany<User> data
    |> Cosmos.execAsync

Many thanks.

[Request] Make compatable with newer Azure Functions versions

Currently the most recent functions versions require .NET Core 6.0, but the currently implementation is pinned to .NET Framework 4.0.