microsoft / fsharplu

This library provides a set of F# helpers for string manipulations, logging, collection data structures, file operations, text processing, security, async, parsing, diagnostics, configuration files and Json serialization.

FSharpLu F# library

This library provides F# lightweight utilities for string manipulations, logging, collection data structures, file operations, text processing, security, async, parsing, diagnostics, configuration files and Json serialization.

This is by no means a full-fledged utility library for F#, but rather a small collection of utilities and other thin wrappers accumulated throughout the development of various internal projects at Microsoft and meant to facilitate development with the .Net framework using the F# programming language.

Some of the provided utilities are just thin let-bindings wrappers around existing .Net libraries (e.g. module FSharpLu.Text or FSharpLu.Parsing) whereas some provide additional features (e.g. Json serialization in module FSharpLu.Json).

Build status

Branch	Status
main

Build requirements

• F# compiler. See https://fsharp.org/use/Windows and https://fsharp.org/use/linux/

• Install .NET Core SDK 6.0 from https://dotnet.microsoft.com/download/visual-studio-sdks.

To build project run dotnet build under the top-level directory or run the script scripts\build.ps1.

Documentation

For the documentation please visit the Wiki

License

MIT

Packages

• FSharpLu: The core set of utilities
• FSharpLu.Json: Json serialization of F# data types implemented as JSon.Net converters and providing more succinct serialization for option types and discriminate unions.
• FSharpLu.Windows: Windows-specific utility functions
• FSharpLu.Azure: Azure Resource Manager helpers
• Test libraries FSharpLu.*.Tests: Unit tests for a specific module

FSharpLu modules

Here is a list of helper modules provided by FSharpLu.

Main module

• FSharpLu.Async
• FSharpLu.AsyncSeq
• FSharpLu.CacheMap
• FSharpLu.Configuration
• FSharpLu.Collection
• FSharpLu.Diagnostics
• FSharpLu.Disposable
• FSharpLu.ErrorHandling
• FSharpLu.ExecutableInfo
• FSharpLu.File
• FSharpLu.FSharpFromCSharp
• FSharpLu.Ini
• FSharpLu.Logger
• FSharpLu.MachineOperations
• FSharpLu.Option
• FSharpLu.Parsing
• FSharpLu.Platform
• FSharpLu.QueueScheduler
• FSharpLu.Security
• FSharpLu.ServiceRequest
• FSharpLu.StateMachine
• FSharpLu.Sysprep
• FSharpLu.Text
• FSharpLu.TraceLogging

Json

• FSharpLu.Json

Azure

• FSharpLu.Azure

Windows

• FSharpLu.Windows.Hyperv
• FSHarpLu.Windows.Security
• FSHarpLu.Windows.Certificate
• FSHarpLu.Windows.Registry
• FSHarpLu.Windows.TraceLoggingConsole
• FSHarpLu.Windows.Smb
• FSHarpLu.Windows.EtwListener
• FSHarpLu.Windows.DirectoryServices
• FSHarpLu.Windows.Wmi
• FSHarpLu.Windows.ManagementWin32
• FSHarpLu.Windows.ManagementHypervisor
• FSHarpLu.Windows.HyperV

Microsoft Open Source Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

fsharplu's Issues

Is it possible to serialize DU cases in camelCase instead of PascalCase?

camelCase serialization is mentioned in this issue and provided fix works nicely for records, but not for discriminated unions. Example:

#r "packages/Microsoft.FSharpLu.Json/lib/net452/Microsoft.FSharpLu.Json.dll" 
#r "packages/Newtonsoft.Json/lib/net45/Newtonsoft.Json.dll" 

open System
open Microsoft.FSharpLu.Json
open Newtonsoft.Json

type Settings =
    static member settings =
        let s = 
            JsonSerializerSettings(
                NullValueHandling = NullValueHandling.Ignore,
                MissingMemberHandling = MissingMemberHandling.Error,
                ContractResolver = Serialization.CamelCasePropertyNamesContractResolver())
        s.Converters.Add(CompactUnionJsonConverter())
        s
    static member formatting = Formatting.None

type MyCamlCaseSerializer = With<Settings>

type Record = { First : int; Second : int }
type DU = | FancyCase of Record

MyCamlCaseSerializer.serialize <| FancyCase { First = 0; Second = 1 }

returns

val it : string =
  "{
  "FancyCase": {
    "first": 0,
    "second": 1
  }
}"

Is there a way to override this behaviour? (from skimming the code it seems to me that there is not)

compact union converter + camel case resolver does not resolve names consistently

The Newtonsoft.Json CamelCasePropertyNamesContractResolver will lower consecutive uppercase letters so properties like FOO become foo instead of fOO.

I propose using the contract resolver to resolve the union case names for better consistency. Will submit a PR shortly.

Here's a type that demonstrates the differences:

type ConsecutiveUppercaseRecord = { BAR : int ; BAZNumber : int }
type ConsecutiveUppercaseDu = FOO | FOOWithRecord of ConsecutiveUppercaseRecord

Current: ["fOO",{"fOOWithRecord":{"bar":2,"bazNumber":3}}]
Expected: ["foo",{"fooWithRecord":{"bar":2,"bazNumber":3}}]

Giraffe JSON serializer on wiki is incorrect

The code listing on wiki for Giraffe JSON serializer is incorrect. The declarations on the top include an extra .Settings:

open Newtonsoft.Json
open System.Threading.Tasks

let Utf8EncodingWithoutBom = System.Text.UTF8Encoding(false)
let DefaultBufferSize = 1024

-let Formatting = Microsoft.FSharpLu.Json.Compact.Settings.TupleAsArraySettings.formatting
-let Settings = Microsoft.FSharpLu.Json.Compact.Settings.TupleAsArraySettings.settings
+let Formatting = Microsoft.FSharpLu.Json.Compact.TupleAsArraySettings.formatting
+let Settings = Microsoft.FSharpLu.Json.Compact.TupleAsArraySettings.settings

Trace log file should be created so that it can be opened for read

By default the log file created by the TextWriterTraceListener does not create the log file with the appropriate FileMode/FileShare options that allows the log file to be opened for read while it's still open by the logger.

Make memorise in FSharpLu.Json thread-safe

I'm occasionally receiving these exceptions in a production API:

2018-03-27 10:59:50.500 +02:00 [Error] Unhandled exception while executing request
System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at <StartupCode$Microsoft-FSharpLu-Json>[email protected](Type key)
[rest omitted]

It seems that there is a NullReferenceException on this line. A quick web search turned up e.g. this SO answer, which suggests that such exceptions may be due to threading issues.

I suggest you make memorise thread-safe, by using ConcurrentDictionary or locking.

Serialize to Stream?

Can I serialize with Microsoft.FSharpLu.Json to a stream? I've asked this question on StackOverflow.

Right now, in Compact, I see a number of methods, some that can deserialize from a stream, but nothing that allows me to serialize to a stream. Is it possible to do with Compact?

How can one use a custom serializer in a BackwardsCompatible serializer?

BackwardsCompatible serializer uses Comapct and Default serializer. Is there a way to use some other (custom) serializer in there, like CamelCaseSerializer in place of Compact?

Maybe current approach to serializers (modules + SRTP) allows for it, but it is simply too hard for me to reason about.

I think creating a composable serializer would be easy if some common interface for the serializers were introduced (in place of SRTP).

type ISerializer =
    abstract member serialize<'T> : 'T->string
    // etc.

type FallbackSerializer(main : ISerializer, fallback : ISerializer) =
    implement ISerializer with
    // etc

What would be the downsides of such approach?

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Unable to run project because fsharplu cause a downgrade (broken build)

I have a issue that with recent versions of FSharp.Core FSharpLu broke my builds. It get this error:

   /Core.fsproj : warning NU1605: Detected package downgrade: FSharp.Core from 4.3.4 to 4.2.0. Reference the package directly from the project to select a different version. 
    Core.fsproj : warning NU1605:  Core (>= 1.0.0) -> Microsoft.FSharpLu (>= 0.10.26) -> FSharp.Core (>= 4.3.4) 

Support single-case discriminated union

It would be nice if single-case discriminated unions was serialized directly:

type PersonId = PersonId of int
type Person = 
  { Id: PersonId
    Name: string }

{"id": 123, "name": "Foo"}

instead of

{"id": {"PersonId": 123}, "name": "Foo"}

---

Edit: I realize there's some cases where this will do something very different from what's intended; e.g. if you have

type AuthType =
   | ApiToken of string

With the intention of adding more cases in the future, and then you'll break the serializer when you add the next case.

So I'm propising that we only do the single-case DU "optimization" when: There's one and only one case of course, and that case only has 1 field(?), and one or both of the following:

1. The name of the case is the same as the type. So type UserId = UserId of string. If it's type UserId = PersonId of string, then that's a no-match. (?)
2. When the argument is a "primitive" – string, number, long, etc. I can't immediatly think of a case where you want a single-case DU for something else, but I might be wrong about that (?).

I think I like 1 more, even though it's slightly more "magical", but I think it's most pragmatic way of doing it. What do you think?

Not deserialize with culture invariant?

I'm trying to read json with financial info like:

{"cost": 0.0, "price": 417.0, "total": 834.0, "doc_id": 2, "status": "Sync", "doc_code": "ORD-5-1", "doc_kind": "O", "username": "02", "createdat": "2019-01-02T17:09:48.478918", "sub_total": 834.0, "tax_value": 0.0, "updatedat": "2019-01-02T17:10:08.00777-05:00", "customer_code": "43915888_37", "delivery_order": 0, "discount_value": 0.0}

But it return, for example:

"cost": 0.0 //json
cost: 281474976710656 //.NET

This happened if the locale of the iOS device is set to Spanish/Colombia. On US locale it work. The weird thing is that is supposed to work with invariant culture:

https://www.newtonsoft.com/json/help/html/P_Newtonsoft_Json_JsonSerializerSettings_Culture.htm

I try to change in the code:

    module Internal =
        /// Serialization settings for the default NetwonSoft Json.Net's serialization format
        type DefaultSettings =
            static member settings =
                let s = JsonSerializerSettings(NullValueHandling = NullValueHandling.Ignore, Culture= System.Globalization.CultureInfo.InvariantCulture)
                s.Converters.Add(Converters.StringEnumConverter())
                s
            static member formatting = Formatting.Indented
//called with
let inline fromJson< ^T> x : ^T  =
    Json.BackwardCompatible.deserialize x

but don't see a change.

`startProcessAsync` deadlocks if process exists before calling AwaitEvent

With the NoTimeout option, startProcessAsync can deadlock when waiting for the process to terminate if the process exits before the call to Async.AwaitEvent:

https://github.com/Microsoft/fsharplu/blob/df50dee9590902a8bb4a15277f22c18711ac2207/FSharpLu/Diagnostics.fs#L136

Repro (extracting minimal code from Diagnostics.fs)

let deadlockRepro =
    async {
        use instance =
            new System.Diagnostics.Process(
                StartInfo =
                    System.Diagnostics.ProcessStartInfo
                        (
                            FileName = "notepad.exe",
                            WorkingDirectory = ".",
                            Arguments = "",
                            CreateNoWindow = false,
                            UseShellExecute = false,
                            RedirectStandardOutput = false,
                            WindowStyle = System.Diagnostics.ProcessWindowStyle.Normal,
                            Verb =   "runas"
                        ),
                EnableRaisingEvents = true)

        let r = instance.Start()
        printfn "killing the process"
        System.Diagnostics.Process.GetProcessById(instance.Id).Kill()
        printfn "waiting for process to exit"
        let! waitAsync = Async.AwaitEvent(instance.Exited)
        printfn "process exited with code: %d" instance.ExitCode

    }
deadlockRepro |> Async.RunSynchronously

Fix:

let startProcessWaitNoHang () =
    async {
        use instance =
            new System.Diagnostics.Process(
                StartInfo =
                    System.Diagnostics.ProcessStartInfo
                        (
                            FileName = "notepad.exe",
                            WorkingDirectory = ".",
                            Arguments = "",
                            CreateNoWindow = false,
                            UseShellExecute = false,
                            RedirectStandardOutput = false,
                            WindowStyle = System.Diagnostics.ProcessWindowStyle.Normal,
                            Verb =   "runas"
                        ),
                EnableRaisingEvents = true)

        let! waitAsync = Async.StartChild(Async.AwaitEvent(instance.Exited))

        if not (instance.Start()) then
            return raise <| System.InvalidOperationException("Could not start the process")
        else
            printfn "killing the process"
            System.Diagnostics.Process.GetProcessById(instance.Id).Kill()
            printfn "waiting for process to exit"
            let! _ = waitAsync
            printfn "process exited with code: %d" instance.ExitCode

    }

startProcessWaitNoHang() |> Async.RunSynchronously

Json: convert tuples to arrays

Currently, even Compact json serialization outputs tuples as multi-element objects

{
    "Item1": "foo",
    "Item2": 12
}

It should output ["foo", 12]. Implementation exists e.g. in https://github.com/haf/Newtonsoft.Json.FSharp (but that project is deprecated)

registerFileAndConsoleTracer file redirection not working in netcoreapp2.1 app

I modified the main function in FSharpLu.Windows.Sample.fsproj (current version in master) to also redirect output to a file, as follows, ran the app and it did not produce any file output.

use logger = System.Diagnostics.Listener.registerFileAndConsoleTracer "myComponentName" (Some @"d:\temp")

Update giraffe-support example for .Net core 3.0 fixing "Synchronous operations are disallowed"

Hi,

I used the following example: https://github.com/Microsoft/fsharplu/wiki/fsharplu.json#giraffe-support.

After upgrading to .Net 3.0 the serialization started to fail with "Synchronous operations are disallowed. Call ReadAsync or set AllowSynchronousIO to true instead." The reason is that Newtonsoft.Json Deserialize is using synchronous methods of the reader.

There is a request for DeserializeAsync for Newtonsoft.Json (JamesNK/Newtonsoft.Json#1193), but it seems that it is not going to happen.

Giraffe has a related issue too: giraffe-fsharp/Giraffe#351
They solved it by reading into intermediate memory stream before calling into Newtonsoft.Json. Here is the change: https://github.com/giraffe-fsharp/Giraffe/pull/354/files

I applied the same changes to the example and it worked. So basically, this is what I ended up with:

module FSharpLuJsonSerialization

open Newtonsoft.Json
open System.Threading.Tasks
open System.IO
open FSharp.Control.Tasks.V2.ContextInsensitive

let Utf8EncodingWithoutBom = System.Text.UTF8Encoding(false)
let DefaultBufferSize = 1024

let Formatting = Microsoft.FSharpLu.Json.Compact.TupleAsArraySettings.formatting
let Settings = Microsoft.FSharpLu.Json.Compact.TupleAsArraySettings.settings
let serializer = JsonSerializer.Create Settings

/// A Giraffe serializer based on FSharpLu.Json
/// See https://github.com/giraffe-fsharp/Giraffe/blob/master/DOCUMENTATION.md#serialization
type FSharpLuJsonSerializer() =
    interface Giraffe.Serialization.Json.IJsonSerializer with
        member __.SerializeToString(o: 'T) = JsonConvert.SerializeObject(o, Formatting, Settings)

        member __.SerializeToBytes<'T>(o: 'T): byte array =
            JsonConvert.SerializeObject(o, Formatting, Settings)
            |> System.Text.Encoding.UTF8.GetBytes

        member __.SerializeToStreamAsync (x : 'T) (stream : Stream) = 
            task {
                use memoryStream = new MemoryStream()
                use streamWriter = new StreamWriter(memoryStream, Utf8EncodingWithoutBom)
                use jsonTextWriter = new JsonTextWriter(streamWriter)
                serializer.Serialize(jsonTextWriter, x)
                jsonTextWriter.Flush()
                memoryStream.Seek(0L, SeekOrigin.Begin) |> ignore
                do! memoryStream.CopyToAsync(stream)
            } :> Task

        member __.Deserialize<'T>(json: string): 'T =
            JsonConvert.DeserializeObject<'T>(json, Settings)

        member __.Deserialize<'T>(bytes: byte []): 'T =
            let json = System.Text.Encoding.UTF8.GetString bytes
            JsonConvert.DeserializeObject<'T>(json, Settings)

        member __.DeserializeAsync(stream: System.IO.Stream): Task<'T> =
            task {
                use memoryStream = new MemoryStream()
                do! stream.CopyToAsync(memoryStream)
                memoryStream.Seek(0L, SeekOrigin.Begin) |> ignore
                use streamReader = new StreamReader(memoryStream)
                use jsonTextReader = new JsonTextReader(streamReader)
                return serializer.Deserialize<'T>(jsonTextReader)
            }

Could you please update the example in the WIKI?

Thank you,
Artem

encodeToBase64 and decodeBase64 are not symmetric

in FSharpLu\Text.cs, encodeToBase64 uses ASCIIEncoding while decodeBase64 uses UTF8Encoding. Recommend using UTF-8 for both.

how to handle missingmember, or nulls

Hi

I am trying to find a jsonparser, besides chiron, that can actually handle deserializing errors. I have tried out FSharpLu which seems nice but with a few flaws, or it could be me ;).

What I'm trying to do is to use "tryDeserialize<'T>" but when trying to parse any json string which is missing a member, the parser gives following structure, while expecting Choice2Of2 error:
{"field1":null } or {"field2" : 0 }, where the last is of type int.

My point is that it doesnt seem to be able to handle nulls or missingmembers.

I have tried with both default, compact and the CamelCaseSerializer way of doing it.

Example comes here:

type CamelCaseSettings =
static member settings =
let s =
JsonSerializerSettings(
NullValueHandling = NullValueHandling.Include,
MissingMemberHandling = MissingMemberHandling.Error,
ContractResolver = Serialization.CamelCasePropertyNamesContractResolver())
s.Converters.Add(CompactUnionJsonConverter(true))
s
static member formatting = Formatting.Indented

    type CamelCaseSerializer = With<CamelCaseSettings>

type C =
| Ompa
| Lompa

type NumberSpecial = NumberSpecial of int

type Ac ={
    number: NumberSpecial
    du: C
    testingOption: string option
    simpleInt : int
}

type Ab = {
    text: string
}

type B =
    | Ab of Ab
    | Ac of Ac

type A = {
    id: System.Guid
    context: B
    name: string
}

//des1 works fine
let des1 ="""{ "id":"f893e695-496d-4e30-8fc7-b2ff59725e6c",
"context": {"ac": {
"number": {
"numberSpecial":2
},
"du":"ompa",
"testingOption":"hejsvejs",
"simpleInt":4}},
"name":"hola"
}"""

//des2 does not throw error when obviously missing members...
//its output is:
// Choice1Of2 {id = 00000000-0000-0000-0000-000000000000;
// context = Ac {number = null;
// du = Ompa;
// testingOption = None;
// simpleInt = 0;};
// name = "hola";}

let des2 =
"""{
"context": {
"ac": {
"du": "ompa"
}
},
"name": "hola"
}"""

Or have i totally missunderstood how to work with FSharpLu?

Best / Johan

Using CompactUnionJsonConverter in ASP.NET MVC results in internal server errors when sending invalid JSON

I have an ASP.NET MVC web api on which I have setup CompactUnionJsonConverter as one of the converters used in deserializing/serializing JSON in the API.

services
    .AddMvc()
    .SetCompatibilityVersion(CompatibilityVersion.Version_2_2)
    .AddJsonOptions(fun options ->
        options.SerializerSettings.Converters <- [| CompactUnionJsonConverter(true) |]
    )

The converter works just fine, but the thing is when you send invalid JSON to an endpoint in the API, instead of resulting in an invalid ModelState in the controller action, as one would expect when using other classes which implement JsonConverter, the server returns an internal server error before the code within the action is actually executed.

From what I gather, the problem is due to the fact that within CompactUnionJsonConverter, instead of raising JsonReaderException or JsonSerializationException, the FSharp utility functions like failwith and failwithf are used. This simply makes it so a basic System.Exception is raised, and this type of exceptions is not caught by the JSON deserialization process in ASP.NET MVC, and a status code 500 is returned instead of normally going through the action as should be expected.

I feel like this is an issue that should be addressed if other people want to use this Converter in the same way.

Plz release new version to Nuget

It'd be nice to have this changes available

FsCheck unit tests broken by F# 4.1: "Unable to cast object of type 'XXX' to type 'Testable ...`"

Four FsCheck-based unit tests are now failing after the upgrade to F# 4.1. This was not caught in the continuous builds because the build definition defaulted to VS 2015 and F# 4.0 even though the the .fsproj files specifically require F# 4.1.

The affected unit tests are:

    member this.``Serialization and deserialization are reciprocal``() =
    member __.``Fuzzing Reciprocal`` () =
    member __.``Deserialization coincides with JSon.Net (Fuzzing)`` () =
    member __.``BackwardCompatible deserializes default Json.Net format and returns same object`` () =

Call stack for the BackwardCompatible test:

Test method Microsoft.FSharpLu.Json.Tests+JsonSerializerTests.BackwardCompatible deserializes default Json.Net format and returns same object threw exception: 
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Exception: BackwardCompatibility.get_x1-Falsifiable, after 1 test (0 shrinks) (StdGen (1333139700,296279097)):
Original:
<null>
with exception:
System.InvalidCastException: Unable to cast object of type 'Arrow@202-1[Microsoft.FSharpLu.Json.Tests+ComplexDu,Microsoft.FSharp.Core.Unit]' to type 'Testable`1[Microsoft.FSharp.Core.FSharpFunc`2[Microsoft.FSharpLu.Json.Tests+ComplexDu,Microsoft.FSharp.Core.Unit]]'.
   at Microsoft.FSharp.Core.LanguagePrimitives.IntrinsicFunctions.UnboxGeneric[T](Object source)
   at FsCheck.Testable.evaluate[a,b](FSharpFunc`2 body, a a) in C:\Users\Kurt\Projects\FsCheck\FsCheck\src\FsCheck\Testable.fs:line 151

Doesn't build with dotnetcore

When I try to build using dotnet build I get:

C:\Program Files\dotnet\sdk\3.0.100\Microsoft.Common.CurrentVersion.targets(1175,5): error MSB3644: The reference assemblies for .NETFramework,Version=v4.6.2 were not found. To resolve this, install the Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Framework Developer Packs at https://aka.ms/msbuild/developerpacks [C:\Development\Libs\fsharplu\FSharpLu.Azure\FSharpLu.Azure.fsproj]
C:\Program Files\dotnet\sdk\3.0.100\Microsoft.Common.CurrentVersion.targets(1175,5): error MSB3644: The reference assemblies for .NETFramework,Version=v4.6.2 were not found. To resolve this, install the Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Framework Developer Packs at https://aka.ms/msbuild/developerpacks [C:\Development\Libs\fsharplu\FSharpLu.Windows\FSharpLu.Windows.fsproj]
C:\Program Files\dotnet\sdk\3.0.100\Microsoft.Common.CurrentVersion.targets(1175,5): error MSB3644: The reference assemblies for .NETFramework,Version=v4.6.2 were not found. To resolve this, install the Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Framework Developer Packs at https://aka.ms/msbuild/developerpacks [C:\Development\Libs\fsharplu\FSharpLu\FSharpLu.fsproj]
C:\Program Files\dotnet\sdk\3.0.100\Microsoft.Common.CurrentVersion.targets(1175,5): error MSB3644: The reference assemblies for .NETFramework,Version=v4.6.2 were not found. To resolve this, install the Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Framework Developer Packs at https://aka.ms/msbuild/developerpacks [C:\Development\Libs\fsharplu\FSharpLu.Tests\FSharpLu.Tests.fsproj]
C:\Program Files\dotnet\sdk\3.0.100\Microsoft.Common.CurrentVersion.targets(1175,5): error MSB3644: The reference assemblies for .NETFramework,Version=v4.6.2 were not found. To resolve this, install the Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Framework Developer Packs at https://aka.ms/msbuild/developerpacks [C:\Development\Libs\fsharplu\FSharpLu.Json\FSharpLu.Json.fsproj]
5 Warning(s)
5 Error(s)

I have these sdk's installed:

λ dotnet --info
.NET Core SDK (reflecting any global.json):
Version: 3.0.100
Commit: 04339c3a26

Runtime Environment:
OS Name: Windows
OS Version: 10.0.18362
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\3.0.100\

Host (useful for support):
Version: 3.0.0
Commit: 7d57652f33

.NET Core SDKs installed:
2.1.801 [C:\Program Files\dotnet\sdk]
2.1.802 [C:\Program Files\dotnet\sdk]
2.2.401 [C:\Program Files\dotnet\sdk]
2.2.402 [C:\Program Files\dotnet\sdk]
3.0.100 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

How to build the project?

1. dotnet build returns this error:

C:\Program Files\dotnet\sdk\1.0.4\Microsoft.Common.CurrentVersion.targets(2630,7): error MSB4057: The target "CreateManifestResourceNames" does not exist in the project. [D:\_\F#\re
pos\fsharplu\FSharpLu\FSharpLu.fsproj]
C:\Program Files\dotnet\sdk\1.0.4\Sdks\Microsoft.NET.Sdk\build\Microsoft.NET.Sdk.Common.targets(73,5): error : Project 'D:\_\F#\repos\fsharplu\FSharpLu.Json\FSharpLu.Json.fsproj' ta
rgets '.NETStandard,Version=v1.6; .NETStandard,Version=v2.0; .NETFramework,Version=v4.5.2; .NETFramework,Version=v4.6.1'. It cannot be referenced by a project that targets '.NETFram
ework,Version=v4.0'. [D:\_\F#\repos\fsharplu\FSharpLu.Json\FSharpLu.Json.fsproj]
C:\Program Files\dotnet\sdk\1.0.4\Microsoft.Common.CurrentVersion.targets(2630,7): error MSB4057: The target "CreateManifestResourceNames" does not exist in the project. [D:\_\F#\re
pos\fsharplu\FSharpLu.Json\FSharpLu.Json.fsproj]
C:\Program Files\dotnet\sdk\1.0.4\Sdks\Microsoft.NET.Sdk\build\Microsoft.NET.Sdk.Common.targets(73,5): error : Project 'D:\_\F#\repos\fsharplu\FSharpLu\FSharpLu.fsproj' targets '.NE
TStandard,Version=v2.0; .NETFramework,Version=v4.5.2; .NETFramework,Version=v4.6.1'. It cannot be referenced by a project that targets '.NETFramework,Version=v4.0'. [D:\_\F#\repos\f
sharplu\FSharpLu\FSharpLu.fsproj]

2. When I open project in VS2015, I get following error message box:
   

Error when deserialising json that was serialised with v0.10.31

I recently upgraded a project's FSharpLu dependency from 0.10.31 to 0.11.7. In this project, we use an eventstore and have years of existing serialised events. I found that upgrading FSharpLu broke deserialisation of specific tuple types. I tracked down a specific repro below. I've fixed the issue for now by changing references from BackwardCompatable to Compact.Legacy in the project (AndrewIOM/global-pollen-project/issues/30). It wasn't clear from the docs whether this is expected behaviour or not, so opening a bug in case!

Edit: some of the earlier events may have been serialised with Newtonsoft, which may explain issues with these events. Later events have a format as follows: """Some ("Novosti Sist. Vyssh. Rast., 41: 93, 2009", None) }"""

#r "nuget:Microsoft.FSharpLu.Json,v=0.11.6"
open Microsoft.FSharpLu.Json

type Url = Url of string
type Test = (string * Url option) option

let testItem = """{ "Item1": "\"Reisen in Europa, Asien und Afrika 1:\", 161, 1847" }"""

let works = Compact.Legacy.deserialize<Test> testItem
let doesntWork = Compact.deserialize<Test> testItem

... and FSI output:

> type Url = Url of string
- type Test = (string * Url option) option
- 
- let testItem = """{ "Item1": "\"Reisen in Europa, Asien und Afrika 1:\", 161, 1847" }"""
- ;;
type Url = | Url of string
type Test = (string * Url option) option
val testItem: string =
  "{ "Item1": "\"Reisen in Europa, Asien und Afrika 1:\", 161, 1847" }"

> let works = Compact.Legacy.deserialize<Test> testItem
- ;;
val works: Test =
  Some (""Reisen in Europa, Asien und Afrika 1:", 161, 1847", None)

> let doesntWork = Compact.deserialize<Test> testItem
- ;;
Newtonsoft.Json.JsonReaderException: Cannot parse legacy tuple value: {
  "Item1": "\"Reisen in Europa, Asien und Afrika 1:\", 161, 1847"
}. Missing property: Item2
   at <StartupCode$Microsoft-FSharpLu-Json>[email protected](String s)
   at [e-mail](b& curr) in D:\a\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 81
   at Microsoft.FSharp.Collections.Internal.IEnumerator.MapEnumerator`1.System.Collections.IEnumerator.MoveNext() in D:\a\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 68
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at Microsoft.FSharp.Collections.SeqModule.ToArray[T](IEnumerable`1 source)
   at Microsoft.FSharp.Collections.ArrayModule.OfSeq[T](IEnumerable`1 source) in D:\a\_work\1\s\src\fsharp\FSharp.Core\array.fs:line 1059
   at Microsoft.FSharpLu.Json.CompactUnionJsonConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.Serialization.JsonSerializerProxy.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize(JsonReader reader, Type objectType)
   at Newtonsoft.Json.Linq.JToken.ToObject(Type objectType, JsonSerializer jsonSerializer)
   at <StartupCode$Microsoft-FSharpLu-Json>.$Compact.ReadJson$cont@172(JsonSerializer serializer, JsonReader reader, Type objectType, FSharpTypeFunc failreadwithf, Unit unitVar)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
   at Microsoft.FSharpLu.Json.Compact.deserialize[T](String json)
   at <StartupCode$FSI_0016>.$FSI_0016.main@()
Stopped due to error
> 

Consider a special case for type `Union option` with an unknown case

I'd like to propose a special case for the type signature where an option wraps a union.

I believe that when external input is not correct (say an unknown case) that if the type is Union option it should be a default or a config option to return None instead of throwing

Currently an unknown case will just throw in the recursive call to the same serializer with
System.Exception: Cannot parse DU field-less value: z. Expected names: x, y

Nested optional algaebric datatype cannot be deserialized

Following code:

type A =
    {
        subField1 : int
        subField2 : int
    }
type B =
    {
        field : A option
    }

let x = { field = Some { subField1 = 1 ; subField2 = 2 } }

let json = Microsoft.FSharpLu.Json.Compact.serialize<B> x
Microsoft.FSharpLu.Json.Compact.deserialize<B> json 

throws the following exception

System.Exception: Nested option types must be serialized as boxed Json objects with attribute 'Some'.
>    at <StartupCode$Microsoft-FSharpLu-Json>.$Compact.parseBox@92(JsonSerializer serializer, Type nestedType, JToken jToken)
   at Microsoft.FSharpLu.Json.CompactUnionJsonConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType, IDictionary`2& extensionData)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
   at Microsoft.FSharpLu.Json.Compact.deserialize[T](String json)
   at <StartupCode$FSI_0035>.$FSI_0035.main@() in D:\sources\Springfield\source\Script1.fsx:line 161

NullReferenceException on Discriminated Union serialization

Consider DU:

type Claim = Read | Edit | Delete | Custom of string

Sometimes during serialization I get NullReferenceException, no matter the case.
Issue is present both for compact and standard modes.
It's a floating issue, I haven't noticed any pattern of it's reproducing

Leaking object in startProcessAsync

Following code in Diagnostics.fs is leaking object instance:

let instance =
    new Process(
            StartInfo =
               ...
...

Preserve formatting when serializing discriminated unions without fields

When serializing discriminated unions without any fields, the current behaviour is to use the Newtonsoft.Json ContractResolver's GetResolvedPropertyName for serialization of the case name.

This means that when serializing using CamelCasePropertyNamesContractResolver

type FooBar = Foo | Bar

{
    Foo: Bar
}

Becomes:

{
    "foo": "bar"
}

I would prefer that the serialization instead produces this:

{
    "foo": "Bar"
}

As I believe it better conveys the value.

The code causing this is likely at or near Compact.fs:151.

Using RequireNonOptionalPropertiesContractResolver causes invalid dictionary serialization

I've noticed a bug that when you serialize a dictionary, e.g. something like

type Dto =
        { ints : IDictionary<string, int> }

in combination with the RequireNonOptionalPropertiesContractResolver serializes json
where all the dictionary keys are converted to lowercase.

Don't use arrays for converters in JsonSerializerSettings instances

There's number of places like this where you use Converters = [|CompactUnionJsonConverter(true, true)|].

Problem is some extension methods like ConfigureForNodaTime adjust existing settings with settings.Converters.Add(...), which leads to a NotSupportedException being thrown.

Solution is very easy and here are some options:

let converters = ResizeArray([CompactUnionJsonConverter(true, false) :> JsonConverter])
JsonSerializerSettings(Converters = converters)

// or

let settings = JsonSerializerSettings(...)
settings.Converters.Add(CompactUnionJsonConverter(true, false))
settings

How can I serialise with no indentation formating?

How can I serialise with no indentation formating?

Add functions taking a System.Type instead of a type parameter

The output of the function deserialize : string -> 'T can be controled with a type parameter. But sometimes, for instance as an ASP.NET Core TextInputFormatter, you only have an instance of System.Type to know the output's type.

I propose to have additional functions, besides deserialize… that additionally takes that System.Type as an argument. They could be named deserializeToType or something along those lines.

The changes to the code are fairly small, as the underlying serializer does already supports that szenario. I also already have an implementation lying around internally and would be willing to contribute this, if I get some advice on the naming of the functions/the API in general.

Internal semaphores created in Microsoft.FSharpLu.Async. should be disposed Synchronization

There are two instances of semaphore created but not being disposed in the Synchronization module.

• The pool class.
• SingleUseEvent class

Those two class should implement Idisposable and properly dispose those resources.

How can I serialize with json keys in camelcase?

How can I serialize with the json keys in camelcase instead of the default pascalcase?
This: { someKey: "value" }
instead of: { SomeKey: "value" }