weichch / system-text-json-jsondiffpatch Goto Github PK

.NET 8 has been released and System.Json.Text has changed its targeting frameworks. Though the current targeting framework net6.0 and netstandard2.x would still work, it would be good to exam over any breaking changes and change the targeting framework accordingly.

We need to:

• Go over breaking changes in the System.Text.Json package of .NET 7
• Change targeting framework accordingly
• Target test projects to .NET 8
• Target benchmark project to .NET 8
• Update Github Actions to build and test in .NET 8
• Remove .NET 5 from all projects

Similar to #9, we need to add new APIs to support JsonElement

• #24
• #27
• Add new APIs to Patch (need to create new JsonElement)
• Add new APIs to JsonValueComparer for values backed by JsonElement

Add JsonDocument support by serializing JsonDocument to JsonNode.

• #24
• #27
• Add new APIs to DeepClone
• Add new APIs to Patch (need to create new JsonDocument)
• #14

Add an automation to test if the main branch is compatible with latest System.Text.Json package.

• Test runs against latest stable version
• Test runs against latest preview version
• Test runs on a schedule

With the following code (excluded the usings for simplicity)

var node1 = JsonNode.Parse("{\"foo\":[ {\"bar1\": \"res1\"} ]}");
var node2 = JsonNode.Parse("{\"foo\":[ {\"bar1\": \"res2\"} ]}");

var x = JsonDiffPatcher.Diff(node1, node2, new JsonPatchDeltaFormatter());
Console.WriteLine(x);

I'm getting follwing:

[
  {
    "op": "remove",
    "path": "/foo/0"
  },
  {
    "op": "add",
    "path": "/foo/0",
    "value": {
      "bar1": "res2"
    }
  }
]

I would like to get something like:

[
  {
    "op": "replace",
    "path": "/foo/0/bar1"
    "value": "res2"
  }
]

I've looked into DefaultFormatter and it seems that JsonDiffDelta has already 2 change entries: one for remove and one for add.

What's the best method to check arrays recursively (the way I want)? Is there some kind of flag or would I need to override FormatArray somehow?

Thanks!

[xUnit.net 00:00:00.32] Imageflow.Test.TestJson.TestAllJob [FAIL]
Failed Imageflow.Test.TestJson.TestAllJob [186 ms]
Error Message:
System.Text.Json.JsonDiffPatch.Xunit.JsonEqualException : JsonAssert.Equal() failure.
Stack Trace:
at System.Text.Json.JsonDiffPatch.Xunit.JsonAssert.HandleEqual(JsonNode expected, JsonNode actual, JsonDiffOptions diffOptions, Func`2 outputFormatter)
at System.Text.Json.JsonDiffPatch.Xunit.JsonAssert.Equal[T](T expected, T actual)
at Imageflow.Test.TestJson.TestAllJob() in C:\Users\lilith\Documents\GitHub\wimageflow-dotnet\tests\Imageflow.Test\TestJson.cs:line 70
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

RFC JsonPatch benchmarking was removed in #7 to speed up releasing version 1.2.0. Now the new version has been released, we should add it back to evaluate future changes in the formatter.

I have type:

class test
{
    public int? ParentId { get; set; }
}

I've generated diff for the change:

{
  "ParentId": [
    1,
    ""
  ]
}

But then I'm not able to Patch object. I've got the exception:

System.Text.Json.JsonException: The JSON value could not be converted to System.Nullable`1[System.Int32]. Path: $.ParentId | LineNumber: 0 | BytePositionInLine: 73.
 ---> System.FormatException: Either the JSON value is not in a supported format, or is out of bounds for an Int32.
   at System.Text.Json.Utf8JsonReader.GetInt32WithQuotes()
   at System.Text.Json.Serialization.Converters.Int32Converter.ReadNumberWithCustomHandling(Utf8JsonReader& reader, JsonNumberHandling handling, JsonSerializerOptions options)
   at System.Text.Json.Serialization.Converters.NullableConverter`1.ReadNumberWithCustomHandling(Utf8JsonReader& reader, JsonNumberHandling numberHandling, JsonSerializerOptions options)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   --- End of inner exception stack trace ---
   at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, Utf8JsonReader& reader, Exception ex)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 utf8Json, JsonTypeInfo jsonTypeInfo, Nullable`1 actualByteCount)
   at System.Text.Json.JsonSerializer.ReadNode[TValue](JsonNode node, JsonTypeInfo jsonTypeInfo)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](JsonNode node, JsonSerializerOptions options)
   at Domain.Services.JsonTools.PatchObject[T](T object, JsonNode diff) in D:\work\inv\src\api\Domain.Services\JsonTools.cs:line 75

When using ArrayObjectItemKeyFinder with test data like below, System.FormatException: 'Invalid patch document.' is thrown

json1
{ "id": 1, "name": "Some name", "projects": [{"id": 88, "name": "bar"}] }

json2
{ "id": 1, "name": "Some name", "projects": [{"id": 77, "name": "foo"}, {"id": 88, "name": "bar z"}] }

Code to reproduce:

using System.Text.Json.JsonDiffPatch.Diffs.Formatters;
using System.Text.Json.JsonDiffPatch;
using System.Text.Json.Nodes;

string json1 = "{ \"id\": 1, \"name\": \"Some name\", \"projects\": [{\"id\": 88, \"name\": \"bar\"}] }";
string json2 = "{ \"id\": 1, \"name\": \"Some name\", \"projects\": [{\"id\": 77, \"name\": \"foo\"}, {\"id\": 88, \"name\": \"bar z\"}] }";

// project with id 77 inserted first in the array, name for project with id 88 changed from "bar" to "bar z"

object? IdKeyFinder(JsonNode? n, int i)
{
    if (n is JsonObject obj
           && obj.TryGetPropertyValue("id", out var value))
    {
        return value?.ToString();
    }

    return null;
}

var diff = JsonDiffPatcher.Diff(
    json1,
    json2,
    new JsonPatchDeltaFormatter(),
    new JsonDiffOptions
    {
        JsonElementComparison = JsonElementComparison.RawText,
        ArrayObjectItemMatchByPosition = false,
        ArrayObjectItemKeyFinder = IdKeyFinder
    });

I am using SystemTextJson.JsonDiffPatch 2.0.0 on .Net 8.

Given this TestClass, I want diffs to include CompareMe and excludeIgnoreMe.

public class TestClass
{
    public int CompareMe { get; set; }
    public int IgnoreMe { get; set; }
}

When both left and right are not null, the diff correctly excludes IgnoreMe.

public void BothNotNull()
{
    TestClass left = new TestClass() { CompareMe = 1, IgnoreMe = 1 };
    TestClass right = new TestClass() { CompareMe = 2, IgnoreMe = 2 };

    string leftJson = JsonSerializer.Serialize(left);
    string rightJson = JsonSerializer.Serialize(right);

    JsonDiffOptions options = new JsonDiffOptions()
    {
        PropertyFilter = (prop, _) => !StringComparer.Ordinal.Equals(prop, nameof(TestClass.IgnoreMe)),
    };

    JsonNode? diff = JsonDiffPatcher.Diff(leftJson, rightJson, options);
    string diffJson = diff!.ToJsonString(); // {"CompareMe":[1,2]}
}

But if one of left and right is null then the JsonDiffOptions.PropertyFilter function doesn't run and the diff includes IgnoreMe.

public void LeftNull()
{
    TestClass? left = null;
    TestClass right = new TestClass() { CompareMe = 2, IgnoreMe = 2 };

    string leftJson = JsonSerializer.Serialize(left);
    string rightJson = JsonSerializer.Serialize(right);

    JsonDiffOptions options = new JsonDiffOptions()
    {
        PropertyFilter = (prop, _) => !StringComparer.Ordinal.Equals(prop, nameof(TestClass.IgnoreMe)),
    };

    JsonNode? diff = JsonDiffPatcher.Diff(leftJson, rightJson, options);
    string diffJson = diff!.ToJsonString(); // [null,{"CompareMe":2,"IgnoreMe":2}]
    // expected diffJson to be [null,{"CompareMe":2}]
}

public void RightNull()
{
    TestClass left = new TestClass() { CompareMe = 1, IgnoreMe = 1 };
    TestClass? right = null;

    string leftJson = JsonSerializer.Serialize(left);
    string rightJson = JsonSerializer.Serialize(right);

    JsonDiffOptions options = new JsonDiffOptions()
    {
        PropertyFilter = (prop, _) => !StringComparer.Ordinal.Equals(prop, nameof(TestClass.IgnoreMe)),
    };

    JsonNode? diff = JsonDiffPatcher.Diff(leftJson, rightJson, options);
    string diffJson = diff!.ToJsonString();  // [{"CompareMe":1,"IgnoreMe":1},null]
    // expected diffJson to be [{"CompareMe":1},null]
}

I'm confused about how to preserve missing values from when diffing two objects.

Take for example these two json blocks:

{
  "Name": "John",
  "Age": 34,
  "Origin": "Scotland",  
  "Pets": [
    {
      "Type": "Cat",
      "Name": "MooMoo",
      "Age": 3.4
    },
    {
      "Type": "Squirrel",
      "Name": "Sandy",
      "Age": 7
    }
  ]
}

{
  "Origin": "Ireland"
}

When I run a patch, I would like to end up with:

{
  "Name": "John",
  "Age": 34,
  "Origin": "Ireland",   // UPDATED HERE  
  "Pets": [
    {
      "Type": "Cat",
      "Name": "MooMoo",
      "Age": 3.4
    },
    {
      "Type": "Squirrel",
      "Name": "Sandy",
      "Age": 7
    }
  ]
}

Basically all the missing values are preserved and only the values that conflict are updated, but I can seem to achieve this.

As part of benchmark improvement(#7), I found the performance of Diff and DeepEquals is not ideal for array comparisons in some scenarios. Further investigation suggests the issue might be related to Lcs where materializing is done N x M times where N and M are the length of the arrays being compared.

Conversely, JToken.DeepEquals performs a lot faster, even faster than comparing JsonNode by RawText, as the tokens are only materialized N+M times and cached in the JToken instances.

I've already benchmarked enabling a global value cache for the entire diff operation, but there is performance setback instead of improvement. I've also benchmarked a custom parser for parsing JSON strings into JsonNode instances in not readonly mode, but the performance gain is once again not worth it, and no improvement in array comparisons.

So, for now, I'm looking at a solution that is just enough for LCS, and hopefully release an enhancement as part of the next version going out.

The benchmarks are confusing as it only compares JToken with JsonNode in readonly mode.

With when comparing JsonNode in readonly mode, we only have two options:

• Raw text (based on string comparison)
• Semantic (materialize JsonElement in DeepEquals and JsonValueComparer)

With RawText mode, it is not expected to output the same result as JToken comparison which is semantic only. Therefore, the benchmarks to compare those two are confusing.

With Semantic mode, it is expected the comparison of JsonNode to be slower than JToken comparison as the materialize processing can happen multiple times during comparison (currently a design to keep memory allocation low). This should ideally be evaluated so that we could refer to the result when improving in this area. We should also add benchmarks for comparing materialized JsonNodes, and compare the benchmark results with JToken comparison, so that it shows the algorithm level benchmarking results. This should not include JSON parsing.

The actions:

• Remove JToken vs readonly JsonNode with RawText option benchmarks
• Add/revise JToken vs readonly JsonNode with Semantic option benchmarks
• Add/revise JsonNode with different options benchmarks

Currently only jsondiffpatch document is supported by Patch and PatchNew methods. This issue is to add support for RFC 6902 document.

The following sub-issues should be addressed:

• Add support for operations produced by JsonPatchFormatter
• Add support for other operations defined in RFC

Once the two APIs are available in newer version of System.Text.Json, the two custom made methods should be decommissioned (if possible).

Track progress in dotnet/runtime#56592.
Also related this dotnet/runtime#33388.

JsonElement and JsonDocument support are easier to add than adding such support to all functionalities. This issue to implement new DeepEquals APIs that compare JsonElement and JsonDocument.

Is there a way to get the different between left and right but only returning the field of one side?
e.g.

A = {
    changedField: "value1",
    unchangedField: "test"
}

B = {
    changedField: "value2",
    unchangedField: "test"
}

A.Diff(B) = {
    changedField: "value2"
}

Add new Diff APIs for JsonDocument and JsonElement

Just reaching out to say hi.

I created and maintain the json-everything project, which aims to ensure support for JSON-related technologies in .Net. While I mainly provide libraries, I also maintain a list of other projects. I'd like to add your project to that list.

Let me know if you'd like that.

Steps to reproduce:

JsonDiffPatcher.DefaultOptions =
	() => new JsonDiffOptions { SuppressDetectArrayMove = true };

var x = JsonDocument.Parse(@"{""a"":[{""b"":1},{""b"":2}]}");
var y = JsonDocument.Parse(@"{""a"":[{""b"":1},{""b"":2}]}");
var z = JsonDocument.Parse(@"{""a"":[{""b"":2},{""b"":1}]}");

var test1 = x.DeepEquals(y);  //test1 is true
var test2 = x.DeepEquals(z);  //test2 is false, expected true

x.Deserialize<JsonNode>().ShouldEqual(y.Deserialize<JsonNode>());  //no exception
x.Deserialize<JsonNode>().ShouldEqual(z.Deserialize<JsonNode>());  //throws JsonEqualException, expected no exception

Either SuppressDetectArrayMove does not work, or I misunderstood what it does? I this case how can I configure JsonDiffPatcher to ignore array element order?

Hi, is there a way to ignore properties from diff? The JsonDiffPatch does it here https://github.com/wbish/jsondiffpatch.net/blob/master/Src/JsonDiffPatchDotNet/JsonDiffPatch.cs#L332

What I am missing in the JsonDiffPatch is that it only allows you to exclude paths instead of properties which is a problem with recursive objects or arrays. To ignore property foo in an array of objects

{ "bar" : [ { "foo": 1 }, { "foo": 2 }, { "foo": 3} ] }

you would have to do

exclude paths: "bar.arr[0].foo", "bar.arr[1].foo", "bar.arr[2].foo"

This is useful in some scenarios, but it would be amazing if there would also be an option to exclude properties by their name

exclude properties: "foo"

The JsonDiffPatch looks abandoned so I am looking for an alternative that would support this scenario.

Is it something doable?

This issue is used to track the tasks to setup for this repository.

• Move benchmarks out of Readme.md
• Add release notes
• Add contributing docs
• Add various templates, e.g. issue template
• Add Github actions and PR checks
  • Add to run on ubuntu-latest for .NET and .NET Core
  • Add to run on windows-latest (or appropriate) for .NET Framework
• Add build status to Readme.md
• Add CODEOWNERS file and restrict PR approval access
• Add Wiki

I have a json which has 2 layers of objects such as

{
  "test": {
      "key1": "value1",
      "key2": "value2",
   }
}

Will this library support diffing and patching with such kind of structure?

Is it intended, or could there be some other flag to enforce it?

Is your feature request related to a problem? Please describe.
I want to minimise dependencies in my project by utilising framework dependencies wherever possible

Describe the solution you'd like
I want the package to not have an explicit dependency on System.Text.Json as it can be provided by the framework.

Describe alternatives you've considered
Accept the additional dependency

Additional context
n/a

We should add more tests and make sure at least 90% (currently 83% in main package, 66%-78% in assert packages) overall test coverage.

• Add more test cases
• Make sure overall test coverage is at least 90%
• Exclude external code (such as Google's diff_match_patch)
• Fix test coverage for .NET Standard 2.0 (currently only .NET Framework 4.6.1 and .NET Standard 2.1 are covered)
• Calculate test coverage in Github Action if possible
• Export test results when needed

Hi there,

I noticed that you included code from https://github.com/google/diff-match-patch and added its license file. From the repository root however it looks like everything under it is MIT. The same goes for the NuGet package which does not include the license file of the third party component. It is pretty common to simply create a file THIRD-PARTY-NOTICES.txt and list all used components, their licenses, copyrights and version numbers. I already prepared a PR for you at #41.

Best regards,
Georg

DeepEquals method currently only implements syntactic equality that compares the raw text of two JsonNode if both backed by JsonElement.

For example, the below test will fail:

var array1 = JsonNode.Parse("[ 1 ]");
var array2 = JsonNode.Parse("[ 1.0 ]");

Assert.True(array1.DeepEquals(array2));

but, can pass for

var array1 = JsonNode.Parse("[         1           ]");
var array2 = JsonNode.Parse("[1]");

Assert.True(array1.DeepEquals(array2));

This is not the case for only one JsonNode backed by JsonElement as the method is already trying to deserialize the JSON representation into .NET types and do semantic comparison.

The following test can pass:

var array1 = new JsonArray(JsonValue.Create(1m));
var array2 = JsonNode.Parse("[1.0]");

Assert.True(array1.DeepEquals(array2));

There should be option/argument to allow using syntactic and semantic IEqualityComparer<JsonElement> implementations when calling the method.

Also see some discussions here.

This may also block adding JsonDocument support.