JsonReaderException when retrieving documents containing DateTime about rethinkdb.driver

[bchavez]

Hi @fiLLLip , thanks for reporting this. After examining the Java driver, I think we may have to impose a restriction to use only DateTimeOffset. The older DateTime struct doesn't support time-zones. RethinkDB natively supports time-zones as well as the data structure in the Java driver.

Let me know what you think.

The problem basicaly comes down to casting a DateTimeOffset -> to -> DateTime which isn't allowed.

[fiLLLip]

I went to Google to find some answers, and found this issue. It seems like we can make Json.NET convert and preserve timezone into a DateTime type.

I mocked up the following test:

using System;
using Newtonsoft.Json;

namespace ConsoleApplication2
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            var strings = new[]
            {"{\"DateTime\": \"1990-08-18T00:00:00\"}", "{\"DateTime\": \"1990-08-18T00:00:00+03:00\"}", "{\"DateTime\": \"1990-08-18T00:00:00+02:00\"}", "{\"DateTime\": \"1990-08-18T00:00:00+01:00\"}"};
            foreach (var s in strings)
            {
                Console.Out.WriteLine(s);
                var dateTimeOffsetClass = JsonConvert.DeserializeObject<DateTimeOffsetClass>(s);
                Console.Out.WriteLine($"DateTimeOffsetClass: {dateTimeOffsetClass.DateTime}");
                var jsonSerializerSettings = new JsonSerializerSettings
                {
                    DateFormatHandling = DateFormatHandling.IsoDateFormat,
                    DateParseHandling = DateParseHandling.DateTimeOffset,
                    DateTimeZoneHandling = DateTimeZoneHandling.Utc
                };
                var dateTimeClass = JsonConvert.DeserializeObject<DateTimeClass>(s, jsonSerializerSettings);
                Console.Out.WriteLine($"DateTimeClass: {dateTimeClass.DateTime}");
            }
            Console.In.ReadLine();
        }
    }

    internal class DateTimeClass
    {
        public DateTime DateTime { get; set; }
    }

    internal class DateTimeOffsetClass
    {
        public DateTimeOffset DateTime { get; set; }
    }
}

This supplied me with the following result:

{"DateTime": "1990-08-18T00:00:00"}
DateTimeOffsetClass: 18.08.1990 00.00.00 +02:00
DateTimeClass: 18.08.1990 00.00.00
{"DateTime": "1990-08-18T00:00:00+03:00"}
DateTimeOffsetClass: 18.08.1990 00.00.00 +03:00
DateTimeClass: 17.08.1990 21.00.00
{"DateTime": "1990-08-18T00:00:00+02:00"}
DateTimeOffsetClass: 18.08.1990 00.00.00 +02:00
DateTimeClass: 17.08.1990 22.00.00
{"DateTime": "1990-08-18T00:00:00+01:00"}
DateTimeOffsetClass: 18.08.1990 00.00.00 +01:00
DateTimeClass: 17.08.1990 23.00.00

This can be somewhat misleading, since the timezone is not specified when using DateTime. The positive thing about using this approach is that we don't get any errors.

[bchavez]

I think we have two kinds of DateTime. DateTimeKind.Utc and DateTimeKind.Utc.Local. The problem is, what does it mean to be local? DB server local DateTime time? App local DateTime?

When the server presents us with a time zone that is not UTC, and we're trying to stuff it into a DateTime we actually lose zone information back to the application.

I'll look at this a bit later, I have to go in for a meeting soon. :)

[bchavez]

we could possibly enforce UTC only with DateTime when serializing also

[fiLLLip]

I think it would be nice to be able to use DateTime if you want, and then we could enforce UTC when serializing/deserializing.

I tested now and updated my model to DateTimeOffset, but I got the same error as when using DateTime.

[fiLLLip]

I added another document to my database, and tried to fetch all again with no success.
When testing even further, it shows that fetching ONE item with .get() works perfectly.

The following uses _table which hides _r.db(_c.Dbname).table(TableName)
It seems like the problem occurs when executing

Cursor<User> result = _table.run<User>(_c.Connection);
return result.BufferedItems;

Edit:
Tested some more, without the .BufferedItems and that also seems to work:

Cursor<User> result = _table.run<User>(_c.Connection);
return result;

Edit2:
Same error occures when executing the following (also here it works without the .BufferedItems:

Cursor<User> result = _table.getAll(email).optArg("index", nameof(User.Email)).run<User>(_c.Connection);
return result.BufferedItems.FirstOrDefault();

[bchavez]

Yea, the problem is with BufferedItems. The response payload isn't getting passed into the Converter. Like the Java driver, Converters job is to translate ReQL native types $reql_type$ like a date and time into deserializable JSON for Newtonsoft's deserialization without $reql_type$. The deserializer throws because DateTime is expected, but it encounters a native $reql_type$ that looks like an Object. Hence the unexpected object exception near Birthdate.

So, when the bug mentioned above is fixed and $reql_type$ tokens get converted by Converter (the default converter uses DateTimeOffset to convert native $reql_type$ date and times because it's the closest BCL type that matches), you get a new exception. The new exception is an invalid conversion between your POCO DateTime and Converter's DateTimeOffset result).

These two issues bring up an implementation code smell. We have two distinct ser/deserialization steps:

1. The Java/C# driver's Converter.java/cs infrastructure. Primarily used for deserializing & extracting $reql_type$ into native objects.
2. Newtonsoft's JsonConverter infrastructure. Primarily used for serialization of POCO objects.

My gut felling here is we need to drop the Java driver's Converter class and move all $reql_type$ ser/deserialization into Newtonsoft JsonConverters. This way, we wont ever have to deal with how Date and Times are ser/deserialized and responsibility will be pushed on to Newtonsoft date & time configuration API. Additionally, we'd also have type information at final deserialization time about what the user is requesting, a DateTime or a DateTimeOffset. Currently, Converter class doesn't know if the final type is DateTime or DateTimeOffset (ie: the crux of the issue here); all it does is convert $reql_type$ into a DateTimeOffset.

Let me know your thoughts. I'll need a few days to work this out.

[fiLLLip]

My first though is to drop the custom deserializing and use the well proven ser/deserialization in Newtonsoft. I do not know how well it handles the $reql_type$ objects, but it seems to me that you might have a plan 👍
How is the deserialization done when using .get()? That seems to work just perfect, so maybe a reuse of that logic would work?

[bchavez]

@fiLLLip Thanks for your feedback. After giving it a lot of thought, I'm going to make a small architecture change so that the driver embraces the more robust Newtonsoft JsonConverter ser/deser infrastructure; and we avoid any/all/future edge case issues like this.

Even as it is now, Converter uses JSONPath to pluck out $reql_type$ to convert types which already differs from the Java Driver.

Overall, I think when we switch to Newtonsoft's JsonConverter, the ser/deser process will be more robust. Just need some time to make these changes.

[fiLLLip]

Totally agree. Newtonsofts's JsonConverter implementation is thoroughly tested, so no need to reinvent the wheel :)

[bchavez]

Hi @fiLLLip ,

alpha6 should be up shortly with the new Newtonsoft JsonConverters implementation.

Minor side effect:

Java driver FormatOptions like time_format: 'raw' don't make sense anymore. If you want the raw $reql_type$, just run a query with .run<JObject>() which bypasses the JsonConverters all together.

We're doing pretty good. Only down 2 unit test categories: https://ci.appveyor.com/project/bchavez/rethinkdb-driver/build/193/tests

Please close the issue once you've confirmed all is well! Thanks! 👍 Again, thanks for all your help.

[fiLLLip]

It looks like something strange is happening with the returned object is deserialized now.
When using alpha5 the the returned object looks like this (deserialized with RethinkDb.Driver and then serialized with ASP.NET):

{
  "TenantPermissions": null,
  "Email": "[email protected]",
  "Phone": "[phone]",
  "Picture": null,
  "Birthday": "2015-11-16T07:41:00.986+00:00",
  "DisplayName": "DemoUser",
  "FirstName": "Demo",
  "MiddleName": null,
  "LastName": "User",
  "Address": null,
  "id": "47e64a83-de3f-407d-8a64-343e06b8c285"
}

When trying out alpha6, the output looks like this (same procedure, nothing changed in code):

{
  "TenantPermissions": null,
  "Email": null,
  "Phone": null,
  "Picture": null,
  "Birthday": "2015-11-16T07:41:00.986+00:00",
  "DisplayName": null,
  "FirstName": null,
  "MiddleName": null,
  "LastName": null,
  "Address": null,
  "id": "1b49b6b9-f467-4364-9933-41c113be356a"
}

[bchavez]

Hey @fiLLLip , could you give alpha7 a try? Let me know how it goes. Thx

[fiLLLip]

I just downloaded alpha9, and it seems to work perfectly! Great work @bchavez 👍