JSON field ordering bug with Scala 3 about play-json

[jonaskoelker-jypo]

My semi-systematic experimentation suggests that the observed field order will be:

• all non-Option fields, in declaration order (relative to one another); then
• all Option fields, in declaration order (relative to one another)

By declaration order, I mean the same order as productElementNames.

So if either all fields are Options or zero fields are Options, the existing code will do the right thing.

Here's a blueprint for a workaround:

def workaround[T <: Product](w: OWrites[T]): OWrites[T] = {
  OWrites.transform(w) { case (product, misorderedObject) =>
    val pairs = misorderedObject.value
    JsObject(
      product.productElementNames
        .map(key => pairs.get(key).map(key -> _))
        .collect { case Some(pair) => pair }
        .toSeq
    )
  }
}

implicit val writes: OWrites[Foo] = workaround(Json.writes[Foo])

This can be simplified—you can use apply instead of get and then remove the collect—if all fields are always present in the output, e.g. if you use JsonConfiguration(optionHandlers = OptionHandlers.WritesNull) in a relatively simple context.

It is of note that the bug seems to still occur if all fields are present in the output—so it must depend on the optionality in the declaration, not in the output.

In version 3.0.3 I did some jump-to-source maneuvers in my application:

• OptionHandlers.WritesNull.writeHandler refers to
• JsPath.writeOptionWithNull which refers to
• Writes.at, which is really PathWrites.at, which refers to
• JsPath.createObj, at https://github.com/playframework/play-json/blob/3.0.3/play-json/shared/src/main/scala/play/api/libs/json/JsPath.scala#L133, which has a note saying "// TODO implement it correctly (doesn't merge )"

That sounds suspicious, but it's no smoking gun. I dug around some more, and found this:

• A list of fields is separated by whether they are optional or not:

  play-json/play-json/shared/src/main/scala-3/play/api/libs/json/JsMacroImpl.scala

  Line 670 in 51bd5d6

  .partition { case WritableField(_, _, t) => isOptionalType(t) }

• Data derived from this partition are concatenated here:

  play-json/play-json/shared/src/main/scala-3/play/api/libs/json/JsMacroImpl.scala

  Line 769 in 51bd5d6

  val fields = values ++ extra

This looks like the cause of exactly the pattern I have observed, i.e. a smoking gun.

Across my work I think I have only seen the bug in those projects that use Scala 3. This makes sense, the apparent smoking gun is in a scala3-specific folder, so one would expect the bug to occur if and only if using Scala 3. This matches what @levinson wrote in the issue-creating post: "TODO: Test fails when using Scala 3".

I hope this helps.

[gbarmashiahflir]

Happens here as well.
Having for example this class:

object MetricStatistics{
  implicit val format: Format[MetricStatistics] = Json.format[MetricStatistics]
}

case class MetricStatistics(from: Instant, to: Instant, min: Option[Double], max: Option[Double], average: Option[Double], count: Long, centroid: Option[GeoLocation])

In scala 2.13, generated json was according to field order, but after upgrading to Scala 3, this is the order of an example:

    {
        "count": 6,
        "min": 10.3,
        "to": "2010-06-24T09:55:00Z",
        "max": 100,
        "from": "1980-06-24T05:55:00Z",
        "centroid": {
            "lat": 0,
            "lon": 0
        },
        "average": 38.926
    }

It breaks a lot of tests we have :(
Is there a fix in the plan?