circe / circe-derivation

Fast type class instance derivation for Circe

circe

circe is a JSON library for Scala (and Scala.js).

Please see the guide for more information about why circe exists and how to use it.

Community

Adopters

Are you using circe? Please consider opening a pull request to list your organization here:

• Abacus
• Anduin Transactions
• Apollo Agriculture
• AutoScout24
• Babylon Health
• Banno inside of Jack Henry
• BBC
• Broad Institute
• Chartboost (sending hundreds of thousands of messages per second on our Ad Exchange)
• CiBO Technologies (using circe to (de)serialize data in support of a sustainable revolution in agriculture)
• ClearScore
• Codacy
• Colisweb
• Connio (creating and managing digital twins with Circe and Akka)
• Coya
• Criteo (collecting 200.000 events per second from our banners)
• Dassault Systèmes
• Datalogue
• Dreamlines
• DriveTribe
• Earnest
• Elastic
• Emmy Sharing
• FOLIO
• GLNGN Server
• Gympass
• Gutefrage
• Hello Soda
• HERE Technologies
• Hireproof
• HolidayCheck
• ImmobilienScout24
• Immozentral
• Indix
• ITV
• Kinoplan
• Latitude Financial Services
• MatchesFashion
• Merit
• Metacommerce
• Mobile GmbH
• MomentFeed
• New Work
• Nike
• Ocado Technology
• On Air Entertainment
• Onfocus
• Opt Technologies
• OVO Energy
• Panaseer
• Permutive
• Prezi
• Project September (using circe to exchange and store data within the platform and serve data using GraphQL with Sangria)
• Raster Foundry
• Ravel Law (using circe to (de)serialize data for search, analytics, and visualization of tens of millions of legal opinions)
• REA Group - realestate.com.au
• Reonomy
• Resilient plc
• Sky
• Snowplow Analytics
• SoundCloud (transforming 200,000,000 JSON events every hour in MapReduce ETLs)
• Spotify (using circe for JSON IO in Scio)
• SpotX
• Stripe
• Stylight
• TabMo (parsing more than 100k events per second with Akka Stream and Spark)
• Tinkoff
• The Guardian
• Threat Stack
• Tranzzo
• Twilio (sending many, many millions of messages a day with Circe and Akka)
• Uberall
• VEACT
• WeWork
• Whisk
• Wide Angle Analytics
• Zalando
• Zendesk

Other circe organization projects

Please get in touch on Gitter if you have a circe-related project that you'd like to discuss hosting under the circe organization on GitHub.

• circe-benchmarks: Benchmarks for comparing the performance of circe and other JSON libraries for the JVM.
• circe-config: A library for translating between HOCON, Java properties, and JSON documents.
• circe-derivation: Experimental generic derivation with improved compile times.
• circe-fs2: A library that provides streaming JSON parsing and decoding built on fs2 and Jawn.
• circe-iteratee: A library that provides streaming JSON parsing and decoding built on iteratee.io and Jawn.
• circe-jackson: A library that provides Jackson-supported parsing and printing for circe.
• circe-spray: A library that provides JSON marshallers and unmarshallers for Spray using circe.
• circe-yaml: A library that uses SnakeYAML to support parsing YAML 1.1 into circe's Json.

Related projects

The following open source projects are either built on circe or provide circe support:

• Actor Messenger: A platform for instant messaging.
• akka-http-json: A library that supports using circe for JSON marshalling and unmarshalling in Akka HTTP.
• akka-stream-json: A library that provides JSON support for stream based applications using Jawn as a parser with a convenience example for circe.
• Argus: Generates models and circe encoders and decoders from JSON schemas.
• Blackdoor JOSE: circe JSON support for blackdoor JOSE and JWT.
• borer: Allows circe encoders/decoders to be reused for CBOR (de)serialization.
• circe-debezium: Circe codecs for Debezium payload types
• circe-geojson: Circe support for GeoJSON (RFC 7946)
• circe-kafka: Implicit conversion of Encoder and Decoder into Kafka Serializer/Deserializer/Serde
• cornichon: A DSL for JSON API testing.
• Cosmos: An API for DCOS services that uses circe.
• crjdt: A conflict-free replicated JSON datatype in Scala.
• diffson: A Scala diff / patch library for JSON.
• elastic4s: A Scala client for Elasticsearch with circe support.
• Enumeratum: Enumerations for Scala with circe integration.
• Featherbed: A REST client library with circe support.
• Finch: A library for building web services with circe support.
• fintrospect: HTTP contracts for Finagle with circe support.
• fluflu: A Fluentd logger.
• Github4s: A GitHub API wrapper written in Scala.
• content-api-models: The Guardian's Content API Thrift models.
• http4s: A purely functional HTTP library for client and server applications.
• IdeaLingua: Staged Interface Definition and Data Modeling Language & RPC system currently targeting Scala, Go, C# and TypeScript. Scala codegen generates models and JSON codecs using circe.
• Iglu Schema Repository: A JSON Schema repository with circe support.
• jsactor: An actor library for Scala.js with circe support.
• jsoniter-scala-circe: A booster for faster parsing/printing to/from circe AST and decoding/encoding of java.time._ and BigInt types.
• jwt-circe: A JSON Web Token implementation with circe support.
• kadai-log: A logging library with circe support.
• msgpack4z-circe: A MessagePack implementation with circe support.
• ohNoMyCirce: Friendly compile error messages for shapeless's Generic, circe's Encoder & Decoder and slick's case class mapping.
• play-circe: circe support for Play!.
• pulsar4s: A Scala client for Apache-Pulsar with circe support.
• Rapture: Support for using circe's parsing and AST in Rapture JSON.
• roc: A PostgreSQL client built on Finagle.
• sangria-circe: circe marshalling for Sangria, a GraphQL implementation.
• scalist: A Todoist API client.
• scala-jsonapi: Scala support library for integrating the JSON API spec with Spray, Play! or Circe
• scala-json-rpc: JSON-RPC 2.0 library for Scala and Scala.js
• scalatest-json-circe: Scalatest matchers for Json with appropriate equality and descriptive error messages.
• Scio: A Scala API for Apache Beam and Google Cloud Dataflow, uses circe for JSON IO
• seals: Tools for schema evolution and language-integrated schemata (derives circe encoders and decoders).
• shaclex: RDF validation using SHACL or ShEx.
• Slick-pg: Slick extensions for PostgreSQL.
• sttp: Scala HTTP client.
• Synapses: A lightweight Neural Network library, for js, jvm and .net.
• telepooz: A Scala wrapper for the Telegram Bot API built on circe.
• Zenith: Functional HTTP library built on circe.

Examples

The following projects provide examples, templates, or benchmarks that include circe:

• https://github.com/alanphillips78/akka-http-microservice-blueprint
• https://github.com/bneil/fcs_boilerplate
• https://github.com/gvolpe/simple-http4s-api
• https://github.com/vitorsvieira/akka-http-circe-json-template
• https://github.com/stephennancekivell/some-jmh-json-benchmarks-circe-jackson
• https://github.com/pauljamescleary/scala-pet-store

Contributors and participation

circe is a fork of Argonaut, and if you find it at all useful, you should thank Mark Hibberd, Tony Morris, Kenji Yoshida, and the rest of the Argonaut contributors.

circe is currently maintained by Darren Gibson and Erlend Hamnaberg. After the 1.0 release, all pull requests will require two sign-offs by a maintainer to be merged.

The circe project is a typelevel affiliate project, and follow the Typelevel Code of Conduct

Please see the contributors' guide for details on how to submit a pull request.

License

circe is licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

circe-derivation's Issues

Derivation fails if there's an .apply() method with different field names

Minimal example:

import io.circe._
import io.circe.derivation._

case class Abc(x: String)

object Abc {
  implicit val dec: Decoder[Abc] = deriveDecoder[Abc]
  implicit val enc: ObjectEncoder[Abc] = deriveEncoder[Abc]

  def apply(otherfield: Int): Abc = new Abc(otherfield.toString)
}

Error:(7, 50) No method otherfield in A$A111.this.Abc (this is probably because a constructor parameter isn't a val)
  implicit val dec: Decoder[Abc] = deriveDecoder[Abc]
                                                ^
Error:(8, 56) No method otherfield in A$A111.this.Abc (this is probably because a constructor parameter isn't a val)
  implicit val enc: ObjectEncoder[Abc] = deriveEncoder[Abc]
                                                      ^

But If it's def apply(x: Int): Abc = new Abc(x) then it works

Clean up published metadata

I just noticed that the published POMs for 0.8.0-M1 have kind of crufty names (derivationBase) and include a test-scoped dependency on e.g. examplesbase_2.11. This should be pretty straightforward clean-up.

Would it be possible to publish this library for circe 0.14?

I see the latest build is already on 0.14 however the badge shows the latest released is for 0.13.

cc @travisbrown

Many thanks!

Renaming with list of renames

Sometimes the derived decoder already does everything necessary apart from a couple of fields which have the wrong name in the json. It is very error prone to implement the custom decoder just for those fields. Something akin to this should exist:

  /** Creates a decoder from `initial` that performs the `renames` to the JsonObject keys before decoding. */
  def withRenames[T](initial: Decoder[T])(renames: (String, String)*): Decoder[T] = initial.prepare {
    _.withFocus {
      _.mapObject { obj =>
        val renamesMap = renames.toMap
        val newMap = obj.toMap.map { case (key, value) =>
          renamesMap.getOrElse(key, key) -> value
        }
        JsonObject.fromMap(newMap)
      }
    }
  }

Example of an use:

object Network {
  implicit val decoder: Decoder[Network] = withRenames(deriveDecoder(renaming.snakeCase))(
    "provider:network_type" -> "provider_network_type",
    "provider:physical_network" -> "provider_physical_network",
    "provider:segmentation_id" -> "provider_segmentation_id",
    "router:external" -> "router_external",
  )
}
case class Network(...)

Defaults for null values

As I was trying to replace circe-generic with circe derivation in a project, I recognized that the behaviour regarding default values was different. When a null value is present the case class will not be constructed with the default value

@ @JsonCodec case class MyData(strings: Seq[String] = Seq.empty)
defined class MyData

@ import io.circe.parser._
import io.circe.parser._

@ decode[MyData]("""{"strings": null}""")
res14: Either[io.circe.Error, MyData] = Left(DecodingFailure(C[A], List(DownField(strings))))

@ decode[MyData]("""{}""")
res15: Either[io.circe.Error, MyData] = Right(MyData(List()))

I expectd MyData to be constructed with an empty string in the first case.

Support sealed trait hierarchies

I'm not 100% sure this should be in scope for this project, for a couple of reasons. First, there are multiple ways to encode sealed trait hierarchies (with a type field, with an object wrapper, without any discriminator, etc.), and the one with no annoying corner cases (object wrappers) generally doesn't seem to be the one people want.

The second reason is that SI-7046 is still a reality for a lot of people, and for everyone to some degree, and there will be cases where derivation for sealed trait hierarchies won't work because of it. I'd like to avoid complications like that in this project.

The primary goal of this project (as opposed to e.g. circe-generic, which is a more general attempt to provide generic derivation) is to help users who have a lot of big case classes and want instances that stay in sync with those definitions without horribly slow compile times.

In general there are likely to be more members in a case class than constructors in a sealed trait hierarchy, and the members are likely to change more often, so focusing on really solid, fast derivation for case classes seems reasonable to me.

For now, then, the recommendation is to use one of the following approaches:

• Derive your case class instances with this library and write your sealed trait instances by hand.
• Derive your case class instances with this library and use circe-generic for your sealed trait instances.

For example, the first approach might look like this:

import io.circe.{ Decoder, ObjectEncoder }, io.circe.syntax._
import io.circe.derivation.{ deriveDecoder, deriveEncoder }

sealed trait Visitor {
  def id: String
}
case class User(id: String, displayName: String) extends Visitor
case class Anon(id: String) extends Visitor

object Visitor {
  implicit val encodeUser: ObjectEncoder[User] = deriveEncoder
  implicit val decodeUser: Decoder[User] = deriveDecoder
  implicit val encodeAnon: ObjectEncoder[Anon] = deriveEncoder
  implicit val decodeAnon: Decoder[Anon] = deriveDecoder

  implicit val encodeVisitor: ObjectEncoder[Visitor] = ObjectEncoder.instance {
    case u @ User(_, _) => u.asJsonObject.add("type", "user".asJson)
    case a @ Anon(_) => a.asJsonObject.add("type", "anon".asJson)
  }

  implicit val decodeVisitor: Decoder[Visitor] = for {
    visitorType <- Decoder[String].prepare(_.downField("type"))
    value <- visitorType match {
      case "user" => Decoder[User]
      case "anon" => Decoder[Anon]
      case other => Decoder.failedWithMessage(s"invalid type: $other")
    }
  } yield value
}

While the second would replace the Visitor instances with the following:

import io.circe.generic.semiauto.{
  deriveDecoder => genericallyDeriveDecoder,
  deriveEncoder => genericallyDeriveEncoder
}

implicit val encodeVisitor: ObjectEncoder[Visitor] = genericallyDeriveEncoder
implicit val decodeVisitor: Decoder[Visitor] = genericallyDeriveDecoder

This limitation could be dropped in future versions of circe-derivation. If you have an opinion about whether sealed trait hierarchies should be supported here, please vote 👍 for yes, they should, or 👎 for no.

ADT class names are lower cased, which is not consistent with other derivation module

Fix code coverage reporting

The 22% currently reported in the README is way off—when I run it locally I get something like 86%.

ADT derivation succeeds on decoder but fails on encoder

The following example ADT fails to derive the Encoder for Foo but can generate the Decoder. Changing this to use generic.semiauto works. This uses the latest 0.13.0-M4 release.

import io.circe._
import io.circe.derivation._

sealed trait Foo {
  def id: String
}
object Foo {
  // could not find implicit value for parameter instance: io.circe.Encoder.AsObject[Bar]
  implicit val encoder: Encoder[Foo] = deriveEncoder[Foo] 
  implicit val decoder: Decoder[Foo] = deriveDecoder[Foo]
}

final case class Bar (
    id: String
) extends Foo 
object Bar {
  implicit val encoder: Encoder[Bar] =
    deriveEncoder[Bar]
  implicit val decoder: Decoder[Bar] =
    deriveDecoder[Bar]
}

final case class Baz(
    id: String
) extends Foo 

object Baz {
  implicit val encoder: Encoder[Baz] = deriveEncoder[Baz]
  implicit val decoder: Decoder[Baz] = deriveDecoder[Baz]
}

Code spliced for handling default parameters mentions companion object, potentially doesn't have to?

Derivation in this code:

final case class RandomPayload(field1: String, randomArray: Set[Int] = List.fill(10)(Random.nextInt(10)).toSet)

object RandomPayload extends WithCirce[RandomPayload]

Will produce the following error:

super constructor cannot be passed a self reference unless parameter is declared by-name
[error]   object RandomPayload extends WithCirce[RandomPayload]
[error]                                ^

Where WithCirce is defined as:

import io.circe.{Decoder, Encoder, derivation}

import scala.language.experimental.macros
import scala.reflect.macros.blackbox

/**
 * Provides circe codecs for case classes and sealed traits
 * {{{
 *   final case class Abc(a: String, b: String, c: String)
 *
 *   object Abc extends WithCirce[Abc]
 * }}}
 */
abstract class WithCirce[A](implicit encoder: DerivationDerivedEncoder[A], decoder: DerivationDerivedDecoder[A]) {
  implicit val enc: Encoder.AsObject[A] = encoder.value
  implicit val dec: Decoder[A] = decoder.value
}

final class MaterializeDerivationMacros(override val c: blackbox.Context) extends derivation.DerivationMacros(c) {
  import c.universe._

  def materializeEncoderImpl[A: c.WeakTypeTag]: c.Expr[DerivationDerivedEncoder[A]] =
    c.Expr[DerivationDerivedEncoder[A]] {
      q"{ ${symbolOf[DerivationDerivedEncoder.type].asClass.module}.apply(${materializeEncoder[A]}) }"
    }

  def materializeDecoderImpl[A: c.WeakTypeTag]: c.Expr[DerivationDerivedDecoder[A]] =
    c.Expr[DerivationDerivedDecoder[A]] {
      q"{ ${symbolOf[DerivationDerivedDecoder.type].asClass.module}.apply(${materializeDecoder[A]}) }"
    }
}

final case class DerivationDerivedEncoder[A](value: Encoder.AsObject[A])
object DerivationDerivedEncoder {
  implicit def materialize[A]: DerivationDerivedEncoder[A] = macro MaterializeDerivationMacros.materializeEncoderImpl[A]
}

final case class DerivationDerivedDecoder[A](value: Decoder[A])
object DerivationDerivedDecoder {
  implicit def materialize[A]: DerivationDerivedDecoder[A] = macro MaterializeDerivationMacros.materializeDecoderImpl[A]
}

If the default parameter is removed, the error disappears:

final case class RandomPayload(field1: String, randomArray: Set[Int])
object RandomPayload extends WithCirce[RandomPayload]

This didn't happen in older versions of circe-derivation and indicates that the code for default value management is splicing additional references to the companion object – which I'm not sure are necessary for the use case, although I may be wrong.

/cc @aparo

Support case object

Circe generic supports it.

Currently if I want to derive codecs for an ADT, say Pet I'll resort to this:

        @JsonCodec
        sealed trait Pet

        case object Bunny extends Pet {
          implicit val e: Encoder.AsObject[Bunny.type] = deriveEncoder
          implicit val d: Decoder[Bunny.type] = deriveDecoder
        }

Would be nice to have

        @JsonCodec
        sealed trait Pet

        @JsonCodec
        case object Bunny extends Pet

Is there a non milestone version of 0.8.0?

Right now we're using circe-generic and core at 0.8.0, but circe-derivation on 0.8.0-M2. This mixture helped us with our compile time quite a bit.

Just wondering if 0.8.0 will be published, so we can sync our versions together in our build. It looks like this project already has a 0.9.0-M1 branch.

Scala 3 support?

From what I can see, this library isn't published for 3, are there any plans to support it?

Nested sealed traits are not ignored as in circe-generic

Example:

object example extends App {
  import io.circe.syntax._
  import io.circe.{Codec, derivation}

  sealed trait Outer
  sealed trait Nested1 extends Outer
  sealed trait Nested2 extends Outer
  final case class Branch(i: Int) extends Nested1 with Nested2
  object Branch { implicit val codec: Codec.AsObject[Branch] = derivation.deriveCodec }

  // derivation
  locally {
    implicit val derivationCodec: Codec.AsObject[Outer] = {
      // necessary due to errors:
      // Error:(18, 27) could not find implicit value for parameter instance: io.circe.Decoder[izumi.idealingua.runtime.circe.example.Nested1]
      //      io.circe.derivation.deriveCodec
      implicit val nested1: Codec.AsObject[Nested1] = io.circe.derivation.deriveCodec
      implicit val nested2: Codec.AsObject[Nested2] = io.circe.derivation.deriveCodec
      io.circe.derivation.deriveCodec
    }

    println(s"derivation: ${(Branch(1): Outer).asJson.noSpaces}")
  }

  // generic
  locally {
    implicit val genericCodec: Codec.AsObject[Outer] = {
      // these are ignored by deriveCodec
      implicit val nested1: Codec.AsObject[Nested1] = io.circe.derivation.deriveCodec
      implicit val nested2: Codec.AsObject[Nested2] = io.circe.derivation.deriveCodec
      io.circe.generic.semiauto.deriveCodec
    }

    println(s"generic: ${(Branch(1): Outer).asJson.noSpaces}")
  }
}

Output:

derivation: {"Nested1":{"Branch":{"i":1}}}
generic: {"Branch":{"i":1}}

i.e. the format becomes incompatible in presence of nested sealed traits (and instances for them)

0.14.0 Compliant Release

Looking to get a circe-derivation release that is compliant w/ 0.14.0

useDefaults parameter is ignored in deriveDecoder

import io.circe.Json
case class Foo(value: Json = Json.obj())
io.circe.derivation.deriveDecoder[Foo]((i: String) => i, false, None).decodeJson(Json.obj())

results Left(DecodingFailure(Attempt to decode value on failed cursor, List(DownField(value)))) in 0.13.0-M1 and before,
results Right(Foo(JObject(object[]))) in 0.13.0-M2 and after.

Support recursive case classes

Support generic case classes

This should be relatively straightforward.

Derivation for generic case classes fails with additional context bounds

Hi!
Found this issue while trying out circe-derivation 0.8.0-M2:

import io.circe.{Decoder, Encoder}
import io.circe.derivation._

trait Typeclass[T]

// notice additional context bound here
case class Test1[T: Typeclass](v: T)
case class Test2[T](v: T)

object CirceDerivation {
  // we have to pass the bound here.
  implicit def encT1[T: Typeclass : Encoder]: Encoder[Test1[T]] = deriveEncoder[Test1[T]]
  implicit def decT1[T: Typeclass : Decoder]: Decoder[Test1[T]] = deriveDecoder[Test1[T]]

  implicit def encT2[T : Encoder]: Encoder[Test2[T]] = deriveEncoder[Test2[T]]
  implicit def decT2[T : Decoder]: Decoder[Test2[T]] = deriveDecoder[Test2[T]]
}

For Test2 all good.
For Test1 derivation fails with this (for both encoder and decoder):

exception during macro expansion:
[error] scala.ScalaReflectionException: value evidence$1 is not a method
[error] 	at scala.reflect.api.Symbols$SymbolApi.asMethod(Symbols.scala:228)
[error] 	at scala.reflect.api.Symbols$SymbolApi.asMethod$(Symbols.scala:222)
[error] 	at scala.reflect.internal.Symbols$SymbolContextApiImpl.asMethod(Symbols.scala:88)
[error] 	at io.circe.derivation.DerivationMacros$$anonfun$membersFromPrimaryConstr$1.$anonfun$applyOrElse$1(DerivationMacros.scala:45)
[error] 	at scala.collection.immutable.List.map(List.scala:287)
[error] 	at io.circe.derivation.DerivationMacros$$anonfun$membersFromPrimaryConstr$1.applyOrElse(DerivationMacros.scala:44)
[error] 	at io.circe.derivation.DerivationMacros$$anonfun$membersFromPrimaryConstr$1.applyOrElse(DerivationMacros.scala:43)
[error] 	at scala.collection.TraversableOnce.collectFirst(TraversableOnce.scala:145)
[error] 	at scala.collection.TraversableOnce.collectFirst$(TraversableOnce.scala:132)
[error] 	at scala.reflect.internal.Scopes$Scope.collectFirst(Scopes.scala:51)
[error] 	at io.circe.derivation.DerivationMacros.membersFromPrimaryConstr(DerivationMacros.scala:43)
[error] 	at io.circe.derivation.DerivationMacros.productRepr(DerivationMacros.scala:52)
[error] 	at io.circe.derivation.DerivationMacros.materializeEncoderImpl(DerivationMacros.scala:221)
[error] 	at io.circe.derivation.DerivationMacros.materializeEncoder(DerivationMacros.scala:87)
[error]   implicit def encT1[T: Typeclass : Encoder]: Encoder[Test1[T]] = deriveEncoder[Test1[T]]

Support constructor name transformations

We support transformMemberNames, for changing the names of case class members in their JSON representation, but don't yet support transformConstructorNames for ADTs. The behavior should be the same as that provided by circe-generic-extras.

Upgrade to circe 0.9.0

add documentation

documentation covering edge cases, default behaviours and limitations would be nice.

would prefer adding it to main circe docs than a separate microsite.
if no objections, i can start making a PR for it.

Support transforming member names

Add 2.13.0-M4 build

Support default values for case class members

There are (at least) two questions about how this should work:

1. Should it always be on, or optional?
2. Should it be possible for defaults to be omitted during encoding? If yes, should that be optional or just the way things are? Also if yes, should it rely on universal equality or cats.kernel.Eq?

JsonKey

Circe includes an annotation @JsonKey to rename fields:
https://circe.github.io/circe/codec.html#custom-key-mappings-via-annotations

It would be nice to have that annotation supported in circe-derivation as well.

relationship to circe-generic's semiauto derivation?

Is this project related to circe-generic's semiauto derivation? They seem to do the same thing and looks like they have the same author(s). The readme explains that this project is different from circe-generic's fully automatic derivation, but doesn't mention circe-generic's semiauto. Would be helpful to explain the relationship there :)