underscoreio / shapeless-guide

The Type Astronaut's Guide to Shapeless

The Type Astronaut's Guide to Shapeless

Copyright 2016 Dave Gurnell. Text and diagrams licensed CC-BY-SA 3.0. Code samples licensed Apache 2.0

Reading the Book

You have three options for grabbing the book:

• read/download the book at the Underscore web site;
• order a print copy from Underscore;
• a french translation is also available.

Related Material

Accompanying code samples can be found here:
https://github.com/underscoreio/shapeless-guide-code

Check the solutions branch for complete versions of each example.

Building the eBook

Install Docker and use go.sh to boot an instance with most of the right dependencies:

bash$ ./go.sh

Then run npm install to install the remaining dependencies:

npm install

And finally use sbt to build the book:

sbt pdf

Building a printable book

To build a black and white, print-ready version of the book, edit src/meta/pdf.yaml and set blackandwhiteprintable to true. Then run sbt pdf as above.

Contributing

Please raise an issue or submit a PR. If you submit a PR, make sure to add yourself to src/pages/intro/constributors.md!

Acknowledgements

Thanks to Miles Sabin, Richard Dallaway, Noel Welsh, Travis Brown, and our fellow space-farers on Github for their invaluable help and feedback.

shapeless-guide's Issues

Could not find tut-plugin 0.4.5-SNAPSHOT

I had to use tut-plugin 0.4.7 to build.

Method "createEncoder" in section 3.2.1

Hi :-)
Thanks for the great guide !
I'm currently reading it and have small remark : in section 3.2.1, the example encoder creation method is def createEncoder[A]

However, the section 3.1.2 explanation gives the method name instance as being the idiomatic. From my understanding, it would then be more idiomatic to name the 3.2.1 method instance too ? Plus, it'd make it even more clear imo (right now, I'm only 90% sure that I'm correct that createEncoder is actually the instance idiom).

WDYT ?

genericObjectEncoder does not work

The implicit function genericObjectEncoder does not work with Coproduct as descibed in the book.

The current definition is as followed:

implicit def genericObjectEncoder[A, H <: HList](
                                                    implicit
                                                    gen: LabelledGeneric.Aux[A, H],
                                                    hEncoder: Lazy[JsonObjectEncoder[H]]
                                                  ): JsonEncoder[A] =
    createObjectEncoder(value => hEncoder.value.encode(gen.to(value)))

To make this work, I have to remove <: HList.

I do not know why this is the case, but I would love to find out more about it.

Other users reported knownDirectSubclasses bug, but I do not see this error message.

Small grammatical correction

In the first chapter, Introduction, the sentence,

Both values are now the same type.

should probably read

Both values are now of the same type.

Generic `Shape` coproduct types reversed in section 2.4

Near the end of section 2.4 there is the following example:

  sealed trait Shape
  final case class Rectangle(width: Double, height: Double) extends Shape
  final case class Circle(radius: Double) extends Shape

  val gen = Generic[Shape]

gen is supposed to be:

Generic[Shape]{type Repr = Rectangle :+: Circle :+: CNil} = ...

But when I try it, it is (note that Rectangle and Circle are in reversed order):

Generic[Shape]{type Repr = Circle :+: Rectangle :+: CNil} = ...

I am going through the book to learn about shapeless, I don't know if this is intended behavior of shapeless and a bug in the documentation or the other way around. Or does this only happen on my system? If this is a bug in shapeless I'll open an issue there (unless someone beats me to it).

I'm using scala 2.12.3 and shapeless 2.3.2, I'm compiling with sbt.

getWrappedValue is not strict enough

On page 48, you say you want to "summon a Generic for a case class of exactly one field". Then on page 49 you present the final and complete implementation of getWrappedValue:

import shapeless.ops.hlist.IsHCons

def getWrappedValue[A, Repr <: HList, Head, Tail <: HList](in: A)(
  implicit
  gen: Generic.Aux[A, Repr],
  isHCons: IsHCons.Aux[Repr, Head, Tail]
): Head = gen.to(in).head

But that method will extract the first field out of any case class. I think you need to change it to:

import shapeless.ops.hlist.IsHCons

def getWrappedValue[A, Repr <: HList, Head, Tail <: HList](in: A)(
  implicit
  gen: Generic.Aux[A, Repr],
  isHCons: IsHCons.Aux[Repr, Head, Tail],
  ev: Tail =:= HNil
): Head = gen.to(in).head

Typo

Page 17:

IN the blue rectangle
"Other product types
It’s worth noting that Scala tuples are actually case classes, so Generic
works with them just fine:"

Noting without 'h'

Use "summoner" in section 8.3

In the random value generator case study a "summoner" method should be added and the created instance used to generate the random values in the following examples.

...
object Random { 
  def apply[A](implicit r: Random[A]): Random[A] = r
}
...
val rc = Random[Cell]
for(i <- 1 to 5) println(rc.get)

Translate this guide to Chinese

Hello everyone
My partner and me had translated this guide to Chinese.We want to make it open source to help more Chinese or others.So can we make some clean work then put it on this repo?Thanks!

Why "Witness"?

From @d6y:

Page 58

I feel there needs some kind of motivation or intuition for the use of the word Witness. One to ask @milessabin about?

5.3.2 Instances for concrete products H is constrained to HList in genericObjectEncoder

The function

import shapeless.LabelledGeneric
implicit def genericObjectEncoder[A, H <: HList]( // <--- this is where I think <: HList should be removed
  implicit
  generic: LabelledGeneric.Aux[A, H],
  hEncoder: Lazy[JsonObjectEncoder[H]]
): JsonEncoder[A] =
  createObjectEncoder { value =>
    hEncoder.value.encode(generic.to(value))
}

has the type constraint H <: HList which I think is too constrained. This works fine for the product IceCream example but in the coproduct Shape example, Scala can't find an implicit encoder since Coproduct is not an HList.

Please let me know if this is a problem or if there is a better way to do this. I just removed <: HList from my code on the H parameter and then everything worked as described.

If this is an issue I can fix it and file a pull request as well just let me know.

Thanks.

Not able to resolve tut-plugin dependency

Cloning, and executing go.sh + sbt pdf yields the following error for me:

[info] Resolving org.tpolecat#tut-plugin;0.4.5-SNAPSHOT ...
[warn] 	module not found: org.tpolecat#tut-plugin;0.4.5-SNAPSHOT
[warn] ==== typesafe-ivy-releases: tried
[warn]   https://repo.typesafe.com/typesafe/ivy-releases/org.tpolecat/tut-plugin/scala_2.10/sbt_0.13/0.4.5-SNAPSHOT/ivys/ivy.xml
[warn] ==== sbt-plugin-releases: tried
[warn]   https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/org.tpolecat/tut-plugin/scala_2.10/sbt_0.13/0.4.5-SNAPSHOT/ivys/ivy.xml
[warn] ==== local: tried
[warn]   /root/.ivy2/local/org.tpolecat/tut-plugin/scala_2.10/sbt_0.13/0.4.5-SNAPSHOT/ivys/ivy.xml
[warn] ==== public: tried
[warn]   https://repo1.maven.org/maven2/org/tpolecat/tut-plugin_2.10_0.13/0.4.5-SNAPSHOT/tut-plugin-0.4.5-SNAPSHOT.pom
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	::          UNRESOLVED DEPENDENCIES         ::
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	:: org.tpolecat#tut-plugin;0.4.5-SNAPSHOT: not found
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] 	Note: Some unresolved dependencies have extra attributes.  Check that these dependencies exist with the requested attributes.
[warn] 		org.tpolecat:tut-plugin:0.4.5-SNAPSHOT (scalaVersion=2.10, sbtVersion=0.13)
[warn]
[warn] 	Note: Unresolved dependencies path:
[warn] 		org.tpolecat:tut-plugin:0.4.5-SNAPSHOT (scalaVersion=2.10, sbtVersion=0.13) (/source/project/plugins.sbt#L1-2)
[warn] 		  +- default:source-build:0.1-SNAPSHOT (scalaVersion=2.10, sbtVersion=0.13)
sbt.ResolveException: unresolved dependency: org.tpolecat#tut-plugin;0.4.5-SNAPSHOT: not found
	at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:313)
	at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:191)
	at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:168)
	at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156)
	at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156)
	at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:133)
	at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:57)
	at sbt.IvySbt$$anon$4.call(Ivy.scala:65)
	at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
	at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
	at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
	at xsbt.boot.Using$.withResource(Using.scala:10)
	at xsbt.boot.Using$.apply(Using.scala:9)
	at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
	at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
	at xsbt.boot.Locks$.apply0(Locks.scala:31)
	at xsbt.boot.Locks$.apply(Locks.scala:28)
	at sbt.IvySbt.withDefaultLogger(Ivy.scala:65)
	at sbt.IvySbt.withIvy(Ivy.scala:128)
	at sbt.IvySbt.withIvy(Ivy.scala:125)
	at sbt.IvySbt$Module.withModule(Ivy.scala:156)
	at sbt.IvyActions$.updateEither(IvyActions.scala:168)
	at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1451)
	at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1447)
	at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$89.apply(Defaults.scala:1482)
	at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$89.apply(Defaults.scala:1480)
	at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:37)
	at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1485)
	at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1479)
	at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:60)
	at sbt.Classpaths$.cachedUpdate(Defaults.scala:1502)
	at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1429)
	at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1381)
	at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
	at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
	at sbt.std.Transform$$anon$4.work(System.scala:63)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
	at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
	at sbt.Execute.work(Execute.scala:237)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
	at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
[error] (*:update) sbt.ResolveException: unresolved dependency: org.tpolecat#tut-plugin;0.4.5-SNAPSHOT: not found

Running a current version of docker-for-mac (not sure if that matters)

5.4 "Deriving coproduct instances with LabelledGeneric" is blank

In the distributed PDF and HTML, the body of section 5.4 is missing. There's only a title.

I'm not sure what the problem was, but I've just tried building the book locally and it worked fine. So just doing a rebuild should be enough to fix it.

dist/pdf missing

dist/pdf went missing in 503b8bf

was that intentional?

Syntax highlighting issue on page 81, 6.4.1 'Selecting fields'

The ' in symbol seemed to mess up the highlighter.

Part 1 is missing?

I just downloaded the pdf version of the book and I see empty pages for Part 1 of the book?

Compiler flags for literal types?

Reported by @d6y:

Box on "Literal types in Scala", p. 56

There's a flag to enable this stuff, right? Worth saying that?

As of Lightbend Scala....., with the -wibble flag enabled, we now have direct syntax support...

LabelledGeneric code okay?

Hi,
Great work on this book. I am learning quite a lot of stuff out of it. I was able to follow the Generic bits pretty well. However, I possibly struggle to understand the convention used in the LabelledGeneric part, where it shows an example of JsonWriter. I believe the complete example from that chapter should look like:

import shapeless.labelled.FieldType
import shapeless.{HList, HNil, LabelledGeneric, Lazy, Witness}

object root {

  sealed trait JsonValue
  case class JsonObject(fields: List[(String, JsonValue)]) extends JsonValue
  case class JsonArray(items: List[JsonValue]) extends JsonValue
  case class JsonString(value: String) extends JsonValue
  case class JsonNumber(value: Double) extends JsonValue
  case class JsonBoolean(value: Boolean) extends JsonValue
  case object JsonNull extends JsonValue

  trait JsonEncoder[A] {
    def encode(value: A): JsonValue
  }
  object JsonEncoder {
    def apply[A](implicit enc: JsonEncoder[A]): JsonEncoder[A] = enc
  }

  def createEncoder[A](func: A => JsonValue): JsonEncoder[A] =
    new JsonEncoder[A] {
      def encode(value: A): JsonValue = func(value)
    }
  implicit val stringEncoder: JsonEncoder[String] =
    createEncoder(str => JsonString(str))
  implicit val doubleEncoder: JsonEncoder[Double] =
    createEncoder(num => JsonNumber(num))
  implicit val intEncoder: JsonEncoder[Int] =
    createEncoder(num => JsonNumber(num))
  implicit val booleanEncoder: JsonEncoder[Boolean] =
    createEncoder(bool => JsonBoolean(bool))

  implicit def listEncoder[A]
  (implicit enc: JsonEncoder[A]): JsonEncoder[List[A]] =
    createEncoder(list => JsonArray(list.map(enc.encode)))
  implicit def optionEncoder[A]
  (implicit enc: JsonEncoder[A]): JsonEncoder[Option[A]] =
    createEncoder(opt => opt.map(enc.encode).getOrElse(JsonNull))

  case class IceCream(name: String, numCherries: Int, inCone: Boolean)
  val iceCream = IceCream("Sundae", 1, false)

  val gen = LabelledGeneric[IceCream].to(iceCream)

  trait JsonObjectEncoder[A] extends JsonEncoder[A] {
    def encode(value: A): JsonObject
  }
  def createObjectEncoder[A](fn: A => JsonObject): JsonObjectEncoder[A] =
    new JsonObjectEncoder[A] {
      def encode(value: A): JsonObject =
        fn(value)
    }

  implicit val hnilEncoder: JsonObjectEncoder[HNil] =
    createObjectEncoder(hnil => JsonObject(Nil))

  implicit def hlistObjectEncoder[K <: Symbol, H, T <: HList](
                                                               implicit
                                                               witness: Witness.Aux[K],
                                                               hEncoder: Lazy[JsonEncoder[H]],
                                                               tEncoder: JsonObjectEncoder[T]
                                                             ): JsonObjectEncoder[FieldType[K, H] :: T] = {
    val fieldName: String = witness.value.name
    createObjectEncoder { hlist =>
      val head = hEncoder.value.encode(hlist.head)
      val tail = tEncoder.encode(hlist.tail)
      JsonObject((fieldName, head) :: tail.fields)
    } }

  implicit def genericObjectEncoder[A, H <: HList](
                                                    implicit
                                                    generic: LabelledGeneric.Aux[A, H],
                                                    hEncoder: Lazy[JsonObjectEncoder[H]]
                                                  ): JsonEncoder[A] =
    createObjectEncoder { value =>
      hEncoder.value.encode(generic.to(value))
    }

  JsonEncoder[IceCream].encode(iceCream)

}

However, when compiling this I get:

> compile
[info] Compiling 4 Scala sources to /Users/rad/dev/my/scala-playground/target/scala-2.12/classes...
[error] /Users/rad/dev/my/scala-playground/src/main/scala/pack/PackTest.scala:65: wrong number of type arguments for ::, should be 1
[error]                                                              ): JsonObjectEncoder[FieldType[K, H] :: T] = {
[error]                                                                                                   ^
[error] /Users/rad/dev/my/scala-playground/src/main/scala/pack/PackTest.scala:67: missing parameter type
[error]     createObjectEncoder { hlist =>
[error]                           ^
[error] /Users/rad/dev/my/scala-playground/src/main/scala/pack/PackTest.scala:82: could not find implicit value for parameter enc: pack.root.JsonEncoder[pack.root.IceCream]
[error]   JsonEncoder[IceCream].encode(iceCream)

This is with shapeless 2.3.2 and scala 2.12.1. I'm not sure if I'm not understanding to retype this into an ide or if the code is targeted at a different version?

Migrating case classes example fails to compile

Hello,

Thanks for this amazing book! While following the examples, I noticed that the example for migrating case classes, namely the third one (adding fields in generic migration) with an external dependency on Cats, failed with "could not find implicit value for parameter migration". This example is not in the accompanying code repo. I also tried different versions of Cats (latest 2.10.0 and version 2.3.0 as used in code repo) but with no fortune.

Best,
Zilu

getWrappedValue to enforces arity assertion

Section 4.3 ("Chaining dependent functions") introduces the getWrappedValue method by asserting: we wanted to summon a Generic for a case class of exactly one field.

However, the given solution does not enforce that assertion:

scala> case class Wrapper2(v: Long, v2: Long)
defined class Wrapper2

scala> foo.Foo.getWrappedValue(Wrapper2(42,12))
res2: Long = 42

Wouldn't it be better (and simpler) to use HNil instead of Tail <: HList, as that will enforce the assertion that the case class has only a single field, eg:

def getWrappedValue[A, Repr <: HList, Head](in: A)(
    implicit gen: Generic.Aux[A, Repr],
    isHCons: IsHCons.Aux[Repr, Head, HNil]
): Head = gen.to(in).head

Now:

scala> case class Wrapper(v: Long)
defined class Wrapper

scala> foo.Foo.getWrappedValue(Wrapper(42))
res1: Long = 42

scala> case class Wrapper2(v: Long, v2: Long)
defined class Wrapper2

scala> foo.Foo.getWrappedValue(Wrapper2(42,12))
<console>:14: error: could not find implicit value for parameter isHCons: shapeless.ops.hlist.IsHCons.Aux[Repr,Head,shapeless.HNil]
       foo.Foo.getWrappedValue(Wrapper2(42,12))

Huge thanks for this guide, learning a lot!

Section 5 needs update regarding `Lazy`?

As far as I recall, there was a relatively recent SIP that improved the implicit resolution process, making the usage of Lazy redundant in at least some cases (quick research suggests this was before 2.12.0).

It would be perhaps advantageous to take it into account as its one less concept to hold in ones head when reading the book, for a number of code snippets. For example, here's a (somewhat modified, mostly replaced explicit trait instance creation with SAM resolution) an example for the code necessary for 5.3 that compiles without using Lazy:

import shapeless.labelled.FieldType
import shapeless.{::, HList, HNil, LabelledGeneric, Witness}

sealed trait JsonValue

case class JsonObject(fields: List[(String, JsonValue)]) extends JsonValue
case class JsonArray(items: List[JsonValue]) extends JsonValue
case class JsonString(value: String) extends JsonValue
case class JsonNumber(value: Double) extends JsonValue
case class JsonBoolean(value: Boolean) extends JsonValue
case object JsonNull extends JsonValue

trait JsonEncoder[A] {
  def encode(value: A): JsonValue
}

object JsonEncoder {
  def apply[A](implicit enc: JsonEncoder[A]): JsonEncoder[A] = enc

  implicit val stringEncoder: JsonEncoder[String] = JsonString.apply
  implicit val doubleEncoder: JsonEncoder[Double] = JsonNumber.apply
  implicit val intEncoder: JsonEncoder[Int] = i => JsonNumber(i)
  implicit val booleanEncoder: JsonEncoder[Boolean] = JsonBoolean.apply

  implicit def listEncoder[A](
      implicit enc: JsonEncoder[A]): JsonEncoder[List[A]] =
    l => JsonArray(l.map(enc.encode))

  implicit def optionEncoder[A](
      implicit enc: JsonEncoder[A]): JsonEncoder[Option[A]] =
    _.map(enc.encode).getOrElse(JsonNull)
}

trait JsonObjectEncoder[A] extends JsonEncoder[A] {
  def encode(value: A): JsonObject
}

object JsonObjectEncoder {

  implicit val hnilEncoder: JsonObjectEncoder[HNil] = _ => JsonObject(Nil)

  implicit def hlistObjectEncoder[K <: Symbol, H, T <: HList](
      implicit fieldKey: Witness.Aux[K],
      hEncoder: JsonEncoder[H], //Lazy was here
      tEncoder: JsonObjectEncoder[T])
    : JsonObjectEncoder[FieldType[K, H] :: T] = {
    val fieldName = fieldKey.value.name

    {
      case (head :: tail) =>
        JsonObject(
          (fieldName -> hEncoder.encode(head)) :: tEncoder.encode(tail).fields)
    }
  }

  implicit def genericObjectEncoder[A, H](
      implicit generic: LabelledGeneric.Aux[A, H],
      hListEncoder: JsonObjectEncoder[H]): JsonEncoder[A] = //and here
    (o: A) => hListEncoder.encode(generic.to(o))

}

case class IceCream(name: String, numCherries: Int, inCone: Boolean)

object test extends App {

  import JsonObjectEncoder._

  val iceCream = IceCream("Sundae", 1, inCone = false)

  println(JsonEncoder[IceCream].encode(iceCream))

}

Lazy, section 3.4.2

After adding following code to example(cvs.scala, object Main) I've got java.lang.StackOverflowError

  sealed trait Tree[A]
  case class Branch[A](left: Tree[A], right: Tree[A]) extends Tree[A]
  case class Leaf[A](value: A) extends Tree[A]
  val tree1 = Branch(
    Branch(Leaf(1), Leaf(2)),
    Branch(Leaf(3), Leaf(4))
  )
  val tree2 = Branch(
    Branch(Leaf(21), Leaf(22)),
    Branch(Leaf(23), Leaf(24))
  )

  println("Trees as CSV:\n" + writeCsv(List(tree1, tree2)))

var x = 42.narrow doesn't compile in scala 2.12

This line just blows up the compiler in scala 2.12.x. In https://github.com/underscoreio/shapeless-guide/blob/master/src/pages/labelled-generic/literal-types.md . Issue here: milessabin/shapeless#674 . Perhaps just a note not to try it with scala 2.12.x?

Download/buy page should be https

Chrome (rightfully) doesn't like me entering credit-card details onto an http:// page like http://underscore.io/books/shapeless-guide/:

It's wrong to think that subsequent https reqs from an http page can be considered secure; no guarantees can be made about the authenticity of the JS running on an http page.