pchiusano / scodec

Scala combinator library for working with binary data

scodec

Scala combinator library for working with binary data.

Design Constraints

This library focuses on contract-first and pure functional encoding and decoding of binary data. The following design constraints are considered:

• Binary structure should mirror protocol definitions and be self-evident under casual reading
• Mapping binary structures to types should be statically verified
• Encoding and decoding should be purely functional
• Failures in encoding and decoding should provide descriptive errors
• Compiler plugin should not be used

As a result, the library is implemented as a combinator based DSL. Performance is considered but yields to the above design constraints.

Acknowledgements

The library uses Scalaz and Shapeless and is heavily influenced by scala.util.parsing.combinator.

Introduction

The primary abstraction is a Codec[A], which supports encoding a value of type A to a BitVector and decoding a BitVector to a value of type A.

The codecs package provides a number of predefined codecs and combinators.

    import scodec._
    import scodec.bits._
    import codecs._
    import scalaz.\/

    // Create a codec for an 8-bit unsigned int followed by an 8-bit unsigned int followed by a 16-bit unsigned int
    val firstCodec = (uint8 ~ uint8 ~ uint16)

    // Decode a bit vector using that codec
    val result: String \/ (Int ~ Int ~ Int) = Codec.decode(firstCodec, BitVector(0x10, 0x2a, 0x03, 0xff))

    // Sum the result
    val add3 = (_: Int) + (_: Int) + (_: Int)
    val sum: String \/ Int = result map add3

Automatic case class binding is supported via Shapeless HLists:

    import shapeless._

    case class Point(x: Int, y: Int, z: Int)
    implicit val pointIso = Iso.hlist(Point.apply _, Point.unapply _)

    val pointCodec = (int8 :: int8 :: int8).as[Point]

    val encoded: String \/ BitVector = pointCodec.encode(Point(-5, 10, 1))
    // \/-(BitVector(24 bits, 0xfb0a01))

    val decoded: String \/ Point = Codec.decode(pointCodec, BitVector(0xfb, 0x0a, 0x01))
    // \/-(Point(-5,10,1))

Codecs can also be implicitly resolved, resulting in usage like:

    // Assuming Codec[Point] is in implicit scope

    val encoded = Codec.encode(Point(-5, 10, 1))
    // \/-(BitVector(24 bits, 0xfb0a01))

    val decoded = Codec.decode[Point](BitVector(0xfb, 0x0a, 0x01))
    // \/-(Point(-5,10,1))

Combinators

New codecs can be created by either implementing the Codec trait or by passing an encoder function and decoder function to the Codec apply method. Typically, new codecs are created by applying one or more combinators to existing codecs.

There are a number of built in combinators:

• "name" | a - creates a Codec[A] that prefixes any error messages with the specified name.
• Tuple Support
  • a ~ b - creates a Codec[(A, B)] that first decodes a and then b.
  • a ~> b - creates a Codec[B] that decodes first with a and then with b and throws away the decoded a. For encoding, the zero value of type As monoid is encoded.
  • a <~ b - creates a Codec[A] that decodes first with a and then with b and throws away the decoded b. For encoding, the zero value of type Bs monoid is encoded.
  • a >>~ f - creates a Codec[(A, B)] that decodes first with a and then with the codec returned from f(decodedA). The non-operator version of this is flatZip(f: A => Codec[B]): Codec[(A, B)].
• HList Support
  • a :: b - creates a Codec[A :: B :: HNil] or if B is an HList, a Codec[A :: B].
  • a :~>: b - creates a Codec[B] (if B is an HList) that decodes and throws away the decoded a and encodes as zero value.
  • a >>:~ f - creates a Codec[A :: B] that decodes first with a and then with the HList codec returned from f(decodedA). The non-operator version of this is flatPrepend[L <: HList](f: A => Codec[L]): Codec[A :: L].
  • l :+ a - creates a codec that decodes first with the HList codec l and then with the codec a. This is the codec equivalent of HList append.
  • k ::: l - creates a codec that decodes first with the HList codec k and then with the HList codec l. This is the codec equivalent of HList concat.
  • a.hlist - creates a Codec[A :: HNil].
• Sizing
  • fixedSizeBits(size, a) and fixedSizeBytes(size, a) - creates a Codec[A] that always encodes/decodes size bits/bytes.
  • variableSizeBits(sizeCodec, a) and variableSizeBytes(size, a) - creates a Codec[A] that encodes the size of the encoded A followed by the encoded A.
• conditional(boolean, a) - creates a Codec[Option[A]] that skips encoding/decoding if the specified boolean is false
• repeated(a) - creates a Codec[IndexedSeq[A]]
• Type Conversions
  • a.xmap(f, g) - creates a Codec[B] given bidirectional functions f: A => B and g: B => A.
  • a.as[B] - creates a Codec[B] if a shapeless.Iso[B, A] is in implicit scope

Examples

There are various examples in the test directory, including codecs for:

• UDP Datagrams
• MPEG Packets
• libpcap Files

Getting Binaries

This library works with Scala 2.10.*.

ScalaDoc for the latest version is available here: http://scodec.github.io/scodec/latest/api.

Releases

The latest released version is 1.0.0-M2. This release is mostly API stable.

For SBT users:

libraryDependencies += "org.typelevel" %% "scodec-core" % "1.0.0-M2"

For Maven users:

<dependencies>
  <dependency>
    <groupId>org.typelevel</groupId>
    <artifactId>scodec-core_2.10</artifactId>
    <version>1.0.0-M2</version>
  </dependency>
</dependencies>

Snapshots

Snapshot builds of the master branch are available on Sonatype's OSS hosting at https://oss.sonatype.org/content/repositories/snapshots/.

For SBT users:

resolvers += "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots/"

libraryDependencies += "org.typelevel" %% "scodec-core" % "1.0.0-SNAPSHOT"

For Maven users:

<repositories>
  <repository>
    <id>sonatype-oss-snapshots</id>
    <name>Sonatype OSS Snapshots</name>
    <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
  </repository>
</repositories>

<dependencies>
  <dependency>
    <groupId>org.typelevel</groupId>
    <artifactId>scodec-core_2.10</artifactId>
    <version>1.0.0-SNAPSHOT</version>
  </dependency>
</dependencies>

Building

This project uses sbt. To build, run sbt publish-local.