twitter / scrooge

A Thrift parser/generator

Scrooge

Scrooge is a thrift code generator written in Scala, which currently generates code for Scala, Java, Cocoa, Android and Lua.

It's meant to be a replacement for the apache thrift code generator, and generates conforming, compatible binary codecs by building on top of libthrift. It integrates with the finagle project, exporting stats and finagle APIs, and makes it easy to build high throughput, low latency, robust thrift servers and clients.

Part of the motivation behind scrooge's scala implementation is that since Scala is API-compatible with Java, you can use the apache thrift code generator to generate Java files and use them from within Scala, but the generated code uses Java collections and mutable "bean" classes, causing some annoying boilerplate conversions to be hand-written. Scrooge bypasses the problem by generating Scala code directly. It also uses Scala syntax so the generated code is much more compact.

There is a comprehensive set of unit tests, which generate code, compile it, and execute it to verify expectations, as well as gold files to make it easy to review the effects of changes to the generator.

Status

This project is used in production at Twitter (and many other organizations), and is actively developed and maintained.

Building the develop branch locally

We are not currently publishing snapshots for Scrooge's dependencies, which means that it may be necessary to publish the develop branches of these libraries locally in order to work on Scrooge's develop branch. To do so you can use our build tool, dodo.

curl -s https://raw.githubusercontent.com/twitter/dodo/develop/bin/build | bash -s -- --no-test scrooge

If you have any questions or run into any problems, please create an issue here, tweet at us at @finagle, or email the Finaglers mailing list.

Full Documentation

https://twitter.github.io/scrooge/

License

Copyright 2013 Twitter, Inc.

Licensed under the Apache License, Version 2.0: https://www.apache.org/licenses/LICENSE-2.0

scrooge's Issues

Exceptions without any params throws a parse error

Scrooge wants a { and a ( to follow an exception definition.

Provide some configuration around the collection types used.

It's often preferable to specify that the types generated are definitely immutable (scala.collection.immutable.Map rather than scala.collection.Map for example), especially when using libraries like scalaz. A config option to specify that would make this easier, as from looking at the code I can't see anything that requires mutability.

"example maven project" and "example pom file" links to the wrong page

In the home page, the following two links point to the wrong address,
"example maven project" should link to
https://github.com/twitter/scrooge/tree/master/demos/scrooge-maven-demo, and
"example pom file" should link to
https://github.com/twitter/scrooge/blob/master/demos/scrooge-maven-demo/pom.xml.

Fully Qualified Type Names are not supported in struct/service defs

We have a best practice of using fully qualified names when referencing a Thrift struct in another Thrift IDL, this practice is not supported by the Scrooge generator which therefore is not backward compatible with the standard Thrift compiler.

The standard Thrift compiler accepts the file name (minus the '.thrift' suffix) as a prefix for the structures used in another file, Scrooge doesn't :-(

file: foo.bar.thrift.data.thrift

namespace java foo.bar.thrift.data

struct A {
1: bool bool_field,
}

file: foo.var.thrift.services.thrift

namespace java foo.bar.thrift.services

include "foo.bar.thrift.data.thrift"

service Svc {
list < foo.bar.thrift.data.A > get(1:string name)
}

Calling thrift --gen java:beans foo.bar.thrift.services.thrift correctly generates code when
scrooge -l java foo.bar.thrift.services.thrift (or foo.bar.thrift.*.thrift) produces the following error:

Exception in thread "main" com.twitter.scrooge.TypeNotFoundException: foo.bar.thrift.data.A
at com.twitter.scrooge.TypeResolver$$anon$1.createMissingEntityException(TypeResolver.scala:74)
at com.twitter.scrooge.EntityResolver.apply(TypeResolver.scala:60)
at com.twitter.scrooge.TypeResolver.apply(TypeResolver.scala:274)
at com.twitter.scrooge.TypeResolver.apply(TypeResolver.scala:201)
at com.twitter.scrooge.TypeResolver.apply(TypeResolver.scala:204)
at com.twitter.scrooge.TypeResolver.apply(TypeResolver.scala:197)
at com.twitter.scrooge.TypeResolver.apply(TypeResolver.scala:185)
at com.twitter.scrooge.TypeResolver$$anonfun$7.apply(TypeResolver.scala:171)
at com.twitter.scrooge.TypeResolver$$anonfun$7.apply(TypeResolver.scala:171)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
at scala.collection.immutable.List.foreach(List.scala:76)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
at scala.collection.immutable.List.map(List.scala:76)
at com.twitter.scrooge.TypeResolver.apply(TypeResolver.scala:171)
at com.twitter.scrooge.TypeResolver$$anonfun$apply$3.apply(TypeResolver.scala:130)
at com.twitter.scrooge.TypeResolver$$anonfun$apply$3.apply(TypeResolver.scala:129)
at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
at scala.collection.immutable.List.foreach(List.scala:76)
at com.twitter.scrooge.TypeResolver.apply(TypeResolver.scala:129)
at com.twitter.scrooge.Compiler$$anonfun$run$2.apply(Compiler.scala:67)
at com.twitter.scrooge.Compiler$$anonfun$run$2.apply(Compiler.scala:61)
at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
at scala.collection.immutable.List.foreach(List.scala:76)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:30)
at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:44)
at com.twitter.scrooge.Compiler.run(Compiler.scala:61)
at com.twitter.scrooge.Main$.main(Main.scala:98)
at com.twitter.scrooge.Main.main(Main.scala)

turn off camel casing

We just started using Scrooge for this example of streamcorpus generation in Scala, and it seems to be converting all of our method names to camel case. For example, "entity_type" becomes "entityType"

https://github.com/trec-kba/streamcorpus/blob/v0.3.0-dev/if/streamcorpus-v0_3_0.thrift

Is there a reason for this? At the very least, it invalidates our documentation. It also makes it painful to port code from other languages that use these thrift interfaces.

https://github.com/trec-kba/streamcorpus/blob/v0.3.0-dev/examples/scala/streamcorpus_example/AddTagging.scala

Can we turn off this camel casing behavior?

Comments cause the Java generator to fail

Comments that begin with /** in the middle of an ENUM or struct seems to make things blow up. Comments like /* do not.

Generate Java classes for thrift

I cannot find a way to tell the scrooge-maven-plugin how to use the --language java.
Here is my pom.xml

com.twitter scrooge-maven-plugin 3.0.8-SNAPSHOT java --finagle -v

            </configuration>
            <executions>
                <execution>
                    <id>thrift-sources</id>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>compile</goal>
                    </goals>
                </execution>
                <execution>
                    <id>thrift-test-sources</id>
                    <phase>generate-test-sources</phase>
                    <goals>
                        <goal>testCompile</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>

Can not use enum consts from included file as default value

It is not possible to use enum constants as default value when the enum is imported from another file.

For example, when I change $/scrooge-generator/src/test/thrift/relative/candy.thrift to:

struct Candy {
  1: i32 sweetness_iso
  2: CandyType candy_type = CandyType.WEIRD   // <---- added default
  3: string headline = include1.HEADLINE
  4: optional string brand = include2.BRAND
  5: i32 count = include3.PIECES
}

You get compilation errors in both Java and Scala output. Here is the Java output:

[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] $/scrooge-generator/target/test-sources/thrift/java_test2/Candy.java:[50,53] cannot find symbol
symbol  : variable CandyType
location: class thrift.java_test2.Candy.Builder
[ERROR] $/scrooge-generator/target/test-sources/thrift/java_test2/Candy.java:[60,24] cannot find symbol
symbol  : variable CandyType
location: class thrift.java_test2.Candy.Builder
[ERROR] $/scrooge-generator/target/test-sources/thrift/java_test2/Candy.java:[131,46] cannot find symbol
symbol: variable CandyType
      thrift.java_test1.CandyType candyType = CandyType.WEIRD;

Here is the wrong java code:

private thrift.java_test1.CandyType _candyType = CandyType.WEIRD;

As you can see the type is missing on the right hand side (but not on the left hand side).

For Scala its the same problem.

Tests no longer running in Travis CI?

See this build for example: https://travis-ci.org/twitter/scrooge/builds/5633433

---

T E S T S

Results :
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] scrooge ........................................... SUCCESS [0.003s]
[INFO] scrooge-runtime ................................... SUCCESS [2:42.640s]
[INFO] scrooge-generator ................................. SUCCESS [4:10.006s]
[INFO] scrooge-maven-plugin .............................. SUCCESS [1:07.597s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8:00.717s
[INFO] Finished at: Tue Mar 19 18:21:34 UTC 2013
[INFO] Final Memory: 28M/181M
[INFO] ------------------------------------------------------------------------
The command "mvn test" exited with 0.
Done. Your build exited with 0.

unable to deal with include with file separator

Currently scrooge cannot deal with:

include "some/directory/my_other_thrift.thrift" inside a thrift file.

Scala 2.9.1 and SBT 0.10

Are there any plans to support Scala 2.9.1 and/or SBT 0.10?

Method names clashes in generated ThriftServer for ostrich server(i.e. 'shutdown')

To reproduce create service IDL file

namespace java ru.exmpl

/**
 * Standard base service
 */
service ExampleService {

  /**
   * Suggest a shutdown to the server
   */
  oneway void shutdown(),

}

generate scala source with flag --ostrich

object ExampleService {
  trait Iface {
    def shutdown(): Unit
  }
.....

trait ThriftServer extends Service with FutureIface {
    val log = Logger.get(getClass)

    def thriftCodec = ThriftServerFramedCodec()
    def statsReceiver: StatsReceiver = new OstrichStatsReceiver
    def tracerFactory: Tracer.Factory = NullTracer.factory
    val thriftProtocolFactory: TProtocolFactory = new TBinaryProtocol.Factory()
    val thriftPort: Int
    val serverName: String

    // Must be thread-safe as different threads can start and shutdown the service.
    private[this] val _server = new AtomicReference[Server]
    def server: Server = _server.get

    def start() {
      val thriftImpl = new FinagledService(this, thriftProtocolFactory)
      _server.set(serverBuilder.build(thriftImpl))
    }

    /**
     * You can override this to provide additional configuration
     * to the ServerBuilder.
     */
    def serverBuilder =
      ServerBuilder()
        .codec(thriftCodec)
        .name(serverName)
        .reportTo(statsReceiver)
        .bindTo(new InetSocketAddress(thriftPort))
        .tracerFactory(tracerFactory)

    /**
     * Close the underlying server gracefully with the given grace
     * period. close() will drain the current channels, waiting up to
     * ``timeout'', after which channels are forcibly closed.
     */
    def shutdown(timeout: Duration = 0.seconds) {
      synchronized {
        val s = server
        if (s != null) {
          s.close(timeout)
        }
      }
    }
  }

Compilation will fail with message

ExampleService.scala:3525: overriding method shutdow
n in trait Service of type ()Unit;
[error] method shutdown in trait FutureIface of type ()com.twitter.util.Future[
Unit] has incompatible type
[error] trait ThriftServer extends Service with FutureIface {
[error] ^

I use scrooge 3.0.2

Deploy to public repositories?

Seems like Scrooge hasn't been deployed to Maven Central for a while... I would love it if you could release some new versions!
For example, the Finagle build fails because it needs scrooge-sbt-plugin version 3.1.5...

Consider using Scala Macros instead of Handlebar

Despite what Stu Hood thinks of them.

escape scala keyword

using a scala keyword in thrift definition should produce an error

NPE when using generated java files for finagle

When I try to create the service client :

 contentServiceClient = new ContentProvider.FinagledClient(
            service, new TBinaryProtocol.Factory(), "contentService",
            new InMemoryStatsReceiver());

I get the following exception :

java.lang.NullPointerException
at net.entropysoft.remoteconnectors2.thrift.generated.ContentProvider$FinagledClient$__Stats.<init>(ContentProvider.java:18698)
at net.entropysoft.remoteconnectors2.thrift.generated.ContentProvider$FinagledClient.<init>(ContentProvider.java:18708)

For each service method the following line is generated in FinagledClient

  private __Stats __stats_addContentChunk = new __Stats("addContentChunk");

__Stats is an inner class in FinagledClient

class __Stats {
  Counter requestsCounter, successCounter, failuresCounter;
  StatsReceiver failuresScope;

  public __Stats(String name) {
    StatsReceiver scope = FinagledClient.this.scopedStats.scope(name);
    this.requestsCounter = scope.counter0("requests");
    this.successCounter = scope.counter0("success");
    this.failuresCounter = scope.counter0("failures");
    this.failuresScope = scope.scope("failures");
  }
}

It accesses scopedStats in FinagledClient but scopedStats is null before the FinagledClient constructor has been called ...

Generator: option -s (skip-unchanged) doesn't work

Main.scala contains the following option definition:

      opt("s", "skip-unchanged", "Don't re-generate if the target is newer than the input", { compiler.skipUnchanged = true; () })

However, the boolean compiler.skipUnchanged is never read. This function doesn't seem implemented.

Upgrade to a modern thrift

I'm attempting to use Scrooge to generate a Java Thrift client, but the generated code does not compile with Thrift 0.7.0 or higher. The generated Client class is declared:

public static class Client implements TServiceClient, Iface {

However, TServiceClient was changed from an interface to an abstract class in Thrift 0.7.0 so this does not compile.

Any thoughts? Thanks.

javac warnings in generated code

To help maintain code health, we build with -Xlint:all -Werror by default. The generated thrift types and services contain warnings such as,

warning: [unchecked] unchecked generic array creation of type T[] for varargs parameter

It would nice to have the code contain SuppressWarnings where appropriate so that the code is lint free.

P.S. This is on JDK6.

Does finagle client only support TBinaryProtocol for Thrift?

Currently I got a problem when change the thrift client to finagle, while the server uses the TCompactProtocol, so I when I create the finagle client, I also applied the TCompactProtocol for it.
But the server keeps throwing exception : org.apache.thrift.protocol.TProtocolException: Expected protocol id ffffff82 but got ffffff80.
When I create a test server using TBinaryProtocol, then the finagle client can work fine, so do I have to change the server side if I want to use finagle client?

support non-finagle servers

Its not currently possible to use scrooge with vanilla thrift.

Build failing

Error - 
>mvn clean package -U
[INFO] Scanning for projects...
Downloading: http://repo1.maven.org/maven2/com/twitter/scala-parent-292/0.0.2/scala-parent-292-0.0.2.pom
[ERROR] The build could not read 2 projects -> [Help 1]
[ERROR]   
[ERROR]   The project com.twitter:scrooge-runtime:3.0.1 (/Users/test/scrooge/scrooge-runtime/pom.xml) has 1 error
[ERROR]     Non-resolvable parent POM: Could not find artifact com.twitter:scala-parent-292:pom:0.0.2 in central (http://repo1.maven.org/maven2) and 'parent.relativePath' points at no local POM @ line 8, column 11 -> [Help 2]
[ERROR]   
[ERROR]   The project com.twitter:scrooge-generator:3.0.1 (/Users/test/scrooge/scrooge-generator/pom.xml) has 1 error
[ERROR]     Non-parseable POM /Users/test/.m2/repository/com/twitter/service-292/0.0.2/service-292-0.0.2.pom: Expected root element 'project' but found 'html' (position: START_TAG seen <html>... @1:6)  @ /Users/test/.m2/repository/com/twitter/service-292/0.0.2/service-292-0.0.2.pom, line 1, column 6 -> [Help 3]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
[ERROR] [Help 3] http://cwiki.apache.org/confluence/display/MAVEN/ModelParseException

Support for sbt 0.11.3?

Scrooge depends on sbt-package-dist which is not available in the maven repo for sbt 0.11.3. It also appears that twitter has moved away from sbt so support should not be expected on that plugin: twitter-archive/sbt-package-dist#4

Any thoughts on having this work for sbt 0.11.3?

Typedefs from an included thrift are neither imported nor referenced by package

Here's an attempt at extracting a minimum repro:

Foo.thrift:

package com.example.foo

struct OneFoo
{
  1: i32 foo_id
}

typedef list<OneFoo> ManyFoos

Bar.thrift

package com.example.bar

include "foo.thrift"

// Generates, but FAILS to compile
struct Bar
{
  1: foo.ManyFoos thefoobars
}

// WORKS
struct Bar
{
  1: list<foo.OneFoo> thefoobars
}

The problems appears to be that the Scala output in bar.scala is this:

class Bar(thefoobars: Seq[OneFoo]) extends ThriftStruct { ... }

It should work the typedef backwards and either prefix:

class Bar(thefoobars: Seq[foo.OneFoo]) extends ThriftStruct { ... }

Or it should import foo._ into bar.scala (there is as include "foo.thrift" in "bar.thrift", so I think it makes sense to generate a corresponding import).

Implement validate()

Scrooge 3.0.2 and 3.0.3 not available in maven repository

Latest available version is 3.0.1 at http://maven.twttr.com/com/twitter/scrooge/

The variables in the startup script should be changed to the approprivate value by the deploy process.

I find the file named scrooge at "${PROJECT_HOME}/scrooge-generator/src/scripts", and here are the contents :

!/bin/bash

HERE=$(dirname $(readlink $0 || echo $0))
java -jar $HERE/../scrooge-@[email protected] "$@"

The variables, such as @Version@ should be changed by the deploy process.

The ThriftStructCodec in the scrooge-runtime doesn't have a method 'decode' .

[ERROR] /Users/bobby/Projects/Cassandro/src/main/scala-gen/com/bithike/cassandra/thrift/Cassandra.scala:11655: error: value decode is not a member of com.twitter.scrooge.ThriftStructCodec[T]
[INFO] codec.decode(iprot)
[INFO] ^

thrift idl containing UTF-8 produces java.nio.charset.UnmappableCharacterException

E.g.

cat echo.thrift
/**

• 测试
  */
  namespace java com.mytest.finagle.thrift

service EchoService {
string ping(string message)
}

java -jar scrooge-generator-3.1.1-jar-with-dependencies.jar --finagle --language JAVA echo.thrift

Exception in thread "main" java.nio.charset.UnmappableCharacterException: Input length = 2
at java.nio.charset.CoderResult.throwException(CoderResult.java:261)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:319)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.read(BufferedReader.java:157)
at scala.io.BufferedSource$$anonfun$iter$1$$anonfun$apply$mcI$sp$1.apply$mcI$sp(BufferedSource.scala:38)
at scala.io.Codec.wrap(Codec.scala:64)
at scala.io.BufferedSource$$anonfun$iter$1.apply(BufferedSource.scala:38)
at scala.io.BufferedSource$$anonfun$iter$1.apply(BufferedSource.scala:38)
at scala.collection.Iterator$$anon$14.next(Iterator.scala:150)
at scala.collection.Iterator$$anon$25.hasNext(Iterator.scala:562)
at scala.collection.Iterator$$anon$19.hasNext(Iterator.scala:400)
...

Problem to Locate Scrooge Binary

Hi,
after cloned and compiled the code with $./sbt +publish-local , I can't find the binary scrooge and scrooge-runtime.
where I find them?

Thanks

scrooge-3.1.1.zip is unavailable for sbt-scrooge

Can you publish to your maven repo this artifact ?

I'm missing http://maven.twttr.com//com/twitter/scrooge/3.1.1/scrooge-3.1.1.zip for sbt-scrooge

Scrooge simply bombs out when an invalid type is used

I accidentally used String as a data type in a thrift file, rather the string. Since I was previously compiling to Java, turns out this just passed through transparently and worked as a Java String type! However, when compiling to Scala, scrooge did this helpful thing:

[error] Exception in thread "main" com.twitter.scrooge.TypeNotFoundException: String
[error]     at com.twitter.scrooge.TypeResolver$EntityResolver$$anonfun$apply$1.apply(TypeResolver.scala:40)
[error]     at com.twitter.scrooge.TypeResolver$EntityResolver$$anonfun$apply$1.apply(TypeResolver.scala:40)
[error]     at scala.Option.getOrElse(Option.scala:104)
[error]     at com.twitter.scrooge.TypeResolver$EntityResolver.apply(TypeResolver.scala:40)
[error]     at com.twitter.scrooge.TypeResolver.apply(TypeResolver.scala:232)
[error]     at com.twitter.scrooge.TypeResolver.apply(TypeResolver.scala:182)
[error]     at com.twitter.scrooge.TypeResolver.apply(TypeResolver.scala:170)
[error]     at com.twitter.scrooge.TypeResolver$$anonfun$apply$2.apply(TypeResolver.scala:154)
[error]     at com.twitter.scrooge.TypeResolver$$anonfun$apply$2.apply(TypeResolver.scala:154)
[error]     at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
[error]     at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
[error]     at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
[error]     at scala.collection.immutable.List.foreach(List.scala:45)
[error]     at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
[error]     at scala.collection.immutable.List.map(List.scala:45)
[error]     at com.twitter.scrooge.TypeResolver.apply(TypeResolver.scala:154)
[error]     at com.twitter.scrooge.TypeResolver.resolve(TypeResolver.scala:118)
[error]     at com.twitter.scrooge.TypeResolver$$anonfun$resolve$2.apply(TypeResolver.scala:96)
[error]     at com.twitter.scrooge.TypeResolver$$anonfun$resolve$2.apply(TypeResolver.scala:95)
[error]     at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
[error]     at scala.collection.immutable.List.foreach(List.scala:45)
[error]     at com.twitter.scrooge.TypeResolver.resolve(TypeResolver.scala:95)
[error]     at com.twitter.scrooge.Main$$anonfun$main$1.apply(Main.scala:92)
[error]     at com.twitter.scrooge.Main$$anonfun$main$1.apply(Main.scala:78)
[error]     at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
[error]     at scala.collection.immutable.List.foreach(List.scala:45)
[error]     at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:44)
[error]     at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:42)
[error]     at com.twitter.scrooge.Main$.main(Main.scala:78)
[error]     at com.twitter.scrooge.Main.main(Main.scala)

Expected: some kind of error message with a line number.

Needs union support

A union is a special case of a struct which has exactly one of its fields set.

https://issues.apache.org/jira/browse/THRIFT-409

3.1.1 not available on Sonatype

I was looking to use Scrooge 3.1.1, but I noticed that 3.1.0 was the latest available on Sonatype.

Are there plans to push 3.1.1 up soon?

args decoding codec fails on enums

In the switch statement that is used to decode an incoming thrift message, when an enum is encountered it is ignored. This is caused by the case for the enum field expecting to get a data type of TType.ENUM but the ruby client transmits the field as TType.I32.

// generated code from scrooge. This leaves the enum field as null in the builder object.
              case 4: /* platform */
                switch (_field.type) {
                  case TType.ENUM:
                    Platform platform_item;
                    platform_item = Platform.findByValue(_iprot.readI32());
                    platform = platform_item;
                    break;
                  default:
                    TProtocolUtil.skip(_iprot, _field.type);
                }
                builder.platform(platform);
                break;

// generated code modified to catch I32 on enum field. This populates the field accordingly.
              case 4: /* platform */
                switch (_field.type) {
+                 case TType.I32:
                  case TType.ENUM:
                    Platform platform_item;
                    platform_item = Platform.findByValue(_iprot.readI32());
                    platform = platform_item;
                    break;
                  default:
                    TProtocolUtil.skip(_iprot, _field.type);
                }
                builder.platform(platform);
                break;

In looking at the code from the Apache Thrift generator, it appears that enums are expected to be i32 there.

            case 4: // RESERVATION_LENGTH
              if (schemeField.type == org.apache.thrift.protocol.TType.I32) {
                struct.reservationLength = ReservationLength.findByValue(iprot.readI32());
                struct.setReservationLengthIsSet(true);
              } else {.
                org.apache.thrift.protocol.TProtocolUtil.skip(iprot, schemeField.type);
              }
              break;

I'm pretty sure this block of code is generated here with the resulting type being generated here.

Generated scala stubs don't compile when the thrift type contains field named 'runtime'

cat src/main/thrift/runtimetest.thrift
struct runtimetest {
1: string runtime;
}

build/generated-sources/thrift/thrift/Runtimetest.scala:124: value ScalaRunTime is not a member of String
override def equals(other: Any): Boolean = runtime.ScalaRunTime._equals(this, other)
^
build/generated-sources/thrift/thrift/Runtimetest.scala:126: value ScalaRunTime is not a member of String
override def hashCode: Int = runtime.ScalaRunTime._hashCode(this)
^
build/generated-sources/thrift/thrift/Runtimetest.scala:128: value ScalaRunTime is not a member of String
override def toString: String = runtime.ScalaRunTime._toString(this)

The maven build fails

1. Not all dependencies can be resolved - I tried adding maven.twttr.com to the repository list, that appeared to do the trick.
2. Maven does not compile scala code by default - or have you in some way configured it to do so? Anyways, I suppose the easiest way to make it happen would be to use the maven-scala-plugin.
3. I tried building the project with the maven-scala-plugin after applying those changes, but it failed due to some compilation/dependency error :-/

When generating java files for finagle, ${GeneratedService}.FinagleService class is protected

This means we must be in the same package to create the server using :
ServerBuilder.safeBuild(
new ${GeneratedService}.FinagledService(processor,
new TBinaryProtocol.Factory()),
ServerBuilder.get().name("serviceName")
.codec(ThriftServerBufferedCodec.get())
.bindTo(new InetSocketAddress(8080)));

mutability issue

Is there an argument to scrooge to cause it to make mutable structures, so we can add more data to the structures that we get out of thrift messages?

Our standard pipelining in other languages takes in a large thrift message, and then augments it with a few small additions, and then serializes it back out to another thrift protocol.

https://github.com/trec-kba/streamcorpus/blob/v0.3.0-dev/examples/scala/streamcorpus_example/AddTagging.scala

Alternatively, is there an efficient scala idiom (proxy?) for making a structure that points to the immutable original message, so we can efficiently augment and serialize it back out without copying the entire message?

Thanks for guidance.

Artifact from scrooge-runtime cannot be pulled in from maven central.

This because it depends on util-codec-5.3.13 which has not been published.

Problem with "include" definitions

Hey there,

Given:

1. A project started created with scala-bootstrapper.
2. A two file thrift IDL definition, where one includes the other:

namespace java com.twitter.testservice
namespace rb TestService

include "user.thrift"

exception TestServiceException {
  1: string description
}

service TestServiceService {
  string get(1: user.User u) throws(1: TestServiceException ex)
}

namespace java com.twitter.testservice

struct User {
       1: string username
}

I'm getting compile errors from the generated code:

Blakes-MacBook-Air-2:test_service blake$ ./sbt compile
[info] Loading project definition from /Volumes/tempodb/test_service/project
No .svnrepo file; no svn repo will be configured.
[info] Set current project to testservice (in build file:/Volumes/tempodb/test_service/)
[warn] Credentials file /Users/blake/.artifactory-credentials does not exist
[info] Generating scrooge thrift for /Volumes/tempodb/test_service/src/main/thrift/testservice.thrift, /Volumes/tempodb/test_service/src/main/thrift/user.thrift ...
[info] + Compiling /Volumes/tempodb/test_service/src/main/thrift/testservice.thrift
[info] + Compiling /Volumes/tempodb/test_service/src/main/thrift/user.thrift
[info] Compiling 8 Scala sources to /Volumes/tempodb/test_service/target/scala-2.9.1/classes...
[error] /Volumes/tempodb/test_service/target/scala-2.9.1/src_managed/main/scala/com/twitter/testservice/TestServiceService.scala:36: not found: value _user_
[error]     def get(`u`: _user_.User): Future[String]
[error]                  ^
[error] /Volumes/tempodb/test_service/target/scala-2.9.1/src_managed/main/scala/com/twitter/testservice/TestServiceService.scala:32: not found: value _user_
[error]     def get(`u`: _user_.User): String
[error]                  ^
[error] /Volumes/tempodb/test_service/target/scala-2.9.1/src_managed/main/scala/com/twitter/testservice/TestServiceService.scala:104: not found: value _user_
[error]     with Product1[_user_.User]
[error]                   ^
[error] /Volumes/tempodb/test_service/target/scala-2.9.1/src_managed/main/scala/com/twitter/testservice/TestServiceService.scala:59: not found: value _user_
[error]         var `u`: _user_.User = null
[error]                  ^
[error] /Volumes/tempodb/test_service/target/scala-2.9.1/src_managed/main/scala/com/twitter/testservice/TestServiceService.scala:73: not found: value _user_
[error]                       _user_.User.decode(_iprot)
[error]                       ^
[error] /Volumes/tempodb/test_service/target/scala-2.9.1/src_managed/main/scala/com/twitter/testservice/TestServiceService.scala:98: not found: value _user_
[error]       val `u`: _user_.User
[error]                ^
[error] /Volumes/tempodb/test_service/target/scala-2.9.1/src_managed/main/scala/com/twitter/testservice/TestServiceService.scala:49: not found: value _user_
[error]       `u`: _user_.User
[error]            ^
[error] /Volumes/tempodb/test_service/target/scala-2.9.1/src_managed/main/scala/com/twitter/testservice/TestServiceService.scala:54: not found: value _user_
[error]     def unapply(_item: get_args): Option[_user_.User] = Some(_item.u)
[error]                                          ^
[error] /Volumes/tempodb/test_service/target/scala-2.9.1/src_managed/main/scala/com/twitter/testservice/TestServiceService.scala:109: not found: value _user_
[error]     def `u`: _user_.User
[error]              ^
[error] /Volumes/tempodb/test_service/target/scala-2.9.1/src_managed/main/scala/com/twitter/testservice/TestServiceService.scala:127: not found: value _user_
[error]       `u`: _user_.User = this.`u`
[error]            ^
[error] /Volumes/tempodb/test_service/target/scala-2.9.1/src_managed/main/scala/com/twitter/testservice/TestServiceService.scala:357: not found: value _user_
[error]     def get(`u`: _user_.User): Future[String] = {
[error]                  ^
[error] 11 errors found
[error] {file:/Volumes/tempodb/test_service/}default-6304ff/compile:compile: Compilation failed
[error] Total time: 5 s, completed Mar 22, 2013 5:15:06 PM

Oddly, when I wrap the user.User inside the testservice IDL in its own struct, I don't get the import errors anymore:

namespace java com.twitter.testservice
namespace rb TestService

include "user.thrift"

exception TestServiceException {
  1: string description
}

struct WrappedUser {
       1: user.User u
}

service TestServiceService {
  string get(1: WrappedUser u) throws(1: TestServiceException ex)
}

Output:

Blakes-MacBook-Air-2:test_service blake$ ./sbt compile
[info] Loading project definition from /Volumes/tempodb/test_service/project
No .svnrepo file; no svn repo will be configured.
[info] Set current project to testservice (in build file:/Volumes/tempodb/test_service/)
[info] Generating scrooge thrift for /Volumes/tempodb/test_service/src/main/thrift/testservice.thrift, /Volumes/tempodb/test_service/src/main/thrift/user.thrift ...
[warn] Credentials file /Users/blake/.artifactory-credentials does not exist
[info] + Compiling /Volumes/tempodb/test_service/src/main/thrift/testservice.thrift
[info] + Compiling /Volumes/tempodb/test_service/src/main/thrift/user.thrift
[info] Compiling 9 Scala sources to /Volumes/tempodb/test_service/target/scala-2.9.1/classes...
[error] /Volumes/tempodb/test_service/src/main/scala/com/twitter/testservice/TestServiceServiceImpl.scala:10: class TestServiceServiceImpl needs to be abstract, since method get in trait FutureIface of type (u: com.twitter.testservice.WrappedUser)com.twitter.util.Future[String] is not defined
[error] (Note that com.twitter.testservice.WrappedUser does not match String)
[error] class TestServiceServiceImpl(config: TestServiceServiceConfig) extends TestServiceService.ThriftServer {
[error]       ^
[error] one error found
[error] {file:/Volumes/tempodb/test_service/}default-6304ff/compile:compile: Compilation failed
[error] Total time: 6 s, completed Mar 22, 2013 5:41:44 PM

Here's my build.sbt:

import com.twitter.sbt._
import com.twitter.scalatest._

seq((
  Project.defaultSettings ++
    StandardProject.newSettings ++
    SubversionPublisher.newSettings ++
    CompileThriftScrooge.newSettings ++
    ScalaTestMixins.testSettings
): _*)

organization := "com.twitter"

name := "testservice"

version := "1.0.0-SNAPSHOT"

libraryDependencies ++= Seq(
  "org.scala-lang" % "jline"                 % "2.9.1",
  "com.twitter"    % "scrooge"               % "3.0.1",
  "com.twitter"    % "scrooge-runtime" % "3.0.8",
  "com.twitter"    % "finagle-core"          % "5.3.6",
  "com.twitter"    % "finagle-thrift"        % "5.3.6",
  "com.twitter"    % "finagle-ostrich4"      % "5.3.1",
  "org.scalatest" %% "scalatest"             % "1.7.1" % "test",
  "com.twitter"   %% "scalatest-mixins"      % "1.1.0" % "test"
)

mainClass in (Compile, run) := Some("com.twitter.testservice.Main")

mainClass in (Compile, packageBin) := Some("com.twitter.testservice.Main")

CompileThriftScrooge.scroogeVersion := "3.0.1"

I even tried upgrading to scrooge 3.0.8, but received some other weird errors:

Blakes-MacBook-Air-2:test_service blake$ ./sbt compile
[info] Loading project definition from /Volumes/tempodb/test_service/project
No .svnrepo file; no svn repo will be configured.
[info] Set current project to testservice (in build file:/Volumes/tempodb/test_service/)
[warn] Credentials file /Users/blake/.artifactory-credentials does not exist
[info] Fetching scrooge 3.0.8 ...
[info] Fetching from: http://maven.twttr.com//com/twitter/scrooge/3.0.8/scrooge-3.0.8.zip
[error] {file:/Volumes/tempodb/test_service/}default-6304ff/*:scrooge-fetch: Error opening http://maven.twttr.com//com/twitter/scrooge/3.0.8/scrooge-3.0.8.zip: java.io.FileNotFoundException: http://maven.twttr.com//com/twitter/scrooge/3.0.8/scrooge-3.0.8.zip
[error] Total time: 0 s, completed Mar 22, 2013 5:20:33 PM

I feel like I'm buried in several different issues here. Can you guys offer some help or insight?

Much thanks!

Blake

Generate JavaBean style classes

Hi,
How to generate JavaBean style classes? I use scrooge --finagle -l java X.thrift, but the class X has not a zero argument constructor.

invalid scala generated for default values

It appears that Scrooge generates invalid Scala when handling default values, like this:

https://github.com/trec-kba/streamcorpus/blob/2bd4562c2deebbc0b13beb2f558edfbde73c39d8/if/streamcorpus-v0_3_0.thrift#L774

The relevant thrift snippet is here:

enum Versions {
v0_2_0 = 0,
v0_3_0 = 1,
}
struct StreamItem {
1: Versions version = 1,

Scrooge generates this:

  version: Versions = 1,

which Scala will then refuse to compile, as 1 is an Integer, not a Versions. This replacement works:

 version: Versions = Versions(1),

Advice on this?
Thanks.

scaladoc warning caused by comments in generated code

I'm using scrooge-sbt-plugin-3.1.5 with sbt-0.12.4, and have met following warning message during sbt doc.

[warn] ....scala:696: Variable result undefined in comment for class Immutable
[warn]     class Immutable(
[warn]           ^

I have found it is caused by the comments in generated code.

This is a part of my generated scala code.

/**
 * The default read-only implementation of putRows$result.  You typically should not need to
 * directly reference this class; instead, use the putRows$result.apply method to construct
 * new instances.
 */
class Immutable(
) extends putRows$result

In this code, scaladoc regards "$result" as a macro and because there is no define for "$result", it shows warnings.
(see Macros in https://wiki.scala-lang.org/display/SW/Tags+and+Annotations)

This issue is not critical but annoying, so I hope to be fixed.

cannot deal with 'typedef'

typedef anotherfile.SomeStruct SomeStruct

will cause an exception

Scrooge compiler doesn't ignore #-style comments

/* this is a comment */
# this is a comment

will not compile, due to the second line. But hash style comments are valid in IDL files. Any chance it can be fixed?

impossible to assign a default value to a bool

It looks like thrift doesn't define any syntax for true/false as a default value, so convention is to use 0 and 1, which may make sense to C people, but is objectively idiotic.

Scrooge should support both true/false and 0/1 as default values for bools.

Example from David Helder:

const bool ECHO_FRAMED = 1;

Issue building with spaces in path

From a jenkins job with spaces in it's title...
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10.182s
[INFO] Finished at: Thu Jun 13 21:00:23 UTC 2013
[INFO] Final Memory: 18M/44M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.twitter:scrooge-maven-plugin:3.1.8:compile (thrift-sources) on project *****-thrift-definitions-scala: An IO error occured: error forming URI for file transfer: java.net.URISyntaxException: Illegal character in path at index 44: file:///mnt/jenkins/jobs/services.Name With Space/workspace/name-with-spaces/target/classes -> [Help 1]

license copyright

look like your forgot to fill this line

https://github.com/twitter/scrooge/blob/master/LICENSE#L190

Casing broken between 3.0.1 and 3.0.6

We have thrift file like so:

exception HCacheException {
  1: string msg;
}

With scrooge 3.0.1, this produced a scala class HCacheException, but with 3.0.6, the name is now HcacheException (notice the lowercase c).

I feel like scrooge shouldn't be mangling this name since it's already using TitleCase...