Its design is adapted from a duplex-transport architecture, where both ends of a transport are peers that can initiate a message. While this works here, it is inappropriate for a basic RPC construct like that defined by Thrift.

We can get away with fewer resources, eliminate the complicated locking, and likely acheive better performance by replacing Reader and Writer threads with one single worker.

#73, on top of everything else, fixed a codegen issue wherein collections of enums were serialized incorrectly - single enums correctly got the typecode TType.I32, but list, set, and map metadata had TType.ENUM, which despite its name is incorrect.

This was a subtle and long-lived bug because not all protocol implementations have a problem with this - we noticed only because of Python's TBinaryProtocolAccelerated, a C module that does actually enforce correct typecodes.

We need tests to ensure that this doesn't regress.

A good approach would be to write a test protocol that records which Protocol methods are invoked and in what order, and uses it to test against code that is generated e.g. in thrifty-integration-tests.

a.thrift:

namespace java com.foo.bar

const i32 foo = 21

b.thrift:

namespace java com.foo.bar

include 'a.thrift'

const i32 bar = a.foo

Loading this fails with Unrecognized identifier: foo

The following typedef reports no annotations when I try to parse it. This appears to be valid thrift ordering in our other language parsers.

typedef i64 (js.type = 'Date') Date

I am deeply concerned about the road that Thrift took during the past years:

• Facebook donated Thrift to the Apache Foundation
• Facebook forked its own incompatible branch for Linux only, adding new features like float support
• Microsoft optimized the Apache Thrift JAVA code

What I fear is that we end up with 4 or 5 different versions of one of the best IPC libraries of the world and that Thrift wil simply die, die, die due to this.

Could you please explain what hinders you to merge Thrifty back into Apache Thrift ? Is the Apache developer team not responsive enough? Are there technical issues? Are there incompatibilities?

Description

Let's say we define a service Foo:

service Foo {
  void foo()
}

and another service Bar which extends Foo:

include 'Foo.thrift'

service Bar extends Foo {
  void bar()
}

The generated code for the BarClient should provide an implementation of the foo method in order for the generated code to compile but currently only an implementation for bar is generated.

My use case involves serializing any struct without knowing the specific class. In order to do this, they need to implement a common interface. In this case, I imagine the interface will be something like the following:

public interface Struct<T, B extends StructBuilder<T>> {
    Adapter<T, B> getAdapter();
}

Then a given generated would look like the following:

public final class Foo implements Struct<Foo, Foo.Builder> {
  public static final Adapter<Foo, Builder> ADAPTER = new FooAdapter();

  ...

  @Override
  public Adapter<Foo, Builder> getAdapter() {
    return ADAPTER;
  }

  ...
}

and you'd be able to serialize a generic struct via something like:

<T extends Struct<T, B>, B extends StructBuilder<T>> void serialize(T struct) {
  Buffer buffer = new Buffer();
  Transport transport = new BufferTransport(buffer);
  Protocol protocol = new BinaryProtocol(transport);

  struct.getAdapter().write(protocol, struct)
}

Since generated structs don't currently extend from or implement any interfaces, I don't imagine this would cause any breakage.

I'm happy to submit a PR that accomplishes this, I believe it would be pretty straight forward.

It would be great if Thrifty could provide something to decorate a Protocol. My usecase is using a multiplexed protocol like TMultiplexedProtocol.

An alternative to adding the implementation below would be to add an accessor method to the transport inside the decorated protocol (so that something equivalent to super(protocol.transport) could be achieved in other packages).

package com.microsoft.thrifty.protocol;

import okio.ByteString;
import java.io.IOException;

public abstract class DecoratingProtocol extends Protocol {

    private final Protocol concreteProtocol;

    public DecoratingProtocol(Protocol protocol) {
        super(protocol.transport);
        concreteProtocol = protocol;
    }

    @Override
    public void writeMessageBegin(String name, byte typeId, int seqId) throws IOException {
        concreteProtocol.writeMessageBegin(name, typeId, seqId);
    }

    @Override
    public void writeMessageEnd() throws IOException {
        concreteProtocol.writeMessageEnd();
    }

    @Override
    public void writeStructBegin(String structName) throws IOException {
        concreteProtocol.writeStructBegin(structName);
    }

    @Override
    public void writeStructEnd() throws IOException {
        concreteProtocol.writeStructEnd();
    }

    @Override
    public void writeFieldBegin(String fieldName, int fieldId, byte typeId) throws IOException {
        concreteProtocol.writeFieldBegin(fieldName, fieldId, typeId);
    }

    @Override
    public void writeFieldEnd() throws IOException {
        concreteProtocol.writeFieldEnd();
    }

    @Override
    public void writeFieldStop() throws IOException {
        concreteProtocol.writeFieldStop();
    }

    @Override
    public void writeMapBegin(byte keyTypeId, byte valueTypeId, int mapSize) throws IOException {
        concreteProtocol.writeMapBegin(keyTypeId, valueTypeId, mapSize);
    }

    @Override
    public void writeMapEnd() throws IOException {
        concreteProtocol.writeMapEnd();
    }

    @Override
    public void writeListBegin(byte elementTypeId, int listSize) throws IOException {
        concreteProtocol.writeListBegin(elementTypeId, listSize);
    }

    @Override
    public void writeListEnd() throws IOException {
        concreteProtocol.writeListEnd();
    }

    @Override
    public void writeSetBegin(byte elementTypeId, int setSize) throws IOException {
        concreteProtocol.writeSetBegin(elementTypeId, setSize);
    }

    @Override
    public void writeSetEnd() throws IOException {
        concreteProtocol.writeSetEnd();
    }

    @Override
    public void writeBool(boolean b) throws IOException {
        concreteProtocol.writeBool(b);
    }

    @Override
    public void writeByte(byte b) throws IOException {
        concreteProtocol.writeByte(b);
    }

    @Override
    public void writeI16(short i16) throws IOException {
        concreteProtocol.writeI16(i16);
    }

    @Override
    public void writeI32(int i32) throws IOException {
        concreteProtocol.writeI32(i32);
    }

    @Override
    public void writeI64(long i64) throws IOException {
        concreteProtocol.writeI64(i64);
    }

    @Override
    public void writeDouble(double dub) throws IOException {
        concreteProtocol.writeDouble(dub);
    }

    @Override
    public void writeString(String str) throws IOException {
        concreteProtocol.writeString(str);
    }

    @Override
    public void writeBinary(ByteString buf) throws IOException {
        concreteProtocol.writeBinary(buf);
    }

    @Override
    public MessageMetadata readMessageBegin() throws IOException {
        return concreteProtocol.readMessageBegin();
    }

    @Override
    public void readMessageEnd() throws IOException {
        concreteProtocol.readMessageEnd();
    }

    @Override
    public StructMetadata readStructBegin() throws IOException {
        return concreteProtocol.readStructBegin();
    }

    @Override
    public void readStructEnd() throws IOException {
        concreteProtocol.readStructEnd();
    }

    @Override
    public FieldMetadata readFieldBegin() throws IOException {
        return concreteProtocol.readFieldBegin();
    }

    @Override
    public void readFieldEnd() throws IOException {
        concreteProtocol.readFieldEnd();
    }

    @Override
    public MapMetadata readMapBegin() throws IOException {
        return concreteProtocol.readMapBegin();
    }

    @Override
    public void readMapEnd() throws IOException {
        concreteProtocol.readMapEnd();
    }

    @Override
    public ListMetadata readListBegin() throws IOException {
        return concreteProtocol.readListBegin();
    }

    @Override
    public void readListEnd() throws IOException {
        concreteProtocol.readListEnd();
    }

    @Override
    public SetMetadata readSetBegin() throws IOException {
        return concreteProtocol.readSetBegin();
    }

    @Override
    public void readSetEnd() throws IOException {
        concreteProtocol.readSetEnd();
    }

    @Override
    public boolean readBool() throws IOException {
        return concreteProtocol.readBool();
    }

    @Override
    public byte readByte() throws IOException {
        return concreteProtocol.readByte();
    }

    @Override
    public short readI16() throws IOException {
        return concreteProtocol.readI16();
    }

    @Override
    public int readI32() throws IOException {
        return concreteProtocol.readI32();
    }

    @Override
    public long readI64() throws IOException {
        return concreteProtocol.readI64();
    }

    @Override
    public double readDouble() throws IOException {
        return concreteProtocol.readDouble();
    }

    @Override
    public String readString() throws IOException {
        return concreteProtocol.readString();
    }

    @Override
    public ByteString readBinary() throws IOException {
        return concreteProtocol.readBinary();
    }

    @Override
    public void flush() throws IOException {
        concreteProtocol.flush();
    }

    @Override
    public void reset() {
        concreteProtocol.reset();
    }

    @Override
    public void close() throws IOException {
        concreteProtocol.close();
    }
}

A pathological-but-believable scenario might be:

http_api.thrift

enum StatusCode {
  NO_ERROR = 1,
  OK = 2,
  // etc
}

other_api.thrift

enum StatusCode {
  NO_ERROR = 1,
  SUCCESS = 2,
  // etc
}

client.thrift

struct Response {
  1: required http_api.StatusCode status = StatusCode.NO_ERROR;
}

There is certainly enough context in the IDL to know that the correct StatusCode is http_api.StatusCode, but presently Constant validation doesn't have enough info completely resolve the identifier StatusCode.NO_ERROR. Current behavior has the type resolved non-deterministically - that is, the first matching type to be linked is chosen - and soon we will force a compilation error suggesting that the identifier be fully-qualified.

Apache Thrift gets this right, incidentally.

Time passes, and bits rot. Lots of Guava things we use have standard-library equivalents in JDK 8, and newer versions of that library explicitly deprecate those things.

We should probably git er done before 1.0.0.

Currently, integration tests are unreliable on Travis CI.

The integration suite will, for each run, open a (local) server socket and a Thrifty client will connect and make RPC calls. The client will time out after 2 seconds, which happens on Travis for reasons unknown. The server doesn't do any real work, so it's likelier to be a timing issue (good) or something intrinsic to the CI environment (bad).

If the former, then it's just a matter of fixing the race. If the latter, then we need a different approach to integration testing.

We use the general form this.x == that.x || (this.x != null && this.x.equals(that.x)), which Android Studio dislikes. We should add @SupressWarnings("NumberEquality") to generated equals methods that use boxed Number types.

For example, say we have:

enum Foo { FOO = 1, BAR = 2 }
struct Baz {
  1: required Foo foo = Foo.FOO;
}

Let's say a client has this definition, but the server adds a new enum field QUX. Right now, clients receiving a Baz with the new enum value will crash. This is because the generated Foo.findByValue method returns null, and the adapter will attempt to set the field with that null value.

It seems wrong to attempt to overwrite a default value with null. Conversely, it also seems wrong to silently ignore incomprehensible data.

What to do?

Linker has had a stub for this forever. Currently, Thrifty will happily accept all manner of nonsensical service definitions, like oneway functions that return data. It will be relatively simple to add validation - just a matter of adding validation on Service and ServiceMethod. Validation should verify that:

For services:

• A service either has no base type, or if it does, that base type is also a service

For service methods:

• A service method has a non-void return type and is not marked one-way or it is marked one-way and has a void return type
• All exceptionTypes are, in fact, exceptions; this can be verified by using Linker#lookupSymbol(ThriftType) and checking the resulting StructType.

Most of the other Named subtypes (e.g. StructType, Constant) have validate(Linker) methods that can serve as an example.

Right now during linking we resolve TypeElement to ThriftType by collapsing multiple instances of the former into a single instance of the latter. This means that the annotations on single instances of TypeElement are not carried through to ThriftType.

A workaround for the problem of lost annotations can be seen in #43, but this is not ideal; it is fragile and not resilient to model changes.

A better solution would be to allow ThriftType to carry annotations. The first step is to eliminate the dependency on each unique thrift type having a single canonical ThriftType instance.

Some new additions to the Okio API (like Pipe) present in newer versions offer interesting possibilities. Can we take advantage without breaking consumers? Are there important fixes or improvements to be had?

Example:

namespace java test;
const bool YES = true
struct Test {
  1: required bool isYes = YES;
}

The default value for the generated field should be test.Constants.YES, but is false.

After linking (i.e. before a user of the parser API ever sees an instance of ServiceMethod), the returnType field is definitively set either to ThriftType.VOID or an actual type; it is never null and so we don't actually get any benefit from exposing it as an optional type.

This is a breaking change, but a good one to make.

We want it, if for no other reason than to log messages received.

We noticed an error like this when running code gen:

E: 'One' is not a member of enum type Foo: members=[One]

with testing_foo.thrift defined as:

namespace java testing 

enum Foo {
  One = 1
}

and testing_bar.thrift defined as:

namespace java testing

include 'testing_foo.thrift'

struct Bar {
  1: required testing_foo.Foo foo = Foo.One;
}

(This happens with required testing_foo.Foo foo = Foo.One and required testing_foo.Foo foo = One)

For reference required testing_foo.Foo foo = 1 seems to work fine.

It seems like the linking happens correctly, just the validation fails.

Some of the File/String things we do are kind of shaky, and would be better accomplished with the newer Path API.

We should support a compiler flag which, when set, causes generated structs to implement android.os.Parcelable

Google's truth testing library is a solid 7/10, not a huge win over Hamcrest. The reason to consider it now, though, is their compile-testing companion library that can assert on generated Java code. It would be interesting to see about using it.

Given how inefficient enums are on Android, would it be possible to leverage enumerated annotations instead of generating plain Java enums?

Given:

struct Foo {
  1: list<list<i32>> ints = [[]]
}

Thrifty generates:

class Builder implements StructBuilder<Foo> {
  // code code code

  public Builder() {
    this.ints = new ArrayList<List<List<Integer>>>();
    List<List<Integer>> list = new ArrayList<List<Integer>>();
    list.add(Collections.emptyList());  // BROKEN
    this.ints.add(list);
  }
}

This won't compile; the call to Collections.emptyList() needs type parameters.

Right now, we unconditionally write out int constants in base 10, so the following transformation happens:

const i32 FOO = 0xF

becomes

public static final int FOO = 16;

Should we preserve the hex-ness of integer constants? It would require threading extra state through from the parser to the code generator - a lot of work, for something that is at best a nice-to-have.

What, if anything, does Jigsaw mean for us? We should figure that out sometime before July.

I am a beginner to the open source community. I want to know how to import this project into Eclipse and build it. Thank you:).

It's hard to imagine anyone actually trying this, but it wouldn't hurt to have a check for this.

Observed Behaviour

The thrifty compiler fails when a Thrift IDL file declares includes using relative paths. So for example: there is a file Foo.thrift in the foo folder that depends on another file Bar.thrift in the bar folder and it declares the dependency like this: include ../bar/Bar.thrift. When running java -jar thrifty-compiler-0.2.0.jar ./foo/Foo.thrift ./bar/Bar.thrift the compiler will fail with the following exception:

Exception in thread "main" java.lang.AssertionError: All includes should have been resolved by now: /some/absolute/path/foo/../bar/Bar.thrift
    at com.microsoft.thrifty.schema.Loader.getAndCheck(Loader.java:274)
    at com.microsoft.thrifty.schema.Loader.resolveIncludedProgram(Loader.java:232)
    at com.microsoft.thrifty.schema.Program.loadIncludedPrograms(Program.java:232)
    at com.microsoft.thrifty.schema.Loader.loadFromDisk(Loader.java:145)
    at com.microsoft.thrifty.schema.Loader.load(Loader.java:107)
    at com.microsoft.thrifty.compiler.ThriftyCompiler.compile(ThriftyCompiler.java:178)
    at com.microsoft.thrifty.compiler.ThriftyCompiler.main(ThriftyCompiler.java:93)

Cause

The loadFromDisk method in the Loader class stores Programs using their canonical paths but getAndCheck is invoked using the absolute path built by the findFirstExisting method, which creates a file using parent and file paths when resolving paths according to the current location or include paths rules.

Fix

Invoking the getAndCheck using canonical paths instead of absolute ones.

The following struct fails to parse:

struct SomeRequest {
    /** Here's a comment. */
    1: required UUID clientUuid

    /**
     * Here's a longer comment.
     * One two lines.
     */
    2: optional string someOtherField

    /**
     * Here's another longer comment.
     * Also on two lines.
     */
    3: optional UUID someField

    /** Here's another single line that works fine. */
    4: optional UUID adminUuid
}

This fails with the following error:

trailing comment must be closed on the same line

Could you explain where the issue is in these comments? It seems like the formatting is fine, and the error message isn't really clear since I see several tests with multiline comments like this.

The exercise of implementing int enums, and realizing just how far-flung were the changes required, led to a realization that the codegen code is not very maintainable. The thing works, but is a jumble of hard-to-understand spaghetti.

General idea: try to encapsulate code generation within a model of the final Java output. That is, map from StructType -> StructClass, exposing TypeSpecs, FieldSpecs, etc. Hypothetically, we may be able to use these to clean up and centralize logic that has just exploded everywhere.

No idea if this is feasible, or if it represents an improvement in maintainability, but we ought to attempt something here.

Hi, whenever I attempt to run a unit test (ex.ThriftyCodeGeneratorTest), I get the following error message on my console (see screenshot). I am using Eclipse IDE. Any idea how I can fix this?

The following fails during parsing:

GetStuffResponse getStuff(
  1: required GetStuffRequest request;
) 
throws (
  1: NotAuthorizedException notAuthorized;
  2: NotFoundException notFound;
  3: InvalidRequestException invalidRequest;
)

but this passes:

GetStuffResponse getStuff(
  1: required GetStuffRequest request;
) throws (
  1: NotAuthorizedException notAuthorized;
  2: NotFoundException notFound;
  3: InvalidRequestException invalidRequest;
)

Specifically, it tries to read throws as an identifier rather than recognize it as a keyword continuing from the previous block. Not sure if this is a parser issue or if thrift specs require throws keywords to be on the same line following the previous block.

Nice work overall on the project! One thing I noticed missing in the library is support for thrift server, similar to TSimpleServer in the apache library. Are there plans to add those or are you assuming most thrift servers will be hosted on actual server and not on the Android client.

A specific use-case for this is when communicating with hardware from the Android device. The Android device may want to act as the server to retrieve data

As time passes, the distinction seems less and less useful.

ThriftType.UserType should subsume Named, and ThriftType in general should become enriched. This would fix all the fiddly Constant validation bugs and simplify lots of requested use cases from @hzsweers and friends.

We currently allow users to customize the generated names of fields with FieldNamingPolicy. It's already available in Program, where we convert the raw parse tree data, we just aren't yet threading it through to Service and ServiceMethod.

We should just add constructor parameters to those classes and pass the user-provided policy through.

After upgrading to 0.3.0 I'm getting this exception when running the compiler:

java.lang.UnsupportedOperationException: cannot unbox java.lang.String
        at com.squareup.javapoet.TypeName.unbox(TypeName.java:160)
        at com.microsoft.thrifty.gen.ThriftyCodeGenerator.buildConst(ThriftyCodeGenerator.java:926)
        at com.microsoft.thrifty.gen.ThriftyCodeGenerator.generateTypes(ThriftyCodeGenerator.java:231)
        at com.microsoft.thrifty.gen.ThriftyCodeGenerator.generate(ThriftyCodeGenerator.java:262)
        at com.microsoft.thrifty.gen.ThriftyCodeGenerator.generate(ThriftyCodeGenerator.java:167)
        at com.microsoft.thrifty.compiler.ThriftyCompiler.compile(ThriftyCompiler.java:217)
        at com.microsoft.thrifty.compiler.ThriftyCompiler.main(ThriftyCompiler.java:96)

This thrift file reproduces the issue

namespace java com.example.test

const string TEST = "test";

We only have hard failures in parsing/codegen at this point, but there are constructs in Thrift that are specified as warnings. For example, adding a namespace for an unrecognized language is specified to still parse, but with a warning:

namespace not.a.language com.foo.bar // This should be ignored and a warning should be shown

This involves:

1. Collecting warnings at parse time
2. Collecting warnings at load/link time
3. Emitting warnings in a sensible way.

Thinking further, a non-ad-hoc approach for emitting errors would be useful here.

The project's actually well-suited for this, all I had to do was make them and put them in the same package space where the normal ones would go. This would allow developers to take advantage of the wonderful parsing support you all have already written, while being able to generate their own models/services that fit their needs.

Now that we've had a production parser for months, let's investigate re-writing it all!

Motivations:

• Manual parsing is manual
• ANTLR would, in theory, do lots of the peekChar-type things for free (i.e., whitespace no longer a fiddly concern)
• With a parser generator, we could turn around bugs much more quickly

Questions:

• With a hand-rolled parser we have ultimate flexibility - case in point, trailing documentation. How might we do that in ANTLR?
• We have a public parser API; can we maintain it with ANTLR?

When marking a field as redacted, that field is always shown as hidden when the structure is printed. It would be useful if "null" was printed for cases where the value is unset. This allows you to at least differentiate between cases where the field was set or not when logging structures.

We need them.

Proposed api for generated methods:

Thrift:

service FooService {
  i32 bar(1: string arg)
}

corresponding Java:

public interface FooService {
  int bar(String arg);
  int bar(String arg, long timeout, TimeUnit unit);
}

Timeouts would be communicated via ServiceMethodCallback#onError(Throwable) - likely via a java.util.concurrent.TimeoutException.

The downside being, of course, that the number of service methods doubles. On the other hand, such methods are typically a small portion of the overall number of generated methods, so it may not be so impactful.