creditkarma / thrift-typescript Goto Github PK

The parser does not guarantee that default values are assigned properly. For example you could give a map<i32,string> the default value of 'foobar'. We should validate these assignments and give the client a useful error.

struct MyResponse { }

generates the following code:

export class MyResponse { constructor(args?: IMyResponseArgs) { } }

which with noUnusedParameters causes errors

I noticed in the proof-of-concept that a success boolean property was being added to structs but it doesn't seem like the thrift binary does that. What is that property used for?

TypeScript supports Promises but doesn't shim them; however, it does shim async/await. This would require a shim library to be loaded (I believe the thrift library already requires & exports the Q promise library).

In researching enum in TypeScript, I found that it is a subtype of number, which makes it very lax during type checking.

For example, the following is valid TypeScript:

enum Operation {
  ADD, SUBTRACT
}

const op: Operation = 4; // Invalid enum value

Should we use the enum type to generate thrift enum definition or should we do something else?

I'm doing some more research into Interface creation for Structs (and other data types) to implement. These interfaces can also be used in the constructor as the typings for args, which fixes a problem I was having with nested container types.

I'm currently thinking the names should just be the Struct name with "Interface" appended. e.g. class MyStruct implements MyStructInterface @nnance thoughts?

On the call with @ian-harshman-ck, I realized that include syntax was only being handled for the top-level file. This needs to be handled recursively for all includes.

We could either use the standard js namespace or maybe a ts namespace? I guess we could use a js.ts or js.typescript if we wanted to be verbose.

Someone submitted typings to the thrift-parser project but I'm not sure how thorough they are. I need to investigate and use them where appropriate (or fix if they are missing things).

Things like: Interfaces/arguments parameter/if-statement for Structs that have no properties

If we decide to use Map/Set data types (#9 and #17), we should probably switch to forEach iteration to normalize iteration behavior between Map/Set/arrays. It would also make a lot of temp variable assignment cleaner.

A lot of my open questions seem to be around features that would make things much simpler (Promises, Maps, etc). What environments are we targeting?

Should we test the result of toAST() is a specific AST shape or should we pass the result into a printer and compare the output text string?

I'm open to other options also.

I some use cases we need to have a generic Thrift Struct Interface defined in typings and added to the code generator so we can identify something as a Thrift Struct.

Once this syntax is implemented in thrift-parser, this module will need to resolve them.

A few decisions to make:

• Should it read the includes by default or only when an option is set (-r flag in the traditional thrift parser)?
• Should separate output files be generated or combined into one? Ref #14
• Should thrift-typescript or thrift-parser be responsible for resolving the location of the file? Currently, I'm resolving in thrift-parser but that might be wrong. This one has been answered by upstream; it seems that thrift-typescript will be doing the resolution.

Looks like it's currently outputting TypeScript source code that then needs to be compiled again. Is this desired? It seems like it would increase the friction for usage.

This is something that stems from the original Apache implementation, but I think it could be improved upon.

In the generated code, thrift structs are currently populated via the read instance method:

const struct = new MyStruct()
struct.read(inputProtocol)

This means that the MyStruct's constructor arg has to be optional, otherwise we wouldn't be able to call the empty constructor. Unfortunately this makes constructors less type-safe for consumers. Even if MyStruct has a bunch of required fields...

struct MyStruct {
  1: required i32 field1
  2: required string field2
   ...

... the constructor can still be called with no arguments because the generated code requires it.

const struct = new MyStruct() // compiles, but should require fields

Alternative Approach

The read methods can be static. Something like

const struct = MyStruct.read(input)

This would free up the constructor signature to require a valid set of input arguments, which would result in required fields being type-checked (If a struct has no fields, we could make an exception in that case).

class MyStruct {
    constructor(args: IMyStructArgs) {...}  // constructor args are no longer optional
}

I've POC'd this by manually editing generated code and it seems to work just fine, though I haven't looked at memory / time complexity.

Anyway, not a high priority, just some food for thought.

When thinking about this effort, I had envisioned transforming the Thrift AST generated by thrift-parser to a JS AST and then use a tool to output the generated JS. I see that a lot of work has been done with handlerbars templates and I'm not sure it's worth it to move away from that. @nnance any thoughts?

The way we are currently resolving:

struct Embed {};

typedef Embed myEmbed;

struct MyStruct {
    1: myEmbed embedded
}

is to:

export type myEmbed = Embed;

export class Embed {};

export class MyStruct {
    ...
    embedded?: Embed; // myEmbed was resolved to Embed
    ...
}

This is resolving the alias to an alias down to the simplest reference. Do we want to keep the myEmbed type alias in the generated output when this happens?

service Caluculator {
i32 add(1: i32 left, 2: i32 right)
i32 subtract(1: i32 left, 2: i32 right)
}

import { Calculator } from './codegen/calculator'
two word type error

While having i64 fields have type number | Int64 is best for clients it complicates handlers for consumers.

@nnance While trying to understand the way node's thrift client library does send/receive for services, I found that it does some really hacky stuff (they even have a TODO to fix it) that would likely be going away. Is it worth the effort to do code generation that interops with their current implementation?

It seems that it's possible to alias an enum in typescript but it can't be referenced from outside the module.

e.g.

export namespace MyThing {
    export type enumAlias = MyEnum;
    export type anotherAlias = MyEnum;
    export enum MyEnum {
        ADD,
        SUBTRACT,
        MULTIPLY,
        DIVIDE
    }
}

MyThing.enumAlias.ADD; // [ts] Property 'enumAlias' does not exist on type 'typeof MyThing'.

What should we do? Ref #30

We should add a docblock header to the top of each source file so we can trace generated source back to a particular version of this tool.

Something like:

/**
 * Generated by @creditkarma/thrift-typescript
 * Version 0.0.2
 */

The parser collects all comments as leading comments.

Should it overwrite them or error if they already exist?

For the static analysis phases (resolver, validator) we should report errors similar to how it is being done in the parser. Don't throw on first error, collect errors and print errors with the line of source that caused the error, highlighting the problem token.

tslint should probably be run as part of the test suite (and I need to get it working in my editor).

Once it's being used, I need to fix lint errors.

namespace * MyNamespace

is recommended in https://www.manning.com/books/programmers-guide-to-apache-thrift-cx (please ignore the cancelled message) and compiles fine using the namespace in all languages (that I've tried) using the Apache Thrift code generator.

With thrift-typescript I encounter:

    Parse Failure: 2 errors found:


    Scan Error:

    Message: Unexpected token: *

    4 | namespace * MyNamespace
                  ^


    Parse Error:

    Message: Unable to find name identifier for namespace

    4 | namespace * MyNamespace

I can bypass it by replacing * with js but then I am going to have to duplicate the namespace line for each language I want to support.

Is this an approach that thrift-typescript could support or is my current approach actually a bad idea in practice?

The thrift binary outputs a bunch of different files (one *_types.js per thrift file and one file per service). Should we generate a single file instead? It seems like it would be easier from an import/usage perspective.

As discussed in #41 - this would cleanup some of the logic.

Just found that Enums will have to be resolved for types.

I am wondering if we should generate the output with assumptions about how the CK team uses the client library.

The best example I found so far is:

// The Thrift tutorial teaches Struct usage like:
var work = new Work();
work.op = tutorial.Operation.DIVIDE;
work.num1 = 1;
work.num2 = 0;

// But the Constructor can (and must for required) take the properties upon construction
var work = new Work({
  op: tutorial.Operation.DIVIDE,
  num1: 1,
  num2: 0
});

Since we are leveraging TypeScript, we can leverage Interfaces as our constructor argument and avoid a lot of if statements in the constructor assignment. However, maybe this pushes the wrong assumptions or we don't want to push them. @nnance thoughts?

function mkdir(dirPath) {
console.log("dirPath:" + dirPath + " " + path.sep);

var parts = dirPath.split(path.sep).filter((val) => val !== '');

// i add this code it can run in windows
if (os.platform() === 'win32') {
parts = parts.slice(1)
}
if (parts.length > 0 && path.isAbsolute(dirPath)) {
createPath(parts, rootDir());
}
else if (parts.length > 0) {
createPath(parts, process.cwd());
}
}

I'll use this simple service for reference.

service MyService {
   Response call(1: Request request)
}

In this case, the tool generates something like

export interface IMyServiceHandler<Context> {
    call: (request: Request, context: Context) => Response;
}

Notice that the return type does not allow for promises to be returned. I believe we can make the return type Response | Promise<Response> as a non-breaking change.

The thrift binary doesn't include the read or write methods on their output types but they have to be publicly accessible to be used within the application.

For thrift files whose dependency tree (the file and all of the includes) can be large node can sometimes run out of memory while running the generator.

The solution for this is to build a file, save it and deallocate as much data as possible before moving on to the next file. Currently we are keeping all of the file's data after the file has been rendered, keeping all data until the entire dependency tree has been rendered.

A temporary solution for this is to increase the memory allocated for node when running the generator:

$ node --max_old_space_size=8192 dist/main/bin/index.js <options>

If we decide to use Maps for #9, we might also want to use Sets too.

I'm thinking their should be some sort of normalization and validation done before the code generation. The reason I think this is beneficial is to resolve custom types and find out if anything used wasn't declared in the definition.

If this makes sense to have, should both be done in a single pass or should it be split into 2 separate phases?

In the genrated Processor code, the process() function looks like the following:

public process(input: thrift.TProtocol, output: thrift.TProtocol, context: Context): Promise<Buffer> {
            return new Promise<Buffer>((resolve, reject): void => {
                const metadata: thrift.IThriftMessage = input.readMessageBegin();
                const fieldName: string = metadata.fieldName;
                const requestId: number = metadata.requestId;
                const methodName: string = "process_" + fieldName;
                switch (methodName) {
                    case "process_foo": {
                        resolve(this.process_foo(requestId, input, output, context));
                    }
                    case "process_bar": {
                        resolve(this.process_bar(requestId, input, output, context));
                    }
                    default: {
                        input.skip(thrift.TType.STRUCT);
                        input.readMessageEnd();
                        const errMessage = "Unknown function " + fieldName;
                        const err = new thrift.TApplicationException(thrift.TApplicationExceptionType.UNKNOWN_METHOD, errMessage);
                        output.writeMessageBegin(fieldName, thrift.MessageType.EXCEPTION, requestId);
                        thrift.TApplicationExceptionCodec.encode(err, output);
                        output.writeMessageEnd();
                        resolve(output.flush());
                    }
                }
            });

You can notice there is no "break;" sentence in the case clauses.
This causes all the handler functions to actually be called, as long as they have no arguments (if they do, argument parsing fails, and the function is not called).

The result is that having functions without arguments in a service is nearly unusable.

While implementing Exceptions, I found that TypeScript doesn't like extending with JS constructors (such as Thrift.TException) but by installing @types/thrift, that constructor is declared as a proper class and TypeScript stops complaining.

How should we handle needing this dependency installed?

Currently breaking due to being unable to call .toUpperCase on the type because it is an object but they have much different logic so it needs to get implemented.

@ian-harshman-ck Putting this here for tracking.

I found a bug with the output in the JS that the thrift binary outputs.

Sample:

map<bool, string> = {false: 'test'}

When this is serialized, it will become {true: 'test'} on the server because false is resolved as a string and checked for truthiness.

if i have something like this:

struct SomeStruct {
  1: optional i16 bpm = 0
  2: optional byte confidence = -1
}

The parser complains:

Parse Failure: 1 errors found:


Parse Error:

Message: FieldType expected but found: IntegerLiteral

174 |   2: optional byte confidence = -1
                                      ^

/usr/local/lib/node_modules/@creditkarma/thrift-typescript/dist/main/utils.js:25
            throw new Error('Unable to parse source');

@ian-harshman-ck @nnance @kevinbgreene do any of you know if this is actually part of the wire spec?

While reviewing container types, I've found some inconsistencies between the thrift.js clientside library and the thrift library on npm. Would it be better to output code that utilizes the npm version of the library?