GithubHelp home page GithubHelp logo

creditkarma / thrift-parser Goto Github PK

View Code? Open in Web Editor NEW
91.0 20.0 21.0 417 KB

A Thrift Parser built in TypeScript that generates a TypeScript AST that retains the Thrift grammar

License: Apache License 2.0

JavaScript 1.66% TypeScript 90.74% Thrift 7.60%
thrift typescript microservices rpc nodejs

thrift-parser's Introduction

TypeScript Thrift Parser

A parser for Thrift written in TypeScript. The resulting AST can be used to codegen JavaScript from a Thrift file, or just to inspect the Thrift structure.

Usage

A successful parse returns a ThriftDocument object. An unsuccessful parse returns a ThriftErrors object.

import { parse, ThriftDocument } from '@creditkarma/thrift-parser'


const rawThrift: string =`
  struct MyStruct {
    1: required i32 id
  }
`;

const thriftAST: ThriftDocument | ThriftErrors = parse(rawThrift);

switch(thriftAST.type) {
  case 'ThriftDocument':
    // Do something with valid AST
  case 'ThriftErrors':
    // Report or recover from errors
}

You can also use Thrift Parser from the command line or npm scripts. When using from the command line the generated AST is saved to file as JSON.

$ thrift-parser --rootDir thrift --outDir thrift-json --fastFail false some_file.thrift

In this usage there are three options:

  • --rootDir: where to initiate file search and save
  • --outDir: relative to rootDir where to save output files
  • --fastFail: If true fail on first error encountered

Build

$ npm install
$ npm run build

Test

$ npm test

AST Structure

The root of the returned AST is either a ThriftDocument (successful parse) or a ThriftErrors (unsuccessful parse).

ThriftDocument

{
  type: "ThriftDocument",
  body: Array<ThriftStatement>
}

ThriftErrors

{
  type: "ThriftErrors",
  errors: Array<ThriftError>
}

ThriftError

A descriptor of what went wrong while parsing the specified Thrift source.

{
  type: "ParseError" | "ScanError",
  message: string,
  loc: TextLocation
}

Thrift Statements

Thrift Statements represent each of the main constructs that can be defined in Thrift source.

NamespaceDefinition

namespace <identifier> <identifier>
{
  type: "NamespaceDefinition",
  scope: Identifier,
  name: Identifier
}

IncludeDefinition

include '<path>'"
{
  type: "IncludeDefinition",
  path: StringLiteral
}

TypedefDefinition

typedef <field-type> <identifier>
{
  type: "TypedefDefinition",
  name: Identifier,
  definitionType: FieldType
}

ConstDefinition

const <field-type> <identifier> = <initializer>
{
  type: "ConstDefinition",
  name: Identifier,
  fieldType: FieldType,
  initializer: ConstValue,
}

EnumDefinition

enum <identifier> { <members> }
{
  type: "EnumDefinition",
  name: Identifier,
  members: Array<EnumMember>
}

StructDefinition

struct <identifier> { <fields> }
{
  type: "StructDefinition",
  name: Identifier,
  fields: Array<FieldDefinition>
}

UnionDefinition

union <identifier> { <fields> }
{
  type: "UnionDefinition",
  name: Identifier,
  fields: Array<FieldDefinition>
}

ExceptionDefinition

exception <identifier> { <fields> }
{
  type: "ExceptionDefinition",
  name: Identifier,
  fields: Array<FieldDefinition>
}

ServiceDefinition

service <identifier> (extends <identifier>)? { <functions> }
{
  type: "ServiceDefinition",
  name: Identifier,
  extends: Identifier | null,
  functions: Array<FunctionDefinition>
}

Viewing with ASTExplorer

ASTExplorer is a web app for visualizing ASTs. You type source. It shows you the resulting syntax tree based on the parser you've selected. I've included the integrations for this parser. To get that up and running you'll need to clone ASTExplorer.

$ git clone https://github.com/fkling/astexplorer.git
$ cd astexplorer/website
$ npm install

You will now need to install thrift-parser for ASTExplorer

$ npm install @creditkarma/thrift-parser

Cool, now we need to copy some stuff into the ASTExplorer project.

If the ASTExplorer project and the @creditkarma/thrift-parser project are siblings you can type this into the temrinal (from astexplorer/website)...

$ cp -r ../../thrift-parser/astexplorer/thrift ./src/parsers/thrift

You'll now need to build ASTExplorer and start the server

$ npm run build
$ npm start

By default this will start ASTExplorer on localhost:8080

There is a dropdown to select the language you want to use, choose 'Thrift IDL'

Note: I have had some trouble getting npm run build to work. However, the watch build is much more reliable.

$ npm run watch

Then in another terminal window run start.

$ npm start

Contributing

For more information about contributing new features and bug fixes, see our Contribution Guidelines. External contributors must sign Contributor License Agreement (CLA)

License

This project is licensed under Apache License Version 2.0

thrift-parser's People

Contributors

daniel-chen-ck avatar danielchencreditkarma avatar kevin-greene-ck avatar kevinbgreene avatar nnance avatar pd4d10 avatar volodymyr-ladnik-ck avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

thrift-parser's Issues

ThriftDocument body has definitions out of order

Hi creditkarma!

Given this example thrift, the body of the ThriftDocument structure is not in order of how the document is laid out.

/* example thrift */
struct Note {
  1: string message
}

struct Measure {
  1: i32 value
}

union DataTypes {
  1: Note note
  2: Measure measure
}

struct Annotation {
  1: string id
  2: DataTypes data
}

using the thrift-parser

$ thrift-parser example.thrift
$ cat thrift-json/test.json | jq '.body[].type'
"StructDefinition"
"StructDefinition"
"StructDefinition"
"UnionDefinition"

This did not match my expectation that the body would be sorted based on how they were defined in the document.

Is this behavior expected? Let me know what you think. I'm willing to make an MR, and there is a simple workaround by sorting the body by the starting line number so its not blocking anything.

Finally, thank you for a great thrift parser; I've been enjoying working with it! ❤️

Support Web-only use case

I'm building a tool similar to AST Explorer. It doesn't have a backend and everything runs in the browser. However, there are code in debugger.ts and index.ts referencing Node.js libraries. This causes errors in the browser environment as these libraries cannot be found. Those code are used to read Thrift code from files.

Is there a workaround? Right now I just comment out those code my usage doesn't need them. But is there a better way I can solve this issue?

Support using namespace *

The Thrift spec allows you to use namespace *, which means that the IDL applies to all target languages: https://thrift.apache.org/docs/idl#namespace

The parser doesn't currently allow for that, because it treats NamespaceScope as an Identifier, which only allows for alpha numeric characters, dots, and underlines: [a-zA-Z0-9\._].

The solution would be to create a NamespaceScope type, which allows for Identifier | *. This would be a breaking change, though. Because it would changes the AST output.

I'm happy to work on a PR for that, if you are ok with the breaking change, and you don't have any other alternative solution to fix this issue.

Add a type-checking pass to the parser

Source like

const i32 test = "hi there"

Should cause an error.

As should using Identifiers that haven't been defined. This would require resolving includes as part of the parsing phase.

parse throws error

version: 1.1.5

a.thrift

/**奖**/
/*****/
struct Hello {
    /*commentthatcauseerror*/
    7: optional i32 A,
}

run thrift-parser a.thrift will throw error like this Message: Invalid start to Thrift statement 7.

token is

{"type":"IntegerLiteral","text":"7","loc":{"start":{"line":5,"column":5,"index":65},"e
nd":{"line":5,"column":6,"index":66}}}

It seems that parser did not recognize continuous comments which contains non-english words.

Allow disabling of organizer

I noticed that the AST that the thrift parser outputs is organized by type. Would you be open to adding a parser option that lets you disable the organization?

Annotation support

How do you folks feel about supporting annotations, e.g.:

struct foo {
  1: i32 bar ( presence = "required" );
} (
  annotation = "Foo",
  another.annotation = "Bar"
)

Currently errors are produced when annotations are encountered.

Tests from #13 lost in rebase

Opening this to track. During the #14 rebase, it seems the tests from #13 were lost. Tests for comments in structs/services should be added back.

Getting error "'thrift' is declared but its value never read" when using typedef

This happened when I were using typedef i64 EpocMillis.

/* tslint:disable */
/* eslint-disable */
/*
 * Autogenerated by @creditkarma/thrift-typescript v3.7.4
 * DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING
*/
import Int64 = require("node-int64");
import * as thrift from "thrift";
export type EpocMillis = Int64;

Working normally with typedef string blahblah or typedef list<common.TError> Errors

Guess this issue only appear with this kind of typedef.

Add comments to the AST?

Can we add the comments to the AST? I think the Thrift binary actually does something with them but we currently don't because the other parser didn't include them in the output.

Parse const map with error

when i parse a struct like this:

enum WithdrawTypeEnum {
    BANK, 
    WECHAT,
    ALIPAY
}

const map<WithdrawTypeEnum, string> WITHDRAW_TYPE_MAP = {
    WithdrawTypeEnum.BANK : 'bank',
    WithdrawTypeEnum.WECHAT : 'weixin'
    WithdrawTypeEnum.ALIPAY : 'alipay'
}

it return a error of 'Closing brace missing from map definition'

Error when running command

When I run the command
thrift-parser --rootDir thrift --outDir thrift-json --fastFail false struct.thrift

I get this error:
internal/fs/utils.js:314
throw err;
^

Error: EINVAL: invalid argument, mkdir 'C:\c:'
at Object.mkdirSync (fs.js:1009:3)
at createPath (C:\Users\1322919\AppData\Roaming\npm\node_modules@creditkarma\thrift-parser\dist\main\bin\mkdir.js:17:12)
at Object.mkdir (C:\Users\1322919\AppData\Roaming\npm\node_modules@creditkarma\thrift-parser\dist\main\bin\mkdir.js:29:9)
at C:\Users\1322919\AppData\Roaming\npm\node_modules@creditkarma\thrift-parser\dist\main\bin\index.js:17:13
at Array.forEach ()
at Object. (C:\Users\1322919\AppData\Roaming\npm\node_modules@creditkarma\thrift-parser\dist\main\bin\index.js:11:29)
at Module._compile (internal/modules/cjs/loader.js:1085:14)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
at Module.load (internal/modules/cjs/loader.js:950:32)
at Function.Module._load (internal/modules/cjs/loader.js:790:14) {
errno: -4071,
syscall: 'mkdir',
code: 'EINVAL',
path: 'C:\c:'
}
struct.txt
I've attached my thrift file as a txt file (thrift files aren't supported as attachments).

Doesn't parse "throws" syntax

Tried to parse:

service Service1 {
    bool ping() throws (1: Exception1 user_exception, 2: Exception2 system_exception)
}

And it fails. When the "throws" syntax is removed, it succeeds.

We need to handle i64 as a value for const and fields.

Right now all numeric values are being parsed using parseInt or parseFloat. We should construct Int64 as the value for i64 fields. We're not handling default values in Scrooge at the moment, but this needs to be fixed.

when the source is 'structt', a bug would be triggered

code:

import * as thrift from '@creditkarma/thrift-parser';
const document = thrift.parse(`structt`);
console.log('doc', document)

error:

TypeError: Cannot read property 'start' of undefined
(node_modules/@creditkarma/thrift-parser/src/main/debugger.ts:69:41)

generate schemas only for referenced struct

When I generated schemas from thrift file, It parse the file and then iterate on each include statement to generate schemas for each included files. As included files may contains other schemas as well which are not referenced in parent file, So I want that it should only generate schemas for referenced structs.

Improve debugging

Right now when there's a scan or parse error things just blow up. We should catch error, print a description of what happened and print the line of source that caused the error.

Parse map const error

My thrift:

...
typedef map<MetadataKey, MetadataValue> Metadata
...

struct X {
  15: Metadata metadata = {}
}

will throw UnhandledPromiseRejectionWarning: TypeError: Type map expected.

My thrift was parsed and worked with Java.
This may relate to many some issue about map const as well.

Thanks.

[discussion] keep block comments' original content?

Take part of this file as example:

/*
 * with the License. You may obtain a copy of the License at
 *
 *   http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 */
 
namespace c_glib TTest

The comment block is parsed to:

        {
          "type": "CommentBlock",
          "value": [
            "with the License. You may obtain a copy of the License at",
            "*",
            " http://www.apache.org/licenses/LICENSE-2.0",
            "*",
            " Unless required by applicable law or agreed to in writing,"
          ]
        }

The indent of http://www.apache.org/licenses/LICENSE-2.0 is intended, which is lost. Maybe it is better to keep block comments' original content?

Trailing comments are lost

Like this:

namespace js test // trailing comment

Output:

{
  "type": "ThriftDocument",
  "body": [
    {
      "type": "NamespaceDefinition",
      "scope": {
        "type": "Identifier",
        "value": "js",
        "loc": {
          "start": {
            "line": 1,
            "column": 11,
            "index": 10
          },
          "end": {
            "line": 1,
            "column": 13,
            "index": 12
          }
        }
      },
      "name": {
        "type": "Identifier",
        "value": "test",
        "loc": {
          "start": {
            "line": 1,
            "column": 14,
            "index": 13
          },
          "end": {
            "line": 1,
            "column": 18,
            "index": 17
          }
        }
      },
      "comments": [],
      "loc": {
        "start": {
          "line": 1,
          "column": 1,
          "index": 0
        },
        "end": {
          "line": 1,
          "column": 18,
          "index": 17
        }
      }
    }
  ]
}

Doesn't parse semicolon in a service

Just tried to parse:

namespace java com.twitter.finagle.example.thriftjava

service Calculator {
  i32 add(1: i32 x, 2: i32 y);
}

And it failed. When I remove the semicolon after the add definition, it parses correctly.

Doesn't parse comments

Just tried to parse:

struct MyStruct {
    1: required i32 id,
    2: required bool field1,
    # 3: required string field,
    4: required i16 field,
}

And it failed. I removed the commented line and it succeeds.

empty BlockComment breaks parser

If you have this IDL:

/* */
const string asdf = 'sdf'

The AST will return an empty document:

{
  "type": "ThriftDocument",
  "body": []
}

Another example:

/* */
some string
*/
const string test = 'asdf'

You get:

{
  "type": "ThriftDocument",
  "body": [
    {
      "type": "ConstDefinition",
      "name": {
        "type": "Identifier",
        "value": "test",
        "loc": {
          "start": {
            "line": 3,
            "column": 14,
            "index": 34
          },
          "end": {
            "line": 3,
            "column": 18,
            "index": 38
          }
        }
      },
      "fieldType": {
        "type": "StringKeyword",
        "loc": {
          "start": {
            "line": 3,
            "column": 7,
            "index": 27
          },
          "end": {
            "line": 3,
            "column": 13,
            "index": 33
          }
        }
      },
      "initializer": {
        "type": "StringLiteral",
        "value": "asdf",
        "loc": {
          "start": {
            "line": 3,
            "column": 21,
            "index": 41
          },
          "end": {
            "line": 3,
            "column": 27,
            "index": 47
          }
        }
      },
      "comments": [
        {
          "type": "CommentBlock",
          "value": [
            "some string"
          ],
          "loc": {
            "start": {
              "line": 1,
              "column": 1,
              "index": 0
            },
            "end": {
              "line": 2,
              "column": 15,
              "index": 20
            }
          }
        }
      ],
      "loc": {
        "start": {
          "line": 3,
          "column": 1,
          "index": 21
        },
        "end": {
          "line": 3,
          "column": 27,
          "index": 47
        }
      }
    }
  ]
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.