bfergerson / arthur

Semantic language-agnostic source code schema

Arthur

Arthur is a semantic language-agnostic UAST (universal abstract syntax tree) schema generator which uses source code as input and outputs unilingual and omnilingual ontologies derived from those language(s). Arthur parses source code using Babelfish and constructs the observed schema for use in a Grakn knowledge graph.

Schemas

Omnilingual Schema

Languages	Segments
Bash, C++, C#, Go, Java, JavaScript, PHP, Python, Ruby	Arthur_Omnilingual_Base_Structure.gql Arthur_Omnilingual_Semantic_Roles.gql

Unilingual Schemas

Language	Segments
Bash	Arthur_Bash_Base_Structure.gql Arthur_Bash_Semantic_Roles.gql
C++	Arthur_Cplusplus_Base_Structure.gql Arthur_Cplusplus_Semantic_Roles.gql
C#	Arthur_Csharp_Base_Structure.gql Arthur_Csharp_Semantic_Roles.gql
Go	Arthur_Go_Base_Structure.gql Arthur_Go_Semantic_Roles.gql
Java	Arthur_Java_Base_Structure.gql Arthur_Java_Semantic_Roles.gql
JavaScript	Arthur_Javascript_Base_Structure.gql Arthur_Javascript_Semantic_Roles.gql
PHP	Arthur_Php_Base_Structure.gql Arthur_Php_Semantic_Roles.gql
Python	Arthur_Python_Base_Structure.gql Arthur_Python_Semantic_Roles.gql
Ruby	Arthur_Ruby_Base_Structure.gql Arthur_Ruby_Semantic_Roles.gql

Supported Concepts

Structural

Conditional

• If/ElseIf/Else
• Switch/SwitchCase

Exception

• Try/Catch/Finally

Loop

• ForLoop
• ForEachLoop
• WhileLoop
• DoWhileLoop

Operator

Logical

• AndOperator
• OrOperator

Misc

• TernaryOperator

Relational

• RelationalOperator

Compare

• IsEqualOperator/IsNotEqualOperator
• IsEqualTypeOperator/IsNotEqualTypeOperator

Define

• DeclareVariableOperator
• InitializeVariableOperator

Misc

• Child
• Function
• InternalRole
• Language
• Literal
• Multi
• Name
• Role
• Token
• Type
• Wildcard

arthur's Issues

File parse time limit

Update StructureFilters to use semantic roles in determing predicate

Currently, the StructureFilter sub-classes all use the internalType/token in determining if the predicate passes or fails. An example of this would be:

class FunctionFilter extends StructureFilter<FunctionFilter, Void> {

    private static final Set<String> functionTypes = new HashSet<>()
    static {
        functionTypes.add("def") //ruby
        functionTypes.add("FuncDecl") //go
        functionTypes.add("MethodDeclaration") //java
        functionTypes.add("FunctionDeclaration") //js
        functionTypes.add("Stmt_Function") //php
        functionTypes.add("FunctionDef") //python
    }

    @Override
    boolean evaluate(SourceNode node) {
        return node != null && node.internalType in functionTypes
    }
}

This results in checking for a specific type per-language. These need to be re-written so the semantic roles are used instead.

Skip adding semantic roles to JavaVariableDeclarationFragmentArtifact

Dynamic attribute types

Ability to name functions

Provide segmented schemas

So you can install different parts of the ontology separately.
Example:

• Entities and attributes
• Entities, attributes, base roles
• Entities, attributes, full roles

Clean up JavaNaming

Roles to indicate multiple roles

Need to look into these roles:

NullLiteralArtifact sub JavaSourceArtifact
	# Semantic
	plays NULL
	plays LITERAL
	plays EXPRESSION
	plays BINARY
	plays RIGHT
	plays CALL
	plays POSITIONAL
	plays ARGUMENT
	plays ASSIGNMENT
	plays IF #null = if?
	plays ELSE
	plays THEN
	plays FUNCTION
	plays BODY
	plays LEFT;

How a null literal can play the role of an IF? Pretty sure it was simply playing a role inside an IF among a couple other roles which signify more concretely what the null literal was doing. Long story short roles of roles may need to be introduced to bring semantic roles back to the realm of being semantic. Something like 'NULL_IF_LITERAL_POSITIONAL_ARGUMENT_EXPRESSION` to mean this artifact can be used as an argument to an if expression.

Switch semantic role relationship/role

So entities plays WHILE instead of plays IS_WHILE

Impressive JavaDocs/GroovyDocs

Definitely lacking in the JavaDocs/GroovyDocs department

Upgrade to Grakn 1.5

Rename project to Authur/Arthor/Arthur/rthor

Add error handling to parse response

Ability to resume observations

Each filter should be able to accept/reject

Rewrite filters so each can accept or reject by their desired predicate. Shouldn't need WhitelistRoleFilter/BlacklistRoleFilter and should have same ability for TypeFilter, etc.

has_statements missing order index

Order of child elements isn't important in all relationships but index should be included in those which it is

Add C# language

• Add to SourceLanguage
• Implement Naming
• Implement Literal
• Write tests

All/any match functionality on RoleFilter

Remove possible semantic roles

Won't scale and is just not an elegant solution. Ideally just build the actual semantic roles more.

Rename roles to _role

Tests for literals

The purpose of this ticket is to ensure StructureLiteral and its descendants properly handle the more complex of literal types.

Below I have some for Java. I doubt these are handled correctly. Tests will need to be made to show that Arthur handles them correctly. Ideally, some complex examples from other supported languages should be found and tests added for them too.

Configurable schema

Ability to provide entities, attributes, roles, etc that you wish to capture and schema is generated to hold that data

Create neo4j/TinkerGraph schema/compatible output

Common language-agnostic filters

Be able to filter by any language on common concepts such as:

• operators
  • and operator
  • or operator
  • ternary / Elvis operator
  • null check operator
• conditionals
  • if
  • else
  • else if
• loops
  • for
  • foreach
  • while
  • do while
• switch/case
• try/catch

Make mulit-threaded at source file level

Correct attribute data type for literals

ElvisOperatorFilter

Implement StructureFilters for:

• https://en.wikipedia.org/wiki/Elvis_operator
• https://en.wikipedia.org/wiki/Safe_navigation_operator

Create tests to prove the filters work.

uuid isn't needed

Fill in all the todo: description

Travis-ci setup

Parent/child root relationship

Add Bash language

• Add to SourceLanguage
• Implement Naming
• Implement Literal
• Write tests

Include function arguments in function name

There are currently no tests for StructureNaming based classes. This means that a lot of function names are going to be returned incorrectly.

The goal here would be to start with testing JavaNaming with a simple function functionName(int var) and confirming that works. Then moving on to more complex function names (var args, generics, etc).

The JavaNaming class is also unnecessarily redundant so if you see something that looks like it could be refactored then it's likely a good idea to go for it.

Fill in all the todo: description

Add C++ language

• Add to SourceLanguage
• Implement Naming
• Implement Literal
• Write tests

Duplicate role in omnilingual schema

NullLiteralArtifact sub OmnilingualSourceArtifact
	# Semantic
	plays CALL
	plays NULL
	plays LEFT
	plays LITERAL
	plays BINARY
	plays FUNCTION
	plays ARGUMENT
	plays ASSIGNMENT
	plays ELSE
	plays RIGHT
	plays THEN
	plays IF
	plays EXPRESSION
	plays BODY;

JavascriptNullLiteralArtifact sub NullLiteralArtifact
	# Structural
	plays has_leading_comments_relation
	# Semantic
	plays VALUE
	plays MAP
	plays INITIALIZATION
	plays BOOLEAN;

JavaNullLiteralArtifact sub NullLiteralArtifact
	# Semantic
	plays POSITIONAL; #dupe

Replace println with log.info

Replace any output to stdout with log.info()

Necessary schema

Ability to point at local/remote repo and create only the necessary schema to store that repo

Rename version

Ability to turn filter logic into graph traversal query

Missing is_ roles

Add is_child/is_parent to SourceArtifact

Rename floatValue to doubleValue

Implement StructureType

This class would be used to get a more strongly typed version of SourceNode. For example, instead of a SourceNode with an internal type of For there would be a ForType which allows for more context around the node.

These would also be the results of the various filters instead of the SourceNode(s) they currently return.

Get rid of actual semantic roles

Make the tests more language-agnostic

When I first started making the tests I just copied one test to the next. This resulted in many tests which look almost the same but scans a different file. I started merging tests down so they look like this:

@Test
void equalOperator_Java() {
    assertEqualOperatorPresent(new File("src/test/resources/same/operators/Operators.java"),
            "Operators.")
}

@Test
void equalOperator_Javascript() {
    assertEqualOperatorPresent(new File("src/test/resources/same/operators/Operators.js"))
}

@Test
void equalOperator_Php() {
    assertEqualOperatorPresent(new File("src/test/resources/same/operators/Operators.php"))
}

This process needs to be repeated for the remaining classes and used going forward. The only reason it isn't just something like:

SourceLanguages.values().each {
    assertEqualOperatorPresent(new File("src/test/resources/same/operators/Operators." + it.getExtension()))
}

would be because sometimes the tests assert slightly different naming. For example, Java expects names to be fully qualified. If it can be made better, go for it.

Natural ordering

Version schemas as x.x

Semantic roles by language

So you can search for all IF or JAVA_IF/PYTHON_IF

Multi-thread observation extractions