andry-tino / rosetta Goto Github PK

A declaration like:

static int example = 42;

throws an error ("An error occurred: Not recognized visibility!!") when executing the definition runner.

Public fields in classes are ignored and not emitted in the definition file.

Source file

public class AppDefinition
    {
        public static AppDefinition Current;
        public Resources Resources;
        public ConfigurationSettings Configuration;
        public int Version;
    }

Output file

declare module OverriddenNamespace {
  export class AppDefinition {
    public current : OverriddenNamespace.AppDefinition;
  
    constructor();
  }
}

The static modifier in source is making that property being emitted. But the static is not emitted so this is another problem.

When translating, it is possible that the source code defines types with the same name as types in MsCoreLib.dll and includes both namespaces, thus generating a clash.

The source code might define an interface:

namespace MyNamespace {
  namespace MySubnamespace {
    public interface INotifyCompletion { }
  }
}

However we have that interface also in MsCoreLibe: System.Runtime.CompilerServices.INotifyCompletion.

Especially when using syntax transformation (which rearranges using directives), we might end up in this situation:

using System.Runtime.CompilerServices;
using MyNamespace.MySubnamespace;

public class MyClass {
  protected INotifyCompletion notifier; // Name clash
}

When processing notifier, the semantic model detects an ambiguity.

When having:

public class MyClass {
  protected TypeInAssembly field;
  protected void DoSome(TypeInAssembly parameter) { }
}

It is emitted like:

class MyClass {
  protected field : TypeInAssembly;
  protected doSome(parameter: Namespace.TypeInAssembly) : void;
}

The field is not emitted correctly as the type is not getting the full name.

Rosetta ScriptSharp Definition Generator can work on DLLs only instead of using source files.

Scenarios

Can be used in contexts where:

• The source code is not available

Benefits

Can cut down dependencies on code.

Downsides

Comments and documentation not available for emitting.

Plan

It is necessary to decouple the ScriptSharp generation components from AST Walkers as not needed. Also more work needed:

• Decouple AST
• Generate components for enumerating types in DLL and get the tree from the SemanticModel.
• Reuse helpers for extracting data from the SemanticModel, here we might get issues as helpers are depending on SyntaxNodes
• Reuse TranslationUnits as they are

Important Evaluate the possibility to generate AST and then feed it to the regular stack.

The definition runner exports enum type names, but the enum values are not included.

Example:

enum Status {
    Inactive = 0,
    Active = 1
};

expected type definition:

declare enum Status {
    Inactive = 0,
    Active = 1,
}

actual type definition:

declare enum Status {
 } 

When generating definition files, add the following header to generated files:

/**
 * Rosetta <version>
 * TypeScript definition file generated on <date>
 */

Maybe also add more info if a bundle was generated.

Depending on #14.

The transformer must be able to process enums as well.

ScriptSharp Reflector emits:

declare class <Module> {
}

It should not be emitted.

We have the ability, after semantic alignment, to detect the ScriptNamespace in referenced types. However this is working only if the referenced type is in the local syntax tree.

If the referenced type is in the referenced assembly, then the original namespace is used even though the ScriptNamespace attribute was specified in that assembly's source code for that type at definition time.

Example

In local syntax tree:

namespace MyNamespace {
  [ScriptNamespace{"NewNamespace"}]
  public class MyClass : MyBase { } // MyBase is in referenced assembly
  public class MySecondClass : MyClass { }
}

In the referenced assembly:

namespace MyNamespace {
  [ScriptNamespace{"NewNamespace"}]
  public class MyBase { }
}

When using Rosetta and passing the assembly through --assembly we get:

declare module DN {
  export class MyClass extends MyNamespace.MyBase { }
  export class MySecondClass extends DN.MyClass { }
}

When using reflector, no separation is added between types. So instead of:

declare module MyModule {
  export class MyClass { ... }
}
declare module MyModule {
  export class MyOtherClass { ... }
}
declare module MyModule {
  export class MyYetAnotherClass { ... }
}

We should have:

// Info here
declare module MyModule {
  export class MyClass { ... }
}

// Info here
declare module MyModule {
  export class MyOtherClass { ... }
}

// Info here
declare module MyModule {
  export class MyYetAnotherClass { ... }
}

The ScriptSharp Reflector does not emit public static methods as static. Only protected members are.

Register Rosetta ScriptSharp definition generator in NuGet.

Nice to have

• Script for creating the package.

Add a flag and logic in ScriptSharp definitions, in order to apply a new namespace to all types.

Performance

Performance are bad when going through the regular AST transformation for handling the [ScriptNamespace("NewNamespace")] script namespace:

Suggested approach

Create a new syntax transformer which renames all namespace into the one passed by the parameter. This will add complexity in memory consumption, but in terms of performance, it will be a much faster process compared to the current transformation algorithm.

AST transformer for ScriptSharp definitions is generating empty modules because of empty namespaces when transforming the AST.

Suggested approach

Check namespace nodes after AST restructuring and remove nodes that do not contain types. Check only namespace nodes inside the program unit.

It is necessary to emit most commonly used types in mscorelib.dll like IDIsposable, object.

Suggested implementation

Using a resource file in C# which contains the definitions in mscorelib and is translated during definition emission process. The bundle will contain the final definitions.

Transformer is only working on classes at the moment, After change ce83661 we need to include interfaces as well.

When emitting for a file like:

#if VAR
[ScriptNamespace("MyNamespace")]
#endif
public class MyClass {
}

The generated output will miss the new namespace. However the referenced type will actually be correct as, there, we use the semantic model.

Enable the usage of semantic model when translating.

Rationale Type references are a problem, the single name is not enough, the full name should be specified.

Must

Scoped for ScriptSharp definition only.

• Have full names in types for classes and interfaces in base lists
• Have full names in methods, getters, setters, properties for parameters and returned values

Implementation details

Allow the possibility to specify an assembly as input when processing input files, this would enable the loading of the semantic model.

Introduce framework to have string text comparison. The system is intended to use our current renderers and compare output with an archetype string.

Progress

So far we have already included Google's DiffMatchPatch in our solution. We need to use it.

Rosetta should support generic types. Generic types can be used for fields, properties, variables, class types and base class types.

public static class Example {		
  private static class Inner
  {
    public const int No = 0;
    public const int Yes = 1;
  }
}

produces

declare class Example {
  public no : number;
  public yes : number;
}

Missing call to new test suites in test runner script in Jenkins.

Setters are emitted as:

set_example(value : Example);

If implicit allow any is enabled that will compile fine; otherwise it is a compile error.
Instead, this should be emitted:

set_example(value : Example): void;

TU Rendering tests found a problem in if statements for which, in body, semicolons are not rendered.

Report
------
Showing test results:
Showing summary:
ConditionalStatements.IfStatement.ts => Fail
ConditionalStatements.IfElseStatement.ts => Fail
ExpressionStatements.ReturnStatement.ts => Pass
ExpressionStatements.ReturnVoidStatement.ts => Pass
ExpressionStatements.ThrowStatement.ts => Pass
ExpressionStatements.ThrowVoidStatement.ts => Pass
KeywordStatements.BreakStatement.ts => Pass
KeywordStatements.ContinueStatement.ts => Pass


Summary
-------
ConditionalStatements.IfStatement.ts
------------------------------------
Found 5 difference(s).
LEFT
====
if (true)
{
  var variable2 : number = 14
}
RIGHT
====
if (true)
{
  var variable2: number = 14;
}

ConditionalStatements.IfElseStatement.ts
----------------------------------------
Found 9 difference(s).
LEFT
====
if (true)
{
  var variable2 : string = 'Hello'
}
else
{
  var variable1 : number = 100 * 21
}
RIGHT
====
if (true)
{
  var variable2: string = 'Hello';
}
else
{
  var variable1: number = 100 * 21;
}

ExpressionStatements.ReturnStatement.ts is OK

ExpressionStatements.ReturnVoidStatement.ts is OK

ExpressionStatements.ThrowStatement.ts is OK

ExpressionStatements.ThrowVoidStatement.ts is OK

KeywordStatements.BreakStatement.ts is OK

KeywordStatements.ContinueStatement.ts is OK

Tests

Tests for conditional statements in TranslationUnits.Renderings.Data have been disabled.

Ability to add a reference to another definition file in the generated definition file.

<ScriptSharpDefinitionRunner> ... --includes ("file.d.ts")

This will cause this line to be emitted at the beginning of the generated definition:

/// <reference path="file.d.ts" />

Scenarios

Include the mscorelib ScriptSharp definitions.

Implementation

Add arguments in runners and task runner.

Base helper TypeReference has Name and FullName. Problem is that base level factories are using FullName which is good for ScriptSharp definitions, but not for normal translation.

1. Add DisplayName to TypeReference.
2. Have factories always use DisplayName when invoking TypeReference.
3. Make DisplayName return Name in Rosetta.AST.Helpers.TypeReference.
4. Make DisplayName return FullName in Rosetta.ScriptSharp.AST.Helpers.TypeReference.

Regression in transformer.

namespace MyOriginalNamespace {
	[ScriptNamespace(""NewNamespace1"")]
	public class Class1 {
		public void Method1(int param1) { }
		public void Method2(int param1, string param2) { }
	}
	public class Class2 {
		public void Method1(int param1) { }
		public void Method2(int param1, string param2) { }
	}
}

Gets rendered like:

declare module MyOriginalNamespace {
  export class Class1 {
    method1(param1 : number) : void;
    method2(param1 : number, param2 : string) : void;
  }

  export class Class2 {
    method1(param1 : number) : void;
    method2(param1 : number, param2 : string) : void;
  }
}

Which is wrong. The new namespace is not considered.

Check rendered properties, as the version we use of TypeScript (0.6.0.0) might render them differently.

When the converter hits a construct that it cannot translate, it throws errors.
Consider to emit the original source instead as commented code. You can add //TODO: convert this as well.

That way, the result may not be a fully converted TypeScript file, but at least you would have a skeleton to work with.

At the moment static members in classes are not emitted correctly, sometimes resulting also in error messages.

What

Scope is classes. We must properly output static members.

public class MyCLass {
  public static void MyMethod() { }
  public static string MyProperty { get; set; }
  public static int MyConst = 12;
}

Class members are not observing ScriptSharp naming conventions:

• First letter must be lowercase
• CamelCase applied

Implement type mappings.

Scenario

In ScriptSharp, some JavaScript types might be mapped into classes like arrays for example:

[Imported]
// This is used to map JavaScriptarrays inside C#
public class ArrayList { ... }

When converting, we should detect that ArrayList is to be converted into [] in TypeScript.

Implementation

Implement a separate component responsible for handling the mapping. The component should act at AST Walking level (preferably, in factories) or in Helpers.

When translating very huge code (around 1300 files), we get up to 8 seconds. This time should be lowered.

Suggested approach

Run profiler and try to catch hot paths.

It is necessary to emit enums in ScriptSharp definitions file generation.

Hi @andry-tino,
Thank you for your awesome project.

Roslyn now supports .NET Core cross platform applications.

Do you have plans to support .NET core in your project?

We are emitting protected keyword with constructors in TypeScript definition files. But TypeScript does not allow that.

Problems might arise when having, for example:

namespace A.B {
  public class MyClass : TypeA { ... }
}

Where:

[ScriptNamespace("NewNamespace")]
public class TypeA { ... }

Suggested approach

Possibility to use transformation to always include the full type name in the syntax before starting the translation process.

Need to add tests for ScriptNamespaceBasedASTTransformer as it is a mojor component in the program and it is still lacking coverage:

• Check allowed types.
• Check tree structure.

So far a separation has been made:

• Rosetta C# translation
• Rosetta ScriptSharp definition generation

Rosetta must be able to translate generic C# code and C# code which has been written specifically for ScriptSharp. Thus the separation must be:

• Rosetta C# translation
• Rosetta C# ScriptSharp translation
• Rosetta ScriptSharp definition generation

In order to do that a new project must be introduced: ScriptSharp. Components need to be re-wired and tests need to be rewired as well.

Projects

Changes to do:

• Rename ScriptSharpDefinition into ScriptSharp.Definition
• Rename ScriptSharpDefinitionRunner into ScriptSharp.Definition.Runner
• Rename ScriptSharpDefinitionBuildTask into ScriptSharp.Definition.BuildTask
• Create project ScriptSharp

In tests:

• Rename ScriptSharpDefinition.ASTWalker.Renderings into ScriptSharp.Definition.ASTWalker.Renderings
• Rename ScriptSharpDefinition.ASTWalker.Renderings.Data into ScriptSharp.Definition.ASTWalker.Renderings.Data
• Rename ScriptSharpDefinition.ASTWalker.Factories.UnitTests into ScriptSharp.Definition.ASTWalker.Factories.UnitTests
• Rename ScriptSharpDefinition.ASTWalker.UnitTests into ScriptSharp.Definition.ASTWalker.UnitTests
• Rename ScriptSharpDefinition.BuildTask.Example into ScriptSharp.Definition.BuildTask.Example

Rearrangement

Components to be rearranged:

• Move ScriptNamespaceBasedASTTransformer into ScriptSharp project
• Move all helpers from ScriptSharpDefinition into ScriptSharp

Rearrange tests accordingly to new structure in source.

Final notes

Rearrangement is only at filesystem level in the file organization. Namespaces in files are fine.

Rosetta relies on Roslyn 1.0, the very first stable version. Upgrading to new version is desirable.

Why

• Better performance
• Expanded API
• Keep up with technology change (.NET Core)

Notes

This is to be couple with #36.

The definition runner ignores the [PreserveName] attribute.

public static class Example
{
  [PreserveName]
  public const string Example = "hello World";
}

Becomes

declare class Example {
    public example: string;
}

Change 86060aa has purposely broken the transformer which had to be deactivate in that same change. Issue #19 has been tracking the feature. This issue arises as a continuation of that and aims to fix the transformer in order to fix the ScriptNamespace attribute handling which is, currently, broken.

Problem

A problem arises concerning syntax transformation.

What Issue is that after transforming, the syntax tree is different and does not correspond anymore to the semantic model.

Solution

A mixture of the mapping approach and the semantic model update.

Available types in AST At transformation time, we transform the AST and then generate an updated model. This will deal with the types available in the local AST (a small part represented in the semantic model). See:

• Modify syntax tree then get updated semantic model
• AST Transofrmation - Roslyn

Referenced unavailable types We still have a problem concerning types referenced by the local AST but not available. The semantic model will refer to them using the not overriden namespace, while we should actually use the overridden one.

To accommodate this, helpers (when provided with semantic model), when providing the full name or every namespace-related quantity, will check the presence of the ScriptNamespace attribute and override the name accordingly. This will impact ONLY helpers in the ScriptSharp Definition area.

Challenge: semantic misalignment

After transforming and updating the semantic model reports a lot of errors relating to missing assemblies.

Solution Collect all using directives and copy in the root level of the new node or in the single namespaces created for each moved type. It is also important to include the namespace of the moved entities in order to have proper semantic alignment for them.

Start work to introduce factories for translation units and use them in all places when constructing a new translation unit. Namely ASTWalkers for example.

Static methods in C# are not emitted as static in definition files.

Ability to emit delegates in definition files. Simple delegates like:

public delegate int ExampleEventHandler(object sender, PersonalEventArgs eventArgs);

Suggestions

Simplification: emit them as empty classes:

declare|export class ExampleEventHandler { }

Evaluate later the possibility of rendering them as proper functions:

declare|export ExampleEventHandler: (object, PersonalEventArgs) => number;

Proper implementation

Allow the ability to specify an external C# file when generating definitions. Normal input files will be converted into definitions and the passed file will be as well. The final definitions will be emitted together in the bundle.

To be considered Evaluate the possibility to have this feature available only when specifying the bundle option.

My team and I have been hacking away at adding more conversions for C# features that we used in our Script# projects. We'd like to contribute it back into the main project.

Please have a look at my branch and let me know what you think. There are several things that are still "work in progress" (like handling the case's inside a switch), but we've handled foreach loops, a rough handling of knockout observables and added a naive handling of csproj files.

There are a number of things that we've identified that we need and will try to implement

• Generics
• Convert List to x[]
• Gracefully handle missing "this"
• Convert delegates to lambdas
• Allow for namespace instead of module
• Call "super" in inherited classes
• and more!

Promote native type transformation. Some examples: ArrayList => any[], Action => function, Func => function. The problem is that we need to map the proper C# type before. Or we can just proceed after the AST walking and detect the type names.

Discovered by newly introduced StringRendering tests. TestSuite: Rosetta\test\renderers\ScriptSharp.Definition.ASTWalker.Renderings.Tests\TestSuite.cs, test: TestClassesWithScriptNamespace.

What

Classes and enums are emitted:

• No declared enclosing module/namespace
• No ScriptNamespace override

Output for classes

Summary
-------
ScriptNamespace.SingleClass.d.ts
--------------------------------
Found 1 difference(s).
LEFT
====
declare class Class1 {
  method1(param1 : number) : void;
  method2(param1 : number, param2 : string) : void;
  method3(param1 : number, param2 : string, param3 : boolean) : void;
  method4(param1 : number, param2 : string, param3 : boolean, param4 : number) : void;
}
RIGHT
====


ScriptNamespace.SingleNamespacedClass.d.ts
------------------------------------------
Found 1 difference(s).
LEFT
====
declare module MyOriginalNamespace {
  export class Class1 {
    method1(param1 : number) : void;
    method2(param1 : number, param2 : string) : void;
    method3(param1 : number, param2 : string, param3 : boolean) : void;
    method4(param1 : number, param2 : string, param3 : boolean, param4 : number) : void;
  }
}
RIGHT
====


ScriptNamespace.TwoClasses.d.ts
-------------------------------
Found 1 difference(s).
LEFT
====
declare class Class1 {
  method1(param1 : number) : void;
  method2(param1 : number, param2 : string) : void;
}
declare class Class2 {
  method1(param1 : number) : void;
  method2(param1 : number, param2 : string) : void;
}
RIGHT
====


ScriptNamespace.TwoNamespacedClasses.d.ts
-----------------------------------------
Found 1 difference(s).
LEFT
====
declare module MyOriginalNamespace {
  export class Class1 {
    method1(param1 : number) : void;
    method2(param1 : number, param2 : string) : void;
  }

  export class Class2 {
    method1(param1 : number) : void;
    method2(param1 : number, param2 : string) : void;
  }
}
RIGHT
====


ScriptNamespace.TwoMixedClasses.d.ts
------------------------------------
Found 1 difference(s).
LEFT
====
declare class Class1 {
  method1(param1 : number) : void;
  method2(param1 : number, param2 : string) : void;
}
declare class Class2 {
  method1(param1 : number) : void;
  method2(param1 : number, param2 : string) : void;
}
RIGHT
====


ScriptNamespace.TwoMixedNamespacedClasses.d.ts
----------------------------------------------
Found 1 difference(s).
LEFT
====
declare module MyOriginalNamespace {
  export class Class1 {
    method1(param1 : number) : void;
    method2(param1 : number, param2 : string) : void;
  }

  export class Class2 {
    method1(param1 : number) : void;
    method2(param1 : number, param2 : string) : void;
  }
}
RIGHT
====

Output for enums

Summary
-------
ScriptNamespace.SingleEnum.d.ts
-------------------------------
Found 1 difference(s).
LEFT
====
declare enum Enum1 {
  Value1
}
RIGHT
====


ScriptNamespace.SingleNamespacedEnum.d.ts
-----------------------------------------
Found 1 difference(s).
LEFT
====
declare module MyOriginalNamespace {
  export enum Enum1 {
    Value1
  }
}
RIGHT
====


ScriptNamespace.TwoEnums.d.ts
-----------------------------
Found 1 difference(s).
LEFT
====
declare enum Enum1 {
  Value1
}
declare enum Enum2 {
  Value1
}
RIGHT
====


ScriptNamespace.TwoNamespacedEnums.d.ts
---------------------------------------
Found 1 difference(s).
LEFT
====
declare module MyOriginalNamespace {
  export enum Enum1 {
    Value1
  }

  export enum Enum2 {
    Value1
  }
}
RIGHT
====


ScriptNamespace.TwoMixedEnums.d.ts
----------------------------------
Found 1 difference(s).
LEFT
====
declare enum Enum1 {
  Value1
}
declare enum Enum2 {
  Value1
}
RIGHT
====


ScriptNamespace.TwoMixedNamespacedEnums.d.ts
--------------------------------------------
Found 1 difference(s).
LEFT
====
declare module MyOriginalNamespace {
  export enum Enum1 {
    Value1
  }

  export enum Enum2 {
    Value1
  }
}
RIGHT
====

Output for interfaces

Summary
-------
ScriptNamespace.SingleInterface.d.ts
------------------------------------
Found 1 difference(s).
LEFT
====
declare interface IInterface1 {
}
RIGHT
====


ScriptNamespace.SingleNamespacedInterface.d.ts
----------------------------------------------
Found 1 difference(s).
LEFT
====
declare module MyOriginalNamespace {
  export interface IInterface1 {
  }
}
RIGHT
====


ScriptNamespace.TwoInterfaces.d.ts
----------------------------------
Found 1 difference(s).
LEFT
====
declare interface IInterface1 {
}
declare interface IInterface2 {
}
RIGHT
====


ScriptNamespace.TwoNamespacedInterfaces.d.ts
--------------------------------------------
Found 1 difference(s).
LEFT
====
declare module MyOriginalNamespace {
  export interface IInterface1 {
  }

  export interface IInterface2 {
  }
}
RIGHT
====


ScriptNamespace.TwoMixedInterfaces.d.ts
---------------------------------------
Found 1 difference(s).
LEFT
====
declare interface IInterface1 {
}
declare interface IInterface2 {
}
RIGHT
====


ScriptNamespace.TwoMixedNamespacedInterfaces.d.ts
-------------------------------------------------
Found 1 difference(s).
LEFT
====
declare module MyOriginalNamespace {
  export interface IInterface1 {
  }

  export interface IInterface2 {
  }
}
RIGHT
====

Projects do not declare independent NuGet deps. Some projects need assemblies and make them as dependencies, however other projects just reference them withput having such deps stated in the app.config.

Action items

• Detect every project having a dependency on NuGet packages (especially Roslyn)
• Make sure the cirresponding app.config exists in those projects.
• If not, create it and set deps there.