Few questions about transcrypt

[JdeH]

Hi Almar,

An important design consideration was to allow for automated back to back testing with CPython.
It has already saved me lots of time and bugs.
In order to allow for this Transcrypt will always require a certain version of Python, currently 3.5.
So for me it was OK to use one specific version of the AST.
Maybe I’ll package Python 3.5 and Transcrypt together, as do libs like Kivy and Fenix/Dolphin, to make installation even easier.

A really big thing for me is the multiple inheritance, it made my code more concise, ever since C++ introduced it.
I’ve always avoided Java for that reason.
Also the bound function pointer (that e.g. RapydScript didn’t support) I use frequently.

As for the operator overloading: I’m still contemplating that. Transcrypt is not yet finished, and even after a first release things may be added.
But to give some insight in my thoughts: Overloading the string * operator seems doable, but e.g. glueing lists together with + would make the JS code bloated.
Since the type of a variable cannot be predicted and changes at runtime, I would be obliged to generate e.g. add (x, y) everywhere, rather than x + y since one day x and y may be arrays rather than numbers.
This would contradict the requirement for readable, simple JavaScript and it would be far too slow for numbers.

One solution I’ve been thinking about is to use type annotations, so that I can generate add (x, y) for lists and x + y for strings.
This would however be non-standard Python, since type annotations are only available for function params, unfortunately.
So, apart from being undesirable in general, it would break back to back testing.

Type inference, on the other hand (like in F#) would make Python less dynamic, for me an unacceptable price to pay.
What I expect is that I will probably in a later version allow overloading [] and () and maybe * for strings, but in general overloading won’t be supported (until, maybe JavaScript acquires an overloading mechanism, which is probably never).

Kind regards
Jacques

From: Almar Klein
Sent: Monday, January 25, 2016 10:40 AM
To: JdeH/Transcrypt
Subject: [Transcrypt] Few questions (#1)

Hi, @datnamer pointed this project out to me (flexxui/flexx#54), nice project! It seems to have a lot of common with what I am trying to do with flexx.pyscript. At least, of all the python-to-js solutions, this is by far the most similar I've seen :)

On the thread on reddit you mentioned that you've checked Flexx out but that it did not have what you needed. As far as I can tell the biggest difference is that Flexx does currently support multiple inheritance or bound functions. I'd be happy to have this changed though. I also want to stress that although the Flexx project itself is about providing a full widget toolkit, flexx.pyscript is self-contained. E.g. we're using it in Bokeh as well.

I was wondering about how you do the ast parsing. I'm using the buildin ast module myself, and have written a sort of compatibility layer (commonast.py) to make it work across different Python versions (support for Python 2.7 to be added like this week). If anything, this would be one place where our projects could potentially share code.

Another question is how you (plan to) handle things like "a string" * 3. Is that going to produce "a string3" as in JS or "a stringa stringa string" as in Python? I chose the latter, but I needed some operator overloading to make that work.

—
Reply to this email directly or view it on GitHub.

[almarklein]

The automated testing of all features seems like a useful thing. I've extensive unit tests for pyscript, but not much that auto-tests that Python and JS produce the same result. Nevertheless, how does this bind you to a specific Python version? And even if it does, could you not restrict only the tests to a certain Python version, and still support other versions?

The __add__(x, y) thing was something I am not overly excited about either. I started out with the idea to not do this kind of stuff (people should be aware that they're writing JS), but I quickly got confused myself when I did not get the results I expected from Python, so I decided that PyScript should behave as much as Python as is reasonably possible.

I don't understand why you would overload multiplication but not addition? Would they not suffer from the same overhead? It would be a drain on performance, yes, but still much faster than CPython I think. I've contemplated a with raw_js: context manager in which such smart overloading stuff would not be applied, for performance critical code. Since PyScript already allows raw JS (by writing a function with only a docstring), I decided against it for the time being.

[JdeH]

I see that I've raised a misunderstanding with regard to the operator overloading. Overloading of + for strings is already a fact, overloading of * for strings seems doable, is what I meant.

In my search for a balance between completeness on one hand and performance/compactness on the other, I came across a problem implementing the full set of *args, **kwargs, default and kwonly args that Python 3.5 has. Since at definition time it is unknown whether a function will ever be called with keyword args, it seemed that I had to build in that expensive mechanism everywhere. So I choose the following solution: There's a command line switch -k to allow keyword args for every function, giving full CPython compatibility, but its use is disadvised. Apart from that there are the pragma ('kwargs') and pragma ('nokwargs') statements, that can switch keyword args on for a certain function. They can be used per module or e.g. per function definition. The stubs.browser module has stubs for those methods, such that CPython will swallow them as well.

Currently I consider expanding the pragma mechanism to other optimizations. This would make operator overloading feasible. The pragma's could be either very specific, or more general. Locally, between e,g. pragma ('overload') pragma ('nooverload') binary operators would be compiled to e.g. add and mul rather than + and , allowing for straightforward overloading, closely matching CPython. I am not sure that would be attractive, though. Suppose M is a matrix, v is a vector and x is a scalar. One could write e.g. x * M*-1 * v rather than mul (x, mul (inverse (M), v) in Python. In JS this would then become mul (mul (x, pow (M, -1)), v) rather than mul (x mul (inverse (M), v)). So the Python would become more readable, but the JS a bit slower if no matrices or vectors are involved. E.g. 3 * 4 would be compiled to mul (3, 4) in between the pragma's.
So the Python would become more readable, but the JS less readable and slower. This is no proper example, since Python has a special operator for matrix multiplication. Conclusion: I do not yet know what's best. For math-intensive pieces of code, overloading may be worth while if the speed penalty of the extra calls doesn't spoil it. If I would introduce it, I would surely use the pragma's. But it seems a lot of complexity for some syntactic sugar. Sometimes may restriction is wiser. Also for intensively numerical code, the speed penalty seems a bad idea. Maybe computation intensive code should have pragma's to compile it asmjs fragments in the long run. Come to think of it maybe this could still be combined with operator overloading, giving better Python readability as well as a performance boost. But I don't know if that's possible,haven't looked much into asm.js. At the time operator overloading for me has low priority. It is a relatively minor issue compared to the (admittedly subjective) gain in expressiveness coming from programming in Python , rather than JavaScript.

As for your remarks about the specific Python version: As long as the deviations weren't to big, I could build some different alternative visitornodes into the parser. But what I expect is that I'll make the newest features of Transcrypt (should they emerge) work only with the newest version of Python. Maybe every now and then I would skip some minor Python versions. This would in my view be reasonable, since the newest features of Python itself will by definition also work only with the newest version of Python. And not using the same version on client (Transcrypt) and server (CPython) would not be such a big deal as long as the language doesn't change to rudely. If it's nothing worse than adding () to the print statement (something that took years to be accepted by the community) I don't worry to much.

[almarklein]

A quick heads-up that commonast.py now also works on Python 2.7. The idea of this module is that it's small enough to include in other projects. Then instead of doing ast.parse() you do commonast.parse(). The produced ast tree is different from what Python produces, but it it's consistent across a wide range of Python versions.

I like the idea of the __pragma__ functions. Preferably, the transpilation just works in a single way, but indeed there seem some difficult choices/compromises wrt speed and Pythonic-ness.

I also like the idea of having bound functions, and might just steal it :P

[JdeH]

Feel free to use anything you like. That's why it's open source. The bound functions thing took some thinking. By making methods properties rather than functions, assigning aBoundMethod = anObject.aMethod actually calls a factory function that produces a bound function. It's much like the CPython descriptor mechanism.

As for the pragma's: Keyword params are expensive in JS but I use them a lot in Python. I think they're a great feature of Python and I wanted to support them. Since it cannot be predicted at def time if a function will be called with keyword params, I came up with the pragma's. They're also used to prevent name conflicts, pragma ('alias', ...) and to insert literal JS, pragma ('JS.... In a later stage Transcrypt will probaby support operator overloading in computation-intensive parts of the code, between pragma ('overload', '+ - * /') and pragma ('nooverload', '+ - * /'). This once again is a thing that indeed should be activated only locally, in matrix computations etc. Another way to go would be a strickt typing pragma to give certain variables a fixed datatype. But that seems more problematic to me. I would not want to embark on a complete type inference mechanism...

[almarklein]

So I guess the bound function mechanism would not work on existing JS methods?

[JdeH]

No, it doesn't. In one case in Transcrypt I rewrote a function directly in JS supporting this mechanism:

// Since it's worthwhile for the 'format' function to be able to deal with *args, it is defined as a property
// __get__ will produce a bound function if there's something before the dot
// Since a call using *args is compiled to e.g. <object>.<function>.apply (null, args), the function has to be bound already
// Otherwise it will never be, because of the null argument
// Using 'this' rather than 'null' contradicts the requirement to be able to pass bound functions around
// The object 'before the dot' won't be available at call time in that case, unless implicitly via the function bound to it
// While for Python methods this mechanism is generated by the compiler, for JavaScript methods it has to be provided manually
Object.defineProperty (String.prototype, 'format', {
    get: function () {return __get__ (this, function (self) {
        var args = tuple ([].slice.apply (arguments).slice (1));            
        var autoIndex = 0;
        return self.replace (/\{(\w*)\}/g, function (match, key) { 
            if (key == '') {
                key = autoIndex++;
            }
            if (key == +key) {  // So key is numerical
                return args [key] == 'undefined' ? match : args [key];
            }
            else {              // Key is a string
                for (var index = 0; index < args.length; index++) {
                    // Find first 'dict' that has that key and the right field
                    if (typeof args [index] == 'object' && typeof args [index][key] != 'undefined') {
                        return args [index][key];   // Return that field field
                    }
                }
                return match;
            }
        });
    });},
    enumerable: true
});

In this case it needed to be compatible with Transcrypts *args mechanism. In most cases simply manually wrapping an unbound method in a bound wrapper would be easier. The wrapper would keep a local reference to the object involved. No need for properties in that case.

[kovidgoyal]

FYI: You dont need a compile time pragma to support keyword args, see how I do it in rapydscript-ng without one (the original rapydscript does use a compile time pragma, but it is unnecessary). https://github.com/kovidgoyal/rapydscript-ng#function-calling-with-optional-arguments

Also using properties for bound methods is a lot of unnecessary overhead, Just do self.mymethod = mymethod.bind(self) [this allows for optimizations in the interpreter]. In RapydScript you can auto-bind all methods of a class with a single call to rebind_all() in the classes __init__ method or a compiler option which inserts this call for you automatically, although I have to say that I dont autobind because of performance so I dont know if the autobind code works well or not :)

[JdeH]

Hi Kovid,

Thanks for your reaction. Bound function calls are done the CPython/descriptor way, which in my view strikes a good balance between handyness and runtime efficiency. Still, to get that last bit of speed they will be controlled by a compiler switch in the next version.

As I make clear in the docs, Transcrypt has it's own niche somewhere on the line between PyPyJs and RapydScript, where it will be closer to RapydScript then to PyPyJs.

There are many other transpilers and each one has its merits. Rather than going into your remarks one by one, I want to state that I am open to any comparison made by a 3rd party, once Transcrypt is ready. I'll postpone mixing in discussions about the weaks and strongs of each project until that point, since otherwise it will take up valuable development time.

Even if I will not enter a discussion rightnow, I have to say that your criticism is very welcome, since it will help me to make Transcrypt better.

As far as I am concerned there's room in the market for diversity. I don't think I'll publish benchmarks myself, since they're inevitibly skewed if I do. But I invite others to do so once the release is out (will be some months).

[kovidgoyal]

To be clear, I am only pointing out better solutions where they exist, so you can improve Transcript, and more generally, the optimal solutions to various categories of problems in the python transpiler world propagate.

You are welcome to continue developing Transcript, do not take my comments as implying otherwise.

[JdeH]

Your remarks are appreciated!