krzema12 / kotlin-python

Python target for the Kotlin Programming Language. See https://github.com/krzema12/kotlin-python/tree/python-backend/python

Hi there 👋

My name is Piotr Krzemiński. I'm a software development engineer, passionate about software quality, automation and constant process improvement. I work full-stack, mainly with JVM languages and React. My programming languge of choice is Kotlin which I use for both backend and frontend, and which I try to spread through a local Kotlin User Group. I try my best to write clean, functional code.

In the open-source world, I own an ecosystem of projects for creating type-safe GitHub workflows:

• github-workflows-kt - a Kotlin-based DSL for defining GitHub Actions workflows
• github-actions-typing - add typings to your actions' API. Used by the above DSL
• github-actions-typing-catalog - a catalog of action typings, if their owners decide to not use github-actions-typing to host the typings together with the actions

How to reach me

Feel free to drop me a line via LinkedIn.

kotlin-python's Issues

Allow adding Python function decorators

Requested in #51, to support e.g. @microptyhon.native in MicroPython.

Constructors e2e test: don't rewrite data in the global var, but append to it. This way we will know that exactly the needed sequence happened

Originally created by @SerVB.

Allow to select the specific e2e test to run. For example, use an argument: `./run.sh [optional arg: pattern of needed tests]`

Originally created by @SerVB.

Understand (and solve if trivial) MemoryErrors when using MicroPython

For example, when running e2e tests with micropython instead of python3, I got:

Traceback (most recent call last):
  File "python/e2e-tests/out/python-consumer.py", line 1, in <module>
  File "/home/piotr/repos/priv/kotlin-python/python/e2e-tests/out/compiled.py", line 12986, in <module>
MemoryError: memory allocation failed, allocating 26032 bytes

Looking at box tests report, >1000 tests fail with this reason. The good thing is that it's fairly reproducible: for a given box test, it always fails with such MemoryError.

My suspicion is that too much code is fed to micropython. Kotlin/Python doesn't do any Dead Code Elimination for now (to be done in #102), and the compiler output weights ~420 KiB.

Support multiple comparisons: `0 <= limit < count` instead of `0 <= limit and limit < count`

Originally created by @SerVB.

Document data flow and steps

The goal is to describe what happens with the input Kotlin code, starting with the CLI, going through the frontend, backend, and ending with the Python code.

Try using multiple agents running tests in parallel to make testing on CI faster

Originally created by @SerVB.

Rework `.toPython()` to use common `StringBuilder`, not plain `String`s

Originally created by @SerVB.

Ensure we don't have JS-related stuff in files, packages, and symbols names; something else like nodejs-something and es6 optimizations

Originally created by @SerVB.

• Rename files.
• Rename packages.
• Rename symbols.
• Get rid of JS-specific things in compiler logic.
• Get rid of JS-specific things in build scripts logic.
• Get rid of JS-specific things in tests logic.
• Stop depending on JS modules (copy what's left).

Run MicroPython e2e tests

With no changes, I get this:

Traceback (most recent call last):
  File "python/e2e-tests/out/python-consumer.py", line 5, in <module>
OSError: [Errno 13] EACCES

When not writing to file, only one test fails, and the rest produces something that is expected:

$ JDK_9="$JAVA_HOME" python/e2e-tests/run.sh
Starting end-to-end tests
===== Executing 'constructors' =====
  Compiling to Python...
python/e2e-tests/out/kotlin-code.kt:3:26: warning: parameter 'a' is never used
class MultiConstructors1(a: Int) {
                         ^
python/e2e-tests/out/kotlin-code.kt:9:17: warning: parameter 'b' is never used
    constructor(b: String) : this(2) {
                ^
python/e2e-tests/out/kotlin-code.kt:24:17: warning: parameter 'a' is never used
    constructor(a: Int) {
                ^
python/e2e-tests/out/kotlin-code.kt:28:17: warning: parameter 'b' is never used
    constructor(b: String) {
                ^
python/e2e-tests/out/kotlin-code.kt:43:17: warning: parameter 'a' is never used
    constructor(a: Int) {
                ^
python/e2e-tests/out/kotlin-code.kt:47:17: warning: parameter 'b' is never used
    constructor(b: String) : this(3) {
                ^
python/e2e-tests/out/kotlin-code.kt:51:17: warning: parameter 'b' is never used
    constructor(b: Boolean) : this("c") {
                ^
python/e2e-tests/out/kotlin-code.kt:70:17: warning: parameter 'i' is never used
    constructor(i: Int) {
                ^
python/e2e-tests/out/kotlin-code.kt:77:17: warning: parameter 'a' is never used
    constructor(a: Int) : super(4) {
                ^
  Running Python to produce the output...
body 11
Traceback (most recent call last):
  File "python/e2e-tests/out/python-consumer.py", line 6, in <module>
  File "/home/piotr/repos/priv/kotlin-python/python/e2e-tests/out/compiled.py", line 13692, in multi12
  File "/home/piotr/repos/priv/kotlin-python/python/e2e-tests/out/compiled.py", line 13670, in MultiConstructors1_init__Create_
AttributeError: type object 'MultiConstructors1' has no attribute '__new__'
===== Finished with 'constructors' =====
===== Executing 'integers' =====
  Compiling to Python...
  Running Python to produce the output...
True
False
120
4
-11
===== Finished with 'integers' =====
===== Executing 'lambdas' =====
  Compiling to Python...
  Running Python to produce the output...
40
20
21
143
===== Finished with 'lambdas' =====
===== Executing '_simple' =====
  Compiling to Python...
  Running Python to produce the output...
4
===== Finished with '_simple' =====
===== Executing 'stdlib' =====
  Compiling to Python...
  Running Python to produce the output...
123
===== Finished with 'stdlib' =====
All done

Use Python's types like `list` at least when an object doesn't go out of a function (when it does, it can be tricky to support compatibility with Kotlin's `List` interface)

Originally created by @SerVB.

Implement `fun main(args: Array<String>)`

The goal is to be able to create a CLI, so that the compiled module can be called like python3 out_ir.py --some --arg 123 and it will be available in Kotlin's fun main(args: Array<String>), in simple cases just fun main().

Introduce Python-specific standard library

https://kotlinlang.slack.com/archives/C0289CS37AS/p1636875290016700

We were advised by JetBrains to look at how kotlin-stdlib-wasm is implemented.

Adjust for no `__new__` method on MicroPython

When trying to run e2e tests for MicroPython, the constructors test fails with:

  File "python/e2e-tests/out/python-consumer.py", line 6, in <module>
  File "/home/piotr/repos/priv/kotlin-python/python/e2e-tests/out/compiled.py", line 13692, in multi12
  File "/home/piotr/repos/priv/kotlin-python/python/e2e-tests/out/compiled.py", line 13670, in MultiConstructors1_init__Create_
AttributeError: type object 'MultiConstructors1' has no attribute '__new__'

Looks like __new__ is another magic method unavailable on MicroPython. We had to use + instead of __add__ already. Not sure if there's an equivalent here, but still it's a must to fix it given we want to support MP.

Support compilation to different versions of Python Bytecode

Originally created by @SerVB.

Related discussion: https://kotlinlang.slack.com/archives/C01EY9ZF9B5/p1625331291002000?thread_ts=1605992072.006800&cid=C01EY9ZF9B5

Ensure our AST to Py conversion is correct, maybe add fuzzing tests with comparison to Python's built-in library

Originally created by @SerVB.

Implement DCE

Something is already inherited from JS backend, maybe it can be reused easily.
I also hope that DCE is a solution to MemoryError in MicroPython.

Understand why some expressions are compiled as `(True) if (limit < 0) else (count <= limit)` but not as `(limit < 0) or (count <= limit)` and fix it

Originally created by @SerVB.

Build on `python-backend` branch occasionally fails

See e. g. https://github.com/krzema12/kotlin-python/runs/4344104341?check_suite_focus=true
It looks like an issue not relevant to our changes. Compiling the compiler fails.

Move box tests reports generation to CI

It takes time and effort to rerun everything locally. Like @SerVB proposed #82, we could move the workflow to GitHub Actions.

As the last step of building a given pull request, there should be an automated commit that updates all the reports (stuff inside python/box.tests/reports).

Be wary of an endless loop: the automation adds the commit, it triggers another build in the PR, which adds another commit, and so on. In theory this loop should be broken as soon as the N-th iteration regenerates the reports and the diff is empty, then I think committing will fail. This case has to be handled gracefully, best if just one build (per test task) is ran for each PR, it commits the reports and that's it.

End-to-end tests never run

See

kotlin-python/.github/workflows/build_and_test.yml

Line 59 in 65b33e9

if: ${{ matrix.testTask == 'pythonTest' }} # TODO: run e2e tests for MicroPython too (#85)

Apparently the condition has a bug because the tests don't run for both Python and MicroPython.
FYI @SerVB

Use Python's typing hints

Originally created by @SerVB.

Run MicroPython in box tests on CI

See #51.

Proposed goal: run box tests for MP next to regular Python, with separate report files. A separate Gradle task can be created: microPythonTest.

• run box tests for MicroPython
• generate SVG git plot, including MicroPython
• run e2e tests with MicroPython - https://github.com/krzema12/kotlin-python/blob/python-backend/python/e2e-tests/run.sh#L25
  • #84
  • #85

Don't remove format of numbers: currently everything is output as decimal, but let's save at least hex representation

Originally created by @SerVB.

Optimize `Unit` returns

Originally created by @SerVB.

If it's just Unit, just return Python's None; if it's nullable (Unit?), then do some more complex logic like null -> None and Unit -> Unit.getInstance(). However doing this will also require being careful with Unit assignments, for example, if there is code like val b = println(), it should be compiled like println(); val b = Unit (achievable via introducing an extra lowering) because println() will return None and it's not what we want to assign to b.

Don't create intermediate collections in chains of functions calls like `.filter.flatMap.zip`

Originally created by @SerVB.

Onboard code quality assurance tools

Like linters, test coverage, mutation coverage.

Optimize `print` and `println` calls: compile to Python's `print`, add `end=""` if needed

Originally created by @SerVB.

Move all the python-related modules under the python dir

For now, python modules are made next to js ones, and that is handy in terms of similarity. However, in terms of developing, looks like having everything in a single dir is more convenient. Since we are going away from js code more and more, maybe we can consider changing the structure of our subprojects and move them.

Assess support for MicroPython

See https://kotlinlang.slack.com/archives/C0289CS37AS/p1637517359061900?thread_ts=1637517359.061900&cid=C0289CS37AS

https://core-electronics.com.au/tutorials/Python_and_MicroPython.html

Let's check how costly it would be to support both regular Python and Micropython (MP).

Ideas/steps:

• check if there's a regular Linux interpreter for MP
• smoke test: try running a sample output from current Kotlin Python compiler and see how it goes
• run box tests with MP interpreter and see what's the success rate comparing to what we have on Python
• hook MP interpreter into the box tests so that CI runs both
• think of cases where it would be challenging to support both at the same time

Minify names

For now, in case of names collisions, we have some ineffective logic. At least for classes, if there are classes named as A in different packages, they become A_0 and A_1.

We can minify those, for example, make them at least A and A_0 or even A and A0.

Profile launching of a single box test (for example, create a 1000-times loop): maybe some long parts of it can be optimized

Originally created by @SerVB.

Remove unneeded lowerings and refactor needed ones

Lowerings are a big part of backend logic, so let's ensure we know what happens there and that we don't have anything surplus or weird.

• #95
• analyze lowerings and dependencies between them
• iteratively remove lowerings to get a minimal list of needed ones
• remove unneeded Gradle module dependencies
• add unit tests for each lowering, if possible. It's to be able to understand what they really do, so it serves documentation purposes
• think outside the box: maybe for some case, the JS way is not the correct one? E. g. constructors, it may be a better way to lower them for Python. Now we base on JS version

"Generate reports" job doesn't regenerage SVG plot

See for example this generated commit: 7fba4c0
The lists of failed tests were updated, but the plot wasn't.

Allow specifying any Python code

Like:

python("any Python code emitted verbatim in Python output")

Requested in #51.

Understand why we have many `if True:` branches and fix it

Originally created by @SerVB.

Support do-while loop fully

Now continue is unsupported. Probably we can support it by compiling do { <body> } while (<condition>) as follows:

firstIterationLoopName = true
while firstIterationLoopName or (<condition>):
    firstIterationLoopName = false
    <body>

We need to introduce loop name, but there is already some logic for it in PyGenerationContext – we can pick the missing parts inside js-ir. Or we could use a concept of tmpVariable, this can be even easier.

Ensure we don't have dependencies to JS modules

Originally created by @SerVB.

Reports are not regenerated for MicroPython

See #115

Decide what we do with `== null` to `== None` compilation. Maybe it should be compiled like `is None`; maybe only `===` should be compiled to `is`

Originally created by @SerVB.

Clean up CLI

This module contains a lot of JS-related stuff: https://github.com/krzema12/kotlin-python/tree/python-backend/compiler/cli/cli-py

Optimize collections methods like `map`: don't use Kotlin's inlining, use Python's functions

Originally created by @SerVB.

OOM when generating box plot after syncing with Kotlin 1.6.0

https://kotlinlang.slack.com/archives/C0289CS37AS/p1637332034038000?thread_ts=1637332034.038000&cid=C0289CS37AS

42ac476#diff-e745cd83095cccd974c16374e666a5617b2f4588fc0ea93807a0db575eac6263

Ideas:

• use sequence when processing commits
• limit the size of fetched git history