In the worst case we'll have to write our own tool. But there are some standalone Python app bundlers that might also do the trick, for example https://py2app.readthedocs.io/en/latest/

All we need is to get all imported modules into one directory so that RustPython's py_freeze works

Right now in our Kybra/Azle compilers, when we come across any kind of inline record in records or enums, we create a new rust struct with a random name, and have to do some extra plumbing to get things to work well. This causes our Candid files to be messier than they could be, and the extra plumbing in our compilers is a cost as well.

If you embraced the enum named fields syntax, we might be able to reduce that plumbing for inline records found within enums. See here for what the syntax looks like: https://stackoverflow.com/questions/52134531/how-to-get-access-to-enum-variant-unnamed-field

We might want to start this project out with Rust as the base language. See here: demergent-labs/azle#151

How to represent records, variants, arrays, and other non-primitives in Python typing, how to represent all types (make an issue for each showing how they should look). I just found this awesome Rust/Python types chart: https://pyo3.rs/v0.13.2/conversions/tables.html

text

# str should be an alias to text
text = str

blob

blob = bytes

# or

blob = bytearray

nat and natN

nat = int
nat64 = int
nat32 = int
nat16 = int
nat8 = int

int and intN

# int is builtin
int64 = int
int32 = int
int16 = int
int8 = int

float32

float32 = float

float64

float64 = float

bool

# bool is builtin

null

# None should be an alias to null
null = None

vec t

T = TypeVar('T')
list[T]

# example

@query
def first_list(list_a: list[int], list_b: list[nat]) -> list[int]:
    return list_a

opt t

T = TypeVar('T')
opt = Optional[T]

# example

@query
def first_opt(opt_a: opt[str], opt_b: opt[bool]) -> opt[str]:
    return opt_a

record

# kybra module
record = TypedDict

# example
from kybra import record

MyRecord = record('MyRecord', {
    'prop1': str,
    'prop2': int
})

# or

class MyRecord(record):
    prop1: str,
   prop2: int

# or ideally I think we want this, behaving like a data class or TypedDict
@record
class MyRecord:
    prop1: str,
    prop2: int

tuple

my_tuple = tuple[str, str]

variant

# TODO consider what to do after figuring out records
# TODO we might want to see what ic-py is doing
# We might just want to do some weird inheritance thing with TypedDict, like Variant
# We might want to do that for records and variants

# kybra module
variant = TypedDict

# example
from kybra import variant

MyVariant = variant('MyVariant', {
    'prop1': str,
    'prop2': int
})

# or

class MyVariant(variant):
    prop1: str,
    prop2: int

# or ideally I think we want this, behaving like a data class or TypedDict
@variant
class MyVariant:
    prop1: str,
    prop2: int

func

# kybra module

def Func(callable: Callable) -> Type[tuple[int, str]]:
    return type((0, ''))

# example
from kybra import Func

MyFunc: TypeAlias = Func(Query[[], str])

service

principal

Use the Principal type from https://github.com/rocklabs-io/ic-py

from ic.principal import Principal
p = Principal() # default is management canister id `aaaaa-aa`
p1 = Principal(bytes=b'') # create an instance from bytes
p2 = Principal.anonymous() # create anonymous principal
p3 = Principal.self_authenticating(pubkey) # create a principal from public key
p4 = Principal.from_str('aaaaa-aa') # create an instance from string
p5 = Principal.from_hex('xxx') # create an instance from hex

reserved

from typing import Any

reserved = Any

empty

from typing import NoReturn

empty = NoReturn

canister

from kybra import canister, Query, Update, Oneway

@canister
class MyCanister:
    read: Query[[int, str], str]
    write: Update[[], str]
    oneway: Oneway[[], None]

It would be awesome to allow canister indexing tools to discover which CDK canisters were written in. This should be easily identifiable by doing something like the following in the Candid files:

type CDK = { kybra };

Right now the build process keeps progressing even if there are for example Rust errors

These are blockers for Kybra that we do not have direct control over. We should prioritize these when discussing with DFINITY, the community, and other projects:

• cycle costs too high (after benchmarking we might find out this is not a problem for RustPython/Kybra)
• Wasm binary limit too small (full RustPython stdlib ~30mb unoptimized and ungzipped)
• RustPython binary with full stdlib (including frozen stdlib) is too large and takes too long to compile
  • We want a module bundler (I have already started working on it)
  • We want a tree shaker (get rid of dead code in the final Python byte code)
• dfx integration (dfx kybra [command] and dfx.json)
• RustPython time: RustPython/RustPython#4302

• Since people only read the absolute bare minimum, put in that bare minimum that they read the import things that they must know
• Copy everything from Azle
• Add documentation for function guards
• We might want to explain in the deployment section how to deal with pip cache purge or other problems with old versions of Kybra being present
• we need to discuss the discrepancy for void on init, pre_upgrade, post_upgrade, heartbeat, etc
• look into pipx for quick start and such
• #298
• #299
• #300

See this forum thread here: https://forum.dfinity.org/t/wasm-module-contains-a-function-that-is-too-complex/15407

We've got to find a way to get around this issue or we're stuck.

Importing the entire stdlib with RustPython, including the Rust-implemented stdlib and the frozen stdlib, creates a binary that is far too large to deploy to the IC. After a lot of experimentation, we have decided to move on for now. We will only include the Rust-implemented stdlib.

There are basically two solutions that we find the most promising so far. One is for RustPython to greatly decrease the size of frozen modules, and the other is for DFINITY to increase the Wasm binary limit:

• RustPython solution
  • RustPython/RustPython#4203
  • RustPython/RustPython#4204
• DFINITY/IC solution
  • https://forum.dfinity.org/t/new-and-improved-rust-cdk-first-class-support-for-rust-canister-development/10399/42

Most important modules to enable

• json
• random
• datetime (probably flesh out the date tests)
• base64
• uuid
• string
• os/sys (probably won't work)
• itertools
• collections
• encodings
• sqlite3??
• urllib
• calendar
• codecs
• typing
• io

Suggest the following change to Readme.md.

Change from:

"Your main.py file should look like this":

Change to:

"Your main.py file should be save in ./src and look like this:"

Preferred order: query methods, update methods, records, variants, system methods, ic object

• Records
• Variants #69
• Canister Methods
  • Query, Update
  • QueryManual, UpdateManual #90
• System canister methods
  • init (not done yet)
  • pre_upgrade (not done yet)
  • post_upgrade
  • heartbeat
  • inspect message
• ic object
• inline types
• tuple types #68
• Arrays of everything
• Type aliases #117
• Types with double quotes need to be handled
• Python keywords need to have their underscores removed: #99
• We need inline funcs

Tests

• bitcoin
• blob_array
• bytes
• call_raw
• candid_encoding
• complex_types (ready for testing)
• counter
• cross_canister_calls - #85
• cycles
• date
• ethereum_json_rpc
• func_types - #106
• generators
• heartbeat (Ben now)
• ic_api
• imports
• init (Ben now)
• inline_types
• inspect_message (Ben now)
• key_value_store
• ledger_canister
• management_canister
• manual_reply
• motoko_examples
  • calc
  • counter
  • echo
  • factorial
  • hello
  • hello-world
  • http_counter
  • minimal-counter-dapp
  • persistent-storage
  • phone-book
  • quicksort
  • simple-to-do
  • superheroes
  • whoami
• notify_raw
• null_example
• optional_types
• outgoing_http_requests
• pre_and_post_upgrade (Ben now)
• primitive_types
• principal
• query
• rejections
• simple_erc20
• simple_user_accounts
• stable_memory
• stable_storage
• tuple_types
• update

The first step here is to get RustPython to compile to wasm32-unknown-unknown and deploy it to a local canister. I've attempted this unsuccessfully but didn't spend too much time on it. See here: RustPython/RustPython#2378

These are stdlib modules that create a binary size far too large (like 20-30mb uncompressed):

• base64
• re

http://www.elebihan.com/posts/using-rust-with-buildroot-pre-built-binaries.html

For Kybra and Azle we'll eventually want to ship some kind of cargo/rustc installation script that can just pull down the prebuilt binaries. This would remove the need for the user to install Rust on their system.

Right now our vectors are all just lists, but especially for Vec we will probably want to use a more efficient conversion, and ensure that we convert to bytes.

One possible way to optimize in the future is to freeze or compile the Python source code we are to execute into Python bytecode. This would save the canister from that whole compilation step I would imagine.

For example, we want our canisters to be interacted with from websites like icscan.io. To achieve this, I think there are two paths. One is to have the Candid data stored automatically in the Wasm binary. Another I believe is the __get_temp_hack thing. Kybra already has __get_temp_hack. We may want to also or instead of that do the Candid service data stored in the binary.

Whatever we do, we should ensure the canisters are discoverable.

We need to elegantly import external libraries

This is architecture that needs to be put in place before the CDK ACT framework and AST manipulations can come in full force:

• Python import bundler (py_freeze import bundler): #25
• get modulegraph to be installed automatically
• use venv instead of virtualenv
• incorporate the use of pyenv to ensure python 3.10 is installed on any system
• explain to the user how to setup VS Code to point to 3.10 installed with pyenv, just for the typechecking...we could also have them use that interpreter specifically to avoid any doubts on how the environment is set up
• Testing framework...we might want to pull azle/examples into its own repo and use the tests across everything?: #22
• Get rid of build.sh and just use main.py
• How to represent records, variants, arrays, and other non-primitives in Python typing, how to represent all types (make an issue for each showing how they should look): #20
• Research diagnostic thing for the bois
• IC object (just hook it up, get that plumbing to work): #21
  • Finish adding primitive types to tests in Kybra
  • Add all tests to Azle
  • Get tests to pass for Kybra in GitHub
• TryFromVmValue and TryIntoVmValue for non-primitives, including the derive macros: #23
• Create many basic examples with tests
• Fix one-tuples in Azle and in Kybra
  • It seems like Boa is treating single-element arrays as the single element, instead of as an array (open issue based on what Boa people say in discord)
  • Fix 1-tuples once reviewing the cdk integration
• Allow types to be defined in multiple Python files, we might want to use modulegraph just to grab all of the raw file paths for analysis
  • Write them to a file and read them from a file instead of trying to pass them on the command line, Azle should do this as well
• stable storage: #24
  • I need the system methods working first
• Examples
  • date #125
  • stable memory
  • generators
  • call_raw
• async functions and cross canister calls: #26
• Try to get rid of ic-cdk-optimizer and ic-wasm, and use the ic-wasm library if possible (need to wait on dfx 0.12.0-beta.4): #42
• All pybaseexceptions should print the good errors, fix this immediately #126

It might be better to just use the builtin venv: https://docs.python.org/3/library/venv.html

• wasm-bindgen
• getrandom
• chrono
• time
• Something about importing and/or using Context or VirtualMachine is causing the canister to break on install

Apparently distutils is deprecated as of Python 3.10: https://docs.python.org/3/whatsnew/3.10.html

• derives
  • into
    • struct
    • enum
  • from
    • struct
    • enum
• TryIntoVmValue
  • basic
  • generic
  • numeric
  • vec
• TryFromVmValue
  • basic
  • generic
  • numeric
  • vec
• Add vec impls for derives

It looks like RustPython does not currently have garbage collection, see these issues:

• RustPython/RustPython#4158
• RustPython/RustPython#4180
• RustPython/RustPython#4181

This should work very similarly to the TypeScript -> Candid compiler found in Azle: https://github.com/demergent-labs/azle/tree/main/src/compiler/typescript_to_candid

This might be a fundamental limitation currently set on the IC. I'm not sure I really have the ability to change this, as RustPython is probably the source of all of these functions.

Until the IC increases its Wasm binary size limit, importing typing is just too much...I think because it imports all of the other libs to do its types? I am not sure...for now we might need to just pull from it what we need, and not allow the user to ever import it directly unfortunately. We should have the user import the types they need from kybra for now

• Multiple file imports
• Beta architecture: #19
• #129
• Finish all Kybra examples with tests: #144
• #127
• #130
• #126
• #146
• Ensure Kybra is on the latest cdk_framework commit
• Make sure all special licenses from Azle are applied to Kybra where appropriate
• Create deploy and testing process similar to Azle's
• external libs
• stdlib
• For stdlib and external libs, I think we want some kind of Python frozen bundler. If its' simple enough we might consider creating one ourselves, but open an issue with RustPython to see if they'd be interested in adding the functionality to py_freeze, because that's ideally where it lies. But if we can't wait, we can just do the bundling ourselves, but all of the code found into the python_source directory, and then freeze that directory
• IC object basics
• TryFromVmValue and IntoVmValue for records and variants
• func type
• service type (not at this time)
• cross canister calls
• async functions
• Generate all of the CdkAct nodes...
• Hook in cdk_act
• Start writing tests
  • Consider just using the tests as they are, maybe pull the examples out into their own repo that can be shared?? It might also be useful to write the tests again but using ic-py
• cdk_act isn't handling void or no return types
• Consider adding in the switch between debug and production mode for warnings and errors and such

It seems like when I try to do from typing import Any and use the Any type, then my code breaks. This might be happening with other types as well, I'm not sure what's going on. I've opened an issue with RustPython: RustPython/RustPython#4173

An initial implementation can probably be very similar to Azle's and use JSON.

Close out this issue as well: demergent-labs/azle#654

Really think about if this type of default stable storage implementation will be worth it to the developer.

• Figure out serde for tuples (seems like RustPython's serde is incorrect)
• Figure out serde for bytes (it would be nice if this were efficient)

• Print anything passed in
• Override the print function

hello.
It's a great project and I've been looking forward to it being available in alpha.

When I tried to deploy, I got the following error.

I'm using Apple Silicon (M1 chip).

$dfx -V       
dfx 0.12.0-beta.2

$cargo -V                        
cargo 1.64.0 (387270bc7 2022-09-16)

$rustc -V     
rustc 1.64.0 (a55dd71d5 2022-09-19)

$python -V        
Python 3.9.7

Deploying all canisters.
Creating canisters...
Creating canister imports...
imports canister created with canister id: rkp4c-7iaaa-aaaaa-aaaca-cai
Building canisters...
Executing 'python -m kybra imports src/main.py src/main.did'
info: component 'rust-std' for target 'wasm32-unknown-unknown' is up to date
    Updating crates.io index
  Installing ic-cdk-optimizer v0.3.4
   Compiling proc-macro2 v1.0.44
   Compiling quote v1.0.21
   Compiling unicode-ident v1.0.4
   Compiling cc v1.0.73
   Compiling memchr v2.5.0
   Compiling version_check v0.9.4
   Compiling syn v1.0.101
   Compiling unicode-segmentation v1.10.0
   Compiling regex-syntax v0.6.27
   Compiling serde v1.0.145
   Compiling autocfg v1.1.0
   Compiling libc v0.2.133
   Compiling serde_json v1.0.85
   Compiling serde_derive v1.0.145
   Compiling hashbrown v0.12.3
   Compiling ryu v1.0.11
   Compiling itoa v1.0.3
   Compiling unicode-width v0.1.10
   Compiling os_str_bytes v2.4.0
   Compiling lazy_static v1.4.0
   Compiling termcolor v1.1.3
   Compiling bitflags v1.3.2
   Compiling strsim v0.10.0
   Compiling vec_map v0.8.2
   Compiling humansize v1.1.1
   Compiling heck v0.3.3
   Compiling cmake v0.1.48
   Compiling proc-macro-error-attr v1.0.4
   Compiling proc-macro-error v1.0.4
   Compiling textwrap v0.12.1
   Compiling indexmap v1.9.1
   Compiling wabt-sys v0.8.0
   Compiling aho-corasick v0.7.19
   Compiling regex v1.6.0
   Compiling atty v0.2.14
   Compiling binaryen-sys v0.12.0
   Compiling clap_derive v3.0.0-beta.2
   Compiling clap v3.0.0-beta.2
   Compiling wabt v0.10.0
error: failed to add native library /var/folders/0h/1jsn6jwn7sz_s6zm5tk2znkw0000gn/T/cargo-installAGJQUZ/release/build/wabt-sys-21af77b6012a4d59/out/build/libwabt.a: file too small to be an archive

error: could not compile `wabt-sys` due to previous error

Looking for a solution, I ran cargo install ic-cdk-optimizer --version 0.3.4 directly in terminal and removed it from build.sh and it worked.

rustup target add wasm32-unknown-unknown
# cargo install ic-cdk-optimizer --version 0.3.4

To run kybra_generate, we need the JS part of Kybra to get a list of all the filenames that are in the bundle, and then we pass those file names into kybra_generate as arguments on the command line. This is a problem however, because importing large libraries causes this list of file names to be too large, and then we run into an ENOBUFS (105) error.

Instead, we need to, grab the list of file names from the rust side of things because that is much smaller and will get around this issue.

We are waiting for all of the number of functions and function complexity limits to be raised to sufficient levels, which should happen in dfx 0.12.0-beta.4. We're currently on dfx 0.12.0-beta.3. Once we have those limits increased, let's switch to ic-wasm. We want to do this because ic-cdk-optimizer is deprecated, and it causes issues when being installed on Mac M1s.

Perhaps our ic-wasm problem will be solved now with the wasm function limit increased.

• Ability to store a RustPython context indefinitely in memory and call portions of code at a time, with the state being maintained between calls (so far this is working, but gc has me concerned)
• Ability to convert results from executing code in the context to any Rust type (working for primitive types)
• Fully functioning Rust type that represents a Python generator. We need to be able to progress through the generator in Rust (we have some example code in RustPython to go off of)
• We need to be able to execute code in the VM, get a result, perform async operations in Rust, and then call into the interpreter again. This is what the generators will be used for
• RustPython/RustPython#4203
• RustPython/RustPython#4173
• RustPython/RustPython#4204
• RustPython/RustPython#4166
• RustPython/RustPython#4178
• RustPython/RustPython#4175

Perhaps we do not have to freeze our modules at all, maybe we can get a dynamic loader to work somehow...we could get a list of all modules required, maybe just a directory using our bundler, and then we just upload it like an asset canister. Then the files can be uploaded after deployment...no, this will be slow and complicated. I'm not sure this is the path we want to go down.

This issue entails allowing the developer to include any of their own files

Our Rust TryFrom and TryInto vm value conversion code does not have a general solution for lists of lists (vectors of vectors). This same problem exists in Azle.

• Figure out how the Python, pip, and Rust environments will work together. We'll want to be able to pip install Kybra and execute the code
• Make sure state can be saved between calls
• Be able to call into any function and get the result...this will require vm.invoke and TryFrom and TryInto
• Get KybraTryFromPyObjectRef and KybraTryIntoPyObjectRef written
• Get function parameters to work
• Make CdkActTryInto and TryFrom generic
• Get python typechecking to work in vs code
• Get all of the Python AST code written
  • Query methods
  • Update methods
• Automatic candid generation
• Get update calls to persist...I thought they were working but they don't seem to be: RustPython/RustPython#4175
  • Try granting access to Interpreter.vm and getting a &mut to it...it might just maintain its state at that point...that would be awesome. And instead of persisting the interpreter we might be able to just persist the vm
  • Try importing as a module, might need to use the py_freeze or py_compile macros. Once that is done, use module.get_attr to get the function and then invoke it. State should persist inside of one enter. If that works, perhaps we can just make vm public to get rid of the enter problem for now
• Alpha should be query and update functions working with primitive types?
  • We should also deploy to PyPi
• #16
• #17
• For both multiple files, external libs, and stdlib I think I want to install all of these things in site-packages so that they can be picked up easily from VS code for type checking
  • Figure out how to get default code in setup.py, so that you can do from kybra import
  • use entry_points or scripts to all a pip kybra...
• async fn is not permitted in Rust 2015 this really needs to be fixed, this error requires the project root directory to have a rustfmt.toml file, something to do with the way we're using rustfmt
• Clean up the bash script
• Add good enough documentation for alpha (explain what types are possible, not to use the stdlib or external modules)
• Find out exactly how the PYTHONPATH or extraPaths stuff is working so that we can guide users in exactly how that works
• Run it through a couple people to work out the bugs
• add pip to the documentation
• move around the python extension area
• explicitly let people know that they must deploy from within a virtualenv
• Since people only read the absolute bare minimum, put in that bare minimum that they read the import things that they must know
• Warn people about exactly following the directions
• explain site packages
• say set the type checking mode to strict, not enable it
• sourcing should be part of the commands in the deploy section
• automatically install wasm32-unknown-unknown target, ic-cdk-optimizer, etc
• Consider adding in the switch between debug and production mode for warnings and errors and such
• get rid of link_section or add no_mangle
• VS Code instructions are specific to Ubuntu
• Explicitly tell the user they have to use Python's static typing
• change crazy dfx build script to pip -m kybra maybe with some arguments
• We need VS Code to automatically pick up the virtual env, and then be able to install external packages there and have them picked up by VS Code (for now I think we can have the user manually change the PYTHONPATH in VS Code or with a .env)

For me I had open ssl version 3.0.2 installed but I needed to install libssl-dev

sudo apt-get install libssl-dev

And that worked out ok.

Deliverable: Document this in the README.

We need to be able to from kybra import query, update. I have this working, but I need to get it to work well with VS Code typechecking. Getting the virtualenv correct with site-packages and such should help stdlib and external libs all work well together.