demergent-labs / cdk_framework Goto Github PK

There is a case to be made that ExternalCanisters are the candid Service type and that they should be renamed and merged together. Additionally, we have ExternalCanisters directly on the ACT. We should consider moving them inside the Candid Data Types grouping.

We'd need to consider how ExternalCanister services are different from the main "this" canister.

Why is Opt not in CandidTypes? Tuples are there, funcs are there, then why not Opts? Something seems inconsistent. Also, we should consider renaming that struct to specify that it's meant to capture the stand-alone defined candid types.

Now that the broad strokes for the refactor are done we can add some final polish

• #50
• Create readme (see comments below)
  • for users
  • for developers
  • see also #42 (comment)
• Get rid of the is_type and as_type: see #27 (comment)
• Make sure our generate lib file has no errors #32
• #45
• Make sure all python keywords are handled correctly demergent-labs/kybra#238

For example right now in Kybra you have to add the ic_cdk and ic_cdk_macros, and ic_cdk_timers, but its actually the cdk_framework that determines what version we need.

Right now we box everything. Rather than just bulk boxing everyting (whether it needs it or not) we should only box things that will cause recursive problems (and actually need to be boxed). So it involves evaluating each box and removing the ones that aren't necessary.

Currently the boxing only happens inside of tuples, records, and variants. Here's where you'd want to look for this code:

• cdk_framework/src/act/node/candid/record/member.rs:17
• cdk_framework/src/act/node/candid/tuple/elem.rs:26
• cdk_framework/src/act/node/candid/variant/member:33

Note: Ben already rewrote some of this with the visitor pattern... maybe that essentially was this ticket and this ticket is no longer needed...

We need to be able to have the CDK Framework accept a prefix so that Azle and Kybra can mark their internal functions with the correct prefix. This should be something that sits on the AbstractCanisterTree.

We need to better handle keyword collisions. Some of this work has been done, but it's spotty. We currently really only handle keyword collisions in record/variant members. But, we don't handle collisions when the record/variant itself has a name that would collide.

This is further complicated by the fact that certain keywords are only problems in certain places. For example, in Azle, TS keywords are valid as object keys, but invalid for Record/Variant names.

So, this will require more planning to come up with a scalable solution.

The readme currently focuses on building an AbstractCanisterTree. However, in addition to creating an ACT, CDK authors need to also generate a Rust project including Cargo.toml and Cargo.lock files. There are lots of other nuances to this process. After this ticket is complete a potential CDK author should understand how this process works in both Azle and Kybra and should understand the general recommended steps.

We should implement a CdkActVM trait on the different VMs. This would allow us to abstract away the body creation entirely, as long as the VM implements the correct trait. It might be too much abstraction though

There are many times when we want to prefix functions with _azle (or whatever the name of the CDK is) when converting to token streams. In order to do that the TS AST should set that on the ACT it instantiates. Then, the to_token_stream method on ACT should pass that property along as a param to all to_token_stream methods that it calls on it's children. This means that the signature of to_token_stream should be changed as well.

There is a lot of cloning going on in the codebase. Unless necessary, we should remove these in favor of using references.

• simplify candid generation? We might be able to use the slightly simplified version now (so far there is no simpler way that I know of for Azle, I'm trying to work with the teams to get this addressed)
• update @dfinity/agent to 0.18.1 in tests, run dfx generate again
• demergent-labs/azle#1053
• demergent-labs/azle#1073
• demergent-labs/azle#1035
  • Check for time, get rid of boa hacks
  • Check for randomness, get rid of all hacks if possible
• VmContext trait
• IcApi trait
• TryIntoVmValue trait
• TryFromVmValue trait
• compiler flags to switch between engines
• move the IC object stuff into the cdk framework, generalize everything
• demergent-labs/azle#981
• demergent-labs/azle#782
• demergent-labs/azle#781
• demergent-labs/azle#506
• demergent-labs/azle#432
• demergent-labs/azle#363
• demergent-labs/azle#1124
• #89
• Figure out randomness
• Get a full suite of benchmarks going, run them on previous versions of Azle and the new versions
• Get async to work great, try to do it in JS as much as possible, perhaps the following will become obsolete
  • demergent-labs/azle#1000
  • demergent-labs/azle#999
• demergent-labs/azle#321
• fix ic-wasi-polyfill transient storage issue
  • wasm-forge/ic-wasi-polyfill#9
  • dfinity/stable-structures#110

Global Principles

• Implement this high level overview
  • 1) Global pass to get all of the canister method, external canisters, etc ast nodes,
  • 2) Call to_act on each of them that will return Results,
• #27
• Clean up actable trait
  • Give it a better name
  • Make it applicable for more than just DataType
  • Just get rid of it

Thoughts on having an intermediate step

• Intermediate representation that is X_Ast_Node has:
  • a name,
  • an ast node and implements
  • a number of methods, such as to_act and get_dependent_types. This trait will be for all ast nodes that will guide you through how to implement any new high level things.
• Don't do an intermediate step

Many of the act data type nodes when they have an inner data type, the name of that field will be type_. Dan would like to rename them all to be data_type instead

I think that the is_type and as_type (eg is_array or as_primitive) was only needed back when we were making a global list of all of the types. We might not need it anymore. Probably after the kybra and azle refactor is done we can assess that and take action at that point.

Ask dan if he uses this as all in azle, see if we use it in kybra, if not, get rid of them, if so, see if we actually need them repeat

@bdemann commented on Thu Feb 23 2023

Determining if something is a type alias is challenging with our new system. Currently there are three things that all look like candid type aliases.

1. Genuine candid type aliases

   MyCandidTypeAlias = MyRecord;
   class MyRecord(Record):
        ...

2. Variable assignments
   1. This includes a variable assigned to another variable and you can't discern that it will eventually be a constant just looking at it

   my_var = 3
   your_var = my_var

3. Python type aliases
   1. This includes aliases to aliases that eventually resolve to be a python type alias but isn't discernible just by looking at it.

   my_float = float
   your_float = my_float

I think the base cases (like my_var and my_float) would be pretty simple to sort out. For the variable assignments we can just check to see if the right hand side of a type alias could be a candid data type annotation. For the type alias we could just check that the right hand side isn't a built in python type or a non-record non-variant class. But the other cases (your_var, and your_float) are going to be more challenging to figure out.

On a related note, we also need to fix this issue with the no None constant enforcement

ExprKind::Constant { value, .. } => match value {
    Constant::None => {
        // TODO the problem is that we need to have None in the actual null and void type alias
        eprintln!("{}", self.none_cant_be_a_type_error());
        Primitive::Null.to_data_type()
    }
    _ => {
        todo!()
    }
},

@bdemann commented on Thu Feb 23 2023

I think the quickest way to fix this would be to have a kybra wrapper to denote that a type alias should be a candid type alias
For example

from kybra import TypeAlias
my_type_alias = TypeAlias[MyRecord]
class MyRecord(Record):
     ...

@dansteren commented on Thu Feb 23 2023

Alright, if we decide to do a tree-traversal type algorithm to make sure type aliases point to correct types. The key idea would be to take the list of all type aliases and shrink it down to things that just resolve to valid candid types. This happens very first, before we do any of the validation for canister methods or what not.

Algorithm:

while(type_aliases.len() hasn't changed) {
    for each item in the type_aliases
        if(the right hand side (or traverse records/variants down to their inner type) is a typeref)
            if so, then make sure the ref's name is in the list of all types (including type_aliases, and records, etc.)
            if the name isn't in the list then remove it from the list, Meaning, that Four isn't a valid type because it references not a valid type
}
// If the size of the list is smaller, then rerun algorithm. You're done when the list size doesn't change.

Here's some test cases in Azle: https://gist.github.com/dansteren/9eb3244101a2bd799a5a79669123249a
And in Kybra: https://gist.github.com/dansteren/b7ecf40c31c80d450c831701705f7426

@bdemann commented on Thu Mar 02 2023

We decided to move forward with a wrapper type. I'm going to leave this ticket open for future consideration, but lower the priority

@bdemann commented on Sat Mar 04 2023

See also demergent-labs/kybra#226

Right now IC object code is tightly coupled with the azle project

The idea is that we will move away from forcing the user to create the ACT and find all of its components, but instead we will create traits that the cdk authors will have to implement on their ACT nodes. If the correct nodes have the correct traits, everything else can be done by the cdk_framework. Just an idea, and it might not be a good idea, but something to consider.

Both of these methods require you to manually pass in a parental prefix a bunch of places. However, that shouldn't be necessary as long as these are implemented on things that have a name property or get_name method on them. So, let's rework these methods to leverage that. That way you can't accidentally pass in the wrong thing.

We should remove all _cdk_ prefixes in favor of always prefixing user-defined methods with _cdk_user_defined. This isn't possible right now because of a bug in the candid export_service.

Note: most of the work for this has been done on ⎇941_prefix_user_defined_functions and ⎇azle/941_prefix_user_defined_functions

PR #75 creates a generic CollectResults trait and implements it for vecs. We should extend that functionality to iters so we don't have to call .collect before using it.

The plugins test has been removed do to complications with the sqlite dependencies compiling to wasm32-wasi. This doesn't really seem like our problem, so I'm punting on it for now. Hopefully when the authors of those libraries get things working we can just update the dependencies in the example and reenable the tests.

• sqlite plugin is broken
  • potential fix: rusqlite/rusqlite#1321
  • potential fix: rusqlite/rusqlite#827

Now that we have redone the cdk_framework I want to make sure that all of the places that could have rust keywords handle them correctly. What I suspect is that we don't have full coverage with our kybra test. For example I don't think function guards are handled or even if they need to be handled.

Right now we use a hard-coded name for system methods, for example _azle_heartbeat. We should consider using the actual name defined by the developer in the source code language.

We need to have a simple Readme to help users and contributors understand how the cdk_framework works. Essentially distill Ben's knowledge into the Readme so when we comeback to it in six months we still understand what on earth he was thinking

Related to https://github.com/demergent-labs/kybra#issuecomment-1473383974
Related to #32 but the opposite problem

The ideal solution for this is for the rust cdk to take a closure like with the timers API instead of a string name of the guard function. Then we wouldn't have the problem from https://github.com/demergent-labs/kybra#issuecomment-1473383974

In the meantime we are going to make a nice error for this and wait for people to actually have issues with this before we try and make an unideal stop gap solution.

As part of this ticket it might be good to explore other things that might conflict besides guard functions. Like if two modules had query methods of the same name and a person wanted to compose those two modules together into one canister.

Description
There are two motivations for this ticket:

The generated lib.rs file is getting rather big and unweildy. The code can logically be broken down into modules and so it makes sense to do that. It would lead to a more readable/navigatable file.
We still have the possibility of name collisions because of the way JS/Python imports work. For example, if a developer imports the managementCanister, they also get all the types/structs/enums that it uses behind the scenes in Rust, whether they use them or not. We can likely prevent this problem by embracing modules.
We already attempted some of this before. See demergent-labs/azle#985 and #64. However, we encountered many problems with generics, and knowing which prefix to add at which level (See demergent-labs/azle#985 (comment)).

This is likely going to be a cross-repository problem/solution, with some changes implemented in the CDK Framework, and some in Azle/Kybra.

The rust canisters that we generate for users are just one really really big lib.rs file. We should be better about how we generate their rust code and make it look more like a manually typed rust project, meaning, break cohesive parts out into separate modules and create multiple files.

The reason for doing this is two-fold:

• It make things more organized
• It may help solve naming collisions between our code and user-defined code

The same thing that we're doing for structs and enums.

Until we implement this, you cannot have self-referencing/recursive funcs.

• Make sure assumption about func inline types is correct
• Clean up the lib file
• Consider getting rid of create_identifier if we aren't using it.
• Rename parental_prefix (See #27 (comment))
• Make a better name for HasEnclosedType (see: #27 (comment))
• Better name for public_canister_methods
• Investigate enforcing certain fields on a rust struct
• data_types folder to candid and DataTypes to CandidTypes

We might want a HasElem for tuples and use the index to generate the inline name. Then for the HasMember instead of using the index we could use property name instead of an index.

CDK Refactor:

Phase 2:

• Parse in a single pass instead of making a tree
• Set up ACT to support new parsing method
  • Reintroduce top level DataTypes
  • Get rid of Proclaim implementation and simplify to just a to_token_stream() function
• Rework ToDeclaration trait
  • Rename Declaration to Proclamation and allow token streams that represent declarations to actually be called declarations.
  • Rename children to be inline declarations
  • Go through all implementations and make sure that only the inline declarations are being created.
• Organize impls and functions in each file (see #27 (comment))
• Address notes from initial review
• Only use Member and HasMember for variants, records, and tuples
  • Make sure tuples are well documented as to how they are different. (see #27 (comment))
  • Create a new thing for TypeAlias, Array, and Opt
• Consider getting rid of the Proclaim impl for Vec<T>
  • Also for Option<T>
• use vec instead of hashmap for the inline types (see: #27 (comment))
• use Declaration and TypeAnnotation type aliases wherever we can to increase clarity
• get rid of the found_types for to_data_type. Now that we are parsing in a single pass we don't need it anymore.
• initial lib.rs clean up
• Get rid of boxed (if we aren't using it. I'm pretty sure we shouldn't be using it)
• Consider having access to both strings and idents as necessary (see #27 (comment))
• Move proclaim and declaration into node mod

Phase 1:

• Remove Act from all of the datatypes and make sure that we use data_type::Option etc
• Make sure all of the canister methods are options if we sometimes don't use them.
  • Init and post we will always need
• Make the ACT nested to reflect the future folder structure
• For data types:
  • Remove type ref as a concept
  • Add a type alias data type (it will be similar to typeref)
    • a type alias will have a name and a data type and that data type could be another type alias so it will be type alias all the way down.
  • Remove the LiteralOrTypeAlias
  • Remove hash and use parent node info to create a unique name.
  • Remove the global list of all types and just have the types be the leaf nodes of canister methods
• Make sure all TODO are gone
• Make sure all contexts that should be keyword_lists are name as such
• Try to get rid of muts for maps
• I am not sure that Has Members is the right fit for Func. Investigate why we have it and if we can get away with out it
  • Address #27 (comment)
  • I think once we get rid of the inline types thing and just get child declarations correctly we should be good to go
  • Also look at
    • opt
    • array
    • type alias
• Imports
  • Make sure we are importing instead of qualifying
  • clean up imports
    • make sure things from super are from super
    • make sure things from crate are from crate
    • make sure they are organized according to the style guide
    • make sure all of the mods are public or have a good reason for private
• Make sure that all of the prefixes are consistent.
  • prefixes should only have to go back as far as the first parent with an actual name
  • when you receive a prefix it's the name of your parent, when you pass a prefix it's your name, plus any positional modifiers that your children wouldn't know about.
  • If you have a name you don't need to pass on your parent's info to your children
• Separate TokenStreams into all of the things that tokenstreams are supposed to represent
  • declarations
  • type annotations
  • just random groups of tokens. (If it's just a random group of tokens then it shouldn't be treated like it might be a type annotation, and a type annotation certainly shouldn't be treated like a declaration. Let's make it more clear when we are create which)
  • get rid of ToTokenStream trait. (It doesn't make sense to group all of them together anymore)
  • update the declaration to have completely flat children, that is a hashmap with tokenstreams instead of hashmap with declarations.
  • Make sure that both children and declaration get called when creating children
  • Address #27 (comment)
    • Same question but for External Canisters
• Make sure that AbstractCanisterTree properly implements to definition.
  • Have a public facing function called to_token_stream() that takes nothing in and creates the declaration and children declarations for the act
• Remove the type ref act data type
• Get rid of the collect_inline_types function. We shouldn't need it anymore
• Look at final Act prefix holdouts
  • Actable trait
  • ToAct trait
  • ToActDataTypeNode
  • ActNode (at the highest level do we need to have ActNode? or is it just Node?)
• get rid of GetAllTypes probably
• Simplify the act Declaration: see #27 (comment)
• Make sure that if we are in a tree loop that we can get out of it. So if a child has a parent as one of it's children or other descendants we need to see that and stop going down that path.
  • address this: #27 (comment)
  • Only box things that need to be boxed
    • I think we can get rid of this and use the boxed data type instead.
• Ask Jordan and Dan about the generate_function function in update and query that are the exact same but duplicated
• rename function_signature to better describe what it actually is and rename the function that generates it to match
  • also bring it out to be called something that represnets both query and update. Maybe generate_public_entry_point_function_body. generate_public_canister_method_body. generate_requestable_canister_method_body.

• We need to traverse the ACT to make sure that all we don't have any undefined types.
  • And by traverse I also mean that any inline types in methods or other methods might use type_refs and also need to report them
• We need to make sure that all generated lib files compile without error.
• Make sure that guard functions exists if we are using them.

Does this mean that we essentially do the the dependency pass on the cdk_framework and exclude things that were never used? If we have a type alias that isn't used and won't compile, where should we catch that? On the cdkframework? Or in kybra and azle? for example type float32 = float

Are we introducing errors at this point? If so how pretty do they need to be?

One of the appealing things about doing as many errors as possible on the cdk end is that we still have access to the source file, and we don't have access to that anymore on the framework end.

Very closely relates to #55