Enhancement

At present, KCL's attribute operators provide idempotent merge:, add +=, and override= three attribute operators (delete by using the Undefined override table) to modify the configuration, and it also defaults that all configuration values in the configuration block are variable. It does not provide semantics similar to Java final attributes, that is, the configuration is not allowed to be modified, and KCL is required to provide corresponding unique configuration semantics and stronger immutability

Sema

Extend immutable semantics to property operators in KCL (analogous to using different keywords in GPL to determine the variability of variables, see reference for details)

• For a declared property/variable starting with _, it is mutable.
• For a property/variable declared not starting with _, its mutability depends on the property operator it uses
  • Properties declared with the = operator are immutable.
  • Properties declared with the : and += operators are mutable.

Examples

• Multiple uses of = assignment are not allowed for properties within the same configuration block.

config = {
     attr1 = "v1"
     attr1 = "v2" # Error, attr1 is an immutable attribute
     attr2.key1 = "v1" # Equivalent to attr2: {key1 = "v1"}, attr2 uses : declaration and is variable
     attr2.key2 = "v2" # Since attr2 is mutable, different key-value pairs can be added multiple times and merged
}

• Multiple uses of = are also not allowed in different merge configuration blocks.

import app

appConfig: app.Config { # appConfig uses : declaration and is a mutable variable
     attr1 = "v1" # attr1 is declared with = and is an immutable attribute
     attr2: "v2" # attr2 is declared with : and is a mutable attribute
}

appConfig: app.Config { # appConfig is a mutable variable that can be merged multiple times
     attr1 = "new_v1" # Error, attr1 has been declared once with = and cannot be modified
     attr2 = "new_v2" # Ok, attr2 is declared with : and can be modified with =
}

• With issues:
  • Use the search tool before opening a new issue.
  • Please provide source code and commit sha if you found a bug.
  • Review existing issues and provide feedback or react to them.

Description

@chai2010
I run './run.sh -a build' in the KCLVM directory and can't get the result.

How to reproduce

git clone 'XXX'
cd KCLVM
./run.sh -a build

Expectations

KCLVM was successfully built.

Actual result

configure:

By default, distutils will build C++ extension modules with "g++".
If this is not intended, then set CXX on the configure command line.

checking for the platform triplet based on compiler characteristics... darwin
configure: error: internal configure error for the platform triplet, please file a bug report

• cd /Users/shijun/Workspace/KCLVM/_build/dist/Darwin/kclvm
• mkdir -p bin
• mkdir -p lib
• cp /Users/shijun/Workspace/KCLVM/_build/dist/Darwin/cpython/bin/python3.7 /Users/shijun/Workspace/KCLVM/_build/dist/Darwin/kclvm/bin/kclvm
  cp: /Users/shijun/Workspace/KCLVM/_build/dist/Darwin/cpython/bin/python3.7: No such file or directory

Environment

• KCLVM version: no kclvm release
• Operating system: MacOS Monterey 12.3.1

Enhancement

The current KCL repository has initially established simple development documents, ISSUE and PR templates, etc. However, there is a lack of detailed developer manuals to help all developers develop better collaboratively, such as https://github.com/rust-lang/rustc-dev-guide in rustc.

To sum up, KCL needs a corresponding developer manual to describe the following things clearly:

• How to participate in the design and discussion of KCL
• How to practice on the basis of existing KCL code
• How to write tests and verify development features
• How to merge code into KCL code repository trunk
• KCL architecture and compiler sub-module principle explanation

Enhancement

According to the content in the Roadmap #29 , use Rust to rewrite the KCL query tool

Enhancement

According to the content in the Roadmap (ref #29 ), use Rust to rewrite the KCL Format tool, the specific method is to compile the format tool based on tools::printer (ref #5 )

Enhancement

Attribute type checking information can be further enhanced. The information for attribute type checking can be further enhanced. For example, for the following KCL code, the following code will report a type error and a runtime error with unclear information respectively.

• kcl code

schema Data:
    id: int = 1

dataList = [{data = 1} for data in [Data {}]]
dataMap = {data = 1 for data in [Data {}]}

• The error message is

KCL Compile Error[E2A31] : Illegal attribute
---> File test.k:4:14
4 |dataList = [{data = 1} for data in [Data {}]]
             14 ^  -> Failure
type 'Data'

KCL Runtime Error[E3M38] : Evaluation failure
---> File test.k:5
5 |dataMap = {data = 1 for data in [Data {}]} -> Failure
unhashable type: 'dict'

Sometimes this is confusing to users, the error message can be improved. The improved error message is as follows:

KCL Compile Error[E2L23] : A complie error occurs during compiling
---> File test.k:4:14
4 |dataList = [{data = 1} for data in [Data {}]]
            14  ^  -> Failure
illegal attribute type, expect str, got 'Data'

KCL Compile Error[E2L23] : A complie error occurs during compiling
---> File test.k:5:12
5 |dataMap = {data = 1 for data in [Data {}]}
          12  ^  -> Failure
illegal attribute type, expect str, got 'Data'

Enhancement

https://github.com/KusionStack/KCLVM/blob/main/kclvm/ast/src/ast.rs#L265

pub struct TypeAliasStmt {
    ...
    #[serde(skip_serializing)]
    pub ty: Option<NodeRef<Type>>, // remove Option
}

• SchemaIndexSignature
• SchemaAttr

Bug Report

1. Minimal reproduce step (Required)

use kclvm_parser::parse_expr;

parse_expr("", Some("".to_string()))

2. What did you expect to see? (Required)

parse_expr should return a Result<Expr> or Option<Expr> instead of panic directly.

3. What did you see instead (Required)

parse_expr raises a panic

Enhancement

All KCLVM Rust code needs to be formatted using the rustfmt tool and pinned to the rustfmt style file. This is done for two purposes:

• Unify KCLVM Rust formatting style using rustfmt.toml.
• Add rust code formatting check points for Github actions later.

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

cd scripts/docker/kclvm-builder-centos8
make

2. What did you expect to see? (Required)

3. What did you see instead (Required)

Error: Failed to download metadata for repo 'appstream': Cannot prepare internal mirrorlist: No URLs in mirrorlist

4. What is your KusionStack components version? (Required)

The KCL language is currently in a stage of rapid development. Therefore, in the future, KCL will continue to iteratively evolve around the goals of stability, ease of use, and ecological expansion. At the same time, the KCL language will be used in more field scenarios for continuous iteration and development. The following are some of the contents of the route planning, welcome to discuss and exchange.

• Documentation improvements: the main content includes KCL specification documentation, KCL Tour documentation, KCL code labs, KCL language courses, KCL best practices Documentation, KCL programming books, etc.
• language improvements
  • Language feature simplification
  • Language stability improvements
  • Complete package management system
  • Constraints and Policy Capabilities Support
• Compiler improvements
  • The compiler supports parallel parsing, incremental compilation, and other features to further improve the performance of the main body of the compiler.
  • The evolution of the front-end architecture of the compiler makes the compiler itself have better scalability. The intermediate IR code is extended based on the abstract syntax tree AST to optimize the code as much as possible.
• Ecological expansion
  • KCL defines API specification: define your own API/CLI/Plugin and other interface definitions through KCL
  • Ecological expansion of multiple domain languages: Support more domain languages ​​and format integration, such as HCL, Jsonnet, Protobuf, Helm, Kustomize, and other language and tool support.
  • Complete multi-language API construction: support KCL compilation, syntax checking, automatic query modification, etc., serialization/deserialization, KCL code writing, and other core API functions, combined with community strength to support Rust/Python/Java/Go and other languages.

KCL v0.7.0

Goals

Growth and developer experience

• Language
  • Developer Experience & Redesigned Lang UI
  • Github KCL code highlighted
• Tools
  • KCL Import Tool
  • KCL Fix Tool
  • KCL Find Ref Tool
  • KCL Format Tool
  • KCL Test Tool
  • KCL Doc Tool
  • KCL IDE
  • Unified convergence of command line interface to kpm
• Integrations
  • 300+ Builtin Models & KPM & Registry & ArtifactHub Integration
  • UI & Engine
  • API Composition Project e.g., Crossplane
• Arch

Milestone

• 2023.10
  • Unified KPM CLI
  • KCL Import Tool
  • KCL Find Ref Tool
  • KCL Fix Tool
  • 60+ Models
  • API Composition Project e.g., Crossplane
• 2023.11
  • KCL Format Tool
  • 150+ Models
  • KCL Test Tool
  • Config UI
• 2023.12
  • 300+ Modules
  • KCL IDE
  • Arch
  • Config UI & Backstage UI Plugin Design

Backlog

• https://github.com/orgs/kcl-lang/projects/1/views/4?layout=roadmap

2024 Roadmap

• #882

Enhancement

We have a core dynamic library at the KCLVM Rust level, which has an API that can execute KCL code and return JSON/YAML results. When other languages call the API, they need the ability to release the memory allocated by the KCL code. Because at this stage KCL does not have the ability to automatically manage memory.

Enhancement

The kcl code (test.k) is:

schema Base(Sub):

schema Sub(Base):

The error message is

KCL Compile Error[E2D33] : Cycle Inheritance is illegal
---> File test.k:3:1
3 |schema Sub(Base):
 1 ^  -> Failure
Sub and Base

The improved error message is

KCL Compile Error[E2D33] : Cycle Inheritance is illegal
---> File test.k:3:1
3 |schema Sub(Base):
 1 ^  -> Failure: There is a circular reference between schema Sub and Base
Cycle Inheritance is illegal

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Restore the "kclvm/runner/test.rs/fn test_kclvm_runner_execute()" method that has been commented out.
In the KCLVM

make sh-in-docker
cd kclvm/runner
cargo test -- --nocapture

2. What did you expect to see? (Required)

running 1 test
test tests::test_kclvm_runner_execute ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.10s

3. What did you see instead (Required)

running 1 test
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/command.rs:391:66
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

src/command.rs:391:66

let txt_path = std::path::Path::new(&executable_root)
      .join(if Self::is_windows() { "libs" } else { "lib" })
      .join("rust-libstd-name.txt");
391 ------>  let rust_libstd_name = std::fs::read_to_string(txt_path).unwrap();
                                                                     ^ 66

Cannot found the file "rust-libstd-name.txt".

4. What is your KusionStack components version? (Required)

kclvm version is 0.4.2; checksum: e07ed7af0d9bd1e86a3131714e4bd20c

Enhancement

kusionstack/kclvm-builder docker image is missing python pip and llvm lld components.

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Write the following KCL code (test.k) and run:

data: [int] = [
    if False:
        *[0]
    else:
        *[1]
]

test.k

2. What did you expect to see? (Required)

The YAML output:

data:
- 1

3. What did you see instead (Required)

The compile error:

KCL Compile Error[E2G22] : The type got is inconsistent with the type expected
---> File test.k:1:1
1 |data: [int] = [
 1 ^  -> got [int(0)|[int(1)]]
expect [int], got [int(0)|[int(1)]]

4. What is your KusionStack components version? (Required)

v0.4.2-alpha.4

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

take str "8_________i" as the input parameter of method "kclvm-parser.parse_expr".

parse_expr("8_________i");

2. What did you expect to see? (Required)

kclvm output syntax error message.

3. What did you see instead (Required)

the rust panic message.

thread '...' panicked at '{"__kcl_PanicInfo__":true,"rust_file":"","rust_line":0,"rust_col":0,"kcl_pkgpath":"","kcl_file":"","kcl_line":1,"kcl_col":0,"kcl_arg_msg":"","kcl_config_meta_file":"","kcl_config_meta_line":0,"kcl_config_meta_col":0,"kcl_config_meta_arg_msg":"","message":"UnexpectedToken { expected: [\"int\"], got: \"8_________i\" }","err_type_code":5,"is_warning":false}', src/session/mod.rs:48:9
stack backtrace:
...
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
UnexpectedToken { expected: ["int"], got: "8_________i" }

4. What is your KusionStack components version? (Required)

kclvm version is 0.4.2; checksum: e07ed7af0d9bd1e86a3131714e4bd20c

Enhancement

At present, several compilation stages in the KCL compiler code, such as parser, resolver, and code generation, lack the complete benchmark test set of e2e, which leads to the need to add new benchmarks when modifying the KCL compiler code. There is no good unity, and cannot reflect the e2e performance comparison.

In short, the KCL compiler needs a benchmark test set that can cover most of the syntactic and semantic e2e tests.

Bug Report

1. Minimal reproduce step (Required)

On ubuntu 20.04, run kcl examples/hello.k --target native

2. What did you expect to see? (Required)

The right YAML output

3. What did you see instead (Required)

__main__.so no such file or directory

CompilerBase-Error is an error handling kit, whose goal is to help compiler developers build the error handling of their own compiler during the development.

google docs:
[WIP] https://docs.google.com/document/d/1oXZr_T76DkL9tkU2RT5anNr615xpLperOghKWdG79NE/edit?usp=sharing
[WIP] https://docs.google.com/document/d/1xFwBOkwD2cJYdb6L-JLKgRMYCy_xfLKgU9-kdPZtqas/edit?usp=sharing

yuque docs:
[WIP] https://www.yuque.com/docs/share/64c4e739-9cbf-4946-8659-5b35b86f5956?# 《CompilerBase-Error》

Enhancement

The script builds KCLVM python package and publishs it to PyPI

Enhancement

KCL Lint

1 Background

• An indispensable tool for programming languages: code checking tools. Code checking is available for most programming languages, and generally compilers have built-in checking tools.
• KCL Lint is designed to help users to coding more compliant with the KCL code specification, detect potential problems in the code, help enforce KCL coding standards, and provide features such as simple refactoring suggestions or automatic fixes.
• Use the lint tool to build ci tests for internal and external Konfig repositories to ensure the quality of the code.

2 Design

2.1 Difference with python ver.

The python version of KCL Lint follows the design of pylint and is a compiler-independent tool. The Lint has indepent command line parsing. In main fuction, Linter calls the parse_program() function to obtain the AST, and then checks the AST, generates Lint messages and output them. Therefore, the python version of KCL Lint needs to deal with many problems by itself, such as parameter parsing, path processing, file traversal, etc. In fact, these work has already been handled once in KCLVM. At the same time, the Lint tool also maintains another error messages and error output system, which is somewhat separated from the main body of KCLVM, and also required cost to maintain. Moreover, for some Warning level information, the user will not be prompted during the compilation process. They need to run the lint command separately to see it.

The KCL Lint rust version is based on the design of Rustc's Resolver and Lint, and executes lint checking during the semantic phase of compilation. Lint shares a common error handling system with the rest of the KCLVM. Lint tool checks AST, generates diagnostics and inserts them into handler.diagnostics, which are handled by the KCLVM. This avoids the need to maintain an additional set of error messages, and also throws the Lint-checked problems at compile time without the need to perform additional lint checks. If a separate lint check is required, the diagnostics are thrown after the lint check and the program exits. This avoids maintaining an additional set of error messages, and also emit the problems detected by Lint at compile time. If only lint checking is required, KCLVM will emit diagnostics after lint checking and exits the main program.

When checking, the python ver. Lint divides different checks into multiple checkers by AST type, such as ImportCheck, BaseChecker etc. Each checker needs to traverse AST at least once. The Rust version is no longer divided into multiple checkers, but collects all lint checks by AST type in a CombinedLintPass structure. When traversing the AST node, Lint calls the check method of CombinedLintPass, which run all lint checks in one traversal.

2.2 Overall

Lint is executed during the semantic analysis phase, and the main structure consists of Lint, LintPass, CombinedLintPass and Linter, which implements the walker methods for traversing AST. When traversing AST node, the corresponding check method in CombinedLintPass is called. Static information and check methods for each lint are defined in Lint and LintPass respectively. CombinedLintPass aggregates the checks in these LintPasses according to the type of AST node.

2.3 Specific design

2.3.1 Lint

Lint is a struct type that defines a lint. It is a global identifier and a description of the lint, which contains some stastic information about the lint (name, level, error message, error code, examples, etc.).

pub struct Lint {
    /// A string identifier for the lint.
    pub name: &'static str,

    /// Level for the lint.
    pub level: Level,

    /// Description of the lint or the issue it detects.
    /// e.g., "imports that are never used"
    pub desc: &'static str,
    
    // Error/Warning code
    pub code: DiagnosticId,
}

2.3.2 LintPass

The LintPass is an implementation of the specific check logic for Lint, containing the check methods that need to be called when traversing the AST tree. LintPass is defined as a trait, which needs to be implemented for each definition of lintpass. Not every Lint needs to check all ASTs, but only the check methods required by the Lint need to be overridden. So when defining trait LintPass, we give all methods a default implementation, i.e. a null check, and when defining LintPass, just override the partial check function.

Every LintPass needs to implement the get_lint() method to generate the corresponding Lint structure.

pub trait LintPass{
    fn name(&self);
    fn get_lint();
    fn check_ident(&mut self, a: ast::Ident, diags: &mut IndexSet<diagnostics>){}
    fn check_module(&mut self, a: ast::Module, diags: &mut IndexSet<diagnostics>){}
    fn check_stmt(&mut self, a: ast::Stmt, diags: &mut IndexSet<diagnostics>){}
    ...
}

pub struct LintPassA{
    name: str
}

pub struct LintPassB{
    name: str
}

impl LintPass for LintPassA{
    fn name(){..}
    fn get_lint(){...}
    fn check_ident(&mut self, a: ast::Ident, diags: &mut IndexSet<diagnostics>){
        ...
    }
}


impl LintPass for LintPassB{
    fn name(){..}
    fn get_lint(){...}
    fn check_stmt(&mut self, a: ast::Stmt, diags: &mut IndexSet<diagnostics>){
        ...
    }
}

2.3.3 CombinedLintPass

Each LintPass defined separate check function for AST node. Traversing the AST once for each Lint individually would cause a relatively large performance overhead. Therefore, the CombinedLintPass structure is defined to aggregate the check methods of all defined LintPasses. By calling the check method of CombinedLintPass when traversing the AST, all Lint checks can be done in one traversal.

pub struct CombinedLintPass {
    LintPassA: LintPassA;
    LintPassB: LintPassB;
    ...
}

impl CombinedLintPass{
    pub fn new() -> CombinedLintPass { 
        CombinedLintPass {
            LintPassA: LintPassA::new(),
            LintPassB: LintPassB::new(),
            ...
        }
    }

}

impl LintPass for CombinedLintPass {
    fn check_ident(&mut self, a: Ident, diags:&mut IndexSet<diagnostics>){
        self.LintPassA.check_ident(a, diags);
        self.LintPassB.check_ident(a, diags);
        ...
    }
    fn check_stmt(&mut self, a: &ast::Stmt, diags: &mut IndexSet<diagnostics>){
        self.LintPassA.check_stmt(a, diags);
        self.LintPassB.check_stmt(a, diags);
        ...
    }
}

2.3.4 Linter

Linter is a structure that traverses the AST and required to implement mothods of Walker. Linter calls the check method of CombinedLintPass when traversing the AST.

pub struct Linter<T: LintPass> {
    pass: T,
    diags: diagnostics
}

impl Checker{
    fn new() -> Checker{
        Checker{
            pass: CombinedLintPass::new()
            diags: Default::default()
        }
    }
}

impl ast_walker::Walker for Linter{
    fn walk_ident(&self, a: ast::Ident, diags: &mut IndexSet<diagnostics>){
        pass.check_ident(a, diags);
        walk_subAST();
    }
    fn walk_stmt(&self, a: ast::Ident, diags: &mut IndexSet<diagnostics>){
        pass.check_stmt(a, diags);
        walk_subAST();
    }
}

2.3.5 lint message

Lint errors are consistent with KCLVM error and are stored as diagnostics types, which are handled by the error handling module.

2.3.6 cli agrs

2.3.7 lint config

There are some lint checks that support custom parameters, such as naming convention, code length, etc., and the lint tool itself requires some configuration, such as allowing the user to ignore some checks. Lint configuration is set according to the following priorities.

1. the config file specified by the cli parameter
2. the .kcllint file in the directory of the file or folder being checked.
3. default config

The configuration file is written in yaml format, e.g.

ignore: ["E0501"]
max_line_length: 120

2.3.8 others

macro

The definitions of Lint, LintPass and CombinedLintPass have a lot of repetitive code that can be generated using macro definitions. For example, declare_lint! and declare_lint_pass! are used in rustc to define Lint (WHILE_TRUE) and LintPass (WHILE_TRUE).

declare_lint! {
    /// The `while_true` lint detects `while true { }`.
    ///
    /// ### Example
    ///
    /// ```rust,no_run
    /// while true {
    ///
    /// }
    /// ```
    ///
    /// {{produces}}
    ///
    /// ### Explanation
    ///
    /// `while true` should be replaced with `loop`. A `loop` expression is
    /// the preferred way to write an infinite loop because it more directly
    /// expresses the intent of the loop.
    WHILE_TRUE,
    Warn,
    "suggest using `loop { }` instead of `while true { }`"
}

declare_lint_pass!(WhileTrue => [WHILE_TRUE]);

impl EarlyLintPass for WhileTrue{
  ...
}

And the definition of BuiltinCombinedEarlyLintPass

early_lint_passes!(declare_combined_early_pass, [BuiltinCombinedEarlyLintPass]);

// Expand all macros
pub struct BuiltinCombinedEarlyLintPass {
    UnusedParens: UnusedParens;
    UnusedBraces: UnusedBraces;
    ...
}

impl BuiltinCombinedEarlyLintPass{
    pub fn new() -> Self {
        UnusedParens: UnusedParens,
        UnusedBraces: UnusedBraces
        ...
    }
    
    pub fn get_lints() -> LintArray {
        let mut lints = Vec::new();
        lints.extend_from_slice(&UnusedParens::get_lints());
        lints.extend_from_slice(&$UnusedBraces::get_lints());
        ...
        lints
    }
}

impl EarlyLintPass for BuiltinCombinedEarlyLintPass {
    fn check_ident(&mut self, context: &EarlyContext<'_>, a: Ident){
        self.UnusedParens.check_ident (context, a: Ident);
        self.UnusedBraces.check_ident (context, a: Ident);
        ...
    }
    fn check_crats(&mut self, context: &EarlyContext<'_>, a: &ast::Crate){
        self.UnusedParens.check_crats (context, a: Crate);
        self.UnusedBraces.check_crats (context, a: Crate);
        ...
    }
}

Ref

• RustcDevGuide: https://rustc-dev-guide.rust-lang.org/
• Rustc: https://github.com/rust-lang/rust.git
• Konfig: https://github.com/KusionStack/konfig
• Pylint: https://github.com/PyCQA/pylint

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

• test.k

a: int = {a = 1}

• cmd

kcl test.k --target native

2. What did you expect to see? (Required)

The error message

expect int, got {str:int(1)}

3. What did you see instead (Required)

The yaml output

a:
  a: 1

4. What is your KusionStack components version? (Required)

kclvm version is 0.4.2; checksum: e07ed7af0d9bd1e86a3131714e4bd20c

Enhancement

Code structure refactoring: sema::eval and sema::ty::parser modules are currently placed in kclvm_sema crate, which is not very suitable, they need to be moved to kclvm_parser for unified management. Files Involving modifications:

• kclvm/sema/src/eval/mod.rs
• kclvm/sema/src/ty/parser.rs

Deps.

• #121
• TypeCast Expression AST Node.

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

take str "fs1_i1re1~s" as the input parameter of method "kclvm-parser.parse_expr".

parse_expr("fs1_i1re1~s");

2. What did you expect to see? (Required)

kclvm output syntax error message.

3. What did you see instead (Required)

the rust panic message.

thread 'main' panicked at 'invalid binary expr: missing binary operation: ()', parser/src/parser/expr.rs:132:22
stack backtrace:
...
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

4. What is your KusionStack components version? (Required)

kclvm version is 0.4.2; checksum: e07ed7af0d9bd1e86a3131714e4bd20c

Good CONTRIBUTING documentation, but there's an outdated introduction in Code Structure chapter: I can't find a spec file or directory anywhere in this repo.

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Take “fh==-h==-” as input to “parse_expr”.

parse_expr("fh==-h==-");

2. What did you expect to see? (Required)

kclvm output syntax error message.

3. What did you see instead (Required)

Rust array out of bounds error

4. What is your KusionStack components version? (Required)

kclvm version is 0.4.2; checksum: e07ed7af0d9bd1e86a3131714e4bd20c

Enhancement

when use override_file API, it needs to support import deduplication for different aliased packages.

• Before override_file call with the parameter import_paths = ["pkg.to.path", "pkg.to.path as alias"]

schema Person:
    name: str

• After override_file call

import pkg.to.path
import pkg.to.path as alias

schema Person:
    name: str

Enhancement

KCLVM needs to be fully tested on different platforms in Github actions including macos, ubuntu and windows. Just like follows:

Bug Report

1. Minimal reproduce step (Required)

Write the following code (test.k) and run:

schema Data:
    [
        a for a in [0, 1, 2]
    ]

kcl test.k --target native

I got the internal parse error

2. What did you expect to see? (Required)

Got a correct output

3. What did you see instead (Required)

Got the internal parse error

4. What is your KusionStack components version? (Required)

The latest main branch. (2022.05.24)

Enhancement

• before

container.envs = [{"name1": "value1"}, {"name2": "value2"}]

• after

container.envs = [
    {"name1": "value1"}
    {"name2": "value2"}
]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

KCL code structure

├── hello.k
└── pkg
    └── pkg.k

• hello.k

import pkg

person = pkg.Person {}

• pkg/pkg.k

schema Name:
    name?: str
    
schema Person:
    name?: Name = Name {name = "Alice"}

run:

kcl hello.k --target native

2. What did you expect to see? (Required)

The YAML output

person:
   name: Alice

3. What did you see instead (Required)

The panic info:

Symbol not found: _$pkg.$Name.11\n Referenced from: /.kclvm/cache/0.4.2-e07ed7af0d9bd1e86a3131714e4bd20c/pkg.dylib

4. What is your KusionStack components version? (Required)

kclvm version is 0.4.2; checksum: e07ed7af0d9bd1e86a3131714e4bd20c

Enhancement

Many errors in the current KCLVM Rust Parser stage are directly panicked, the location information of some AST nodes is not completed, and there is no good error handling and error disclosure. It is necessary to further collect and reveal errors. Such as the following places:

• bump_token
• bump_keyword
• struct_span_error
• struct_compiler_bug
• panic call in the parser::ty module
• panic call in the parser::stmt module
• error message about on indent is un-friendly.
  • Invalid syntax: fatal: logic error on dedenting. -> Unindent 3 does not match any outer indentation level.

Enhancement

KCLVM Resolver checking for import statements should be enhance. Need to check for user-defined but unused , and duplicate defined import statements. Prompting users to remove useless importstmt to improve compilation speed.

• kcl code

import a
import a

schema Data:
    id: int = 1

• The warning message like

KCL Warning [ReimportWarning]
---> File /path_to_file/main.k:2:1
1 |import a
a is reimported multiple times.

KCL Warning [UnusedImportWarning]
---> File /path_to_file/main.k:2:1
1 |import a
a imported but unused.

Enhancement

Since Ubuntu 20.10 comes with libff8 instead of libffi7, libffi7 used by KCLVM Python Plugin built on ubuntu20 may not work on ubuntu22.

Ref：https://askubuntu.com/questions/1286772/libffi-so-7-cannot-open-shared-object-file-no-such-file-or-directory

Enhancement

Semantic

• The visibility/reference of a Mixin to its host variables can only be achieved through the Protocol (similar to the interface for property access). When the host uses a Mixin, it must satisfy the Protocol property constraints (satisfying the type partial order relationship), and it is one-way, that is Mixin is visible to the properties of the host individually, different mixin attributes are invisible to each other, and the host is invisible to the properties in the mixin.
• Mixin can only do merged incremental modification, cannot reassign the existing properties of the host, but can define the properties that the host does not have.
• Automatic merging of configuration blocks with the same name in mixins and schemas that use the merge operator :.

The KCL code is

schema Person:
    mixin [FullNameMixin]
    firstName: str
    lastName: str
    age: int
    id?: int

# protocol specifies what properties the host of the mixin must have
protocol FullNameProtocol:
    firstName: str
    lastName: str

mixin FullNameMixin for FullNameProtocol:
    fullName: str = "{} {}".format(firstName, lastName)
    
alice = Person {
    firstName = "Alice"
    lastName = "Green"
    age = 10
    id = 1
}

The output YAML is

alice:
firstName: Alice
lastName: Green
age: 10
id: 1
  fullName: Alice Green

Enhancement

According to the content in the Roadmap #29 , use Rust to rewrite the KCL validation tool.

Enhancement

KCL code meta information needs to have a specification.

The current KCL packet error information is based on 1-based column number, such as the following code, but the code writing allows setting the column number to 0, there is no good specification and verification, so what needs to be done is:

• Clearly marked column numbers starting from 1 in the KCL documentation
• Write the kclvm_spec package to store these specification information and add verification

Note: These specifications need to be placed in the compiler_base Implementation.

Appendix

A sample kcl error message is

KCL Compile Error[E2L23] : A complie error occurs during compiling
---> test.k:1:1
1 |import not_found_pkg
 1 ^  -> Failure
Cannot find the module not_found_pkg

Enhancement

KCLVM Rust Github CI has not added corresponding unit test coverage card points and statistical tools, and CI need to be used to further add code stability.

In addition to rust unit tests, some rust code related tests are not added to github actions, such as kclvm runtime lib test and grammar test.

Feature Request

Is your feature request related to a problem? Please describe:

Describe the feature you'd like:

Users can call kclvm's cli api through the C language dynamic link library to more efficiently support kcl cross-language calls

Describe alternatives you've considered:

The dynamic link library will be responsible for two aspects, one of which is to perform cross-language memory management, and the other to call functions inside rust according to user parameters

Teachability, Documentation, Adoption, Migration Strategy:

Enhancement

At present, the generation and execution of dynamic link libraries are all in the main function, and they need to be refactored and encapsulated separately.

Enhancement

the "extended attributes" stored in appledouble file, that causes the kclvm-Go/scripte create many .filename file, some time will causes the python start failed.

the kclvm build script should remove "extended attributes", and the kclvm-Go/script.untargz function should skip the appledouble file.

Enhancement

What and Why:
At present, when the kcl handler does semantic analysis, it will generate struct diagnostic, which contains the string type error message and the position of the error. But diagnostic is missing the necessary parameter information to describe the error, such as the name of token. For example, for import error, we get an error message like "pkgpath xx not found in the program" and the location of xx. However, in continue (lint, or other) analysis, the name of xx may be needed for concatenation other error messages, or continue to output as parameters (sarif report).

How:

• add a string list (maybe more than one parameter) field to diagnostic
• modify code where new Dagnostic{...}

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Parallel multi-file compilation code looks like follows:

1. let pool = ThreadPool::new(self.thread_count);
2. let (tx, rx) = channel();
3. for .... {
4.   let tx = tx.clone();
5.    pool.execute(move || {
6.        ...
7.    tx.send(dylib_path).expect("channel will be there waiting for the pool");
8.    });
9. }
10. rx.iter().take(prog_count).collect::<Vec<String>>()

If panic occurs when executing lines 5 to 8, the corresponding thread will be killed immediately, then "tx" in line 7 will not send the result, and "rx" in line 10 will wait for the result from "tx" and the main thread will never stop .

2. What did you expect to see? (Required)

The program outputs a normal error message and stops.

3. What did you see instead (Required)

The program cannot stop after outputting panic information.

4. What is your KusionStack components version? (Required)

kclvm version is 0.4.2; checksum: e07ed7af0d9bd1e86a3131714e4bd20c

Enhancement

1. The temporary LLVM IR file *.ll and lock file *.lock generated by KCL are not deleted and need to be deleted.
2. The temporary LLVM IR file generated by the KCL main package is named main.ll. It may cause data races in concurrent compilation mode, so it needs to be named differently.

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

The kcl code is

a: str

2. What did you expect to see? (Required)

A syntax error.

KCL Syntax Error[E1001] : Invalid syntax
---> File /Users/_Code/KusionOpenSource/KCLVM/a.k:1:7
1 |a: str
       7 ^  -> Expected one of ['=']
Invalid syntax

3. What did you see instead (Required)

thread 'main' panicked at 'called Option::unwrap() on a None value', sema/src/resolver/node.rs:280:47

4. What is your KusionStack components version? (Required)

kclvm version is 0.4.2; checksum: e07ed7af0d9bd1e86a3131714e4bd20c

Enhancement

1. The back end of the compiler should include code generation, dynamic link library generation and linking, executable file generation.

2. The problem with the current version of KCLVM is that the modules Assembler and Linker/Loader in the above figure are not individually encapsulated.

3. This destroys the readability of the code and reduces the reusability of these modules. For example: issue #67 can be solved by the above modules.

Therefore, the Assembler, Linker/Loader and Executor need to be encapsulated separately.

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Take "-" combine with an unicode char as input for the method parse_expr.

parse_expr("-\u{feff}sjda");

2. What did you expect to see? (Required)

The KCLVM should tell me the exception information such as “illegal characters”.

3. What did you see instead (Required)

The panic info:

running 1 test
thread 'tests::test_parse_expr_invalid' panicked at 'byte index 2 is not a char boundary; it is inside '\u{feff}' (bytes 1..4) of `-sjda`', library/core/src/str/mod.rs:127:5
stack backtrace:
...
failures:
    tests::test_parse_expr_invalid

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 92 filtered out; finished in 0.03s

error: test failed, to rerun pass '-p kclvm-parser --lib'
The terminal process "cargo 'test', '--package', 'kclvm-parser', '--lib', '--', 'tests::test_parse_expr_invalid', '--exact', '--nocapture'" terminated with exit code: 101.

4. What is your KusionStack components version? (Required)

kclvm version is 0.4.2; checksum: e07ed7af0d9bd1e86a3131714e4bd20c

Enhancement

All versions of kclvm release larger than 150MB, smaller binaries are always welcome.

Introduction

KCL provides the schema settings meta attribute for planning YAML. In addition, you can also write KCL code to customize the way of outputting YAML, so the settings meta attribute function can be removed, reducing the burden of understanding.

Background

• At present, the YAML output of kcl is hardcode in KCLVM, and some variables will be undefined and filtered according to the schema's settings: {"output_type": } to control the output to stdout or file, and all yaml will be generated in the same one In the file, different YAML uses --- to split
• In order to control the order or pattern of YAML output, users need to understand the YAML output logic in KCLVM and write a lot of KCL code, such as modifying the settings of the schema or controlling the instantiation order of resources, it is better to directly get all the resources for processing, such as Sorting and outputting is intuitive, and the type of the subsequent schema output_type can not be written, and the output to stdout and files can be controlled at the same time by writing code, and it can be output directly inside the schema, etc., flexible use
• On the other hand, after customizing the output YAML, the Konfig front-end can be freely written, in line with user best practices and the needs of upper-level operation and maintenance automation, the corresponding output module can be adjusted according to the needs
• For example, in the current large library, OutputMixin is used to control the output style of all k8s resources. It can be seen that the code is more complex and needs to be used as a mixin. It is better to output directly in the back-end model render.

Design

schema ManifestsYamlStreamOptions:
    sort_keys: bool = False
    ignore_private: bool = True
    ignore_none: bool = False
    separator: str = "\n---\n"

manifests.yaml_stream(values: [any], * , opts: ManifestsYamlStreamOptions = ManifestsYamlStreamOptions {})

The function is to serialize a list of KCL objects to YAML and output using the style with the --- delimiter, and add it to the existing YAML output stream.

Note: When the yaml.manifest_stream function is called, the output result of plan inside KCLVM is subject to this function, and global variables are not output by default.

Examples

import manifests

# resources.k
schema Deployment:
    apiVersion: str = "v1"
    kind: str = "Deployment"
    metadata: {str:} = {
        name = "deploy"
    }
    spec: {str:} = {
        replica = 2
    }

schema Service:
    apiVersion: str = "v1"
    kind: str = "Service"
    metadata: {str:} = {
         name = "svc"
    }
    spec: {str:} = {}    
        
deployments = [Deployment {}, Deployment {}]
services = [Service {}, Service {}]

manifests.yaml_stream(deployments + services)

The output YAML is

apiVersion: v1
kind: Deployment
metadata:
  name: deploy
spec:
  replica: 2
---
apiVersion: v1
kind: Deployment
metadata:
  name: deploy
spec:
  replica: 2
---
apiVersion: v1
kind: Service
metadata:
  name: svc
spec: {}
--
apiVersion: v1
kind: Service
metadata:
  name: svc
spec: {}

Reference

• Ksonnet: https://github.com/ksonnet/ksonnet
• Jsonnet Manifest Functions: https://jsonnet.org/ref/stdlib.html (using std.)

std.manifestYamlStream(
  ['a', 1, []],
  indent_array_in_object=false,
  c_document_end=true)
{
    a: 1,
    b: 2
}

The output is

"---\n\"a\"\n---\n1\n---\n[]\n...\n{\"a\": 1, \"b\": 2}"

• CUE Kubernetes Integration: https://cuelang.org/docs/integrations/k8s/ (using yaml.MarshalStream(objects))

package kube

objects: [ for v in objectSets for x in v {x}]

objectSets: [
	service,
	deployment,
	statefulSet,
	daemonSet,
	configMap,
]

package kube

import (
	"encoding/yaml"
	"tool/cli"
)

command: dump: {
	task: print: cli.Print & {
		text: yaml.MarshalStream(objects)
	}
}

Note

schema __settingss__ attrtibute will be removed in KCLVM v0.4.6