arcalot / arcaflow-expressions Goto Github PK

To improve code quality, uncomment or enable the remaining golang linters (gocritic, godot, goconst, dupl, testpackage, typecheck, etc.) in the golangci-lint config, .golangci.yaml, and fix the resulting errors.

Please describe what you would like to see in this project

We have determined that the visitor pattern is likely better than using switch cases for the evaluation of expressions.

By switching to the visitor pattern, the code will become more compartmentalized, which should improve maintainability and flexibility.

        var grammarErr *InvalidGrammarError
	ok := errors.As(err, &grammarErr)
	if !ok {
		t.Fatalf("Returned error is not InvalidGrammarError")
	}

It seems like it would good to refactor this into

	assert.ExpectedError(&grammarErr)

with the code moved to go-assert/error.go.

Originally posted by @webbnh in #23 (comment)

Describe the bug

When parsing this expression, the error message is less than helpful: $.input.

The error message is the following:

failed to parse expression: $.input. (%!v(PANIC=Error method: runtime error: invalid memory address or nil pointer dereference))

In the Arcaflow tokenizer, the pattern for BooleanLiteralToken is incorrect: since anchoring (^/$) is higher precedence than alternation (|), we need to explicitly group the alternation so that the anchoring applies to both alternates (which is why lines 110 and 112 contain groupings). As the code stands at the moment, it will match a string which begins with true or ends with false, since each alternation uses separate anchoring.

This issue awaits approval of proposal in roundtable.

Please describe what you would like to see in this project

Function calls should be added to the expression language.

There are two types of functions that should be added to the expression language: preprocessor and runtime functions.

• Runtime functions are the usual types of functions, and can take input from steps.
• Preprocessor functions can only take input from literals, or potentially input variables. Will be used for enhancing security.

The syntax for a standard runtime parameter-less function call will be identifier(), where you replace identifier with the name of the function.
Parameters will be comma-separated expressions, including literals. Non-literals will be evaluated before execution.

The syntax will change for preprocessor functions by adding an ! after the function name, but before the parameter parentheses. For example, identifier!()

The changes to the BNF (Backus–Naur form) are:

• Adding function calls and literals to the root of the expression: root ::= identifier | "$" | function_call | preprocessor_call
• Adding function calls: function_call ::= identifier "(" parameter_list ")"
• Adding preprocessor calls: preprocessor_call ::= identifier "!" "(" parameter_list_literal")"
• Adding parameter_list ::= "" | expression | expression, parameter_list (This is recursive as shown. Nothing is a valid value)
• Adding parameter_list_literal ::= "" | value_literal | value_literal, parameter_list (This is recursive as shown. Nothing is a valid value)
• Changing sub-expressions to not require () since that will just be confusing: key ::= value_literal | "(" expression ")" -> key ::= root
• Adding literals to root: root ::= value_literal | identifier | "$" | function_call

Please describe your use case

The simplest use case is in place of an expression, so the value can be retrieved straight from the function.

Another use case is a function with an input of an expression. In this case, the inner expression would need to be evaluated before the function. The function would then be called with the input from the inner expression.

A more complex use case expands it further and allows an expression to be used as an input to a key in a map.

The preprocessor function calls will be used for instances where we want to run something before the rest of the workflow. For example, accessing a file outside of the workflow working directory. That file would likely be accessed in a way like file!("~/.kube/config"), and we could then prompt the user at the beginning of the workflow, rather than stalling the workflow at another point.

Implementation details

A registry of functions will be created. The structures will be similar to the ones shown below.

First, we need to define a schema for the functions, which give the inputs and outputs. These store the parameters and return types.

type ArcaflowFunctionSchema[InputType any, OutputType any] struct {
    InputSchema schema.TypedType[InputType]
    OutputSchema schema.TypedType[OutputType]
}

Next, using that, the function interface will have a function that returns the schema, and a run function that has the type and the workflow context passed in, and returns the output type or an error.

type ArcaflowFunction[InputType any, OutputType any] interface {
    Schema() ArcaflowFunctionSchema[InputType, OutputType]
    Run(input InputType, workflowContext map[string][]byte) (OutputType, error)
}

This data will be stored in a function registry. Individual functions will be defined in a sub-package of this.

Additional context

Functions available to call will be known before running the expression. That will allow there to be type info that would permit static type analysis.

Please describe what you would like to see in this project

Binary comparison operators should be added to the expression language. This will allow comparing values to create a boolean result.
Binary comparison operators include:

• >
• <
• >=
• <=
• ==
• !=

It may also make sense to add the not operator, which can likely be implemented as !

This will be implemented as a new grammar rule that can be used in the root of an expression, and therefore anywhere where a sub-expression is explicitly or implicitly allowed.

Typing

Mismatched types will cause a type error. Due to all inputs being strongly typed, the only way for a type to change is for the plugin's schema to change. In that scenario, the workflow should fail until the workflow maintainer fixes the typing so that it matches on each side.

Please describe your use case

The main use case for this will be for conditional execution, so that the values being used to determine whether to execute the sub-workflow do not need to be boolean, but can instead be something compared to create a boolean.

This will also be useful for creating boolean results for step inputs, and workflow output.

Please describe what you would like to see in this project

Two foreach sub-workflows produce list outputs:

$.steps.sub1.outputs.success.data
$.steps.sub2.outputs.success.data

I would like the ability to concatenate these lists into one when passing the data to a step or output, hopefully with something very simple liike:

my_step:
    plugin:
      deployment_type: image
      src: my_plugin_path
    input:
      my_list: !expr $.steps.sub1.outputs.success.data + $.steps.sub2.outputs.success.data

Where the my_list input to the step is a single list object consisting of all of the list items from the outputs of both sub-workflows.

At the moment, bracketAccessorDependencies's else path has similarities to dependenciesBracketKey. It is worth looking into ways of reducing redundancy, and if not, documenting clearly why not.

          This would be a great place to handle the opening parenthesis, if it's present, _before_ calling `parseArgs()` (and then to handle the closing parenthesis, after it returns)...like it says in the function header comment.  😉

Originally posted by @webbnh in #23 (comment)

Describe the bug

When a property is not found on an object during schema evaluation, the error message is rather unhelpful:

object greet does not have a property named outputs

Instead, the error message should list the possible properties that the user could have meant.

Please describe what you would like to see in this project

The expression language should support basic mathematical expressions.
They would be supported within expressions where literals and variables were allowed before. They are binary operators, so there is one variable or literal each on the left and right, with the binary operator in the middle.

The standard PEMDAS order of operations will be supported.

Supported binray mathematical operations:

• Add: +
• Subtract -
• Multiply: *
• Divide: /
• Modulus/remainder: %

Potential supported binary operation, if easy to add:

• Exponent: **

Potential supported operation: Order of operation parentheses.

• ( )

Typing

Non-numeric types

Math is exclusively for numeric types, so Integer and Float. Upon input of a non-numeric type, it will output a type error.

Mismatching Numeric Types

Due to the inherit loss of precision between float and integer, a type error will be raised in the event of ints and floats being used together.
Type conversions would require calling a type conversion function, as would be enabled by #1

Unit data

In the case of units, it will manipulate using the base unit. It may make sense to enforce units that are being interacted with. For example, it may be okay to multiply seconds by 5, but not okay to add 5 seconds to 5 bytes.

Please describe your use case

The most obvious use case is to allow mathematical manipulation from input or step output, to be used for some other input or output.
For example, if you have two steps that have a duration input, and you want one step to last 10 seconds longer than the other, it makes more sense to add 10 to the duration input than to have the workflow designers and users manage two variables.

Additional context

This is a lower priority task than #1 and #2. A lot of the work for #2 will likely be similar to this work because both tasks are requesting the implementation of binary operators. Therefore, the additional work created by this task will likely not be too great.