effekt-lang / effekt Goto Github PK
View Code? Open in Web Editor NEWA research language with effect handlers and lightweight effect polymorphism
Home Page: https://effekt-lang.org
License: MIT License
A research language with effect handlers and lightweight effect polymorphism
Home Page: https://effekt-lang.org
License: MIT License
Can the work done here be directed so it can be added to a production language like Scala. It would be good to have it sooner than later hence can the research be adopted and contributed to Scala 3.+?
Currently, block arguments always have to be fully applied:
map([1, 2, 3]) { a => f(a) }
Instead, we could also support
map([1, 2, 3]) { f }
which could desugar to the above version.
Similarly, we could allow partial application:
f: (Int)(Int) -> T
map([1, 2, 3]) { f(1) }
desugaring to
map([1, 2, 3]) { x=> f(1)(x) }
The desugaring is type directed and should always lead to fully applied blocks. This is necessary to eventually adapt capabilities.
There are multiple possible designs to where the desugaring can take place. Since it is type directed, we further need to investigate interaction with overload resolution.
Namer already generates fresh names (if there are multiple definitions in the same scope).
This is way overloaded definitions can be uniquely addressed.
However, this is confusing responsibilities. Making generated names unique should be a part of code-generation.
The job of namer is to resolve names to unique symbols, not to also introduce unique names.
We should consider adding polymorphism not only for value types, but also block types.
Before adding it to the language, we should formalize it in our Coq proofs.
It would be nice to have a language specification similar to the Koka book. For starters it is good to collect examples in examples/
. The website could include some tutorials that introduce the syntax of features and explain a bit our philosophy on the go.
The examples
folder has a features
subfolder that could contain small examples show casing the individual features.
I'd like to write this:
effect Iterate[A] {def emit(element: A): Unit}
effect Reduce[A] {def consume(): Option[A]}
effect Transduce[A, B] = {Reduce[A], Iterate[B]}
But I get a parse error pointing at the =
in Transduce[A, B] = {...}
saying '{' expected but '=' found
. Is this intentional? It looks more like a parsing oversight than an actual limitation.
Separate out finding and loading files of dependencies into
--> -->
main ---> Dependency Discovery --> Compiler --> CompilationUnit --> Backend
| --> --> |
v v
Reading Files Writing Files
The compiler should always receive a single file to compile and all compiled (typed, lowered, ...) dependencies.
In consequence, it could in principle be parallelized to a certain degree. However, this refactoring is more important for reasoning and testing, than it is for performance.
Processing the files in topological ordering of their dependencies would remove the dependency of Namer
on Compiler.frontend
. In consequence, Compiler
and Context
wouldn't need to be mutually recursive anymore.
If we use such an architecture, then we can't use the Task infrastructure anymore.
At the moment, the frontend task itself invokes frontend on other files, which registers the dependency. Instead, the dependencies would need to be tracked manually by the driver (or, intelligence, or REPL).
I have some questions about the section effect-polymorphism.
def eachLine(file: File) { f: String => Unit / {} }: Unit / {}
Is this equivalent to a generic effect:
def eachLine[E](file: File) { f: String => Unit / E }: Unit / E
For example, I want to express that for any effect E, there must be an output.
Can it be expressed as:
def eachLine[E](file: File) { f: String => Unit / E }: Unit / { Console }
Currently only effect operations can take type parameters:
effect Foo {
def op[A](...): ...
}
To express a state effect with operations get and put, we would like to also support type parameters on effects
effect State[S] { ... }
In the SystemC branch, Substitution
has largely been reimplemented / refined. In particular, we moved away from using mutable references within region unification variables in favor of explicitly substituting.
We should backport those changes to the Effekt compiler (favourably without chaning region inference, for now).
In papers we always talk about "block types", since there we do not support interfaces, yet.
One proposal for a naming scheme is:
Right now the traces generated by region errors are displayed as individual error / info / warning messages.
Currently the Problems panel lists the trace like:
Also "Go to next Problem" already is a quite convenient interface.
This could be heavily improved by:
As part of our work on the OOPSLA'22 paper
https://se-tuebingen.github.io/oopsla-2022-artifact/
, we developed System C, a language similar to Effekt, but with explicit capability passing.
As an important next step, we need to integrate System C with Effekt master.
This means:
The effect system of System C is already largely implemented in Effekt (as a region checker). However, lessons learnt from our work on the paper need to be backported and incorporated into Effekt.
It is not just a matter of merging the two implementations. Instead, we need to find a design that integrates explicit and implicit capability passing nicely. Ideally it also should be integrated with a module system and explicit name spacing.
The System C code base currently lives in the branch: https://github.com/effekt-lang/effekt/tree/system-c-steps
In PR #59 I started cherry-picking some commits -- but there is much more to do.
On hover, currently a summary is shown together with the declared type. Also show the actual type at that position.
We want a module system for Effekt. At first we should focus on a simple and clear basis on which we then build interesting extensions.
Some features we might want are:
We look at namespaces in Flix and objects as modules in Scala for inspiration.
We already have interfaces so a first step would be to make the following work:
interface A {
def f(): Unit
}
def A = new A {
def f() = ()
}
def main() = {
A.f()
}
The second step is to introduce sugar for the above:
module A {
def f(): Unit = ()
}
def main() = {
A.f()
}
We add syntactic sugar with keyword module
. It introduces the same name on the type level and on the term level.
As a next step we make the following work:
interface B {
def f(): Unit
}
interface A {
def B : B
}
def A = new A {
def B = new B {
def f() = ()
}
}
def main() = {
A.B.f()
}
As a final step we make the module
sugar work within modules.
Files are completely orthogonal to modules. There is a keyword include
that copy-pastes the content of the given file at this position. Each file with a certain path is only copied once. The order is topologically sorted.
Unsupported
Also see #68.
Consider:
module Http {
type Request { Get(url: String) }
def send(r: Request): Unit = ...
}
def getExample(): Http.Request = Http.Get("example.com")
def main() = Http.send(getExample())
Could this be sugar for:
interface Http[R] {
def send(r: R): Unit
}
type Request { Get(url: String) }
def Http = new Http[Request] {
def send(r: Request): Unit = ...
}
def getExample(): Request = Get("example.com")
def main() = Http.send(getExample())
Does this always work?
It would be great, if we could have existential type and effect members on interfaces.
We should consider adding typeclasses or similar to prevent repeating code such as sorting datastructures, etc.
Hi again! This isn't a bug report or anything. I wrote a basic implementation of an asynchronous IO system that uses a Suspend
effect for cooperative multitasking. I thought it might make a good example for the website. Here's the code:
type Condition = Int
effect Suspend {
def newCondition(): Condition
def wait(condition: Condition): Unit
def signal(condition: Condition): Unit
}
type ThreadStatus[T] {
Uninitialized();
Finished(result: T);
Waiting(condition: Condition);
Signalling(condition: Condition)
}
def combineAsync[A, B, C] {f: => A / Suspend} {g: => B / Suspend} {combiner: (A, B) => C / Suspend}
: C / Suspend = {
var firstComputation = fun() { Uninitialized() }
firstComputation = fun() {
try {
val result = f()
firstComputation = fun() { Finished(result) }
Finished(result)
} with Suspend {
def newCondition() = resume(do newCondition())
def wait(condition) = {
firstComputation = fun() { resume(()) }
Waiting(condition)
}
def signal(condition) = {
firstComputation = fun() { resume(()) }
Signalling(condition)
}
}
}
var secondComputation = fun() { Uninitialized() }
secondComputation = fun() {
try {
val result = g()
secondComputation = fun() { Finished(result) }
Finished(result)
} with Suspend {
def newCondition() = resume(do newCondition())
def wait(condition) = {
secondComputation = fun() { resume(()) }
Waiting(condition)
}
def signal(condition) = {
secondComputation = fun() { resume(()) }
Signalling(condition)
}
}
}
def loop(firstStatus: ThreadStatus[A], secondStatus: ThreadStatus[B]): C / Suspend = {
(firstStatus, secondStatus) match {
// These two cases should be impossible because we already assigned these variables.
case (Uninitialized(), _) => loop(firstComputation(), secondStatus)
case (_, Uninitialized()) => loop(firstStatus, secondComputation())
case (Finished(firstResult), Finished(secondResult)) => combiner(firstResult, secondResult)
case (Finished(_), Waiting(condition)) =>
do wait(condition)
loop(firstStatus, secondComputation())
case (Finished(_), Signalling(condition)) =>
do signal(condition)
loop(firstStatus, secondComputation())
case (Waiting(condition), Finished(_)) =>
do wait(condition)
loop(firstComputation(), secondStatus)
case (Signalling(condition), Finished(_)) =>
do signal(condition)
loop(firstComputation(), secondStatus)
case (Waiting(firstCondition), Waiting(secondCondition)) =>
do wait(firstCondition)
do wait(secondCondition)
loop(firstComputation(), secondComputation())
case (Signalling(firstCondition), Signalling(secondCondition)) =>
do signal(firstCondition)
do signal(secondCondition)
loop(firstComputation(), secondComputation())
case (Waiting(waitCondition), Signalling(signalCondition)) =>
do signal(signalCondition)
if (waitCondition != signalCondition) {
do wait(waitCondition)
}
loop(firstComputation(), secondComputation())
case (Signalling(signalCondition), Waiting(waitCondition)) =>
do signal(signalCondition)
if (waitCondition != signalCondition) {
do wait(waitCondition)
}
loop(firstComputation(), secondComputation())
}
}
loop(firstComputation(), secondComputation())
}
type ExecutionResult[A] {
Success(value: A);
Deadlock(condition: Condition)
}
def execute[A] { f: => A / Suspend }: ExecutionResult[A] = {
var nextCondition = 0
try {
Success(f())
} with Suspend {
def newCondition() = {
val c = nextCondition
nextCondition = c + 1
resume(c)
}
def wait(condition) = {
Deadlock(condition)
}
def signal(_) = resume(())
}
}
def main(): ExecutionResult[Unit] / Console = {
execute {
val condition = do newCondition()
def firstThread(): Unit / Suspend = {
println("First thread sending signal")
do signal(condition)
do wait(condition)
println("First thread received signal")
}
def secondThread(): Unit / Suspend = {
do wait(condition)
println("Second thread received signal")
println("Second thread sending signal")
do signal(condition)
}
// These both print out messages in the same order:
// - First thread sending signal
// - Second thread received signal
// - Second thread sending signal
// - First thread received signal
println("Starting first thread first")
combineAsync { firstThread } { secondThread } { (a, b) => () }
println("Starting second thread first")
combineAsync { secondThread } { firstThread } { (a, b) => () }
}
}
The following repl session crashes:
val x = 1 // enter
val x = 1 // enter
with the following stack trace
[error] Internal Compiler Error: Cannot find symbol for IdDef(x)
[error] at effekt.util.messages$CompilerPanic$.apply(Messages.scala:24)
[error] at effekt.util.messages$ErrorReporter.panic(Messages.scala:59)
[error] at effekt.util.messages$ErrorReporter.panic$(Messages.scala:50)
[error] at effekt.context.Context.panic(Context.scala:40)
[error] at effekt.util.messages$ErrorReporter.panic(Messages.scala:58)
[error] at effekt.util.messages$ErrorReporter.panic$(Messages.scala:50)
[error] at effekt.context.Context.panic(Context.scala:40)
[error] at effekt.context.AnnotationsDB.symbolOf$$anonfun$1(Annotations.scala:366)
[error] at scala.Option.getOrElse(Option.scala:201)
We should reconsider the design of records and datatypes / variants. Some form of structural subtyping could prove to be more convenient / flexible to use.
At the moment we only have support for a very limited form of pattern matching. Implement pattern matching and also matching value binders like:
val (x, y) = (1, 2)
In VSCode, sometimes, after trying to type check an invalid program and then fixing the error the message:
"Internal Compiler Error: Already set exports on module"
appears. This indicates that the presentation compiler is in an invalid state and VSCode needs to be reloaded (Cmd+R on Mac OSX).
We should make sure that the Server always starts in a fresh state or has some means to reset its state for that matter.
Kiama has support for language servers. Figure out what they actually support and how we can easily use it to offer IDE integration.
The file path (eg. /my/inlude/path/to/file.effekt
) and the module path (e.g. module path/file
) can differ.
Now assume we include an import statement in another file:
import path/to/file
Typechecking and compilation might succeed since the file is correctly resolved. The generated JavaScript file however uses the declared
path on the module, not the actual path. That is, it generates a file path_file.js
. The import will look for path_to_file.js
, which it cannot find.
We either should error / warn if there is this mismatch or exclude paths in module declarations altogether.
Currently, the binary for the JIT is a blob in the git repository at https://github.com/effekt-lang/effekt/blob/jit/bin/x86_64/rpyeffect-jit.
We should find a better solution for this, that makes updates easier and puts a smaller load on git.
type Foo = Int
Just brings Foo
into scope while resolving to Int
.
Right now, the only two people how know how to update the Effekt compiler on our website are @jiribenes and I. It is a manual process of:
sbt fullOptJS
)webpack
dist
files generated by webpack
. (for example: effekt-lang/effekt-website@547ca60)Maybe we want to automate this process? I don't know exactly which form of automation we want. Two trigger options come to mind:
effekt/master
effekt/master
One thing to keep in mind is that I haven't created a "release" in a while. If we keep that release schedule (which is random and almost never :) ), then the website won't be updated. It is nice to have nightlys on the website, however it could also break the website and we do not have sufficient tests to rule that out.
WDYT?
Right now every effect operation corresponds to one effect. It is planned to relax this and allow multiple operations. For this we would change declaration syntax from:
effect Flip(): Boolean
to:
effect Amb {
def flip(): Boolean
}
The old syntax could be syntactic sugar for:
effect Flip {
def flip(): Boolean
}
Effect invocation could be
do Amb.flip()
where
def flip(): Boolean / Amb = do Amb.flip()
could automatically beingbegenerated, so typically effect operations would be invoked as:
flip()
effect Stream = { Emit, Stop }
{ Stream, Stream }
This can be observed by visiting
https://effekt-lang.org/docs/casestudies/smc
then in the run dialog on the bottom of the page enter ()
and click run again. I observe compile times of around 10second. Also clicking a second time doesn't speed up, which indicates that caching is not working...
Currently the tests are all in folders pos
and neg
.
Instead, we could organize them by topic:
and add additional README.md files to the folders guiding newcomers. Additionally, the tests could have comments explaining the features.
This way we could have a low cost documentation that is automatically checked
We should rearrange tests to make it easier for newcomers to the project to only run the tests they are interested in.
After having merged the WIP LLVM Backend, we now have three different platforms and corresponding tests.
If we run test
then all of the tests will be executed. This requires
node
for running the JS testsscheme
for running the chez testsopt-12
etc. for running the LLVM testsThere are also some unit tests which are independent of the backend (hopefully more in the future).
Frontend testing (including neg
tests) are performed in the context of the (still main) JS backend.
In our System C implementation, we have builtin support for local regions:
region r {
var x in r = 1;
val closure = box { () => x };
()
}
As a next step (in System C or Effekt), we could allow users to overload var
as follows:
effect Variable[T] { def get(): T; def set(value: T): Unit }
def state[R, S](init: Int) { prog: {Variable[S]) => R }: R
def user() {
var x in state = 4;
... x = x + x
}
will be desugared to
def user() {
state(4) { {x} =>
... x.set(x.get() + x.get()) ...
}
}
(I think I have seen something like this in Kotlin)
Currently we cannot have two names in the same scope. We could implement type-directed overload resolution to allow for instance the use of data/list
and data/option
in one module (both define map
).
the other backends perform whole program generation. In order to use the generated program from javascript again (other than just calling main
) we need some form of exports (like with ScalaJS).
I'm trying to model lazy streams in Effekt. I've got an effect representing a push-based producer of values and an effect representing a pull-based consumer of values, but I don't know how to combine these two things. It seems to require starting two computations and switching between them with resume
somehow. Is what I'm trying to do possible? Here's the code I've got so far:
type Option[A] {
Present(value: A);
Absent()
}
effect Iterate[A] {
def yield(element: A): Unit
}
effect Reduce[A] {
def receive(): Option[A]
}
def inRange(start: Int, end: Int): Unit / Iterate[Int] = {
do yield(start);
if (start != end) inRange(start + 1, end);
}
def intoFold[A, S](initState: S) { f: (S, A) => S } : S / Reduce[A] =
do receive[A]() match {
case Present(a) => intoFold(f(initState, a)) { (s, a) => f(s, a) }
case Absent() => initState
}
def intoSum(): Int / Reduce[Int] = intoFold(0) { (s, a) => s + a }
def reduce[A, B]() { iterator: Unit => Unit / Iterate[A] } { reducer: Unit => B / Reduce[A] } = {
try {
reducer();
} with Reduce[A] {
def receive() = {
try {
iterator();
} with Iterate[A] {
def yield(element) = {
// how to resume the outer reducer?
}
}
}
}
}
Some unstructured notes about things we might want to support at some point:
Possible IDE features:
Questions:
def bar(x: Int) = {
...
}
def foo(): Int / Print = <?
val x = foo();
x
?> // ?? =. <? () ?>
def bar() = {
val position = 42
position.<? { x => x + 1 } ?>
({ x => 42 })(5)
()
}
There is some (to me) strange interaction between effect handlers and mutable variables which leads to a slowdown in runtime performance. The following example works just fine and finishes pretty much immediately
effect Effect(): Unit
def main() = {
val num = 15000;
var count = 0;
try {
while (count < num) {
do Effect();
do Effect();
count = count + 1
}
} with Effect { () =>
resume(())
}
}
If, however, a mutable variable is used inside the loop, as is done here,
effect Effect(): Unit
def main() = {
val num = 15000;
var count = 0;
try {
while (count < num) {
var unused = ();
do Effect();
do Effect();
count = count + 1
}
} with Effect { () =>
resume(())
}
}
execution slows down considerably, it takes several seconds to finish and each additional call to the Effect
operation increases execution time by additional seconds.
On the other hand, having the handler inside the loop, while still using the mutable variable, does not lead to a performance drop:
effect Effect(): Unit
def main() = {
val num = 15000;
var count = 0;
while (count < num) {
try {
var unused = ();
do Effect();
do Effect();
count = count + 1
} with Effect { () =>
resume(())
}
}
}
Importing a file helloWorld.effekt
def hello() = {
println("Hello World!")
}
to the REPL using
import helloWorld
causes the REPL to load the file, however, as no module name is defined, the generated outputfile will have no name an thus, calling
hello()
in the REPL will fail, as no file helloWorld.js
can be found. The following file, however, works perfectly fine:
module helloWorld
def hello() = {
println("Hello World!")
}
Expected behavior: import
should work the same in Windows and OSX/*nix.
Attached: crash report of the above example.
crashReport.txt
a - b - c
parses as a - (b - c)
, which is clearly wrong.
Can another symbol be chose instead of /
There is two things we should do:
For the later, we probably need to move to manual whitespace handling in the parser
On Ubuntu 16.04 it has been reported that Effekt cannot be exeucted since
java -jar
is treated as one command instead of a command and an argument.
The Effekt "binary" is created by prefixing:
#! /usr/bin/env java -jar
to the jar file itself. Installing it with npm makes the binary available (almost) platform independently. In particular on Windows, npm analyses the shebang and automatically generates scripts that correctly start Effekt by invoking java -jar
.
We could change the shebang to
#! java -jar
which seems to work on MacOS X, Windows 10 and Ubuntu 16.04 -- however, now we cannot start Effekt as a subprocess from node. This is necessary to start the language server in the VSCode extension.
When hovering on records, the language server usually provides useful details.
Unfortunately, because list literals are immediately desugared into applications of Cons
and Nil
,
the language server shows details about Cons
or Nil
on hover:
Sometimes the displayed information is for Cons
no matter to which point of the list literal one points,
in other cases, the displayed information is for Nil
.
The code handling for showing hover information is here:
https://github.com/effekt-lang/effekt/blob/master/effekt/shared/src/main/scala/effekt/Intelligence.scala#L144-L150
I tried to understand effekt_runtime.js but encountered some difficulties.
Is there an implementation of typescript version?
In the following example the generated error message refers to an implicitly inferred capability. It is never bound and impossible to understand:
effect Yield(): Unit
def foo() = {
var x in global = 42;
x = x + 1
def bar(): Unit / {} = do Yield();
println(x)
fun() { bar() }
}
def main() = try {
foo()
} with Yield { resume(()) }
The inferred type of foo
also mentions the capability (which is equally hard to understand)
We need a proper way to refer to capabilities that are introduced for effects.
The third runnable example in section Lightweight Effect Polymorphism demonstrates a case where not all effects are correctly handled by the user. Unfortunately, the error message seems to be just an internal, serialized representation instead of the nice, rendered version.
Compare this to the Effekt REPL (current master
) output for the same query:
There seems to be a problem with undetected capabilities escaping their delimiters when using first class functions closing over them, if the handler for the corresponding effect is defined in a separate function. In the following example the escaping capability for Effect2
is detected (fcfCaughtErr.txt):
effect Effect1(): Unit
effect Effect2(): Unit
def escape { prog: () => Unit / { Effect1, Effect2 } }: Unit = {
var k: (Unit) => Unit / { escape } = fun (x: Unit) { () }
try {
try {
prog()
} with Effect1 { () =>
k = fun (x: Unit) { resume(x) };
resume(())
}
} with Effect2 { () => resume(()) }
k(())
}
def main() = {
escape { do Effect1(); do Effect2() }
}
However, if the handler for Effect2
is defined in the function handle2
as is done here
effect Effect1(): Unit
effect Effect2(): Unit
def handle2 { prog: () => Unit / { Effect2 } }: Unit = {
try { prog() }
with Effect2 { () => resume(()) }
}
def escape { prog: () => Unit / { Effect1, Effect2 } }: Unit = {
var k: (Unit) => Unit / { escape } = fun (x: Unit) { () }
handle2 {
try {
prog()
} with Effect1 { () =>
k = fun (x: Unit) { resume(x) };
resume(())
}
}
k(())
}
def main() = {
escape { do Effect1(); do Effect2() }
}
this is not detected and results in a runtime error (fcfUncaughtErr.txt).
We could use / infer the module name from the file name, if none is given.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.