nim-lang / fusion

Fusion is for now an idea about how to grow Nim's ecosystem without the pain points of more traditional approaches.

Nim Fusion

Docs: https://nim-lang.github.io/fusion/theindex.html

Fusion contains Nim modules that are to be bundled with the Nim installation in order to give us something like the "Nim distribution". Fusion fullfills multiple purposes:

• It contains candidates for inclusion into the stdlib. However these modules only move into the stdlib if it makes sense to maintain them with the rest of the core of Nim and currently I cannot imagine why that ever would need to be the case.
• Fusion has a broader perspective about what is considered useful. For example an HTML parser or a simple UI library that doesn't support every OS that Nim itself supports is acceptable for Fusion.
• Fusion aims to be an immutable code repository, once a module is included it stays. Like Nim itself, Fusion uses a .since annotation. New modules which contain few procs are preferred over larger modules that have more procs.
• Fusion is also a Nimble package and it is compatible with Nim version 1 as well as the latest Nim.

Fusion is for now an idea about how to grow Nim's ecosystem without the pain points of more traditional approaches. Time will tell if the idea is a good one.

fusion's Issues

CI: don't build nim from scratch (can take majority of the CI time)

there are better options, eg:

      - uses: jiro4989/setup-nim-action@v1
        with:
          nim-version: ${{ matrix.nimVersion }}

further, for devel (which changes on every nim commit), we should use caching; somehow it rebuilds nim even though there is caching in the yml file specified, I need to investigate further this point

open(f: var File, filehandle: FileHandle) does not work on Windows

Opening a File from a FileHandle using

proc open*(f: var File, filehandle: FileHandle,
           mode: FileMode = fmRead): bool {.tags: [], raises: [], benign.}

does not work on Windows but does on Linux.

Example

  import std/ioutils

  let file1 = open("dummy.txt", mode=fmWrite)
  let dupfile1fd = duplicate(file1.getFileHandle())
  var f : File
  doAssert(isNil(f))
  let res = open(f, dupfile1fd, mode=fmWrite)
  # Both fails
  doAssert(not isNil(f)) # f == nil
  doAssert(res) # res == false
  let msg = "This is a test message that will be displayed ! \n"
  f.write(msg)
  f.close()

Current Output

Error: unhandled exception: test.nim(9, 9) `not isNil(f)`  [AssertionDefect]
Error: execution of an external program failed'

Expected Output

This is a test message that will be displayed ! 

Additional Information

Works on Linux

On branch devel commit ea6c28249ae7107fba954f90dac31132564dcaad

Using caseStmtMacros inside a generic function fails

Using caseStmtMacros inside a generic function fails:

proc works() = 
  let a = some 2
  case a:
  of Some(@t):
    echo "some"
  of None():
    echo "none"

proc willFail[T]() = 
  let b = some 2
  case b:
  of Some(@t):
    echo "some"
  of None():
    echo "none"

Fails with: Error: undeclared identifier: 'Some' in devel and 1.6.6

btreesets

Fusion has a btreetables module, which mimics the tables module from the Nim standard library. Are there any plans to also add a btreesets module (which would possibly mimic the sets module from the standard library)? An implementation could probably use btreetables internally or just copy the implementation.

[matching] assertMatch not raise exception when failed

According to document, assertMatch should raise MatchError on failure, but it actually didn't raise (and leaves a unused variable ok)

where is fusion.since?

readme says:

Fusion uses a .since annotation

but I don't see since being defined nor used at all.

I propose we reuse stdilib's private since instead of duplicating its implementation, a lesser evil in this case:

import std/private/since

macos CI queues for a very long time

macos jobs seem to take at least 30mn to start running (queueing time)

example 1:

in #64:
Fusion CI / macos-amd64 (1.0.0) (pull_request) Successful in 8m
Fusion CI / linux-i386 (1.0.0) (pull_request) Successful in 1m
ditto with all other jobs, macos is always the straggler by far

example 2:

in #67 all macos jobs queue for a long time (even after i restart all CI)

fusion/matching should also use some and none instead of Some and None:

Ref nim-lang/Nim#17147 (comment)

htmlparser test fails

... with xmltree.nim(118, 53) Error: invalid pragma: since: (1, 3)

nim -v
Nim Compiler Version 1.3.5 [Linux: amd64]

instead of copyDir(fustion, stdlib/lib), update --path

@alaviss
continuation of #22 (comment) to avoid mingling unrelated topics

instead of copyDir(distDir / "fusion" / "src" / "fusion", "lib" / "fusion"), we should probably use the usual nimble logic which allows adding import dirs to nim's --path, so that nimble develop fusion works as intended (NIMBLE_DIR=customdir can always be used during bootstrap to avoid user's environment messing up with bootstrap)

I decided against this and make use of copyDir simply because nimble does not support "system-installed" dependencies. Also it's widely incompatible with how Linux distributions package Nim, so unless we got those sorted out copying is the foolproof way.

• wouldn't NIMBLE_DIR=pathto/Nim/dist work?
• if not, why not add --path:dist/fusion/src to Nim/config/nim.cfg ?
  if the sources are there it'll pick them up, else, it will be a noop

fusion CI should run tests and docs for js

refs #59 (review)

otherwise PR's like this #59 have tests that are never run.

Furthermore, we need to publish docs for js-only modules

/cc @juancarlospaco

regression: fusion CI doesn't run

fusion CI isn't running since #27

we should fix this before merging any new PR

found this while investigating #64 (comment)

`matching` bug with `{orc,arc}`

The following matching example:

import options
import fusion/matching
{.experimental: "caseStmtMacros".}
type
  OKind* = enum O1 O2
  O* = object
    case kind*: OKind
    of O1: o1*: int
    of O2: o2*: Option[string]

case O(kind:O2,o2:some[string]("hello world")):
  of O2(o2:Some(@mystring)): echo mystring

produces the error:

/home/deech/NimScratch/match.nim(11, 1) template/generic instantiation of `case` from here
/home/deech/.nimble/pkgs/fusion-#4f0de4c2c50c8c204fa18d6f46b3bc38f7ff873e/fusion/matching.nim(2249, 16) Error: type mismatch: got <O>
but expected one of: 
proc get[T](self: Option[T]): lent T
  first type mismatch at position: 1
  required type for self: Option[get.T]
  but expression 'expr_369098795' is of type: O
proc get[T](self: Option[T]; otherwise: T): T
  first type mismatch at position: 1
  required type for self: Option[get.T]
  but expression 'expr_369098795' is of type: O
proc get[T](self: var Option[T]): var T
  first type mismatch at position: 1
  required type for self: var Option[get.T]
  but expression 'expr_369098795' is of type: O

expression: get(expr_369098795)

nim-compile exited abnormally with code 1 at Sun Mar  7 11:25:45

but only when compiled with gc:orc or gc:arc.

Nim version:

$ nim --version
Nim Compiler Version 1.5.1 [Linux: amd64]
Compiled at 2021-03-07
Copyright (c) 2006-2021 by Andreas Rumpf

git hash: d1e093207acdd10ad375705bd1732fc6f0db9f55
active boot switches: -d:release

matching and tmatching trigger a ton of styleCheck hints

/cc @haxscramper

tests/tmatching.nim(1874, 11) Hint: 'doASsert' should be: 'doAssert' [Name]

a fix for this issue should also include this change:

switch("styleCheck", "hint")
=>
switch("styleCheck", "error")

in tests/config.nims

fusion CI should run js tests that require browser

#63 was fixed but lots of js tests require a browser environment, not nodejs.

This issue is fixable, refs timotheecour/Nim#197

Support qualified variant selectors with matching?

The following fails:

import options
import fusion/matching
{.experimental: "caseStmtMacros".}
type
  OKind* = enum O1
  O* = object
    case kind*: OKind
    of O1: o1*: int

case O(kind:O1,o1:1):
  of OKind.O1(o1: @i): echo i

with:

nim c --forceBuild:on --gc:orc -r --verbosity\:0 --hint\[Processing\]\:off --excessiveStackTrace\:on /tmp/matchingtest.nim
/tmp/matchingtest.nim(10, 1) template/generic instantiation of `case` from here
/tmp/matchingtest.nim(11, 11) Error: undeclared field: 'O1' for type matchingtest.O [type declared in /tmp/matchingtest.nim(6, 3)] 
  found 'O1' [enumField declared in /tmp/matchingtest.nim(5, 17)]

nim-compile exited abnormally with code 1 at Sat Mar 27 11:49:16

If I remove the variant qualification it works:

case O(kind:O1,o1:1):
  of O1(o1: @i): echo i

I have variant kinds with name clashes so being able to fully qualify the enum selector would be useful.

Matching example at top of docs fails to compile

import fusion/matching
{.experimental: "caseStmtMacros".}

case [(1, 3), (3, 4)]:
  of [(1, @a), _]:
    echo a

  else:
    echo "Match failed"

Results in:

benjaminlee@Benjamins-Mac-mini ~ [1]> nim c  tmatching.nim
Hint: used config file '/opt/homebrew/Cellar/nim/1.6.0/nim/config/nim.cfg' [Conf]
Hint: used config file '/opt/homebrew/Cellar/nim/1.6.0/nim/config/config.nims' [Conf]
.................................................................................
stack trace: (most recent call last)
/Users/benjaminlee/.nimble/pkgs/fusion-1.0/fusion/matching.nim(2346, 49) case
/Users/benjaminlee/.nimble/pkgs/fusion-1.0/fusion/matching.nim(2307, 12) matchImpl
/Users/benjaminlee/.nimble/pkgs/fusion-1.0/fusion/matching.nim(1189, 44) parseMatchExpr
/Users/benjaminlee/.nimble/pkgs/fusion-1.0/fusion/matching.nim(1006, 31) parseSeqMatch
/Users/benjaminlee/.nimble/pkgs/fusion-1.0/fusion/matching.nim(1360, 12) parseMatchExpr
/Users/benjaminlee/tmatching.nim(4, 1) template/generic instantiation of `case` from here
/Users/benjaminlee/tmatching.nim(5, 7) Error: Malformed DSL - found (1, @a) of kind nnkTupleConstr.

Nim info:

benjaminlee@Benjamins-Mac-mini ~ [1]> nim -V
Nim Compiler Version 1.6.0 [MacOSX: arm64]
Compiled at 2021-10-19
Copyright (c) 2006-2021 by Andreas Rumpf

active boot switches: -d:release -d:nimUseLinenoise

Run CI tests with arc/orc in addition to default GC

Right now all fusion tests pass with orc/arc enabled (with #77 pattern matching passes too), but I think it is necessary to also test compilation with non-default GC.

deprecate or remove fusion/[jsformdata,jsheaders] because of the new [std/jsformdata,jsheaders]

/cc @juancarlospaco

• deprecate or remove fusion/[jsformdata,jsheaders] because of the new [std/jsformdata,jsheaders] (since nim-lang/Nim#12531)

• what about fusion/js/jsxmlhttprequest.nim? is there equivalent in stdlib?

RFC: fusion should be importable by compiler sources, and even stdlib and koch

/cc @Araq

rationale

avoid code duplication and re-inventing the wheel

how it'd work

• build_all.sh builds csources
• the it builds koch
• then it calls koch boot
• koch boot clones dependencies in dist including fusion at a fixed hash baked in koch.nim (equivalent to a git submodule)
• import fusion/foo is made available for import, in fact it already is implemented in:

proc bundleFusion(latest: bool) =
  let commit = if latest: "HEAD" else: FusionStableCommit
  cloneDependency(distDir, "https://github.com/nim-lang/fusion.git", commit,
                  allowBundled = true)
  copyDir(distDir / "fusion" / "src" / "fusion", "lib" / "fusion")

• then nim c compiler/nim can use import fusion/foo
• likewise with stdlib modules; it's just a question of avoiding cyclic dependencies as usual except now there are 2 base dirs.

koch importing fusion

• that's also possible if fusion is cloned before koch.nim is built, ie via a git submodule or in build_all.sh; see for example nim-lang/RFCs#261 where kochdocs could import fusion/globs

note

instead of copyDir(distDir / "fusion" / "src" / "fusion", "lib" / "fusion"), we should probably use the usual nimble logic which allows adding import dirs to nim's --path, so that nimble develop fusion works as intended (NIMBLE_DIR=customdir can always be used during bootstrap to avoid user's environment messing up with bootstrap)

Type naming for `btreetables`

tl;dr - rename btreetables.Table & related to something like BTreeTable. Why? Easy to confuse with tables.Table and leads to name clashes.

Main reason for name change: it is really confusing to have two identically named types, which have different asymptotics and behaviour (e.g log n + sorted vs unsorted & constant). Using the name naming not only leads to user confusion but also creates severe name clashes for all type declarations and constructor procs.

I would suggest renaming type to something like TreeTable / BTreeTable (consistent with module name, but probably too long to type) or BTable (shorter, but may still be confusing - not too much different from regular Table)

`min` example for fusion/astdsl.nim is wrong

The fusion/astdsl has a usage example in https://nim-lang.github.io/fusion/src/fusion/astdsl.html
for the min macro that is wrong.

How to reproduce

replace the assertion in the example with

assert min("d", "c", "b", "a") == "a"

Output

Error: unhandled exception: XXXX/min.nim(25, 8) `min("d", "c", "b", "a") == "a"`  [AssertionDefect]

The reason being that min("d", "c", "b", "a") returns "c" actually.

The min algorithm as implemented is wrong. It should not be a sequence of elif branches that stops the first time a smaller element is found. A loop over all the elements is needed to find the minimum.

RFC: adopt semver, like almost all nimble packages

right now fusion.nimble is stuck at 1.0 (I wished it didn't start out at 1.0, packages should always start at some 0.x.y).
This proposal is to start adopting semver; anything is better than the current approach, eg:

• breaking changes likely to break code (which should be minimized but sometimes are the lesser evil) bump minor
• api additions bump patch
• nim should start referring to point releases instead of arbitrary looking FusionStableCommit = "319aac4d43b04113831b529f8003e82f4af6a4a5"
• we can also distinguish odd vs even minor and patch as is done in nim, with rationale that fusion 1.2x.2y even is more stable than 1.(2x+1).(2y+1)

note that any change (including an API addition) can be considered breaking changes in nim (eg, causing a redefinition error when fusion defines bar which was otherwise in some 3rd party import), so practicality trumps 0 breaking changes.

links

• timotheecour/Nim#393

Releases

Do you have an idea about how fusion releases will work, if it's composed of many submodules that are updated independently? I'm guessing the submodules are independently versioned and that "fusion" is a nimble-installable module.

Compiler crashes when else match NimNode

import macros, fusion/matching

{.experimental: "caseStmtMacros".}

case newLit 0
else: discard

current output

Nim/compiler/nim.nim(138) nim
Nim/compiler/nim.nim(94) handleCmdLine
Nim/compiler/main.nim(275) mainCommand
Nim/compiler/main.nim(245) compileToBackend
Nim/compiler/main.nim(101) commandCompileToC
Nim/compiler/modules.nim(178) compileProject
Nim/compiler/modules.nim(98) compileModule
Nim/compiler/passes.nim(180) processModule
Nim/compiler/passes.nim(73) processTopLevelStmt
Nim/compiler/sem.nim(653) myProcess
Nim/compiler/sem.nim(639) semStmtAndGenerateGenerics
Nim/compiler/sempass2.nim(1514) trackStmt
Nim/compiler/sempass2.nim(1089) track
Nim/compiler/sempass2.nim(735) trackBlock
Nim/compiler/sempass2.nim(1088) track
Nim/compiler/sempass2.nim(707) trackIf
Nim/lib/system/fatal.nim(53) sysFatal
Error: unhandled exception: index 1 not in 0 .. 0 [IndexDefect]

(stack trace from nim compiler, compiled without -d:release)

expected output
compiles and do nothing