StackOverflow when refining long string at compile time about iron

[Iltotore]

After further investigations, it looks like a bug (or is it expected?) from Dotty when returning a big Expr/Term. I submitted a bug report with a more minimal example: scala/scala3#17218.

The only immediate workaround I see is to increase the max depth of the JVM running the compiler.

[gvolpe]

No problems, thanks for looking into it @Iltotore ! For now, the workaround is to directly use asInstanceOf[MyConstraint] 👍🏽

[Iltotore]

In 2.1.0, you will be able to use .assume instead.

[Iltotore]

I cannot reproduce the error on Scala CLI and Scastie does not shows the stacktrace. Can you send it?

[gvolpe]

@Iltotore that's true, I can't reproduce with scala-cli either, but the error appears on Scastie. For more context, this particular code is part of a test suite, so I wonder if that makes any difference.

I'm afraid the stack trace is not particularly useful, though. Here's a truncated version, as it keeps repeating over and over.

[info] compiling 1 Scala source to /home/gvolpe/foo/target/scala-3.2.2/test-classes ...
java.lang.StackOverflowError while running inlining on /home/gvolpe/foo/ServiceTest.scala
java.lang.StackOverflowError while compiling /home/gvolpe/foo/ServiceTest.scala
[error] ## Exception when compiling 3 sources to /home/gvolpe/foo/target/scala-3.2.2/test-classes
[error] java.lang.StackOverflowError
[error] dotty.tools.dotc.core.Uniques$.unique(Uniques.scala:33)
[error] dotty.tools.dotc.typer.ProtoTypes$IgnoredProto$.apply(ProtoTypes.scala:160)
[error] dotty.tools.dotc.typer.ProtoTypes$.shallowSelectionProto(ProtoTypes.scala:286)
[error] dotty.tools.dotc.inlines.Inliner$InlineTyper.typedSelect(Inliner.scala:763)
[error] dotty.tools.dotc.typer.Typer.typedNamed$1(Typer.scala:2864)
[error] dotty.tools.dotc.typer.Typer.typedUnadapted(Typer.scala:2957)
[error] dotty.tools.dotc.typer.ReTyper.typedUnadapted(ReTyper.scala:126)
[error] dotty.tools.dotc.inlines.Inliner$InlineTyper.typedUnadapted(Inliner.scala:900)
[error] dotty.tools.dotc.typer.Typer.typed(Typer.scala:3025)
[error] dotty.tools.dotc.typer.Typer.typed(Typer.scala:3029)
[error] dotty.tools.dotc.typer.Typer.typedExpr(Typer.scala:3145)
[error] dotty.tools.dotc.typer.Applications.realApply$1(Applications.scala:928)
[error] dotty.tools.dotc.typer.Applications.typedApply(Applications.scala:1070)
[error] dotty.tools.dotc.typer.Applications.typedApply$(Applications.scala:346)
[error] dotty.tools.dotc.inlines.Inliner$InlineTyper.typedApply(Inliner.scala:812)
[error] dotty.tools.dotc.typer.Typer.typedUnnamed$1(Typer.scala:2895)
[error] dotty.tools.dotc.typer.Typer.typedUnadapted(Typer.scala:2958)
[error] dotty.tools.dotc.typer.ReTyper.typedUnadapted(ReTyper.scala:126)
[error] dotty.tools.dotc.inlines.Inliner$InlineTyper.typedUnadapted(Inliner.scala:900)
[error] dotty.tools.dotc.typer.Typer.typed(Typer.scala:3025)
[error] dotty.tools.dotc.typer.Typer.typed(Typer.scala:3029)
[error] dotty.tools.dotc.typer.Typer.typedInlined(Typer.scala:1922)
[error] dotty.tools.dotc.typer.Typer.typedUnnamed$1(Typer.scala:2917)
[error] dotty.tools.dotc.typer.Typer.typedUnadapted(Typer.scala:2958)

I can try making a full fledge sbt project this weekend to see if I can reproduce. We use the Weaver test framework.

[gvolpe]

I could reproduce it using scala-cli as well, basically making the string value 3x bigger. It just hangs, though, doesn't give SO.

//> using scala "3.2.2"
//> using lib "io.github.iltotore::iron:2.1.0-RC1"

import io.github.iltotore.iron.*
import io.github.iltotore.iron.constraint.all.*

object Main:

  type NonBlank = Not[Blank DescribedAs "Textual value must not be blank"]
  type NonBlankText = String :| NonBlank

  val privateKey: NonBlankText =
    "1SUT8Dv40Ay78zl26xJJrpkP1cDPsLk1V4eVoRQuGceS7hIo8gmKxbAVWdn15lPYQ1fcifRaTINXg6tDH4WNhxBLbgF+Yqd3ouP7kS978LlFUZq3cAC8UAekvIcnJwzxaqsHBG7+mC1Adk3TqlJt+5JELo5r8apwChkxRzeFaybcRD6xUmb8GGJj/FTpatATVAMh1SrkQsaJhucByHxxt9EVUT2GaITsebwdyTRxtGnam5m+ZrspaAkvXCwXgvfVwZEN5wbvwU5dcZ5279KCPZY/HnW9s8SlS8qbTst5z248l0cALe+hyNkNC1Y05u3uEUlhsJ1AD0MLCjBlroHNjvWrWRDoz93qEbakuPb9/Rr0Z6l09V6exmJ2PXGnSUtV1hYo2DAQbOklQZdEZBkhsY4wirsCTKhOainF5UvtLljHDtap9UaMXtoT6g/gazq1SUT8Dv40Ay78zl26xJJrpkP1cDPsLk1V4eVoRQuGceS7hIo8gmKxbAVWdn15lPYQ1fcifRaTINXg6tDH4WNhxBLbgF+Yqd3ouP7kS978LlFUZq3cAC8UAekvIcnJwzxaqsHBG7+mC1Adk3TqlJt+5JELo5r8apwChkxRzeFaybcRD6xUmb8GGJj/FTpatATVAMh1SrkQsaJhucByHxxt9EVUT2GaITsebwdyTRxtGnam5m+ZrspaAkvXCwXgvfVwZEN5wbvwU5dcZ5279KCPZY/HnW9s8SlS8qbTst5z248l0cALe+hyNkNC1Y05u3uEUlhsJ1AD0MLCjBlroHNjvWrWRDoz93qEbakuPb9/Rr0Z6l09V6exmJ2PXGnSUtV1hYo2DAQbOklQZdEZBkhsY4wirsCTKhOainF5UvtLljHDtap9UaMXtoT6g/gazq441SUT8Dv40Ay78zl26xJJrpkP1cDPsLk1V4eVoRQuGceS7hIo8gmKxbAVWdn15lPYQ1fcifRaTINXg6tDH4WNhxBLbgF+Yqd3ouP7kS978LlFUZq3cAC8UAekvIcnJwzxaqsHBG7+mC1Adk3TqlJt+5JELo5r8apwChkxRzeFaybcRD6xUmb8GGJj/FTpatATVAMh1SrkQsaJhucByHxxt9EVUT2GaITsebwdyTRxtGnam5m+ZrspaAkvXCwXgvfVwZEN5wbvwU5dcZ5279KCPZY/HnW9s8SlS8qbTst5z248l0cALe+hyNkNC1Y05u3uEUlhsJ1AD0MLCjBlroHNjvWrWRDoz93qEbakuPb9/Rr0Z6l09V6exmJ2PXGnSUtV1hYo2DAQbOklQZdEZBkhsY4wirsCTKhOainF5UvtLljHDtap9UaMXtoT6g/gazq4"

  @main
  def run: Unit =
    print(privateKey)

[gvolpe]

Okay, that was using scala-cli v2.1.0. I upgraded to v1.0.0-RC1 and I get the SO error.

Scala CLI version: 1.0.0-RC1
Scala version (default): 3.2.2
$ scala-cli run main.scala
Starting compilation server
Downloading compilation server 1.5.6-sc-4
Compiling project (Scala 3.2.2, JVM)
Error: Encountered a StackOverflowError coming from the compiler. You might need to restart your Bloop build server:
java.base/java.lang.invoke.DirectMethodHandle.allocateInstance(DirectMethodHandle.java:501)
dotty.tools.dotc.core.Symbols$.mapSymbols(Symbols.scala:778)
dotty.tools.dotc.ast.TreeTypeMap.withMappedSyms(TreeTypeMap.scala:199)
dotty.tools.dotc.ast.TreeTypeMap.transformDefs(TreeTypeMap.scala:153)
dotty.tools.dotc.ast.TreeTypeMap.transformInlined(TreeTypeMap.scala:90)
dotty.tools.dotc.ast.TreeTypeMap.transform(TreeTypeMap.scala:125)
dotty.tools.dotc.ast.TreeTypeMap.transformInlined(TreeTypeMap.scala:91)
dotty.tools.dotc.ast.TreeTypeMap.transform(TreeTypeMap.scala:125)
dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1406)
dotty.tools.dotc.ast.TreeTypeMap.transform(TreeTypeMap.scala:145)
dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1412)
dotty.tools.dotc.ast.TreeTypeMap.transform(TreeTypeMap.scala:145)
dotty.tools.dotc.ast.TreeTypeMap.transformInlined(TreeTypeMap.scala:91)
dotty.tools.dotc.ast.TreeTypeMap.transform(TreeTypeMap.scala:125)
dotty.tools.dotc.ast.TreeTypeMap.transformInlined(TreeTypeMap.scala:91)
dotty.tools.dotc.ast.TreeTypeMap.transform(TreeTypeMap.scala:125)
dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1406)
dotty.tools.dotc.ast.TreeTypeMap.transform(TreeTypeMap.scala:145)
dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1412)
dotty.tools.dotc.ast.TreeTypeMap.transform(TreeTypeMap.scala:145)

[Iltotore]

Might be hard to investigate since the stacktrace does not mention anything about Iron. I still have an idea about the cause: maybe the generated Expr is too long and the compiler fails because of that.

[Iltotore]

Refining at runtime works:

//> using scala "3.2.2"
//> using lib "io.github.iltotore::iron:2.1.0-RC1"

import io.github.iltotore.iron.*
import io.github.iltotore.iron.constraint.all.*

object Main:

  type NonBlank = Not[Blank DescribedAs "Textual value must not be blank"]
  type NonBlankText = String :| NonBlank

  val privateKey: NonBlankText =
    "1SUT8Dv40Ay78zl26xJJrpkP1cDPsLk1V4eVoRQuGceS7hIo8gmKxbAVWdn15lPYQ1fcifRaTINXg6tDH4WNhxBLbgF+Yqd3ouP7kS978LlFUZq3cAC8UAekvIcnJwzxaqsHBG7+mC1Adk3TqlJt+5JELo5r8apwChkxRzeFaybcRD6xUmb8GGJj/FTpatATVAMh1SrkQsaJhucByHxxt9EVUT2GaITsebwdyTRxtGnam5m+ZrspaAkvXCwXgvfVwZEN5wbvwU5dcZ5279KCPZY/HnW9s8SlS8qbTst5z248l0cALe+hyNkNC1Y05u3uEUlhsJ1AD0MLCjBlroHNjvWrWRDoz93qEbakuPb9/Rr0Z6l09V6exmJ2PXGnSUtV1hYo2DAQbOklQZdEZBkhsY4wirsCTKhOainF5UvtLljHDtap9UaMXtoT6g/gazq1SUT8Dv40Ay78zl26xJJrpkP1cDPsLk1V4eVoRQuGceS7hIo8gmKxbAVWdn15lPYQ1fcifRaTINXg6tDH4WNhxBLbgF+Yqd3ouP7kS978LlFUZq3cAC8UAekvIcnJwzxaqsHBG7+mC1Adk3TqlJt+5JELo5r8apwChkxRzeFaybcRD6xUmb8GGJj/FTpatATVAMh1SrkQsaJhucByHxxt9EVUT2GaITsebwdyTRxtGnam5m+ZrspaAkvXCwXgvfVwZEN5wbvwU5dcZ5279KCPZY/HnW9s8SlS8qbTst5z248l0cALe+hyNkNC1Y05u3uEUlhsJ1AD0MLCjBlroHNjvWrWRDoz93qEbakuPb9/Rr0Z6l09V6exmJ2PXGnSUtV1hYo2DAQbOklQZdEZBkhsY4wirsCTKhOainF5UvtLljHDtap9UaMXtoT6g/gazq441SUT8Dv40Ay78zl26xJJrpkP1cDPsLk1V4eVoRQuGceS7hIo8gmKxbAVWdn15lPYQ1fcifRaTINXg6tDH4WNhxBLbgF+Yqd3ouP7kS978LlFUZq3cAC8UAekvIcnJwzxaqsHBG7+mC1Adk3TqlJt+5JELo5r8apwChkxRzeFaybcRD6xUmb8GGJj/FTpatATVAMh1SrkQsaJhucByHxxt9EVUT2GaITsebwdyTRxtGnam5m+ZrspaAkvXCwXgvfVwZEN5wbvwU5dcZ5279KCPZY/HnW9s8SlS8qbTst5z248l0cALe+hyNkNC1Y05u3uEUlhsJ1AD0MLCjBlroHNjvWrWRDoz93qEbakuPb9/Rr0Z6l09V6exmJ2PXGnSUtV1hYo2DAQbOklQZdEZBkhsY4wirsCTKhOainF5UvtLljHDtap9UaMXtoT6g/gazq4"

  @main
  def run: Unit =
    print(privateKey)

The SO probably comes from this code:

expr.value match
  case Some(value) =>
    value
      .map(Expr.apply)
      .map(applyConstraint(_, constraintExpr))
      .foldLeft(Expr(true))((e, t) => '{ $e && $t })

EDIT: The issue comes from '{ $e && $t }. I suppose it is a bug/limitation of Scala quoting for big expressions. I will try "manually" using the AST this weekend or next week.

[Iltotore]

We can work around the problem for string-related constraints by creating custom Constraint instances instead of reusing ForAll but I'm afraid it would add much more (compared to the simplicity of the current string.scala) and it would not solve the problem for ForAll, Exists etc...

[Iltotore]

Update: Expr.betaReduce seems to be the way. However, while it works for basic examples and simple constraints, it does not for constraint involving a runtime and a compile-time part (like char.Whitespace). Expr.betaReduce does not work for new Foo().myInlineMethod() (like Constraint#test).

Again, looks like a bug/missing feature to me since the above code is correctly reduced by the compiler. I will open another issue and eventually work on a PR myself when possible.

I don't think this will get fixed for 2.1.0 but I think it should be easily fixable once the "bug" described above is fixed. Using Expr beta-reduction should also improve compilation performances for big values.

[Iltotore]

Closed for the moment. The compiler does not permit us to fix this bug without adding a lot of complexity might be re-opened when Expr.betaReduce is fixed.