Function with LOOP fails with "Compiler failed [...] break target does not exist" when run on worker about trino

[findepi]

This seems related to use of IdentityHashMap for io.trino.sql.routine.SqlRoutineCompiler.BytecodeVisitor#breakLabels.
There would be no "break target does not exist', ie the entry would be found if it was a non-identity based set.
The map keys are records and per @martint 's comment #21632 (comment) right, the records are generally not supposed to be used for identity-based identification. Maybe the fix would be just that?

I don't understand why sometimes there are two (equal) IrLabels and sometimes there is one. Something to do with function deserialization?

cc @electrum

[findepi]

I don't understand why sometimes there are two (equal) IrLabels and sometimes there is one.

Compilation does not fail on coordinator. in case of WITH FUNCTION, LanguageFunctionManager that is used on the coordinator has access to function definitions that did not go thru serialization.
Setting node-scheduler.include-coordinator=false forces execution to be always scheduled on the worker. There, WorkerLanguageFunctionProvider is used which compiles function code that arrived thru task updates. There the object identity is lost and function cannot be compiled again.

A deterministic repo does not use VALUES and thus forces worker distribution

$ trino --debug
trino> WITH FUNCTION leave_loop(s varchar)
    -> RETURNS integer
    -> BEGIN
    ->     loop_label: LOOP
    ->         LEAVE loop_label;
    ->     END LOOP;
    ->     RETURN 42;
    -> END
    -> SELECT leave_loop(name) FROM tpch.tiny.nation;
Query 20240424_135915_00036_erzet failed: Compiler failed. Possible reasons include: the query may have too many or too complex expressions, or the underlying tables may have too many columns
io.trino.spi.TrinoException: Compiler failed. Possible reasons include: the query may have too many or too complex expressions, or the underlying tables may have too many columns
	at io.trino.sql.planner.LocalExecutionPlanner$Visitor.visitScanFilterAndProject(LocalExecutionPlanner.java:2038)
	at io.trino.sql.planner.LocalExecutionPlanner$Visitor.visitProject(LocalExecutionPlanner.java:1926)
	at io.trino.sql.planner.LocalExecutionPlanner$Visitor.visitProject(LocalExecutionPlanner.java:870)
	at io.trino.sql.planner.plan.ProjectNode.accept(ProjectNode.java:81)
	at io.trino.sql.planner.LocalExecutionPlanner$Visitor.visitOutput(LocalExecutionPlanner.java:959)
	at io.trino.sql.planner.LocalExecutionPlanner$Visitor.visitOutput(LocalExecutionPlanner.java:870)
	at io.trino.sql.planner.plan.OutputNode.accept(OutputNode.java:82)
	at io.trino.sql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:638)
	at io.trino.sql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:542)
	at io.trino.execution.SqlTaskExecutionFactory.create(SqlTaskExecutionFactory.java:83)
	at io.trino.execution.SqlTask.tryCreateSqlTaskExecution(SqlTask.java:563)
	at io.trino.execution.SqlTask.updateTask(SqlTask.java:514)
	at io.trino.execution.SqlTaskManager.doUpdateTask(SqlTaskManager.java:558)
	at io.trino.execution.SqlTaskManager.lambda$updateTask$9(SqlTaskManager.java:490)
	at io.trino.$gen.Trino_testversion____20240424_135447_36.call(Unknown Source)
	at io.trino.execution.SqlTaskManager.updateTask(SqlTaskManager.java:490)
	at io.trino.server.TaskResource.createOrUpdateTask(TaskResource.java:153)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:263)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:266)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:242)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:697)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:312)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:736)
	at org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1614)
	at io.airlift.http.server.TraceTokenFilter.doFilter(TraceTokenFilter.java:62)
	at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
	at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
	at io.airlift.http.server.TimingFilter.doFilter(TimingFilter.java:51)
	at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
	at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
	at org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1547)
	at org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:819)
	at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:431)
	at org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:464)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:597)
	at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:851)
	at org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)
	at org.eclipse.jetty.server.handler.EventsHandler.handle(EventsHandler.java:81)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)
	at org.eclipse.jetty.server.Server.handle(Server.java:179)
	at org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:619)
	at org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:411)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)
	at org.eclipse.jetty.util.thread.MonitoredQueuedThreadPool$1.run(MonitoredQueuedThreadPool.java:73)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.base.VerifyException: break target does not exist
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2087)
	at com.google.common.cache.LocalCache.get(LocalCache.java:4019)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4042)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5024)
	at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:5031)
	at com.google.common.cache.ForwardingLoadingCache.getUnchecked(ForwardingLoadingCache.java:54)
	at io.trino.sql.gen.ExpressionCompiler.compileCursorProcessor(ExpressionCompiler.java:85)
	at io.trino.sql.planner.LocalExecutionPlanner$Visitor.visitScanFilterAndProject(LocalExecutionPlanner.java:2003)
	... 73 more
Caused by: com.google.common.base.VerifyException: break target does not exist
	at com.google.common.base.Verify.verify(Verify.java:126)
	at io.trino.sql.routine.SqlRoutineCompiler$BytecodeVisitor.visitBreak(SqlRoutineCompiler.java:374)
	at io.trino.sql.routine.SqlRoutineCompiler$BytecodeVisitor.visitBreak(SqlRoutineCompiler.java:283)
	at io.trino.sql.routine.ir.IrBreak.accept(IrBreak.java:29)
	at io.trino.sql.routine.ir.IrNodeVisitor.process(IrNodeVisitor.java:20)
	at io.trino.sql.routine.SqlRoutineCompiler$BytecodeVisitor.visitBlock(SqlRoutineCompiler.java:343)
	at io.trino.sql.routine.SqlRoutineCompiler$BytecodeVisitor.visitBlock(SqlRoutineCompiler.java:283)
	at io.trino.sql.routine.ir.IrBlock.accept(IrBlock.java:41)
	at io.trino.sql.routine.ir.IrNodeVisitor.process(IrNodeVisitor.java:20)
	at io.trino.sql.routine.SqlRoutineCompiler$BytecodeVisitor.lambda$visitLoop$2(SqlRoutineCompiler.java:420)
	at io.trino.sql.routine.SqlRoutineCompiler$BytecodeVisitor.compileLoop(SqlRoutineCompiler.java:447)
	at io.trino.sql.routine.SqlRoutineCompiler$BytecodeVisitor.visitLoop(SqlRoutineCompiler.java:416)
	at io.trino.sql.routine.SqlRoutineCompiler$BytecodeVisitor.visitLoop(SqlRoutineCompiler.java:283)
	at io.trino.sql.routine.ir.IrLoop.accept(IrLoop.java:32)
	at io.trino.sql.routine.ir.IrNodeVisitor.process(IrNodeVisitor.java:20)
	at io.trino.sql.routine.SqlRoutineCompiler$BytecodeVisitor.visitBlock(SqlRoutineCompiler.java:343)
	at io.trino.sql.routine.SqlRoutineCompiler$BytecodeVisitor.visitBlock(SqlRoutineCompiler.java:283)
	at io.trino.sql.routine.ir.IrBlock.accept(IrBlock.java:41)
	at io.trino.sql.routine.ir.IrNodeVisitor.process(IrNodeVisitor.java:20)
	at io.trino.sql.routine.SqlRoutineCompiler$BytecodeVisitor.visitRoutine(SqlRoutineCompiler.java:312)
	at io.trino.sql.routine.SqlRoutineCompiler$BytecodeVisitor.visitRoutine(SqlRoutineCompiler.java:283)
	at io.trino.sql.routine.ir.IrRoutine.accept(IrRoutine.java:35)
	at io.trino.sql.routine.ir.IrNodeVisitor.process(IrNodeVisitor.java:20)
	at io.trino.sql.routine.SqlRoutineCompiler.generateRunMethod(SqlRoutineCompiler.java:230)
	at io.trino.sql.routine.SqlRoutineCompiler.compileClass(SqlRoutineCompiler.java:174)
	at io.trino.sql.routine.SqlRoutineCompiler.compile(SqlRoutineCompiler.java:135)
	at io.trino.metadata.WorkerLanguageFunctionProvider.specialize(WorkerLanguageFunctionProvider.java:57)
	at io.trino.metadata.FunctionManager.getScalarFunctionImplementationInternal(FunctionManager.java:108)
	at io.trino.metadata.FunctionManager.getScalarFunctionImplementation(FunctionManager.java:93)
	at io.trino.sql.gen.BytecodeUtils.lambda$generateFullInvocation$3(BytecodeUtils.java:219)
	at io.trino.sql.gen.BytecodeUtils.generateFullInvocation(BytecodeUtils.java:255)
	at io.trino.sql.gen.BytecodeUtils.generateFullInvocation(BytecodeUtils.java:212)
	at io.trino.sql.gen.BytecodeGeneratorContext.generateFullCall(BytecodeGeneratorContext.java:101)
	at io.trino.sql.gen.RowExpressionCompiler$Visitor.visitCall(RowExpressionCompiler.java:91)
	at io.trino.sql.gen.RowExpressionCompiler$Visitor.visitCall(RowExpressionCompiler.java:78)
	at io.trino.sql.relational.CallExpression.accept(CallExpression.java:93)
	at io.trino.sql.gen.RowExpressionCompiler.compile(RowExpressionCompiler.java:75)
	at io.trino.sql.gen.RowExpressionCompiler.compile(RowExpressionCompiler.java:70)
	at io.trino.sql.gen.CursorProcessorCompiler.generateProjectMethod(CursorProcessorCompiler.java:290)
	at io.trino.sql.gen.CursorProcessorCompiler.generateMethods(CursorProcessorCompiler.java:86)
	at io.trino.sql.gen.ExpressionCompiler.compileProcessor(ExpressionCompiler.java:157)
	at io.trino.sql.gen.ExpressionCompiler.compile(ExpressionCompiler.java:141)
	at io.trino.sql.gen.ExpressionCompiler.lambda$new$0(ExpressionCompiler.java:70)
	at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:169)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3576)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2191)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2081)
	... 80 more