API for resetting the emulator state about flow-emulator

[sideninja]

You can create a snapshot or multiple of them (by specifying different names) at any point in blockchain, so you can create the first one before doing any work by calling /emulator/snapshot/init then the second one after seeding by calling /emulator/snapshot/seed and then you can at any point revert to any of them by calling the same endpoint again.

[sideninja]

Thank you for suggesting this. There is one possible way to do this already (however don't have enough context to know if it will solve your problem). You can use the snapshot feature of the emulator (our bad on not documenting this yet enough), where you can use the HTTP API and do GET on endpoint /emulator/snapshot/{name} right after the emulator is started anytime you want to go back to that state you can just again call the same endpoint with the same name and it will revert to that (initially saved) state.

[sampullman]

Thanks! I will give it a try and see if we can speed up our reset procedure. Just to double check, if blocks have been created before we make a snapshot, will they be saved? It would be great if we could revert to a state where the test data has already be seeded.

[sideninja]

@sampullman Let me know if that did the job and we can close this and just make sure we have the docs expanded as per #115

[sampullman]

Will follow up over the weekend when I get the chance to test it out, but I'm fine closing now. I'll re-open if it doesn't work out for some reason.

[sampullman]

The snapshot feature works, though it would be neat to have without persistence. I understand that would be tricky, since it looks like a local git repo is used for snapshotting a badger db. Deleting the flowdb folder on test startup is an easy workaround.

I have noticed two issues (on the latests CLI for reference, so emulator v0.27.3). Should I open separate issues for these?

1. Warnings to stderr when using either emulator/snapshot/:id or emulator/newBlock:

2022/01/24 00:02:27 http: superfluous response.WriteHeader call from github.com/onflow/flow-emulator/server.EmulatorApiServer.Snapshot (emulatorApi.go:124)
2022/01/24 00:03:04 http: superfluous response.WriteHeader call from github.com/onflow/flow-emulator/server.EmulatorApiServer.CommitBlock (emulatorApi.go:82)

2. Emulator crashes eventually. Still working on isolating the issue, but it's with tests that ran fine before we implemented snapshotting. The response always something like:

error: fatal error: slab (0xf8d6e0586b0a20c7.51) not found: slab not found for stored value

It doesn't seem related to any particular action, it just happens eventually after running a bunch of tests.

Full emulator error log:

ERROR: panic: fatal error: slab (0xf8d6e0586b0a20c7.59) not found: slab not found for stored value [recovered]
	panic: fatal error: slab (0xf8d6e0586b0a20c7.59) not found: slab not found for stored value [recovered]
	panic: fatal error: slab (0xf8d6e0586b0a20c7.59) not found: slab not found for stored value [recovered]
	panic: fatal error: slab (0xf8d6e0586b0a20c7.59) not found: slab not found for stored value [recovered]
	panic: fatal error: slab (0xf8d6e0586b0a20c7.59) not found: slab not found for stored value [recovered]
	panic: fatal error: slab (0xf8d6e0586b0a20c7.59) not found: slab not found for stored value

goroutine 2594 [running]:

ERROR: github.com/onflow/flow-go/fvm.(*VirtualMachine).Run.func1(0x14001114f90)
	github.com/onflow/[email protected]/fvm/fvm.go:64 +0x190
panic(0x1055c3320, 0x140013ae050)
	runtime/panic.go:965 +0x14c
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).RecoverErrors(0x14001383c20, 0x14001113ea0)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter.go:871 +0x294
panic(0x1055c3320, 0x140013ae050)
	runtime/panic.go:965 +0x14c
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).RecoverErrors(0x14001383c20, 0x14001113df0)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter.go:871 +0x294
panic(0x1055c3320, 0x140013ae050)
	runtime/panic.go:965 +0x14c
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).RecoverErrors(0x14001383c20, 0x10578da00)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter.go:871 +0x294
panic(0x1055c3320, 0x140013ae050)
	runtime/panic.go:965 +0x14c
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).RecoverErrors(0x14001394000, 0x10578da00)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter.go:871 +0x294
panic(0x1055c3320, 0x140013ae050)
	runtime/panic.go:965 +0x14c
github.com/onflow/cadence/runtime/interpreter.(*DictionaryValue).Transfer(0x14001375800, 0x14001394000, 0x140013a8480, 0x0, 0x106219d00, 0x0, 0x0, 0x14001122ea7, 0x12e0af138)
	github.com/onflow/[email protected]/runtime/interpreter/value.go:9331 +0x530
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).transferAndConvert(0x14001394000, 0x1057f63d8, 0x14001375800, 0x1057f9280, 0x140004e92c0, 0x1057f9280, 0x140004e94c0, 0x140013a8480, 0x14001112d01, 0x14002a9e100)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter.go:1869 +0x54
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).VisitReturnStatement(0x14001394000, 0x140004e8080, 0x1, 0x1)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter_statement.go:73 +0x1f0
github.com/onflow/cadence/runtime/ast.(*ReturnStatement).Accept(0x140004e8080, 0x1058015c8, 0x14001394000, 0x104164a28, 0x140013a4528)
	github.com/onflow/[email protected]/runtime/ast/statement.go:40 +0x38
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).evalStatement(0x14001394000, 0x1057e8038, 0x140004e8080, 0x0, 0x0)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter_statement.go:43 +0xa0
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).visitStatements(0x14001394000, 0x14001041720, 0x1, 0x1, 0x12dfe3a01, 0x140013a4528)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter_statement.go:49 +0x68
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).invokeInterpretedFunctionActivated.func1(0x14001394000, 0x0)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter_invocation.go:149 +0x44
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).visitFunctionBody(0x14001394000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x14001113000, 0x0, 0x0, ...)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter.go:1084 +0x2e0
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).invokeInterpretedFunctionActivated(0x14001394000, 0x1400138f280, 0x10627b100, 0x0, 0x0, 0x0, 0x0)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter_invocation.go:145 +0xe4
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).invokeInterpretedFunction(0x14001394000, 0x1400138f280, 0x1057f8ef0, 0x14001396b40, 0x1057f9150, 0x14001307200, 0x10627b100, 0x0, 0x0, 0x10627b100, ...)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter_invocation.go:130 +0xec
github.com/onflow/cadence/runtime/interpreter.(*InterpretedFunctionValue).invoke(0x1400138f280, 0x1057f8ef0, 0x14001396b40, 0x1057f9150, 0x14001307200, 0x10627b100, 0x0, 0x0, 0x10627b100, 0x0, ...)
	github.com/onflow/[email protected]/runtime/interpreter/function.go:104 +0x4c
github.com/onflow/cadence/runtime/interpreter.BoundFunctionValue.invoke(0x1057f8828, 0x1400138f280, 0x14001396b40, 0x1057f8ef0, 0x14001396b40, 0x1057f9150, 0x14001307200, 0x10627b100, 0x0, 0x0, ...)
	github.com/onflow/[email protected]/runtime/interpreter/function.go:345 +0xf4
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).invokeFunctionValue(0x14001383c20, 0x12d9e2bd8, 0x140013a44b0, 0x1057f9150, 0x14001307200, 0x10627b100, 0x0, 0x0, 0x10627b100, 0x0, ...)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter_invocation.go:112 +0x254
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).VisitInvocationExpression(0x14001383c20, 0x14000fe1730, 0x14001383c20, 0x12d735898)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter_expression.go:579 +0x3cc
github.com/onflow/cadence/runtime/ast.(*InvocationExpression).AcceptExp(...)
	github.com/onflow/[email protected]/runtime/ast/expression.go:467
github.com/onflow/cadence/runtime/ast.(*InvocationExpression).Accept(0x14000fe1730, 0x1058015c8, 0x14001383c20, 0x10, 0x14001113778)
	github.com/onflow/[email protected]/runtime/ast/expression.go:456 +0x58
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).evalExpression(...)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter_expression.go:162
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).assignmentGetterSetter.func1(0x14001383c00, 0x1057f1660, 0x14000fe1730)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter_expression.go:48 +0x44
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).visitVariableDeclaration(0x14001383c20, 0x14000fd0500, 0x140011137c8)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter_statement.go:392 +0x10c
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).VisitVariableDeclaration(0x14001383c20, 0x14000fd0500, 0x1, 0x1041a4f10)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter_statement.go:353 +0x4c
github.com/onflow/cadence/runtime/ast.(*VariableDeclaration).Accept(0x14000fd0500, 0x1058015c8, 0x14001383c20, 0x104164a28, 0x140013a4200)
	github.com/onflow/[email protected]/runtime/ast/variable_declaration.go:59 +0x38
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).evalStatement(0x14001383c20, 0x1057e8138, 0x14000fd0500, 0x0, 0x0)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter_statement.go:43 +0xa0
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).visitStatements(0x14001383c20, 0x14000fe8640, 0x3, 0x4, 0x14000011c01, 0x1400135d540)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter_statement.go:49 +0x68
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).invokeInterpretedFunctionActivated.func1(0x14001383c20, 0x0)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter_invocation.go:149 +0x44
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).visitFunctionBody(0x14001383c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x14001113a40, 0x0, 0x0, ...)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter.go:1084 +0x2e0
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).invokeInterpretedFunctionActivated(0x14001383c20, 0x1400138f480, 0x1400135d520, 0x2, 0x2, 0x0, 0x0)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter_invocation.go:145 +0xe4
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).invokeInterpretedFunction(0x14001383c20, 0x1400138f480, 0x0, 0x0, 0x0, 0x0, 0x1400135d520, 0x2, 0x2, 0x0, ...)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter_invocation.go:130 +0xec
github.com/onflow/cadence/runtime/interpreter.(*InterpretedFunctionValue).invoke(0x1400138f480, 0x0, 0x0, 0x0, 0x0, 0x1400135d520, 0x2, 0x2, 0x0, 0x0, ...)
	github.com/onflow/[email protected]/runtime/interpreter/function.go:104 +0x4c
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).prepareInvoke(0x14001383c20, 0x1057f8828, 0x1400138f480, 0x140013617a0, 0x1400135d4c0, 0x2, 0x2, 0x14001113c78, 0xb4, 0x104feabe8, ...)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter.go:828 +0x1a0
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).invokeVariable(0x14001383c20, 0x104fe941d, 0x4, 0x1400135d4c0, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0)

ERROR: github.com/onflow/[email protected]/runtime/interpreter/interpreter.go:764 +0x144
github.com/onflow/cadence/runtime/interpreter.(*Interpreter).Invoke(0x14001383c20, 0x104fe941d, 0x4, 0x1400135d4c0, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0)
	github.com/onflow/[email protected]/runtime/interpreter/interpreter.go:839 +0xa0
github.com/onflow/cadence/runtime.scriptExecutionFunction.func1(0x14001383c20, 0x0, 0x0, 0x0, 0x0)
	github.com/onflow/[email protected]/runtime/runtime.go:332 +0x130
github.com/onflow/cadence/runtime.(*interpreterRuntime).interpret.func1()
	github.com/onflow/[email protected]/runtime/runtime.go:372 +0xa0
github.com/onflow/cadence/runtime.reportMetric(0x140011140c0, 0x105800f78, 0x1400017d180, 0x140011140e8)
	github.com/onflow/[email protected]/runtime/runtime.go:143 +0x7c
github.com/onflow/cadence/runtime.(*interpreterRuntime).interpret(0x140014c60c0, 0x1400138a2c0, 0x105800f78, 0x1400017d180, 0x1057e4558, 0x140010db998, 0x0, 0x0, 0x0, 0x140010f8bd0, ...)
	github.com/onflow/[email protected]/runtime/runtime.go:366 +0x174
github.com/onflow/cadence/runtime.(*interpreterRuntime).ExecuteScript(0x140014c60c0, 0x140008ba900, 0x2fc, 0x300, 0x140010f8930, 0x2, 0x2, 0x105800f78, 0x1400017d180, 0x1057e4558, ...)
	github.com/onflow/[email protected]/runtime/runtime.go:256 +0x454
github.com/onflow/flow-go/fvm.ScriptInvoker.Process(0x140014c60d0, 0x1057f4728, 0x106213ac0, 0x1057bcf68, 0x140014c60b0, 0x1057e3250, 0x10627b100, 0x0, 0x0, 0x186a0, ...)
	github.com/onflow/[email protected]/fvm/script.go:83 +0x154
github.com/onflow/flow-go/fvm.(*ScriptProcedure).Run(0x1400775ce70, 0x140014c60d0, 0x1057f4728, 0x106213ac0, 0x1057bcf68, 0x140014c60b0, 0x1057e3250, 0x10627b100, 0x0, 0x0, ...)
	github.com/onflow/[email protected]/fvm/script.go:51 +0xa8
github.com/onflow/flow-go/fvm.(*VirtualMachine).Run(0x140014c60d0, 0x1057f4728, 0x106213ac0, 0x1057bcf68, 0x140014c60b0, 0x1057e3250, 0x10627b100, 0x0, 0x0, 0x186a0, ...)
	github.com/onflow/[email protected]/fvm/fvm.go:68 +0x158
github.com/onflow/flow-emulator.(*Blockchain).ExecuteScriptAtBlock(0x1400150a000, 0x140008ba900, 0x2fc, 0x300, 0x140010f8930, 0x2, 0x2, 0x9, 0x0, 0x0, ...)
	github.com/onflow/[email protected]/blockchain.go:1040 +0x2dc
github.com/onflow/flow-emulator/server/backend.(*Backend).executeScriptAtBlock(0x1400a5b7400, 0x140008ba900, 0x2fc, 0x300, 0x140010f8930, 0x2, 0x2, 0x9, 0x301, 0x12dff1c00, ...)
	github.com/onflow/[email protected]/server/backend/backend.go:565 +0x74
github.com/onflow/flow-emulator/server/backend.(*Backend).ExecuteScriptAtLatestBlock(0x1400a5b7400, 0x1057e26f0, 0x140010f8900, 0x140008ba900, 0x2fc, 0x300, 0x140010f8930, 0x2, 0x2, 0x1055daae0, ...)
	github.com/onflow/[email protected]/server/backend/backend.go:372 +0x1d8
github.com/onflow/flow-emulator/server/backend.(*Adapter).ExecuteScriptAtLatestBlock(0x1400a2e82e8, 0x1057e26f0, 0x140010f8900, 0x140008ba900, 0x2fc, 0x300, 0x140010f8930, 0x2, 0x2, 0x1057d0a30, ...)
	github.com/onflow/[email protected]/server/backend/adapter.go:159 +0x6c
github.com/onflow/flow-go/access.(*Handler).ExecuteScriptAtLatestBlock(0x1400a5b7480, 0x1057e26f0, 0x140010f8900, 0x14000641f90, 0x1400a5b7480, 0x3, 0x12e0c7998)
	github.com/onflow/[email protected]/access/handler.go:305 +0x80
github.com/onflow/flow/protobuf/go/flow/access._AccessAPI_ExecuteScriptAtLatestBlock_Handler.func1(0x1057e26f0, 0x140010f8900, 0x1056ca1c0, 0x14000641f90, 0x31, 0x140010fc000, 0x0, 0x0)
	github.com/onflow/flow/protobuf/go/[email protected]/access/access_grpc.pb.go:700 +0x7c
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1(0x1057e26f0, 0x140010f8900, 0x1056ca1c0, 0x14000641f90, 0x14007e13d60, 0x140010db848, 0x105699560, 0x140010f8900, 0x58, 0x1056e1240)
	github.com/grpc-ecosystem/[email protected]/server_metrics.go:107 +0x8c
github.com/onflow/flow/protobuf/go/flow/access._AccessAPI_ExecuteScriptAtLatestBlock_Handler(0x105730820, 0x1400a5b7480, 0x1057e26f0, 0x140010f8900, 0x14007bfdda0, 0x140002d7190, 0x1057e26f0, 0x140010f8900, 0x140002f8a80, 0x34d)
	github.com/onflow/flow/protobuf/go/[email protected]/access/access_grpc.pb.go:702 +0x12c
google.golang.org/grpc.(*Server).processUnaryRPC(0x1400845bdc0, 0x1057f2f78, 0x140010f6240, 0x140010d7e60, 0x1400a5bad80, 0x10622bbd0, 0x0, 0x0, 0x0)
	google.golang.org/[email protected]/server.go:1282 +0x3e8
google.golang.org/grpc.(*Server).handleStream(0x1400845bdc0, 0x1057f2f78, 0x140010f6240, 0x140010d7e60, 0x0)
	google.golang.org/[email protected]/server.go:1616 +0xa50
google.golang.org/grpc.(*Server).serveStreams.func1.2(0x140006375d0, 0x1400845bdc0, 0x1057f2f78, 0x140010f6240, 0x140010d7e60)
	google.golang.org/[email protected]/server.go:921 +0x94
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/[email protected]/server.go:919 +0x1f8

[bluesign]

Hey @sampullman, thanks for the feedback. Recently cadence made some changes to storage layer, I will have a look at it today if I can figure out what went wrong.

Although it is possible to make it work without persistence, it is a bit tricky as memory grows on each snapshot significantly. (Also you lose a bit about branching capabilities)

For deleting flowdb part, you can always use a initial empty state snapshot. And rollback to emulator to initial state when starting new test suite.

[sampullman]

Recently cadence made some changes to storage layer, I will have a look at it today if I can figure out what went wrong.

Got it, thanks!

For deleting flowdb part, you can always use a initial empty state snapshot. And rollback to emulator to initial state when starting new test suite.

Right, that's how we set it up at first, but it's slightly fragile since we'd be relying on state external to our Jest tests. In practice it's probably fine.

[sideninja]

@sampullman just to confirm, the above crash happen while also using the snapshot feature? and if snapshot is not used it doesn't? I feel like there might be some invalid manipulation of the storage.

[bluesign]

@sampullman about the crash, any chance you can provide flowdb after crash ? Also is there a possibility somehow running concurrent snapshots in tests?

[sampullman]

@sideninja @bluesign Sorry for the late followup, I didn't have much time to test this further during the week.

the above crash happen while also using the snapshot feature? and if snapshot is not used it doesn't?

I refactored our tests somewhat to work with snapshots, but they are hitting the same features in our contract code. We've been running the tests in CI for several months without any issues.

any chance you can provide flowdb after crash ? Also is there a possibility somehow running concurrent snapshots in tests?

I attached the flowdb folder, let me know if there's anything else I can provide. I'm not sure exactly what you mean by concurrent snapshots, but the pattern I'm attempting is:

• Take an initial snapshot
  • In the first test file, this initializes the snapshot. In later test files, it should rest to the initial state
• In each test file, set up some basic seed data (accounts, tokens, contracts)
• In Jest beforeEach, reset to the "seed" snapshot

flowdb.zip