Error with Collator - Binary v2.19.0-24c7810f200 about acala

[paradox-tt]

Built binaries using cargo build --locked --features with-karura-runtime --profile production --bin acala thus far all is working well but I'm holding for observation for another 24hrs.

[xlc]

what’s the commit you using and can you share bit more logs before the error?

[paradox-tt]

Here's a bit more detail before the error and we're using 2.19 commit 24c7810f200

Dec 20 03:59:48 Kurura karura[477345]: 2023-12-20 03:59:48 [Parachain] Starting collation. relay_parent=0x81f70140e85dd4b36ead510ac46b2ec763768ea171736f94c8faca418b712534 at=0x21a6dff64d3c8a228377ecac6fd01ac1448e5bb724a96caf769c0b5d3728a92f
Dec 20 03:59:49 Kurura karura[477345]: 2023-12-20 03:59:48 [Relaychain] ✨ Imported #21065049 (0xc323…3966)
Dec 20 03:59:49 Kurura karura[477345]: 2023-12-20 03:59:49 [Parachain] Starting collation. relay_parent=0xc3238f2f3271f76d417ce10d8e6976544228e20f6fd539326d9df014c0dd3966 at=0x21a6dff64d3c8a228377ecac6fd01ac1448e5bb724a96caf769c0b5d3728a92f
Dec 20 03:59:49 Kurura karura[477345]: 2023-12-20 03:59:49 [Relaychain] 💤 Idle (7 peers), best: #21065049 (0x81f7…2534), finalized #21065045 (0x45d2…16ee), ⬇ 92.3kiB/s ⬆ 172.5kiB/s
Dec 20 03:59:49 Kurura karura[477345]: 2023-12-20 03:59:49 [Parachain] ✨ Imported #5888895 (0x3be7…4bd5)
Dec 20 03:59:49 Kurura karura[477345]: 2023-12-20 03:59:49 [Parachain] ✨ Imported #5888895 (0x5107…0f1e)
Dec 20 03:59:52 Kurura karura[477345]: 2023-12-20 03:59:51 [Parachain] 💤 Idle (28 peers), best: #5888894 (0x21a6…a92f), finalized #5888893 (0xb0af…798f), ⬇ 11.6kiB/s ⬆ 190.9kiB/s
Dec 20 03:59:54 Kurura karura[477345]: 2023-12-20 03:59:54 [Relaychain] 💤 Idle (6 peers), best: #21065049 (0x81f7…2534), finalized #21065046 (0x77c8…2844), ⬇ 410.2kiB/s ⬆ 454.8kiB/s
Dec 20 03:59:54 Kurura karura[477345]: 2023-12-20 03:59:54 [Relaychain] ✨ Imported #21065050 (0xf736…e638)
Dec 20 03:59:57 Kurura karura[477345]: 2023-12-20 03:59:56 [Parachain] 💤 Idle (29 peers), best: #5888894 (0x21a6…a92f), finalized #5888893 (0xb0af…798f), ⬇ 1.6kiB/s ⬆ 1.5kiB/s
Dec 20 03:59:59 Kurura karura[477345]: 2023-12-20 03:59:59 [Relaychain] 💤 Idle (7 peers), best: #21065050 (0xf736…e638), finalized #21065047 (0x5eca…5937), ⬇ 149.5kiB/s ⬆ 156.8kiB/s
Dec 20 04:00:00 Kurura karura[477345]: 2023-12-20 04:00:00 [Relaychain] ✨ Imported #21065051 (0x48df…6c46)
Dec 20 04:00:00 Kurura karura[477345]: 2023-12-20 04:00:00 [Parachain] Starting collation. relay_parent=0x48dfadd63b138aa86d6afcbc4ea5c49a91815949d4fa6e49974f3b62216d6c46 at=0x51079c54942995fc3c840d7ad0ececaee77b2b07e16e23608d8ebffcd51b0f1e
Dec 20 04:00:00 Kurura karura[477345]: 2023-12-20 04:00:00 [Parachain] 🙌 Starting consensus session on top of parent 0x51079c54942995fc3c840d7ad0ececaee77b2b07e16e23608d8ebffcd51b0f1e
Dec 20 04:00:00 Kurura karura[477345]: 2023-12-20 04:00:00 [Parachain] panicked at /home/builder/cargo/registry/src/index.crates.io-6f17d22bba15001f/cumulus-pallet-parachain-system-0.4.1/src/lib.rs:626:18:
Dec 20 04:00:00 Kurura karura[477345]: Invalid messaging state in relay chain state proof: RelayDispatchQueueRemainingCapacity(Proof)
Dec 20 04:00:00 Kurura karura[477345]: 2023-12-20 04:00:00 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
Dec 20 04:00:00 Kurura karura[477345]: 2023-12-20 04:00:00 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
Dec 20 04:00:00 Kurura karura[477345]: 2023-12-20 04:00:00 [Parachain] ❗️ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed

[xlc]

we wasn’t expect breaking change and will investigate. but can you upgrade to 2.23?

[Poggi-Luca]

We had the same identical issue with Karura collator v2.19.0.
Updated both Acala and Karura to v2.23.0.
Let's keeping monitored!

[xlc]

v2.19.0 is known that contains multiple issues. everyone should upgrade to v2.23.0