Orderly shutdown and recovery of nodes about akkadotnet-code-samples

[tstojecki]

Thanks for the feedback @Aaronontheweb but it looks like this goes further than that.
I removed Topshelf from WebCrawler.CrawlService and wired-up console shut down event, calling ClusterSystem.Terminate(). Console shutdown is captured as described in this post (ugly, but it does work as verified in debugger) https://stackoverflow.com/questions/474679/capture-console-exit-c-sharp?noredirect=1&lq=1

This is still happening. I have shut down the crawler console app either through ctrl+c or by clicking X. In both cases after a few seconds it reports

Akka.Cluster.ClusterCoreDaemon: Leader can currently not perform its duties, reachability status: [Reachability([akka.tcp://[email protected]:56374 -> UniqueAddress: (akka.tcp://[email protected]:16666, 689080532): Reachable [Reachable] (4)], [akka.tcp://[email protected]:56374 -> UniqueAddress: (akka.tcp://[email protected]:4053, 894445096): Reachable [Reachable] (5)], [akka.tcp://[email protected]:56374 -> UniqueAddress: (akka.tcp://[email protected]:56373, 1593674278): Unreachable [Unreachable] (3)][akka.tcp://[email protected]:16666 -> UniqueAddress: (akka.tcp://[email protected]:56374, 310250411): Unreachable [Unreachable] (7)][akka.tcp://[email protected]:4053 -> UniqueAddress: (akka.tcp://[email protected]:56374, 310250411): Unreachable [Unreachable] (7)][akka.tcp://[email protected]:56373 -> UniqueAddress: (akka.tcp://[email protected]:56374, 310250411): Unreachable [Unreachable] (7)])], member status: [$akka.tcp://[email protected]:4053 $Up seen=$True, $akka.tcp://[email protected]:16666 $Up seen=$True, $akka.tcp://[email protected]:56373 $Up seen=$False, $akka.tcp://[email protected]:56374 $Up seen=$False]

It never recovers after that. Starting a crawler again, LH will report the node joining, but then it keeps throwing association errors and nothing works anymore.

Had this scenario worked before switching to dotnetty? Is that scenario (nodes being shut-down, crashing) something that is being tested regularly in akka.cluster? Sorry for asking naive questions like that, but I have been out of the loop for a bit and I am a little surprised that this behaves this way as I would consider this a show stopper for production use.

Also, even if the node (in this case crawler) isn't doing a clean up work properly, why is it having such damaging effects on the rest of the cluster? This can't be by design, can it? LH could report that there is an issue with a node, but why is it entering this "cannot perform its duties" state? Can something be done to make that more resilient?

Please let me know how I can be of help in troubleshooting this further.

[Aaronontheweb]

@tstojecki currently there's an issue where the Topshelf services don't properly allow the CoordinatedShutdown to run for WebCrawler. I noticed that this week after we did the upgrade to 1.2 earlier. One thing you have to do is Control + C instead of just clicking X, otherwise the ServiceStop method for Topshelf never runs (it says this at startup. )