Grain Timer stop firing after a while about orleans

[sc-starman]

Upgraded to 8.0.0-rc2 but still got the same problem.
Still, there is no log or evidence that I could use to find out what happened.

[ReubenBond]

How are you verifying that the timer is not firing? Could you show the relevant code? When you say that the grain continues to function, I assume this means that there are no logs showing that the grain has been deactivated. Is that correct?

[sc-starman]

This is the timer function code:

        private async Task PublishOrderbook(object arg)
        {
            if (_orderbookUpdatedTimestamp <= _orderbookSentTimestamp)
            {
                if ((DateTime.UtcNow - _orderbookUpdatedTimestamp).TotalSeconds == 5)
                    _logger.LogCritical("Orderbook not Updated Recently, Pair={Pair}, LastUpdateInSeconds={LastUpdateInSeconds}, Hostname={Hostname}",
                        Pair, (DateTime.UtcNow - _orderbookUpdatedTimestamp).TotalSeconds, Dns.GetHostName());
                return;
            }

            _orderbookSentTimestamp = DateTime.UtcNow;

                    _logger.LogCritical("Orderbook Updated Successfuly, Pair={Pair}, LastUpdateInSeconds={LastUpdateInSeconds}, Hostname={Hostname}",
                        Pair, (DateTime.UtcNow - _orderbookUpdatedTimestamp).TotalSeconds, Dns.GetHostName());


            int count = 40;
            var book = _engine.Book;
            var asks = book.AskSide;
            var bids = book.BidSide;

            _orderBook = new OrderBookStream()
            {
                Pair = Pair,
                Asks = asks.Take(count).ToDictionary(a => (decimal)a.Value.Price, a => (decimal)a.Value.Quantity),
                Bids = bids.Take(count).ToDictionary(a => (decimal)a.Value.Price, a => (decimal)a.Value.Quantity),
                BestAskPrice = book.BestAskPrice,
                BestAskQuantity = book.BestAskQuantity,
                BestBidPrice = book.BestBidPrice,
                BestBidQuantity = book.BestBidQuantity,
                BestStopAskPrice = book.BestStopAskPrice,
                BestStopBidPrice = book.BestStopBidPrice,
                Timestamp = book.LastBookUpdate
            };

            var subject = $"{typeof(OrderBookStream).FullName}_{Pair}";


            await _natsCommand.PublishAsync(subject, _orderBook);
        }

Yeah not only I'm sure the grain is not deactivated (no log) but also I have a grain method that I can query the difference between when the timer fired and how much time passed from the grain activation. Everything seems fine except the timer.

[sc-starman]

Further investigation, I realized that this occurred on the silo level not at the grain level, which means that all timers which belong to each grain on a specific silo won't fire at all. Even if I registered them again (when the grain was already active), they would fire only for the first time.