Could not find an implementation for interface error when using grain persistence about orleans

[attilah]

Perhaps unrelated, but are the grains are in a separate project and that project used the Microsoft.Orleans.Sdk package? (if .Client or .Server is not used in that project.

[ReubenBond]

This issue does not seem related to grain persistence. How is the client connecting to the cluster where the grain is hosted.

[macfma01]

I'm running both the silo and the client locally for debugging. So, the client contacts a running instance of the silo on a localhost address.

Did a test and verified an older grain and it's working fine when the client calls _clusterClient.GetGrain<...> so I can confirm the connection between the client and the silo is working.

[macfma01]

Couple other simple things I've verified

• The grain interface inherits IGrainWithStringKey.
• The grain class inherits Grain.

[ReubenBond]

Yes, please confirm that you have the packages which @attilah mentioned on both the client, server, and grains/interfaces. Your grain/grain interface project should all reference Microsoft.Orleans.Sdk. Your client should reference Microsoft.Orleans.Client, and your server should reference Microsoft.Orleans.Server.

[macfma01]

There are 3 projects in the silo solution. One contains the interfaces and data models classes for request and response types. We call it Abstractions. The implementation of the grains are a different projects called Grains. Then there is the code gen project. All 3 projects have a reference to the Microsoft.Orleans.Sdk package version 7.2.4.

[macfma01]

Yes, the client project has Microsoft.Orleans.Client 7.2.4 and the silo project has Microsoft.Orleans.Server 7.2.4.

[ReubenBond]

@macfma01 is the client definitely connected to the silo before you call GetGrain? If not, that may be the issue. How is the client's host being configured right now and when is the GetGrain call being made in the client host's lifetime? If you can share some code snippets, that will be quite helpful

[attilah]

@macfma01 ah, wanted to confirm as that was what I missed...it is something I did not remember/changed since I was doing Orleans ;-)

[macfma01]

I'm confident in the client/silo connection because I can activate other grains. Just not this newer one. The code for the client app is a vast ASP.NET app with calls out to Orleans spread around in the different classes. Each class that calls Orleans has an IClusterClient injected. We use both Guid and String based grains. We ask for a grain instance using GetGrain

For guid types
var grainRef = _clusterClient.GetGrain<IMyGrainInterface>(Guid.NewGuid());

For string string
var grainRef = _clusterClient.GetGrain<IMyGrainInterface>(Guid.NewGuid().ToString()); or some other relevant string key we're tracking.

I did a fresh build on the Orleans project and update the packages it produced on the client just verify that was up to date but it didn't make a difference. There seems to be something about the new grain that off. I just can't spot it yet.

Here are couple excerpts from Program.cs on the client in case that's relevant.

services.AddSingleton<ClusterClientHostedService>(); // orleans client host,
services.AddSingleton<IHostedService>(_ => _.GetRequiredService<ClusterClientHostedService>());
services.AddSingleton(_ => _.GetRequiredService<ClusterClientHostedService>()?.Client.Services.GetService<IClusterClient>());

Excepts from ClusterClientHostedService. I don't fully understand this class but I think it's designed to start up a new ClusterClient if it dies. I haven't delved into all this too deeply yet since I have grains that are working fine with all this it hasn't seemed like a likely cause.

public class ClusterClientHostedService : IHostedService
    {
        public IHost Client { get; }

        public ClusterClientHostedService

        {
            var clientBuilder = new HostBuilder()
                .ConfigureLogging(loggingBuilder =>
                    loggingBuilder.SetMinimumLevel(LogLevel.Information).AddProvider(loggerProvider))
                .UseOrleansClient((context,siloBuilder) =>
                {
                    siloBuilder.AddOutgoingGrainCallFilter<ClientGrainCallFilter>();
                    siloBuilder.AddActivityPropagation();
                    siloBuilder.Configure<ClusterOptions>(clusterOptions =>
                    {
                        clusterOptions.ClusterId = MyClusterInfo.GetClusterId(GetSiloEnvironment());
                        clusterOptions.ServiceId = MyClusterInfo.ServiceId;
                    }).ConfigureServices(services => { services.AddSingleton(DistributedContextPropagator.Current); });
                if (GetSiloEnvironment() == "local")
                {
                    siloBuilder.UseLocalhostClustering(
                        int.Parse((Environment.GetEnvironmentVariable("GATEWAYPORT"))),
                        MyClusterInfo.ServiceId,
                        MyClusterInfo.GetClusterId(GetSiloEnvironment())
                    );
                }
                else
                {
                    siloBuilder.UseDynamoDBClustering(dynamoDbGatewayOptions =>
                    {
                        dynamoDbGatewayOptions.Service = AwsHelpers.GetAwsRegion();
                        dynamoDbGatewayOptions.TableName = ErsClusterInfo.MembershipTableName;
                        dynamoDbGatewayOptions.UpdateIfExists = false;
                    });
                }
            });
            Client = clientBuilder.Build();
        }

        public async Task StartAsync(CancellationToken cancellationToken)
        {
            try
            {
                while (true)
                {
                    try
                    {     
                        if (cancellationToken.IsCancellationRequested)
                        {
                            throw new OperationCanceledException("Orleans cluster connection cancelled");
                        }
                        await Client.StartAsync(cancellationToken);
                        return;
                    }
                    catch (Exception exception) 
                    {
                        _logger.LogWarning(
                            exception,
                            "Failed to connect to Orleans cluster on attempt {attempt} of {maxAttempts}.", attempt, maxAttempts);
                        
                        if (++attempt <= maxAttempts)
                        {
                            await Task.Delay(delay, cancellationToken);
                        }
                        else
                        {
                            throw new Exception("Unable to connect to Orleans cluster");
                        }
                    }
                }
            }
            catch (Exception exception)
            {
            }
        }

[macfma01]

I'm pretty sure it's a bug in the cluster client class. I'll double check everything and update this issue tomorrow. The env variable isn't returning local as expected
if (GetSiloEnvironment() == "local")

I might be getting the local interfaces and trying to grains from another dev silo where they haven't been deployed yet.

[ReubenBond]

Does the issue resolve itself after the client has been connected to the cluster for some time?

[macfma01]

I've confirmed this was a misconfiguration in our launch settings. The client wasn't using the right settings for local connections as I had thought 😳 Since some of the calls were working I mistakenly assumed it was good but the client was talking to another silo on our dev environment (I think). The grain I was getting an error on is loading up fine now. So, there was no Orleans issue at all. Thanks for your questions and suggestions for things to check. That led me to the root cause.

[ReubenBond]

Glad to see that you got it fixed!