AutoMapper throws EntryPointNotFoundException when profiler is enabled on Linux about dd-trace-dotnet

[pgrm]

@lucaspimentel - we could figure out that it happens when we define a mapping for generic collections such as List<T> or Dictionary, but it doesn't happen for any other mappings. Funny enough, arrays, such as T[] work just fine.

The error seems to happen in this line of automapper: https://github.com/AutoMapper/AutoMapper/blob/9903a3d2997c993e116c29d5f43c4ba89fac87d4/src/AutoMapper/Mappers/Internal/CollectionMapperExpressionFactory.cs#L65

unfortunately, we didn't manage to reproduce it in an empty project yet. All that we could see, is, if we remove the mapping for those generic collection properties, it worked and if we put just one back, it breaks.

We'll try to figure out, which other part of the application might influence those collections. In case you have any more ideas, how we could figure out, which assembly might affect the issue, we're open for suggestions.

[lucaspimentel]

Just a heads up that I just finished up some other work and will start investigating this issue today. If it doesn't make it in time for the 1.2 release in the next few days, we will follow up with a 1.2.1 for this issue.

[simongottschlag]

We are seeing the same issue with automapper 8.0.0 and dd-trace-dotnet 1.1.0. As soon as we disable tracing, it starts working.

We are using IdentityServer4.

[pgrm]

@lucaspimentel here you go with a repo reproducing the issue: https://github.com/pgrm/DD_EntryPointNotFoundException/

As I mentioned before, we still didn't manage reproducing it starting from scratch, but now we took out pretty much everything from the broken project and the error is still happening. The README should detail all the findings, hope it will help to fix it. Thx

[HitDaCa]

We are facing the same issue using the .net core client 1.2.0 with Identity Server 4.

For Identity server it's definitely to do with the EntityFrameworkCore -> AutoMapper integration (conflicting and throwing the same type of error as described above and in the provided example).

Additionally, I only need to add the APM package into docker and include 'Datadog.Trace.ClrProfiler.Managed' package into our project to start seeing the error. Setting the config DD_TRACE_ENABLED to true or false doesn't influence the exception occurring.

[daniel-white]

@moyuanhui this is going to be fixed in a future version of .net core 2.1 and 2.2.

[lucaspimentel]

This issue was fixed in .NET Core 2.1.12 and 2.2.6, which were released a few days ago. If your apps are running on these versions of .NET Core or newer, you can disable our workaround by setting environment variable DD_CLR_DISABLE_OPTIMIZATIONS=false. (Linux only. The default is already false on Windows.)

As more users start using these newer versions of .NET Core, we will change the workaround so it is disabled by default. Users on older versions of .NET Core will still be able to opt-in by setting DD_CLR_DISABLE_OPTIMIZATIONS=true.

[pgrm]

FYI - just tried it out with new version 1.0 and it's still the same error. We'll try to give you a simple tool where the error can be reproduced

[lucaspimentel]

Thanks for reporting this issue, @pgrm. A small project that reproduces this issue would be very useful.

[pgrm]

right @lucaspimentel we're on it. We couldn't reproduce it when we tried to build a project from scratch so we're stripping down the broken micro service instead and found now 4 files, of which one or more contribute to that bug. We'll try to move them into a new project, that can be made open to link it here. Should be done next week

[lucaspimentel]

Thanks, @pgrm. This repro will be very useful in debugging the issue.

[SamuelMcAravey]

Same here. Using the latest 1.1.0 version of the .Net Datadog library with automatic instrumentation throws the same Automapper error. Disabling tracing on the app solves this for now.

[daniel-white]

Any progress?

[lucaspimentel]

Looks like this issue is caused by a bug in the JIT compiler on some platforms (including Linux on x64 and arm64, but not Windows).

See the following github issues:

• AutoMapper/AutoMapper#3089
• https://github.com/dotnet/coreclr/issues/24676
• https://github.com/dotnet/coreclr/issues/22330
• https://github.com/dotnet/coreclr/issues/19256
• https://github.com/dotnet/coreclr/issues/12468

Seems to be fixed in more recent version of the .NET Core 3.0 preview and might also be back-ported to .NET Core 2.2 and 2.1 (the latter being a LTS release).

(edit 2020-02-06: removed github issues statuses since they are now all closed)

[lucaspimentel]

See dotnet/coreclr#24858 for the .NET Core 2.1 fix.

[lucaspimentel]

A workaround for this issue was added in release 1.3 and is enabled by default.

[moyuanhui]

We have same problem.
untime environment (please complete the following information):

Instrumentation mode: automatic on .net core linux
Tracer version: 0.8.2
OS: linux in docker container
CLR: .NET Core 2.2

Application startup exception: System.EntryPointNotFoundException: Entry point was not found.
   at AutoMapper.IMemberMap.get_UseDestinationValue()
   at AutoMapper.Execution.ExpressionBuilder.NullCheckSource(ProfileMap profileMap, Expression sourceParameter, Expression destinationParameter, Expression objectMapperExpression, IMemberMap memberMap)
   at AutoMapper.Execution.ExpressionBuilder.MapExpression(IConfigurationProvider configurationProvider, ProfileMap profileMap, TypePair typePair, Expression sourceParameter, Expression contextParameter, IMemberMap propertyMap, Expression destinationParameter)
   at AutoMapper.Mappers.NullableDestinationMapper.MapExpression(IConfigurationProvider configurationProvider, ProfileMap profileMap, IMemberMap memberMap, Expression sourceExpression, Expression destExpression, Expression contextExpression)
   at AutoMapper.Execution.ExpressionBuilder.MapExpression(IConfigurationProvider configurationProvider, ProfileMap profileMap, TypePair typePair, Expression sourceParameter, Expression contextParameter, IMemberMap propertyMap, Expression destinationParameter)
   at AutoMapper.Mappers.NullableSourceMapper.MapExpression(IConfigurationProvider configurationProvider, ProfileMap profileMap, IMemberMap memberMap, Expression sourceExpression, Expression destExpression, Expression contextExpression)
   at AutoMapper.Execution.ExpressionBuilder.MapExpression(IConfigurationProvider configurationProvider, ProfileMap profileMap, TypePair typePair, Expression sourceParameter, Expression contextParameter, IMemberMap propertyMap, Expression destinationParameter)
   at AutoMapper.Execution.TypeMapPlanBuilder.CreatePropertyMapFunc(IMemberMap memberMap, Expression destination, MemberInfo destinationMember)
   at AutoMapper.Execution.TypeMapPlanBuilder.TryPropertyMap(PropertyMap propertyMap)
   at AutoMapper.Execution.TypeMapPlanBuilder.CreateAssignmentFunc(Expression destinationFunc)
   at AutoMapper.Execution.TypeMapPlanBuilder.CreateMapperLambda(HashSet`1 typeMapsPath)
   at AutoMapper.TypeMap.CreateMapperLambda(IConfigurationProvider configurationProvider, HashSet`1 typeMapsPath)
   at AutoMapper.TypeMap.Seal(IConfigurationProvider configurationProvider)
   at AutoMapper.MapperConfiguration.Seal()
   at AutoMapper.MapperConfiguration..ctor(MapperConfigurationExpression configurationExpression)
   at AutoMapper.MapperConfiguration..ctor(Action`1 configure)
   at AutoMapper.Mapper.Initialize(Action`1 config)
   at Abp.AutoMapper.AbpAutoMapperModule.CreateMappings()
   at Abp.AutoMapper.AbpAutoMapperModule.PostInitialize()
   at Abp.Modules.AbpModuleManager.<>c.<StartModules>b__15_2(AbpModuleInfo module)
   at System.Collections.Generic.List`1.ForEach(Action`1 action)
   at Abp.Modules.AbpModuleManager.StartModules()
   at Abp.AbpBootstrapper.Initialize()
   at Abp.AspNetCore.AbpApplicationBuilderExtensions.InitializeAbp(IApplicationBuilder app)
   at Abp.AspNetCore.AbpApplicationBuilderExtensions.UseAbp(IApplicationBuilder app, Action`1 optionsAction)
   at Abp.AspNetCore.AbpApplicationBuilderExtensions.UseAbp(IApplicationBuilder app)
   at HKERP.BasicData.Web.Host.Startup.Startup.Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory) in /home/jenkins/workspace/API-Basicdata/src/HKERP.Web.Host/Startup/Startup.cs:line 43
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.Configure(IApplicationBuilder app)
   at Microsoft.AspNetCore.Builder.CapStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app)
   at Microsoft.AspNetCore.Mvc.Internal.MiddlewareFilterBuilderStartupFilter.<>c__DisplayClass0_0.<Configure>g__MiddlewareFilterBuilder|0(IApplicationBuilder builder)
   at Microsoft.AspNetCore.HostFilteringStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app)
   at Microsoft.AspNetCore.Hosting.Internal.AutoRequestServicesStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder builder)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
crit: Microsoft.AspNetCore.Hosting.Internal.WebHost[6]
      Application startup exception
System.EntryPointNotFoundException: Entry point was not found.
   at AutoMapper.IMemberMap.get_UseDestinationValue()
   at AutoMapper.Execution.ExpressionBuilder.NullCheckSource(ProfileMap profileMap, Expression sourceParameter, Expression destinationParameter, Expression objectMapperExpression, IMemberMap memberMap)
   at AutoMapper.Execution.ExpressionBuilder.MapExpression(IConfigurationProvider configurationProvider, ProfileMap profileMap, TypePair typePair, Expression sourceParameter, Expression contextParameter, IMemberMap propertyMap, Expression destinationParameter)
   at AutoMapper.Mappers.NullableDestinationMapper.MapExpression(IConfigurationProvider configurationProvider, ProfileMap profileMap, IMemberMap memberMap, Expression sourceExpression, Expression destExpression, Expression contextExpression)
   at AutoMapper.Execution.ExpressionBuilder.MapExpression(IConfigurationProvider configurationProvider, ProfileMap profileMap, TypePair typePair, Expression sourceParameter, Expression contextParameter, IMemberMap propertyMap, Expression destinationParameter)
   at AutoMapper.Mappers.NullableSourceMapper.MapExpression(IConfigurationProvider configurationProvider, ProfileMap profileMap, IMemberMap memberMap, Expression sourceExpression, Expression destExpression, Expression contextExpression)
   at AutoMapper.Execution.ExpressionBuilder.MapExpression(IConfigurationProvider configurationProvider, ProfileMap profileMap, TypePair typePair, Expression sourceParameter, Expression contextParameter, IMemberMap propertyMap, Expression destinationParameter)
   at AutoMapper.Execution.TypeMapPlanBuilder.CreatePropertyMapFunc(IMemberMap memberMap, Expression destination, MemberInfo destinationMember)
   at AutoMapper.Execution.TypeMapPlanBuilder.TryPropertyMap(PropertyMap propertyMap)
   at AutoMapper.Execution.TypeMapPlanBuilder.CreateAssignmentFunc(Expression destinationFunc)
   at AutoMapper.Execution.TypeMapPlanBuilder.CreateMapperLambda(HashSet`1 typeMapsPath)
   at AutoMapper.TypeMap.CreateMapperLambda(IConfigurationProvider configurationProvider, HashSet`1 typeMapsPath)
   at AutoMapper.TypeMap.Seal(IConfigurationProvider configurationProvider)
   at AutoMapper.MapperConfiguration.Seal()
   at AutoMapper.MapperConfiguration..ctor(MapperConfigurationExpression configurationExpression)
   at AutoMapper.MapperConfiguration..ctor(Action`1 configure)
   at AutoMapper.Mapper.Initialize(Action`1 config)
   at Abp.AutoMapper.AbpAutoMapperModule.CreateMappings()
   at Abp.AutoMapper.AbpAutoMapperModule.PostInitialize()
   at Abp.Modules.AbpModuleManager.<>c.<StartModules>b__15_2(AbpModuleInfo module)
   at System.Collections.Generic.List`1.ForEach(Action`1 action)
   at Abp.Modules.AbpModuleManager.StartModules()
   at Abp.AbpBootstrapper.Initialize()
   at Abp.AspNetCore.AbpApplicationBuilderExtensions.InitializeAbp(IApplicationBuilder app)
   at Abp.AspNetCore.AbpApplicationBuilderExtensions.UseAbp(IApplicationBuilder app, Action`1 optionsAction)
   at Abp.AspNetCore.AbpApplicationBuilderExtensions.UseAbp(IApplicationBuilder app)
   at HKERP.BasicData.Web.Host.Startup.Startup.Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory) in /home/jenkins/workspace/API-Basicdata/src/HKERP.Web.Host/Startup/Startup.cs:line 43
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.Configure(IApplicationBuilder app)
   at Microsoft.AspNetCore.Builder.CapStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app)
   at Microsoft.AspNetCore.Mvc.Internal.MiddlewareFilterBuilderStartupFilter.<>c__DisplayClass0_0.<Configure>g__MiddlewareFilterBuilder|0(IApplicationBuilder builder)
   at Microsoft.AspNetCore.HostFilteringStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app)
   at Microsoft.AspNetCore.Hosting.Internal.AutoRequestServicesStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder builder)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()

(edit: formatting)

[lucaspimentel]

Update: with release 1.15.0, we have flipped the DD_CLR_DISABLE_OPTIMIZATIONS switch so it always defaults to false (opt-in). Users running applications on affected versions of the .NET Core runtime on Linux/x64 are strongly encouraged to update to a newer version. Windows users are not affected by this issue.

• .NET Core 2.0: All versions are affected. Note that .NET Core 2.0 reached end of life before this bug was fixed and will not receive further updates.
• .NET Core 2.1: Versions from 2.1.0 to 2.1.11 are affected. .NET Core 2.1.12 and later do not have this issue.
• .NET Core 2.2: Versions 2.2.0 to 2.2.5 are affected. .NET Core 2.2.6 and later do not have this issue. Note that .NET Core 2.2 reached end of life and will not receive further updates.
• .NET Core 3.0: No version is affected. The bug was fixed during the preview cycle and was not present in the 3.0.0 release. Note that .NET Core 3.0 reached end of life and will not receive further updates.
• .NET Core 3.1: No version is affected.

See .NET Core downloads for instructions on updating your SDK and runtime.

If your applications run on an affected runtime version and you cannot update, you should set the environment variable DD_CLR_DISABLE_OPTIMIZATIONS=true to opt-in to our workaround to avoid throw unexpected exceptions.

(edit: added link to release page, typos)