Hi Yvan
We are experiencing this issue where users inteminnently are getting HTTP error 500 with message "The server is busy now. Try again later".
This happens after user is authenticatet and redirected to https://site/_trust/
In ULS I can see that calls to securitytoken.svc time out after 60 sec. It happens on both frontends.
The servers are not under pressure, doesn't look like a hardware capacity issue. Restarting SecurityTokenServiceAppPool seem to help but not for a long period.
This farm is using LDAPCP, and I'm wondering if our LDAPCP's configuration somehow contribute to the issue, specially the augmentation part of it.
-
LDAPCP the only thing that is unique comparing to other farms
-
Users can be members of up to 500 AD groups
-
The token issued by ADFS contains claim "Role" with users group membership, but this is ignored and LDAPCP augmentation of "Role" is used, getting the same memberships from identity providers directly using LDAP
-
Can see LDAP errors occuring in ULS, LDAP timeouts or LDAP Unknown error (0x8000500c). Usually when it happens users get "This page is not shared with you..." message
Is there anything with this configuration and symptoms that can cause securitytoken.svc timeouts when converting external to internal token og should I search for a solution somewhere else?
:: Sharepoint configuration
- Sharepoint 2016 RTM on Windows Server 2016
- Using claims authentication
- Federation provider: ADFS
- Identity providers: 2 active directories AD.DOMAIN1.ORG and AD.DOMAIN2.ORG
- Claim provider: LDAPCP
- Identity claim: UPN
:: LDAPCP configuration
-
LDAP connections:
LDAP://OU=UsersOU1,OU=Accounts,DC=AD,DC=DOMAIN1,DC=ORG
LDAP://OU=UsersOU2,OU=Accounts,DC=AD,DC=DOMAIN1,DC=ORG
LDAP://OU=UsersOU3,OU=Accounts,DC=AD,DC=DOMAIN1,DC=ORG
LDAP://OU=UsersOU4,OU=Accounts,DC=AD,DC=DOMAIN1,DC=ORG
LDAP://OU=UsersOU5,OU=Accounts,DC=AD,DC=DOMAIN1,DC=ORG
LDAP://OU=Groups,DC=AD,DC=DOMAIN1,DC=ORG (used for groups augmentation in DOMAIN1)
LDAP://AD.DOMAIN2.ORG (used for groups augmentation in DOMAIN2)
-
Augmentation: Enabled
-
Augmented Group Claim Type: http://schemas.microsoft.com/ws/2008/06/identity/claims/role
-
LDAP connections used for augmentation ("This is an AD server" selected):
LDAP://OU=Groups,DC=AD,DC=DOMAIN1,DC=ORG
LDAP://AD.DOMAIN2.ORG
-
LDAP query timeout: increased to 30, does not seem to help
-
Additional LDAP filter for user attributes: Defined and present
:: Full exception from ULS
Claims Saml Sign-In: Could not get local token for trusted third party token. Exception: 'System.TimeoutException: The request channel timed out while waiting for a reply after 00:01:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to 'http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc' has exceeded the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.WebException: The operation has timed out at System.Net.HttpWebRequest.GetResponse() at System.ServiceModel.Channels.HttpChannelFactory1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) - -- End of inner exception stack trace --- at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason) at System.ServiceModel.Channels.HttpChannelFactory1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) - -- End of inner exception stack trace --- Server stack trace: at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Microsoft.IdentityModel.Protocols.WSTrust.IWSTrustContract.Issue(Message message) at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst, RequestSecurityTokenResponse& rstr) at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst) at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForContext(Uri context, Boolean bearerToken, SecurityToken onBehalfOf, SecurityToken actAs, SecurityToken delegateTo, SPRequestSecurityTokenProperties properties) at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForOnBehalfOfContext(Uri context, SecurityToken onBehalfOf) at Microsoft.SharePoint.IdentityModel.SPFederationAuthenticationModule.ExchangeArgumentTrustedThirdPartySessionSecurityTokenForLocalToken(SecurityToken thirdPartyToken, SessionSecurityTokenCreatedEventArgs arguments)'. Stack: ' Server stack trace: at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Microsoft.IdentityModel.Protocols.WSTrust.IWSTrustContract.Issue(Message message) at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst, RequestSecurityTokenResponse& rstr) at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst) at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForContext(Uri context, Boolean bearerToken, SecurityToken onBehalfOf, SecurityToken actAs, SecurityToken delegateTo, SPRequestSecurityTokenProperties properties) at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForOnBehalfOfContext(Uri context, SecurityToken onBehalfOf) at Microsoft.SharePoint.IdentityModel.SPFederationAuthenticationModule.ExchangeArgumentTrustedThirdPartySessionSecurityTokenForLocalToken(SecurityToken thirdPartyToken, SessionSecurityTokenCreatedEventArgs arguments)'.