Data export job get paused with the following error: RecurringIntegrationsScheduler.Common.Helpers.HttpClientHelper.<GetExecutionSummaryStatus>d__19.MoveNext() about recurring-integrations-scheduler

[TomekMelissa]

please use the latest version and report back

[ChristiaanSu]

Hey Tomek,

Thanks for the quick response.
Dynamics 365FO
Installed platform version : Update15 (7.0.4841.35259)
RIS Version 2.3.0.0

Ive installed the latest version but randomly getting the following error for export jobs. This is happening for 7 different jobs thats using different data entities. Same error for all jobs.

Quartz.Core.JobRunShell - Job Export jobs. threw a JobExecutionException:
Parameters: refire = False, unscheduleFiringTrigger = False, unscheduleAllTriggers = False
Quartz.JobExecutionException: Export job: Export jobs. failed ---> System.NullReferenceException: Object reference not set to an instance of an object.
at RecurringIntegrationsScheduler.Common.Helpers.HttpClientHelper.d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at RecurringIntegrationsScheduler.Job.Export.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at RecurringIntegrationsScheduler.Job.Export.d__6.MoveNext()
--- End of inner exception stack trace ---
at RecurringIntegrationsScheduler.Job.Export.d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Quartz.Core.JobRunShell.d__9.MoveNext() [See nested exception: System.NullReferenceException: Object reference not set to an instance of an object.
at RecurringIntegrationsScheduler.Common.Helpers.HttpClientHelper.d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at RecurringIntegrationsScheduler.Job.Export.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at RecurringIntegrationsScheduler.Job.Export.d__6.MoveNext()]

[TomekMelissa]

Can you try to delete and schedule one of the job from scratch?

[ChristiaanSu]

Ive created new data projects on Dynamics and new jobs on the RIS but still getting the same error

Quartz.Core.JobRunShell - Job Export jobs.CustomerExportINTAU_CS threw a JobExecutionException:
Parameters: refire = False, unscheduleFiringTrigger = False, unscheduleAllTriggers = False
Quartz.JobExecutionException: Export job: Export jobs.CustomerExportINTAU_CS failed ---> System.NullReferenceException: Object reference not set to an instance of an object.
at RecurringIntegrationsScheduler.Common.Helpers.HttpClientHelper.d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at RecurringIntegrationsScheduler.Job.Export.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at RecurringIntegrationsScheduler.Job.Export.d__6.MoveNext()
--- End of inner exception stack trace ---
at RecurringIntegrationsScheduler.Job.Export.d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Quartz.Core.JobRunShell.d__9.MoveNext() [See nested exception: System.NullReferenceException: Object reference not set to an instance of an object.
at RecurringIntegrationsScheduler.Common.Helpers.HttpClientHelper.d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at RecurringIntegrationsScheduler.Job.Export.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at RecurringIntegrationsScheduler.Job.Export.d__6.MoveNext()]

Thanks for your inputs

[TomekMelissa]

Please add section below in RecurringIntegrationsScheduler.exe.config file
after line </configSections>
Then start Standalone scheduler and schedule one export job. Wait for it to fail and shut down RIS.
Find file network.log that should be present in RIS main folder. Send it to me directly [email protected]

<system.diagnostics>  
    <sources>  
      <source name="System.Net" tracemode="includehex" maxdatasize="1024">  
        <listeners>  
          <add name="System.Net"/>  
        </listeners>  
      </source>  
      <source name="System.Net.Cache">  
        <listeners>  
          <add name="System.Net"/>  
        </listeners>  
      </source>  
      <source name="System.Net.Http">  
        <listeners>  
          <add name="System.Net"/>  
        </listeners>  
      </source>  
      <source name="System.Net.Sockets">  
        <listeners>  
          <add name="System.Net"/>  
        </listeners>  
      </source>  
      <source name="System.Net.WebSockets">  
        <listeners>  
          <add name="System.Net"/>  
        </listeners>  
      </source>  
    </sources>  
    <switches>  
      <add name="System.Net" value="Verbose"/>  
      <add name="System.Net.Cache" value="Verbose"/>  
      <add name="System.Net.Http" value="Verbose"/>  
      <add name="System.Net.Sockets" value="Verbose"/>  
      <add name="System.Net.WebSockets" value="Verbose"/>  
    </switches>  
    <sharedListeners>  
      <add name="System.Net"  
        type="System.Diagnostics.TextWriterTraceListener"  
        initializeData="network.log"  
      />  
    </sharedListeners>  
    <trace autoflush="true"/>  
  </system.diagnostics>

[ChristiaanSu]

Ive done a test using download job and not the export job.
When I use the download job I dont get the same error as when im using the export job.

[TomekMelissa]

I am trying to repro that error.
Could you guys tell me what data entities you are using? what is the volume of exported data?
Looks like this happens on GetExecutionSummaryStatus request, but I need to define more precise conditions.

[abhishekbaluni]

Hi Melissa ,
my observations below , will keep you posted if i have more pointers

1. all the entities we are using are custom,

2. the data entities are incremental, so most of the times the files that come out are 1 Kb , meaning no records.

3. the job frequency is about 10-15 minutes

4. i actually thought this occurs due to the status check interval, when it tries to see if the job went through , i tried playing with the values there
   e,g
   status check interval -- if i set this to 500 i,e 5 mins, the job runs for 5 mins no matter what,
   meaning the time from Blocked to Normal is 5 mins
   status check interval -- if i set this 180 , 3 mins m then the job runs for 3 mins
   it seems that this value actually drive how long the job runs, is this correct??

5. i got a similar error , when i scheduled a export job, and put in a data project which no longer existed , not sure if this helps

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Quartz.Core.JobRunShell.d__9.MoveNext() [See nested exception: System.NullReferenceException: Object reference not set to an instance of an object.
at RecurringIntegrationsScheduler.Common.Helpers.HttpClientHelper.d__17.MoveNext()

[TomekMelissa]

Check v.2.4.0 It should fix this

[abhishekbaluni]

Thank you Tomek for the quick response, i will install this version

[ChristiaanSu]

Hi Tomek,

I installed the latest version v.2.4.0 and created export jobs with he below parameters

Recurrence: 10 mins
Status Check Interval: 120 sec
Retry policy 1:
Delay:60
Pause job when error occurs: yes

For some reason the jobs goes into a blocking state but does not release the blocking at all. It worked fine for a few hours where it went from Normal > Blocking > Normal but now its stuck in blocking.
Its been in a blocked state now for 12 hours.

Ive deleted the jobs and created them again and the same happened.

Any help would be appreciated.

[TomekMelissa]

run it with debug mode enabled in config. Then check logs. I guess it is problem with status message again. Seems like D365 does not reliably responds with it. For few previous version the first request with ExportToPackage was actually synchronous, and then first check for status of export always responded with "Succeded". Now it is finally asynchronous operation, but sometimes all you get from D365 is "Bad request" after query about status of export job.
I decided to try 1000 times before I break the loop of status query. So 1000*2 minutes is actually over 33 hours... and all you get is bad request response probably. Need to think how to workaround this, but the main problem is reliability of D365 response...
Debug mode and logs should be able to verify this.

[TomekMelissa]

This is code snippet responsible for this
do
{
attempt++;
if(attempt != 1)
System.Threading.Thread.Sleep(_settings.Interval);
executionStatus = await _httpClientHelper.GetExecutionSummaryStatus(executionId);
if (Log.IsDebugEnabled)
Log.Debug(string.Format(Resources.Job_0_Checking_if_export_is_completed_Try_1_Status_2, _context.JobDetail.Key, attempt, executionStatus));
if (attempt == 1000)
break;
}
while ((executionStatus == "NotRun" || executionStatus == "Executing" || executionStatus == "Bad request"));

You could argue that I should not include "Bad request" in the condition, but I saw cases when this kind of response was only random and the next request returned proper status. In your case, I guess it is permanent. Try to use 5 seconds interval. In worst scenario it will only block your job for ca. 1h 30m.

[TomekMelissa]

Please send me environment IDs of instances where you are facing this issue. Send it by email please.

[abhishekbaluni]

Hi Tomek

any more update on this, we have started hitting this issue more frequently,sometimes it would be an hour before a successful export ,

Regards

[TomekMelissa]

See previous comment.

[abhishekbaluni]

thanks Tomek, totally skipped the previous comment,

[abhishekbaluni]

hi Tomek,

an update on this , we had MS change the flag to sync instead of async, as you had mentioned.
past 24 hours hardly any job failure due to the above issue

there another error, that i havent seen before that popped up
does the following error mean, that the dowload failed

Quartz.Core.JobRunShell - Job Export.TestExport schedule threw a JobExecutionException: Parameters: refire = False, unscheduleFiringTrigger = False, unscheduleAllTriggers = False  Quartz.JobExecutionException: Export job: Export.11 Delivery schedule failed ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at Polly.Policy.<>c__DisplayClass208_01.<<ExecuteAsyncInternal>b__0>d.MoveNext()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at Polly.RetrySyntaxAsync.<>c__DisplayClass21_1.<<WaitAndRetryAsync>b__1>d.MoveNext()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at Polly.Retry.RetryEngine.<ImplementationAsync>d__11.MoveNext()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   at Polly.Retry.RetryEngine.d__11.MoveNext()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at Polly.Policy.<ExecuteAsyncInternal>d__2081.MoveNext()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at RecurringIntegrationsScheduler.Common.Helpers.HttpClientHelper.d__10.MoveNext()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at RecurringIntegrationsScheduler.Common.Helpers.HttpClientHelper.d__23.MoveNext()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at RecurringIntegrationsScheduler.Job.Export.d__7.MoveNext()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at RecurringIntegrationsScheduler.Job.Export.d__6.MoveNext()   --- End of inner exception stack trace ---   at RecurringIntegrationsScheduler.Job.Export.d__6.MoveNext()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at Quartz.Core.JobRunShell.d__9.MoveNext() [See nested exception: System.Threading.Tasks.TaskCanceledException: A task was canceled.   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at Polly.Policy.<>c__DisplayClass208_01.<<ExecuteAsyncInternal>b__0>d.MoveNext()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at Polly.RetrySyntaxAsync.<>c__DisplayClass21_1.<<WaitAndRetryAsync>b__1>d.MoveNext()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at Polly.Retry.RetryEngine.<ImplementationAsync>d__11.MoveNext()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   at Polly.Retry.RetryEngine.d__11.MoveNext()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at Polly.Policy.<ExecuteAsyncInternal>d__2081.MoveNext()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at RecurringIntegrationsScheduler.Common.Helpers.HttpClientHelper.d__10.MoveNext()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at RecurringIntegrationsScheduler.Common.Helpers.HttpClientHelper.d__23.MoveNext()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at RecurringIntegrationsScheduler.Job.Export.d__7.MoveNext()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at RecurringIntegrationsScheduler.Job.Export.d__6.MoveNext()]ReplyForward		ReplyForward
	ReplyForward

[TomekMelissa]

Is this reproducible error? What are the conditions? Looks like an error after few retries handled by Polly, but I cannot figure out what exactly is going on from this alone.

[abhishekbaluni]

it started occuring one of the export job, after MS switched the exporttopackage call to sync,
the project executes okay in D365, and then RIS logs this error exactly an hour later, the job starts at 2 :00 am and this error is logged at 3:00am

[TomekMelissa]

Looks like you are trying to export a lot of data and 1 hour timeout of HttpClient set in RIS is just not enough. You can increase it, but I would suggest to first evaluate why it is taking so long.
Lines 52-55 in HttpClientHelper.cs

_httpClient = new HttpClient(httpClientHandler)
{
    Timeout = TimeSpan.FromMinutes(60) //Timeout for large uploads or downloads
};