An established connection was aborted by the software in your host machine about javascript.nodejs

[JeremyTCD]

Thanks for the extra info. My "possible causes" seem unlikely. Anyway I've released 5.2.1 with 2 changes:

• Removed NodeJS connection timeout. No harm doing this since NodeJS is removing it anyway. Might solve the issue if it is caused by requests somehow taking longer than usual.

• Added a listener in NodeJS for client errors. The listener should tell us why NodeJS is dropping connections - look out for the exception printed in log messages:

  An invocation attempt failed. Retries remaining: 1. <exception>
  

---

Incidentally, while running benchmarks for Microsoft.AspNetCore.NodeServices, NodeJS dropped a connection and the same exception, System.Net.Http.HttpRequestException, was thrown:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The response ended prematurely.
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.AspNetCore.NodeServices.HostingModels.HttpNodeInstance.InvokeExportAsync[T](NodeInvocationInfo invocationInfo, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.NodeServices.HostingModels.OutOfProcessNodeInstance.InvokeExportAsync[T](CancellationToken cancellationToken, String moduleName, String exportNameOrNull, Object[] args)
   at Microsoft.AspNetCore.NodeServices.NodeServicesImpl.InvokeExportWithPossibleRetryAsync[T](String moduleName, String exportedFunctionName, Object[] args, Boolean allowRetry, CancellationToken cancellationToken)
   at Jering.Javascript.NodeJS.Performance.RealWorkloadBenchmarks.INodeServices_RealWorkload() in C:\Projects\Jering.Javascript.NodeJS\perf\NodeJS\RealWorkloadBenchmarks.cs:line 102
   at BenchmarkDotNet.Autogenerated.Runnable_1.<.ctor>b__3_2() in C:\Projects\Jering.Javascript.NodeJS\perf\NodeJS\bin\Release\netcoreapp3\e831b542-5daa-4111-833a-3c6d7fdeb24b\e831b542-5daa-4111-833a-3c6d7fdeb24b.notcs:line 829
   at BenchmarkDotNet.Autogenerated.Runnable_1.WorkloadActionUnroll(Int64 invokeCount) in C:\Projects\Jering.Javascript.NodeJS\perf\NodeJS\bin\Release\netcoreapp3\e831b542-5daa-4111-833a-3c6d7fdeb24b\e831b542-5daa-4111-833a-3c6d7fdeb24b.notcs:line 1122
   at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData data)
   at BenchmarkDotNet.Engines.EngineStage.RunIteration(IterationMode mode, IterationStage stage, Int32 index, Int64 invokeCount, Int32 unrollFactor)
   at BenchmarkDotNet.Engines.EngineStage.Run(IStoppingCriteria criteria, Int64 invokeCount, IterationMode mode, IterationStage stage, Int32 unrollFactor)
   at BenchmarkDotNet.Engines.EngineWarmupStage.Run(Int64 invokeCount, IterationMode iterationMode, Int32 unrollFactor, RunStrategy runStrategy)
   at BenchmarkDotNet.Engines.EngineWarmupStage.RunWorkload(Int64 invokeCount, Int32 unrollFactor, RunStrategy runStrategy)
   at BenchmarkDotNet.Engines.Engine.Run()
   at BenchmarkDotNet.Autogenerated.Runnable_1.Run(IHost host, String benchmarkName) in C:\Projects\Jering.Javascript.NodeJS\perf\NodeJS\bin\Release\netcoreapp3\e831b542-5daa-4111-833a-3c6d7fdeb24b\e831b542-5daa-4111-833a-3c6d7fdeb24b.notcs:line 812
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in C:\Projects\Jering.Javascript.NodeJS\perf\NodeJS\bin\Release\netcoreapp3\e831b542-5daa-4111-833a-3c6d7fdeb24b\e831b542-5daa-4111-833a-3c6d7fdeb24b.notcs:line 49

Heads up for the guys cloning INodeServices @johnnyoshika.

[DaniilSokolyuk]

Problem not in Content-Length, node.js always return Content-Length, when response.end used (idk why)

[JeremyTCD]

Note: I deleted a couple of comments because my points weren't concise.

Clarification and Dirty Fix

Your trace is from a log message. It is logged before retrying an invocation request:

Javascript.NodeJS/src/NodeJS/NodeJSServiceImplementations/OutOfProcess/OutOfProcessNodeJSService.cs

Line 296 in 2cd5d66

Logger.LogWarning(string.Format(Strings.LogWarning_InvocationAttemptFailed, numRetries < 0 ? "infinity" : numRetries.ToString(), exception.ToString()));

From the log message, we're retrying because a connection was aborted by NodeJS while the .Net process was sending data to it:

System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.

Does the retry succeed? If it fails, an InvocationException is thrown by Jering.Javascript.NodeJS.

Dirty fix: I'd suggest increasing the number of retries for now (OutOfProcessNodeJSServiceOptions.NumRetries) to be safe.

Possible Causes

1. By default (pre-v13), NodeJS connections time out after 120 seconds (nodejs/node#27556). What is your workload like? Is it possible a request has to wait that long?
2. A malformed request can cause a connection to drop (nodejs/node#4543). This is not the issue if the request works on retry.
3. You're hitting some concurrent connection limit - Node.js maxing out at 1000 concurrent connections. Please provide more information about your environment - OS, Node.js version and .Net target framework.

Possible Fix (#61)

1. Manually remove connection timeout.
2. Add client error listener so if the issue persists, we might get more information about why the connection is dropping.

[DaniilSokolyuk]

1. Average request time is about ~10-30ms
2. Hosted on windows, node.js process have 2-3 opened connections
3. Retry count is 1. Thats works perfect

Error occurs 1-2 times per hour.

[aKzenT]

I'm also getting the "The response ended prematurely." error frequently. Since the errors are also triggered from a web test that always sends the same data to node, I know that it cannot be from a malformed request. Any ideas?

[JeremyTCD]

Hey @aKzenT, could you could provide the full stack trace? Let me know if you have any problems capturing it.

[aKzenT]

System.Net.Http.HttpRequestException:An error occurred while sending the request. The response ended prematurely.

   at System.Net.Http.HttpConnection+<SendAsyncCore>d__53.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

   at System.Net.Http.HttpConnectionPool+<SendWithNtConnectionAuthAsync>d__48.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

   at System.Net.Http.HttpConnectionPool+<SendWithRetryAsync>d__47.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

   at System.Net.Http.RedirectHandler+<SendAsync>d__4.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

   at System.Net.Http.DiagnosticsHandler+<SendAsync>d__2.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)

Inner exception System.IO.IOException handled at System.Net.Http.HttpConnection+<SendAsyncCore>d__53.MoveNext:

   at System.Net.Http.HttpConnection+<SendAsyncCore>d__53.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)

[JeremyTCD]

Thanks for the extra details. We'll need a bit more information, System.Net.Http.HttpRequestException is quite a high level exception. The following inner exception's message would help:

Inner exception System.IO.IOException handled at System.Net.Http.HttpConnection+d__53.MoveNext:

Are you able to add logging to your project? Check out the first comment, it includes this library's log output, which lists retry count and the hierarchy of inner exceptions and their log messages.

Also is it possible for you to reproduce the issue in a public repository?

[aKzenT]

This is the log output from the library:

An invocation attempt failed. Retries remaining: 1.
Exception:
  System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: The response ended prematurely.
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Jering.Javascript.NodeJS.HttpNodeJSService.TryInvokeAsync[T](InvocationRequest invocationRequest, CancellationToken cancellationToken)
   at Jering.Javascript.NodeJS.OutOfProcessNodeJSService.TryInvokeCoreAsync[T](InvocationRequest invocationRequest, CancellationToken cancellationToken)

I don't think I can reproduce this in a pulic repository. It is worth noticing that this affects about on in a thousand calls.

[JeremyTCD]

Did some reading up on the .Net side. Found an identical issue: dotnet/runtime#31528. Also took a look at the code throwing the IOException:
https://github.com/dotnet/runtime/blob/c06810bf654e09f016fe34dd2b4ac31fb73370a1/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs#L477-L490.

Noticed that HttpConnection uses NetEventSource to log, e.g. https://github.com/dotnet/runtime/blob/c06810bf654e09f016fe34dd2b4ac31fb73370a1/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs#L160. If you could enable event source tracing, it might give us a clue. I haven't tried tracing HttpConnection, but from this article it seems straightforward: https://developers.redhat.com/blog/2019/12/23/tracing-net-core-applications/.

Short-term, I suggest increasing retries to 3: OutOfProcessNodeJSServiceOptions.NumRetries.

[blushingpenguin]

This call stack seems to be very similar to what you get if the node process dies for some reason, e.g.

System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Jering.Javascript.NodeJS.HttpNodeJSService.TryInvokeAsync[T](InvocationRequest invocationRequest, CancellationToken cancellationToken)
   at Jering.Javascript.NodeJS.OutOfProcessNodeJSService.TryInvokeCoreAsync[T](InvocationRequest invocationRequest, CancellationToken cancellationToken)
   at Jering.Javascript.NodeJS.OutOfProcessNodeJSService.InvokeFromStringAsync[T](String moduleString, String newCacheIdentifier, String exportName, Object[] args, CancellationToken cancellationToken)

which can be easily reproduced by running the node process out of ram using a combination of:

        let bigThing = {}
        for (let i = 0; i < 1024000; ++i) {
            bigThing[`prop${i}1`] = "a".repeat(1024 * 1024);
        }

and

            services.Configure<NodeJSProcessOptions>(
                options => options.NodeAndV8Options = "--max-old-space-size=512"); // 512Mb

When this happens node produces detailed diagnostic messages, but these are lost making it difficult to know what actually went wrong. I think collating the messages from node's stderr/stdout would likely help diagnose the issues above.

[JeremyTCD]

6.0.0-beta.2 now uses HTTP/2 which should mitigate connection issues.

It also exposes a new option, OutOfProcessNodeJSServiceOptions.NumProcessRetries that retries in a new process if retries in the original process fail:

Javascript.NodeJS/src/NodeJS/NodeJSServiceImplementations/OutOfProcess/OutOfProcessNodeJSServiceOptions.cs

Lines 27 to 39 in b96c2f9

	/// <summary>
	/// <para>The number of times we create a new NodeJS process to retry an invocation.</para>
	/// <para>Invocations are retried <see cref="NumRetries"/> times in the existing NodeJS process. Once <b>existing process retries</b> are exhausted,
	/// if any <b>process retries</b> remain, a new NodeJS process is created and invocations are tried <see cref="NumRetries"/> times in the new process.</para>
	/// <para>For example, consider the situation where <see cref="NumRetries"/> and this value are both 1. An invocation is first tried in the existing process.
	/// If it fails it is retried in the same process. If it fails again, a new process is created and the invocation is retried there once. In total, the
	/// invocation is attempted 3 times.</para>
	/// <para>If set to a negative value, new NodeJS processes are recreated for retries indefinitely.</para>
	/// <para>If the module source of an invocation is an unseekable stream, the invocation is not retried.
	/// If you require retries for such streams, copy their contents to a <see cref="MemoryStream"/>.</para>
	/// <para>Defaults to 1.</para>
	/// </summary>
	public int NumProcessRetries { get; set; } = 1;