Followup to Authorization Header question about serilog-sinks-http

[qodex2]

The URL and authCode look kinda like this:

		string myURL = "https://myserver.mycompany.com/v1/request/gwa.diag";
		string authCode = "XXXXXXXXXXXXXXXXXXXXfWlGUtYXXXXXXXXHj3l6cKo=";

[FantasticFiasco]

If I replace DurableHttp with Console (and an output template), it works as expected. But when I use DurableHttp, I get nothing.

The Console sink will print the log events in console, but the HTTP sink wont. Its only purpose is to transport the log events over the network to the receiving log server. If you look at the log server, does it not show the log events posted by the HTTP sink?

If it doesn't, please configure the Serilog SelfLog and tell me if anything is printed there.

[qodex2]

Sorry I wasn't clear. When I said "I get nothing", I didn't mean "at the console." I meant that the server at URL myURL does not receive any data from Serilog. Nothing was POSTed. I assume that at some point the PostAsync method in AuthorizedHttpClient would be called. It never is. Also, I note that a simple WebRequest to myURL along with authCode works fine; it's not a server or firewall or any other problem like that.

I already have SelfLog enabled as follows:

	protected void ConfigureSerilogErrorLogging()
	{
		string path = GetStandardLogFolder();
		string filename = Path.Combine(path, "GWAErrorLog.txt");
		var file = File.CreateText(filename);
		Serilog.Debugging.SelfLog.Enable(TextWriter.Synchronized(file));
	}

Nothing is being written to GWAErrorLog.txt

[FantasticFiasco]

Please try the sample application serilog-sinks-http-sample-dotnet-core, but change ./sample/appsettings.json to use the sink DurableHttp instead of Http, since that is the sink you wish to use.

Start the sample by running docker-compose up --build in the root folder.

Does the sample work as expected on your machine?

[FantasticFiasco]

Could you also please validate that your application has the adequate permissions to write a file in the folder the application is residing? By default the buffer file is named Buffer-{Date}.json. Does there exist such a file on disk?

[qodex2]

That's interesting. The file Buffer-20190402.json IS present in the same folder as the .exe. The app has write permission in this test environment, but it will NOT once deployed. The file has all the data that wasn't posted to my server. So, a side question: I tried the bufferpathFormat argument, but it won't let me specify a path, or at least I didn't do it right. How can I specify a path for the event buffer? I thought durability would be nice, but it's not a requirement for me, so I can use the Http sink if I have to.

I'll try the sample app.

[qodex2]

I don't have docker installed, so I ran them the old-fashioned way. Looks like it's working.

[qodex2]

BTW, nothing was appearing in the log-server if that was expected.

[qodex2]

While I was at it, I changed CustomHttpClient to add my Authorization header, and changed requestUri in appsettings.json to use my server. Still nothing appearing there. The event buffer file did have all the events that weren't showing up on my server.

[qodex2]

I don't know if this will help. To verify my url and authcode, I used this method, which worked perfectly. Data appeared on my server within a couple seconds.

	private static void WebRequestTest(string url, string authCode, string data)
	{
		WebRequest webRequest = WebRequest.Create(url);
		webRequest.Method = "POST";
		webRequest.ContentType = "application/json";
		webRequest.Headers[HttpRequestHeader.Authorization] = authCode;

		using (StreamWriter reqStream = new StreamWriter(webRequest.GetRequestStream()))
		{
			reqStream.Write(data);
			reqStream.Flush();
			reqStream.Close();
		}

		using (var response = (HttpWebResponse)webRequest.GetResponse())
		{
			using (var responseString = new StreamReader(response.GetResponseStream()))
			{
				Console.WriteLine(responseString.ReadToEnd());
				responseString.Close();
			}
			response.Close();
		}

	}

[qodex2]

Maybe I should try a test with an HttpClient....

[qodex2]

Maybe I should try a test with an HttpClient....

Which fails. It's returning 403. Where a WebRequest POST succeeds, an HttpClient POST fails. So, this probably has nothing to do with the Http sink. I think I may have wasted your time, sorry!

[FantasticFiasco]

Back from a nights sleep.

Thank you for your findings. Ok, so it seems that there are differences regarding WebRequest and HttpClient, interesting...

If you wouldn't mind, when you find the problem, could you share it with me? I currently don't know why there would be differences between the two clients either.

I'll investigate why the HTTP error isn't written to the Serilog SelfLog, it ought to be.

[qodex2]

I used Fiddler to find the difference in the POST headers. This is it:

Works: Authorization: Yxm5FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXj3l9cKo=

Fails: Authorization: Basic Yxm5FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXj3l9cKo=

Looks like an issue on the server side. Thanks for all your help!

[qodex2]

There's still a problem. Can you think of any reason why PostAsync would never be called?

[qodex2]

Another question: in appsettings, what is "Sample" at the end of httpClient? A namespace, I guess, but why is it needed?

{
"Serilog": {
"WriteTo": [
{
"Name": "DurableHttp",
"Args": {
"requestUri": "https://mq-gateway.grantstreet.com/v1/request/gwa.diag",
"httpClient": "Sample.Sink.CustomHttpClient, Sample"
}
},
{
"Name": "Console"
}
]
}
}

[FantasticFiasco]

Basic authentication dictates that Basic should be a part of the header value, i.e. the request with the header that fails seems to be the most correct one.

I've modified the sample, created a branch called feat/basic-authentication, and have implemented Basic Authentication on the server receiving the log events. Please try it out and validate that it works.

Then make any of the following changes in the class Sample.Program and restart the application. With any of these changes the error will be sent to Serilog SelfLog:

• Change the log server address to http://invalid-log-server:5000/log-events
• Change the password to wrong-password

So, I think the sink is logging to SelfLog as it is expected to, and cannot explain your issues. I think you will have to show me a complete code sample that behaves as you've described.

[FantasticFiasco]

Another question: in appsettings, what is "Sample" at the end of httpClient?

Sample.Sink.CustomHttpClient, Sample is the full name of the class CustomHttpClient, residing in namespace Sample.Sink in assembly Sample.

[FantasticFiasco]

You can also implement IHttpClient using a WebRequest since you've got that working.

[qodex2]

I think I found the problem. My test app creates a logger, logs some events, and exits. If I add a Console.ReadLine() at the bottom, events start appearing in my server. I guess Serilog was in a thread queuing up events, but never got a chance to send them. There doesn't appear to be anything like logger.Flush().

Wasn't expecting that. But I think my problem is solved. Thanks again for all your help!

[FantasticFiasco]

Ah, that would explain it. The ordinary, as well as the durable, sink is batching the events, and after a certain time it is triggered to send the batch over the network. This due to performance, it wouldn't be feasible to send each event over the network.

Regarding flushing the events, Serilog is supporting it. Either call Log.CloseAndFlush() if you configure the Singleton logger, or Dispose the logger if you are saving it as a variable in your code. You can try it and see if it works. I don't remember if I prevent the logger from disposing before all cached events are transmitted, or if I just gracefully dispose my own resources, thus not flushing all logs over the network. Flushing all log events over the network could potentially never finish.

Are you satisfied enough to close the issue?

[qodex2]

I disposed the logger, that seemed to work as well as pausing the program. All queued events were delivered.

Yes, I'm satisfied, please close the issue.