serilog-contrib / serilog.sinks.network

A serilog network sink. Designed with logstash and the Elastic stack in mind

Serilog Network Sink

Writes the JSON output from serilog log event to either UDP or TCP

Versions

Serilog Network Sink is targeted NetStandard 1.3 from version 2.x. It can be used for .NET Core based projects.

1.x versions are still targeted to standard .Net Framework 4.5

Usage

Set up to log via TCP

var ip = IPAddress.Parse("1.3.3.7");
var log = new LoggerConfiguration()
    .WriteTo.TCPSink(ip, 1337)
    .CreateLogger();

var urlLogger = new LoggerConfiguration()
    .WriteTo.TCPSink("some.url.com", 1337)
    .CreateLogger();

var urlLogger = new LoggerConfiguration()
    .WriteTo.TCPSink("tls://some.fqdn.com:12435")
    .CreateLogger();

// you can provide any specific formatter ...
var urlLogger = new LoggerConfiguration()
    .WriteTo.TCPSink("tls://some.fqdn.com:12435", new RawFormatter())
    .CreateLogger();

// ... otherwise this will use the default provided LogstashJsonFormatter (described below)
var urlLogger = new LoggerConfiguration()
    .WriteTo.TCPSink("tls://some.fqdn.com:12435")
    .CreateLogger();

Or maybe UDP

var ip = IPAddress.Parse("1.3.3.7");
var log = new LoggerConfiguration()
    .WriteTo.UDPSink(ip, 1337)
    .CreateLogger();

var urlLogger = new LoggerConfiguration()
    .WriteTo.UDPSink("some.url.com", 1337)
    .CreateLogger();

// you can provide any specific formatter for UDP too ...
var urlLogger = new LoggerConfiguration()
    .WriteTo.UDPSink("some.url.com", 1337, new RawFormatter())
    .CreateLogger();

Configure from the config file

<add key="serilog:minimum-level" value="Verbose" />
<add key="serilog:using:TCPSink" value="Serilog.Sinks.Network" />
<add key="serilog:write-to:TCPSink.uri" value="192.165.25.55" />
<add key="serilog:write-to:TCPSink.port" value="3251" />

JSON structure (LogstashJsonFormatter)

Serilog log JSON tends to look like this:

{
  "Timestamp": "2016-11-03T16:28:55.0094294+00:00",
  "Level": "Information",
  "MessageTemplate": "ping: {ping} and pong: {pong}",
  "message": "ping: 972 and pong: 973",
  "Properties": {
    "ping": 972,
    "pong": 973,
    "application": "ping ponger",
    "type": "example",
    "environment": "production"
  }
}

The LogstashJsonFormatter flattens that structure so it is more likely to fit into an existing logstash infrastructure.

{
  "timestamp": "2016-11-03T16:28:55.0094294+00:00",
  "level": "Information",
  "message": "ping: 972 and pong: 973",
  "ping": 972,
  "pong": 973,
  "application": "ping ponger",
  "type": "example",
  "environment": "production",
}

Acknowledgements

Adapted from Serilog Splunk Sink and Splunk .Net Logging both Apache 2.0 licensed

serilog.sinks.network's Issues

LogStashFormatter can produce json output with duplicate properties

Assume the following log format string (sample code in .NET):

log.Information("Received {message} at {timestamp}.", aMessage, aTimestamp);

The SeriLog json looks like this, no problem:

{
  "Timestamp": "2016-11-03T16:28:55.0094294+00:00",
  "Level": "Information",
  "MessageTemplate": "Received {message} at {timestamp}",
  "message": "Received foobar at 2016-10-03T 15:28:55.0094294+00:00",
  "Properties": {
    "message": "foobar",
    "timestamp": "2016-10-03T 15:28:55.0094294+00:00",
  }
}

When going through the LogStashFormatter, the flattened json has duplicate properties:

{
  "timestamp": "2016-11-03T16:28:55.0094294+00:00",
  "level": "Information",
  "message": "Received foobar at 2016-10-03T 15:28:55.0094294+00:00",
  "message": "foobar",
  "timestamp": "2016-10-03T 15:28:55.0094294+00:00",
}

This is invalid json. Depending on the behavior of the json parser used downstream, either the first or the last occurrence of properties with the same name wins, or the parser throws an error.

Of course this can be fixed in the application by forbidding the usage of log format string parameters names "message", "timestamp" and "level". This requires the application to know which logging sink is used downstream. With another logging sink downstream, the constraint does not exist (or would be different). Also, this does not solve the problem. Assume the application uses the log format string parameter "applicationClass", and somewhere downstream the log events are annotated by configuration with a property named "applicationClass" - the problem is the same.

The root of the problem is the fact that the LogStashFormatter flattens all properties from different levels into the top most json namespace. This mixes the LogStash format meta level properties, the format string parameter level, and the property annotation level into a single flat namespace.

LogStashFormatter should handle this in a better way. Knowing this is easily said: the obvious fixes (keeping the separation between the aspects by introducing structure, or adding prefixes/postfixes to the property names) will break existing downstream processing.

Unable to configure TCPSink through App/web.config in case of using IP address

.NET Standard support

Currently it's not possible to reference this sink from .NET Core project. Any plans for .NET Standart support?

how to output message about an object to TCP sink in Json format

Hello,
I am using Serilog TCP sink like this -

//Object
var nitin = new Employee {Name = "Nitin", Ssn = "112"};
 //Serilog Client
                var ip = IPAddress.Parse(ServerIp);
                .WriteTo.TCPSink(ip, PortNo, new JsonFormatter())
                    .CreateLogger();
 log.Information("{@nitin}",nitin);

the output is -
{"host":"10.208.11.71","Timestamp":"2019-12-06T12:04:45.5160298-06:00","MessageTemplate":"{@nitin}","timestamp":"2019-12-06T18:04:45.536092500Z","Properties":{"nitin":{"Ssn":"112","_typeTag":"Employee","Name":"Nitin"}},"Level":"Information"}

which is not in json format I want . How to I change the above message in Json format to -

{"timestamp":"2019-12-06T09:45:33.7410418-06:00","level":"Information","message":{
"nitin":{"Ssn":"112","Name":"Nitin"}
}}

Recommend a good tcp/udp listener for windows

Hi,

Just wondering what tools you would use this serilog extension with in order to see the logs

Cheers

Invalid JSON generated by LogstashJsonFormatter when no properties on LogEvent

The LogstashJsonFormatter always appends a comma in the method WritePropertyAndValue.
This means if there are no items in the logEvent.Properties collection, then you end up with a trailing comma.
E.g.

{"timestamp":"2017-12-01T12:22:20.7748558+00:00","level":"Information","message":"My simple log message",}

Add PackageProjectUrl and RepositoryUrl to Serilog.Sinks.Network NuGet package metadata

The Serilog.Sinks.Network NuGet package currently doesn't display hyperlinks to Project Site and Source repository on nuget.org

  <PropertyGroup>
    <PackageProjectUrl>https://github.com/serilog-contrib/Serilog.Sinks.Network</PackageProjectUrl>
    <RepositoryType>git</RepositoryType>
    <RepositoryUrl>https://github.com/serilog-contrib/Serilog.Sinks.Network.git</RepositoryUrl>
  </PropertyGroup>

Configuring sink using JSON

Hi!

There is a proposal over at Serilog.Settings.Configuration where sink developers would be able to specify an extension point specially designed to support JSON configuration. If this is something you would find beneficial for your sink, please give a thumbs up or something. If not, have a nice day and continue being awesome!

Error: failed inside TCP socket

Hello.

I have problem this lib.

I get error:
"failed inside TCP socket: Unable to read data from the transport connection..."

My solution:
I called another constructor. My error was clear.

Uri scheme must be tcp or tls

According to the documentation configuring the Sink using a hostname and port should be valid:

var urlLogger = new LoggerConfiguration()
    .WriteTo.TCPSink("some.url.com", 1337)
    .CreateLogger();

Howerver, when I attempt this I get the following exception:

An unhandled exception of type 'System.UriFormatException' occurred in Serilog.Sinks.Network.dll: 'Uri scheme must be tcp or tls' at Serilog.Sinks.Network.NetworkLoggerConfigurationExtensions.BuildUri(String s) at Serilog.Sinks.Network.NetworkLoggerConfigurationExtensions.TCPSink(LoggerSinkConfiguration loggerConfiguration, String uri, ITextFormatter textFormatter, LogEventLevel restrictedToMinimumLevel)

Is this an error in the documentation, or a bug?

Error: Thread was being aborted

I experienced log messages lost. The error message I received is "Error: Thread was being aborted. "

Set property casing

Is it possible to add an option that we can choose between camelCasing (current, default) and as-is (leave casing alone).

This can be a boolean, or an enum to be future proof (kebab-case, PascalCase, ...)

TCP Connection aborted and throws IOException

I keep seeing the following exception when my app uses the TCP network sink. On some desktops, the connection works, but throws this intermittent, however on one server where we run the app, this always occurs and we're never able to send the logs.

We are connecting to a Logstash endpoint via TCP.

System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
   at System.Net.Sockets.Socket.BeginSend(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
   at System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   at System.IO.Stream.<>c.<BeginEndWriteAsync>b__53_0(Stream stream, ReadWriteParameters args, AsyncCallback callback, Object state)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncTrim[TInstance,TArgs](TInstance thisRef, TArgs args, Func`5 beginMethod, Func`3 endMethod)
   at System.IO.Stream.BeginEndWriteAsync(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.IO.Stream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count)
   at Serilog.Sinks.Network.Sinks.TCP.TcpSocketWriter.<>c__DisplayClass8_1.<<-ctor>b__1>d.MoveNext()

TcpSink doesn't have a finalizer

We sometimes see in our production that we have a lot of TcpSocketWriters and small number of TcpSinks. VM can go out of sockets, effectively preventing our application from writing logs or make outbound connections. Seems thats somewhere we have a lot of TcpSinks born and die w/o Dispose.
We understand that's probably bug in our code, and we are working on it, but I think that TcpSink will benefit from having finalizer and implementing old-style dispose pattern, disposing TcpSocketWriter on finalization. What do you think? If yes, we can provide a PR.

P.S. @pauldambra I wonder if the project is maintained still or it's recommended to move to other sinks? We are happy to contribute and fix bugs — but only there is project has active mantainer and there are other users :-)

should Serilog be excluded from Nuget package

UDPSink has no async buffer

Hello

I have made several benchmarks for your TCPSink and UDPSink with BenchmarkDotNet
I have tested both single-thread and multi-thread applications to write log messages

My results are the following

TCP (validate single message sending (N = 1)):

Method	Job	Runtime	N	Mean	Error	StdDev	Ratio	RatioSD
SendTcpLogsSync	Clr	Clr	1	4.302 us	0.0851 us	0.0946 us	1.00	0.00
SendTcpLogsSync	Core	Core	1	3.277 us	0.0600 us	0.0561 us	0.76	0.02
SendTcpLogsAsync	Clr	Clr	1	12.202 us	0.2096 us	0.1960 us	1.00	0.00
SendTcpLogsAsync	Core	Core	1	9.027 us	0.1308 us	0.1224 us	0.74	0.02

TCP (validate 100 message sending (N = 100)):

Method	Job	Runtime	N	Mean	Error	StdDev	Ratio	RatioSD
SendTcpLogsSync	Clr	Clr	100	418.3 us	3.261 us	3.050 us	1.00	0.00
SendTcpLogsSync	Core	Core	100	316.5 us	4.051 us	3.789 us	0.76	0.01
SendTcpLogsAsync	Clr	Clr	100	217.7 us	3.368 us	3.150 us	1.00	0.00
SendTcpLogsAsync	Core	Core	100	174.3 us	2.941 us	2.751 us	0.80	0.02

UDP (validate single message sending (N = 1)):

Method	Job	Runtime	N	Mean	Error	StdDev	Ratio
SendUdpLogsSync	Clr	Clr	1	8.864 us	0.0839 us	0.0701 us	1.00
SendUdpLogsSync	Core	Core	1	6.750 us	0.0597 us	0.0529 us	0.76
SendUdpLogsAsync	Clr	Clr	1	20.914 us	0.2737 us	0.2561 us	1.00
SendUdpLogsAsync	Core	Core	1	14.905 us	0.1999 us	0.1870 us	0.71

UDP (validate 100 message sending (N = 100)):

Method	Job	Runtime	N	Mean	Error	StdDev	Ratio	RatioSD
SendUdpLogsSync	Clr	Clr	100	860.4 us	9.958 us	8.827 us	1.00	0.00
SendUdpLogsSync	Core	Core	100	681.4 us	3.936 us	3.489 us	0.79	0.01
SendUdpLogsAsync	Clr	Clr	100	277.4 us	7.458 us	6.977 us	1.00	0.00
SendUdpLogsAsync	Core	Core	100	216.3 us	1.827 us	1.709 us	0.78	0.02

As you can see Mean value is two times bigger for UDPSink.

Source code review showed me that the UDPSink send data to socket as soon as evant was emitted

void Emit(LogEvent logEvent)
{
    Emitted++;
    var sb = new StringBuilder();

    using (var sw = new StringWriter(sb))
        _formatter.Format(logEvent, sw);

    sb.Replace("RenderedMessage", "message");

    _socket.Send(Encoding.UTF8.GetBytes(sb.ToString()));
}

Quesions:

1. Is there any reasons to skip buffering for UDPSink?
2. Is it possible to add a async buffer as well as it done for TCPSink?

Add Serilog Sink icon to Serilog.Sinks.Network NuGet package

The Serilog.Sinks.Network NuGet package doesn't yet have an icon.

I suggest using the standard icon for sinks:

---

More details: https://github.com/serilog-contrib/brand

Run tests against Serilog 2.x

TCPSink tls mode don't allow to pass certificate for SslStream

Hello. Using your lib i found that we can't pass certificates to SslStream constructor. Is there some another way to use certificates or should it be implemented? I need your help

https://github.com/pauldambra/Serilog.Sinks.Network/blob/8fcb0cefae6c5c54536716b7bb254fb0328226ad/Serilog.Sinks.Network/Sinks/TCP/TCPSocketWriter.cs#L84-L86

Azure Appservices are not capable of stream logs

Use case:

Deploy any application that uses Serilog.Sinks.network to Azure Appservices. You will notice that the application is not streaming any log and throw "Error: An address incompatible with the requested protocol was used " exception.

Sink gives surprising output, possible corruption

TCPSink/UDPSink perform a string replacement on opaque data: the formatter output.
Users expecting the same output from the their formatter on this sink will be surprised when the output keys are not as expected. Additionally, if the string RenderedMessage appears anywhere in the output it will be modified, effectively corrupting the output.

Be able to run tests against 2.x _and_ 1.x

Add publick key to the assembly

I'm working with Sharepoint and i need all dll's to have publickeytoken. Serilog has, every sink has except yours, It would be very helpful.
I need it only for 1.* version

TcpSocketWriter maxQueueSize is not configurable

Issue:
TCPSink constructor does not use maxQueueSize of TcpSocketWriter.

public TCPSink(IPAddress ipAddress, int port, ITextFormatter formatter)
{
    _socketWriter = new TcpSocketWriter(new Uri($"tcp://{ipAddress}:{port}"));
    _formatter = formatter;
}

In our application we want to extend a queue size.

Request:

1. Let's add maxQueueSize optional parameter for TCPSink constructor
2. Let's add maxQueueSize to NetworkLoggerConfigurationExtensions extension methods