ais-dotnet / ais.net.receiver

A simple .NET AIS Receiver for capturing the Norwegian Coastal Administration AIS network data (available under Norwegian license for public data (NLOD)) and persisting in Microsoft Azure Blob Storage. Sponsored by endjin.

AIS.NET Projects

Package	Status
Ais.Net
Ais.Net.Models
Ais.Net.Receiver
Ais.Net.Receiver.Storage.Azure.Blob
IP Maturity Model Score
Build Status

The AIS.NET project contains a series of layers, from a low-level high performance NMEA AIS sentence decoder, to a rich high-level C# 9.0 models of AIS message types, a receiver component that can listen to TCP streams of NMEA sentences and expose them as an IObservable<string> of raw sentences or an decoded IObservable<IAisMessage>, and finally a Storage Client implementation to persisting the raw NMEA sentence stream to Azure Blob storage for future processing.

Ais.Net.Receiver

A simple .NET 5.0 AIS Receiver for capturing the Norwegian Coastal Administration's marine Automatic Identification System (AIS) AIVDM/AIVDO NMEA message network data (available under Norwegian license for public data (NLOD)) and persisting in Microsoft Azure Blob Storage.

The Norwegian Costal Administration provide a TCP endpoint (153.44.253.27:5631) for broadcasting their raw AIS AIVDM/AIVDO sentences, captured by over 50 base stations, and covers the area 40-60 nautical miles from the Norwegian coastline.

This project contains a NmeaReceiver which consumes the raw NetworkStream, a NmeaToAisMessageTypeProcessor, which can decode the raw sentences into IAisMessage, and ReceiverHost which manages the process and exposes an IObservable<string> for raw sentences and an IObservable<IAisMessage> for decoded messages. ReceiverHost can be hosted in a console application or other runtime environments like .NET Interactive.

The project also includes a demo console which shows how the various pieces can fit together, including subscribing to the IObservable<string> and IObservable<IAisMessage> streams and displaying the results or batch the AIVDM/AIVDO sentences and write them to Azure Blob Storage using the Append Blob feature, to create timestamped hour-long rolling logs.

The purpose of this application is to provide sample data for Ais.Net - the .NET Standard, high performance, zero allocation AIS decoder. The majority of raw AIS data is only available via commercial sources, and thus creating AIS datasets large enough to test / benchmark Ais.Net is almost impossible.

The Norwegian Costal Administration TCP endpoint produces:

• ~2.9 KB per second
• ~10.3 MB per hour
• ~248 MB per day
• ~1.7 GB per week
• ~7 GB per month
• ~81.4 GB per year

Ais.Net.Models

Ais.Net.Receiver bridges the gap between the high performance, zero allocation world of Ais.Net and the real world need for types to perform meaningful operations. Ais.Net.Models provides a series of C# 9.0 records which define the the message types, a series of interfaces that define common behaviours, and extension methods to help with type conversions & calculations.

The table below shows the messages, their properties and how they are mapped to interfaces.

Show AIS Message Types and .NET Interfaces

	Message Type 1 to 3	Message Type 5	Message Type 18	Message Type 19	Message Type 24 Part 0	Message Type 24 Part 1
IAisMessageType5		AisVersion
ICallSign		CallSign				CallSign
IAisMessageType18			CanAcceptMessage22ChannelAssignment
IAisMessageType18			CanSwitchBands
IVesselNavigation	CourseOverGround10thDegrees		CourseOverGround10thDegrees	CourseOverGround10thDegrees
IAisMessageType5		Destination
IAisMessageType18			CsUnit
IVesselDimensions		DimensionToBow		DimensionToBow		DimensionToBow
IVesselDimensions		DimensionToPort		DimensionToPort		DimensionToPort
IVesselDimensions		DimensionToStarboard		DimensionToStarboard		DimensionToStarboard
IVesselDimensions		DimensionToStern		DimensionToStern		DimensionToStern
IAisMessageType5		Draught10thMetres
IAisMessageType5		EtaMonth
IAisMessageType5		EtaDay
IAisMessageType5		EtaHour
IAisMessageType5		EtaMinute
IAisMessageType18			HasDisplay
IIsAssigned			IsAssigned	IsAssigned
IAisMessageType18			IsDscAttached
IAisMessageType5		ImoNumber
IAisIsDteNotReady		IsDteNotReady		IsDteNotReady
IVesselNavigation	Latitude10000thMins		Latitude10000thMins	Latitude10000thMins
IVesselNavigation	Longitude10000thMins		Longitude10000thMins	Longitude10000thMins
IAisMessageType1to3	ManoeuvreIndicator
IAisMessageType24Part1						MothershipMmsi
IAisMessageType	MessageType	MessageType	MessageType	MessageType	MessageType	MessageType
IVesselIdentity	Mmsi	Mmsi	Mmsi	Mmsi	Mmsi	Mmsi
IAisMessageType1to3	NavigationStatus
IAisMultipartMessage					PartNumber	PartNumber
IVesselNavigation	PositionAccuracy		PositionAccuracy	PositionAccuracy
IAisPositionFixType		PositionFixType		PositionFixType
IAisMessageType18			RadioStatusType
IAisMessageType1to3	RadioSlotTimeout
IAisMessageType1to3	RadioSubMessage
IAisMessageType1to3	RadioSyncState
IAisMessageType19				RegionalReserved139
IAisMessageType19				RegionalReserved38
IRaimFlag	RaimFlag		RaimFlag	RaimFlag
IAisMessageType18			RegionalReserved139
IAisMessageType18			RegionalReserved38
IAisMessageType1to3	RateOfTurn
IRepeatIndicator	RepeatIndicator	RepeatIndicator	RepeatIndicator	RepeatIndicator	RepeatIndicator	RepeatIndicator
IAisMessageType24Part1						SerialNumber
IAisMessageType19				ShipName
IShipType		ShipType		ShipType		ShipType
IAisMessageType1to3	SpareBits145
IAisMessageType24Part0					Spare160
IAisMessageType24Part1						Spare162
IAisMessageType5		Spare423
IAisMessageType19				Spare308
IVesselNavigation	SpeedOverGroundTenths		SpeedOverGroundTenths	SpeedOverGroundTenths
IVesselNavigation	TimeStampSecond		TimeStampSecond	TimeStampSecond
IVesselNavigation	TrueHeadingDegrees		TrueHeadingDegrees	TrueHeadingDegrees
IAisMessageType24Part1						UnitModelCode
IAisMessageType24Part1						VendorIdRev3
IAisMessageType24Part1						VendorIdRev4
IVesselName		VesselName

The C# record types then implement the relevant interfaces, which enables simpler higher level programming constructs, such as Rx queries over an IAisMessage stream:

IObservable<IGroupedObservable<uint, IAisMessage>> byVessel = receiverHost.Messages.GroupBy(m => m.Mmsi);

IObservable<(uint mmsi, IVesselNavigation navigation, IVesselName name)>? vesselNavigationWithNameStream =
    from perVesselMessages in byVessel
    let vesselNavigationUpdates = perVesselMessages.OfType<IVesselNavigation>()
    let vesselNames = perVesselMessages.OfType<IVesselName>()
    let vesselLocationsWithNames = vesselNavigationUpdates.CombineLatest(vesselNames, (navigation, name) => (navigation, name))
    from vesselLocationAndName in vesselLocationsWithNames
    select (mmsi: perVesselMessages.Key, vesselLocationAndName.navigation, vesselLocationAndName.name);

Azure Blob Storage Taxonomy

The AIS data is stored using the following taxonomy

<USER DEFINED CONTAINER NAME>/raw/yyyy/MM/dd/yyyyMMddTHH.nm4

An example directory listing, with a user defined container name of nmea-ais would look as follows:

\---nmea-ais
    \---raw
        \---2021
            \---07
                +---12
                | 20210712T00.nm4 |
                | 20210712T01.mm4 |
                | 20210712T02.nm4 |
                | 20210712T03.nm4 |
                | 20210712T04.nm4 |
                | 20210712T05.nm4 |
                | 20210712T06.nm4 |
                | 20210712T07.nm4 |
                | 20210712T08.nm4 |
                | 20210712T09.nm4 |
                | 20210712T10.nm4 |
                | 20210712T11.nm4 |
                | 20210712T12.nm4 |
                | 20210712T13.nm4 |
                | 20210712T14.nm4 |
                | 20210712T15.nm4 |
                | 20210712T16.nm4 |
                | 20210712T17.nm4 |
                | 20210712T18.nm4 |
                | 20210712T19.nm4 |
                | 20210712T20.nm4 |
                | 20210712T21.nm4 |
                | 20210712T22.nm4 |
                | 20210712T23.nm4 |
                +---20210713
                | 20210713T00.nm4 |

To Run

Update the values in the settings.json file:

{
  "Ais": {
    "host": "153.44.253.27",
    "port": "5631",
    "retryAttempts": 100,
    "retryPeriodicity": "00:00:00:00.500"
  },
  "Storage": {
    "connectionString": "<YOUR AZURE STORAGE CONNECTION STRING>",
    "containerName": "nmea-ais",
    "writeBatchSize": 500
  }
}

From the command line: dotnet Ais.Net.Receiver.Host.Console.exe

Raspberry Pi

As the AIS.NET stack is written in .NET 6.0 and .NET Standard you can publish the Ais.Net.Recevier.Host.Console application with a target runtime of Portable. This will allow you to run the recevier on a Raspberry Pi if you want to capture your own AIS data.

For reliability you can run Ais.Net.Recevier.Host.Console.dll as daemon.

Installation

The combination of Windows Terminal, .NET and PowerShell make a Raspberry Pi a very productive environment for .NET Devs.

Install Windows Terminal. You can download Windows Terminal from the Microsoft Store or from the GitHub releases page.

Open Windows Terminal and use ssh pi@<Raspberry PI IP Address> to connect to your Pi.

Install .NET 6.0

Use the following commands to install .NET 6.0 on your Pi.

1. curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin --channel Current
2. echo 'export DOTNET_ROOT=$HOME/.dotnet' >> ~/.bashrc
3. echo 'export PATH=$PATH:$HOME/.dotnet' >> ~/.bashrc
4. source ~/.bashrc
5. dotnet --version

Install PowerShell 7.x

Use the following commands to install PowerShell on your Pi.

1. Download the latest package wget https://github.com/PowerShell/PowerShell/releases/download/v7.2.7/powershell-7.2.7-linux-arm32.tar.gz
2. Create a directory for it to be unpacked into mkdir ~/powershell
3. Unpack tar -xvf ./powershell-7.2.7-linux-arm32.tar.gz -C ~/powershell
4. Give it executable rights sudo chmod +x /opt/microsoft/powershell/7/pwsh
5. Create a symbolic link sudo ln -s /opt/microsoft/powershell/7/pwsh /usr/bin/pwsh

use the command pwsh to enter the PowerShell session.

Install Ais.Net.Receiver.Host.Console

2. From the solution root, open a command prompt and type dotnet publish -c Release .\Solutions\Ais.Net.Receiver.sln
3. Add your Azure Blob Storage Account connection string to settings.json
4. Transfer (I use Beyond Compare as it has native SSH support) the contents of .\Solutions\Ais.Net.Receiver.Host.Console\bin\Release\net5.0\publish to a folder called aisr in the home/pi directory on your Raspberry Pi (assuming you still have the default set up.)
5. Copy Solutions\Ais.Net.Receiver.Host.Console.RaspberryPi\aisr.service to /lib/systemd/system/aisr.service
6. run sudo chmod 644 /lib/systemd/system/aisr.service
7. run sudo systemctl enable aisr.service
8. run sudo reboot

You can use journalctl -u "aisr" to view the console output of Ais.Net.Recevier.Host.Console.dll

You can use sudo systemctl restart aisr to restart the service.

If you need to look at / edit the deployed aisr.service use sudo nano /lib/systemd/system/aisr.service make your edits then use Ctrl+O and Ctrl+X to save the file and exit.

Use Azure Storage Explorer to browse to where files are captured.

Configuration

Configuration is read from settings.json and can also be overridden for local development by using a settings.local.json file.

{
  "Ais": {
    "host": "153.44.253.27",
    "port": "5631",
    "loggerVerbosity": "Minimal", 
    "statisticsPeriodicity": "00:01:00",
    "retryAttempts": 100,
    "retryPeriodicity": "00:00:00:00.500"
  },
  "Storage": {
    "enableCapture": true,
    "connectionString": "DefaultEndpointsProtocol=https;AccountName=<ACCOUNT_NAME>;AccountKey=<ACCOUNT_KEY>",
    "containerName": "nmea-ais-dev",
    "writeBatchSize": 500
  }
}

AIS

These settings control the ReceiverHost and its behaviour.

• host: IP Address or FQDN of the AIS Source
• port: Port number for the AIS Source
• loggerVerbosity: Controls the output to the console.
  • Quiet = Essential only,
  • Minimal = Statistics only. Sample rate of statistics controlled by statisticsPeriodicity,
  • Normal = Vessel Names and Positions,
  • Detailed = NMEA Sentences,
  • Diagnostic = Messages and Errors
• statisticsPeriodicity: TimeSpan defining the sample rate of statistics to display
• retryAttempts: Number of retry attempts when a connection error occurs
• retryPeriodicity: How long to wait before a retry attempt.

Storage

These settings control the capturing NMEA sentences to Azure Blob Storage.

• enableCapture: Whether you want to capture the NMEA sentences and write them to Azure Blob Storage
• connectionString: Azure Storage Account Connection String
• containerName: Name of the container to capture the NMEA sentences. You can use this to separate a local dev storage container from your production storage container, within the same storage account.
• writeBatchSize: How many NMEA sentences to batch before writing to Azure Blob Storage.

Licenses

AIS.Net.Receiver is also available under the Apache 2.0 open source license.

The Data ingested by the AIS.Net.Receiver is licensed under the Norwegian license for public data (NLOD).

Project Sponsor

This project is sponsored by endjin, a UK based Microsoft Gold Partner for Cloud Platform, Data Platform, Data Analytics, DevOps, Power BI Partner, and .NET Foundation Corporate Sponsor.

For more information about our products and services, or for commercial support of this project, please contact us.

We produce two free weekly newsletters; Azure Weekly for all things about the Microsoft Azure Platform, and Power BI Weekly.

Keep up with everything that's going on at endjin via our blog, follow us on Twitter, or LinkedIn.

Our other Open Source projects can be found on GitHub

Code of conduct

This project has adopted a code of conduct adapted from the Contributor Covenant to clarify expected behavior in our community. This code of conduct has been adopted by many other projects. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

IP Maturity Model (IMM)

ais.net.receiver's Issues

Handle Azure Storage Authentication Exception

When the AIS Receiver was running for over 7 days a fatal exception occurred while attempting to write a batch of messages to blob storage:

Unhandled Exception: Microsoft.Azure.Storage.StorageException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
at Microsoft.Azure.Storage.Core.Executor.Executor.ExecuteAsync[T](RESTCommand 1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
at Microsoft.Azure.Storage.Core.Executor.Executor.<>cDisplayClass0_0 1.<ExecuteSync>b0()
at Microsoft.Azure.Storage.Core.Uti1.CommonUti1ity.RunWithoutSynchronizationContext[T](Func'1 actionToRun) at Microsoft.Azure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand 1 cmd, IRetryPolicy policy, OperationCon text operationContext)
at Endjin.Ais.Receiver.Program.GetAppendBlob() in C:\_Proj ects\OSS\Endjin.Ais.Receiver\Endjin.Ais.Receiver\Program.cs :line 98
at Endiin.Ais.Receiyer.Program.OnMessageReceived(lList 1 messages) in C:\Endjin.Ais.Receiver\Program.cs:line 83
at System.Reactive.AnonymousSafeObserver 1.OnNext(T value) in D:\a\l\s\Rx.NET\Source\src\System.Reactive\AnonymousSafeObserver.csrline 44
at System.Reactive.Subjects.Subject 1.OnNext(T value) in D:\a\l\s\Rx.NET\Source\src\System.Reactive\Subjects\Subject.cs:line 148
at Endjin.Ais.Receiver.NmeaReceiver.RecieveAsync() in C:\Endjin.Ais.Receiver\NmeaReceiver.csrline 59
at Endjin.Ais.Receiver.Program.LongRunningOperationAsync() in C:\Endjin.Ais.Receiver\Program.cs:1ine 74
at Endjin.Ais.Receiyer.Program.LonqRunningOperationWithCancel1ationTokenAsync(Cancel1ationToken cancel1ationToken) in C:\Endjin.Ais.Receiver\Program.cs:1ine 61
at Enajin.Ais.Receiver.Program.Main(String[] args) in C:\Endjin.Ais.Receiver\Progra m.cs:line 33
at Endjin.Ais.Receiver.Program.<Main>(String[] args)

NmeaPayloadParser.PeekMessageType returns a signed int

Message type values are always in the range 0-47 inclusive, because there are only 6 bits available to hold the message type, and by definition they are non-negative.

While all of the various message parsers reflect this by defining a property of the form public uint MessageType => ..., the NmeaPayloadParser.PeekMessageType method inexplicably returns an int.

I can't think of any good reason why I would have done that, so I believe this was a simple mistake.

Since we're still in v0.x versioning, it's not technically a semver violation to fix this.

Interfaces should inherit from IAisMessage

While there are many different AIS message types, the features in IAisMessage are available in all message types. Although this fact is reflected in the hierarchy of concrete message types (all the various record types representing specific message types such as AisMessageType1Through3 and AisMessageType5 derive from AisMessageBase which implements IAisMessage) it is not reflected in the interfaces themselves.

This leads to the following slightly annoying situation. Suppose I have some source of IAisMessages:

IObservable<IAisMessage> messages = GetAisMessageSource()

and suppose I filter this down to messages that provide navigation information:

IObservable<IVesselNavigation> navigationMessages =
    messages.OfType<IVesselNavigation>();

the resulting navigationMessages makes it awkward to get hold of universal AIS features such as the MMSI.

navigationMessages .Subscribe(m => Console.WriteLine(
    $"Ship {((IAisMessage)m).Mmsi} status is {m.NavigationStatus}");

This code has had to cast the message to IAisMessage in order to use its Mmsi property.

The MMSI is a universal feature of AIS messages. (That's why it's safe for that code to cast all messages to IAisMessage.) This should be reflected in the various interface definitions. We should never have to cast back to an IAisMessage—the availability of those features should be reflected by the interface type hierarchy.

(Other fixed relationships like this should also be represented. For example, we define IAisMessageType1to3. Messages of this kind will invariably offer IRaimFlag, IRepeatIndicator, and IVesselNavigation as well as IAisMessage. This is not currently evident from the IAisMessageType1to3 definition, but it should be.)

Exception: Index was out of range.

Must be non-negative and less than the size of the collection. Arg_ParamName_Name

   at System.ThrowHelper.ThrowArgumentOutOfRange_IndexException()
   at System.Collections.Generic.List`1.get_Item(Int32 index)
   at Ais.Net.Receiver.Host.Console.ReceiverHostExtensions.<>c.<GetStreamStatistics>b__0_1(IList`1 window) in \Ais.Net.Receiver\Solutions\Ais.Net.Receiver.Host.Console\Ais\Net\Receiver\Host\Console\ReceiverHostExtensions.cs:line 34
   at System.Reactive.Linq.ObservableImpl.Select`2.Selector._.OnNext(TSource value) in /_/Rx.NET/Source/src/System.Reactive/Linq/Observable/Select.cs:line 39
--- End of stack trace from previous location ---
   at System.Reactive.AnonymousSafeObserver`1.OnError(Exception error) in /_/Rx.NET/Source/src/System.Reactive/AnonymousSafeObserver.cs:line 62
   at System.Reactive.Sink`2.OnError(Exception error) in /_/Rx.NET/Source/src/System.Reactive/Internal/Sink.cs:line 94
   at System.Reactive.Sink`1.ForwardOnError(Exception error) in /_/Rx.NET/Source/src/System.Reactive/Internal/Sink.cs:line 61
   at System.Reactive.Linq.ObservableImpl.Select`2.Selector._.OnNext(TSource value) in /_/Rx.NET/Source/src/System.Reactive/Linq/Observable/Select.cs:line 41
   at System.Reactive.Linq.ObservableImpl.Buffer`1.TimeHopping._.Tick() in /_/Rx.NET/Source/src/System.Reactive/Linq/Observable/Buffer.cs:line 468
   at System.Reactive.Concurrency.Scheduler.<>c__67`1.<SchedulePeriodic>b__67_0(ValueTuple`2 t) in /_/Rx.NET/Source/src/System.Reactive/Concurrency/Scheduler.Services.Emulation.cs:line 79
   at System.Reactive.Concurrency.DefaultScheduler.PeriodicallyScheduledWorkItem`1.<>c.<Tick>b__5_0(PeriodicallyScheduledWorkItem`1 closureWorkItem) in /_/Rx.NET/Source/src/System.Reactive/Concurrency/DefaultScheduler.cs:line 127
   at System.Reactive.Concurrency.AsyncLock.Wait(Object state, Delegate delegate, Action`2 action) in /_/Rx.NET/Source/src/System.Reactive/Concurrency/AsyncLock.cs:line 93
   at System.Threading.TimerQueueTimer.<>c.<.cctor>b__27_0(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.TimerQueueTimer.CallCallback(Boolean isThreadPool)
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.TimerQueue.FireNextTimers()
   at System.Threading.TimerQueue.AppDomainTimerCallback(Int32 id)

ReceiverHostExtensionsGetStreamStatistics line 34:

return runningCounts.Buffer(period)
                     .Select(window => (window[0], window[^1])

NmeaAisLongRangeAisBroadcastParser constructor docs name the wrong type

The NmeaAisLongRangeAisBroadcastParser constructor docs say that it creates a new NmeaAisPositionReportClassAParser!

Handle unexpected connection closure of the AIS Transmitter

The AIS Transmitter sometimes hangs and needs a reboot, when this happens the AIS Receiver crashes with the following exception.

Unhandled Exception: System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: No connection could be made because the target machine actively refused it 153.44.253.27:5631 at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult) at System.Net.Sockets.TcpClient.EndConnect(IAsyncResult asyncResult) at System.Net.Sockets.TcpClient.<>c.<ConnectAsync>b	28_l(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory'1.FromAsyncCoreLogic(IAsyncResult iar, Func'2 endFunction, Action'1 endAction, Task'1 promise. Boolean requiresSynchronization)
— End of stack trace from previous location where exception was thrown —
at Endjin.Ais.Receiver.NmeaReceiver.InitaliseAsync() in C:\Ais.Net.Receiver\NmeaReceiver.cs:line 48 at Endjin.Ais.Receiver.Program.Main(String[] args) in C:\Ais.Net.Receiver\Program.cs:line 33 at Endjin.Ais.Receiver.Program.<Main>(String[] args)

Handle System.Net.Internals.SocketExceptionFactory+ExtendedSocketException

Unhandled Exception: System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 153.44.253.27:5631
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.Sockets.TcpClient.EndConnect(IAsyncResult asyncResult)
   at System.Net.Sockets.TcpClient.<>c.<ConnectAsync>b__28_1(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at Endjin.Ais.Receiver.NmeaReceiver.InitaliseAsync() in C:\Ais.Net.Receiver\Solutions\Ais.Net.Receiver\NmeaReceiver.cs:line 48
   at Endjin.Ais.Receiver.Program.Main(String[] args) in C:\Ais.Net.Receiver\Solutions\Ais.Net.Receiver\Program.cs:line 33
   at Endjin.Ais.Receiver.Program.<Main>(String[] args)

ReceiverHost retry policy causes Negative timespan

public Task StartAsync(CancellationToken cancellationToken = default)
        {
            return Retriable.RetryAsync(() =>
                        this.StartAsyncInternal(cancellationToken),
                        cancellationToken,
                        new Backoff(maxTries: 100, deltaBackoff: TimeSpan.FromSeconds(5)),
                        new AnyExceptionPolicy(),
                        false);
        }

retry time 18 = -24.20:31:22.6480000
Backoff is an exponentially increasing retry delay, which is not very suitable here and will lead to long retry waiting

Decide what to do with bad messages in Ais.Net.Receiver.Host.Console

Once #141 is implemented, we need to go on to decide what the correct behaviour is for the app that persists messages received via the NetworkStreamNmeaReceiver.

Should it record the lines anyway? Should it somehow include information in its recorded output saying that the line is bad? Should it include information from the exception? Should it record bad messages separately?

Update to new Azure Storage client library

The library currently in use has now been marked as deprecated. Microsoft wants everyone to move over to the new Azure.* libraries.

Handle parse errors in ReceiverHost

Bad AIS messages are a fact of life. The NmeaLineParser constructor detects various clear indications of bad data and throws an ArgumentException. The ReceiverHost does not handle these exceptions, causing it to crash if a bad message is received.

We should handle ArgumentException from this line:

Ais.Net.Receiver/Solutions/Ais.Net.Receiver/Ais/Net/Receiver/Receiver/ReceiverHost.cs

Line 59 in 88f5211

lineStreamProcessor.OnNext(new NmeaLineParser(lineAsAscii), 0);

We should consider adding a new IObservable<T> property reporting these bad lines, so that applications can still do something with them. (E.g., if we're recording all incoming messages, it may be useful to include incomplete/corrupt ones, because manual inspection might be able to infer something from the data.)

Add support for Ais Message Type 27

The readme of the Ais.Net project claims support for message type 27 but as far as I can tell there is not any support for this meesage type, would be great to add it.

Handle Storage Exception "The append position condition specified was not met"

Unhandled Exception: Microsoft.Azure.Storage.StorageException: The append position condition specified was not met.
   at Microsoft.Azure.Storage.Blob.BlobWriteStreamBase.ThrowLastExceptionIfExists()
   at Microsoft.Azure.Storage.Blob.BlobWriteStream.FlushAsync(CancellationToken token)
   at Microsoft.Azure.Storage.Blob.BlobWriteStream.CommitAsync()
   at Microsoft.Azure.Storage.Blob.BlobWriteStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at Microsoft.Azure.Storage.Blob.CloudAppendBlob.UploadFromStreamHelper(Stream source, Nullable`1 length, Boolean createNew, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext)
   at Microsoft.Azure.Storage.Blob.CloudAppendBlob.AppendFromStream(Stream source, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext)
   at Microsoft.Azure.Storage.Blob.CloudAppendBlob.AppendFromByteArray(Byte[] buffer, Int32 index, Int32 count, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext)
   at Microsoft.Azure.Storage.Blob.CloudAppendBlob.AppendText(String content, Encoding encoding, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext)
   at Endjin.Ais.Receiver.StorageClient.AppendMessages(IList`1 messages) in C:\Ais.Net.Receiver\Solutions\Ais.Net.Receiver\StorageClient.cs:line 43
   at System.Reactive.AnonymousSafeObserver`1.OnNext(T value) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\AnonymousSafeObserver.cs:line 44
   at System.Reactive.Subjects.Subject`1.OnNext(T value) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\Subject.cs:line 148
   at Endjin.Ais.Receiver.NmeaReceiver.RecieveAsync() in C:\Ais.Net.Receiver\Solutions\Ais.Net.Receiver\NmeaReceiver.cs:line 62
   at Endjin.Ais.Receiver.Program.Main(String[] args) in C:\Ais.Net.Receiver\Solutions\Ais.Net.Receiver\Program.cs:line 31
   at Endjin.Ais.Receiver.Program.<Main>(String[] args)

Runtime Failure

Process terminated with the following stracktrace

at System.Reactive.Concurrency.ConcurrencyAbstractionLayerImpl.PeriodicTimer.<>c.<.ctor>b__2_0(Object this) in /_/Rx.NET/Source/src/System.Reactive/Concurrency/ConcurrencyAbstractionLayerImpl.cs:line 207
at System.Threading.TimerQueueTimer.<>c.<.cctor>b__27_0(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
at System.Threading.TimerQueue.FireNextTimers()
at System.Threading.TimerQueue.AppDomainTimerCallback(Int32 id)

Handle network connectivity outage

The Receiver encountered a transient error whereby Azure Storage could not be reached. The following exception was thrown:

Unhandled Exception: Microsoft.Azure.Storage.StorageException: No such host is known —> System.Net.Http.HttpRequestException: No such host is known —> System.Net.Sockets.SocketException: No such host is known
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
	 End of inner exception stack trace 	
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken) at System.Threading.Tasks.ValueTask 1.get_Result()
at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask 1.get_Result()
at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask 1 creationTask) at System.Threading.Tasks.ValueTask 1.get_Result()
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, Cancellat ionToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 senaTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Microsoft.Azure.Storage.Core.Executor.Executor.ExecuteAsync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
	 End of inner exception stack trace 	
at Microsoft.Azure.Storage.Blob.BlobWriteStreamBase.ThrowLastExceptionlfExists() at Microsoft.Azure.Storage.Blob.BlobWriteStream.FlushAsync(Cancel1ationToken token) at Microsoft.Azure.Storage.Blob.BlobWriteStream.CommitAsync() at Microsoft.Azure.Storage.Blob.BlobWriteStream.Dispose Boolean disposing) at System`10.Stream.Close()
at Microsoft.Azure.Storage.Blob.CloudAppendBlob.UploadFromStreamHelper(Stream source, Nullable`1 length, Boolean createNew, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext)
at Microsoft.Azure.Storage.Blob.CloudAppendBlob.AppendFromStream(Stream source, AccessCondition accessCondition, Blob RequestOptions options, OperationContext operationContext)
at Microsoft.Azure.Storage.Blob.CloudAppendBlob.AppendFromByteArray(Byte[] buffer, Int32 index, Int32 count, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext)
at Microsoft.Azure.Storage.Blob.CloudAppendBlob.AppendText(String content, Encoding encoding, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext)
at Endjin.Ai s.Receiver.StorageClient.AppendMessages(IList`1 messages) in C:\Ais.Net.Receiver\Solutions\Ais.Net.Receiver\StorageClient.cs :1ine 43
at Endjin.Ais.Receiver.Program.OnMessageReceived(IList`1 messages) in C:\Ais.Net.Receiver\Solutions\Ais.Net.Receiver\Program.cs:line 51
at System.Reactive.AnonymousSafeObserver`1.OnNext(T value) in D:\a\l\s\Rx.NET\Source\src\System.Reactive\AnonymousSafeObserver.cs:line 44
at System.Reactive.Subjects.Subject`1.OnNext(T value) in D:\a\l\s\Rx.NET\Source\src\System.Reactive\Subjects\Subject.cs:line 148
at Endjin.Ais.Receiver.NmeaReceiver.RecieveAsync() in C:\Ais.Net.Receiver\Solutions\Ais.Net.Receiver\NmeaReceiver.cs:line 61
at Endiin.Ais.Receiver.Program.Main(String[] args) in C:\Ais.Net.Receiver\Solutions\Ais.Net.Receiver\Program.es:line 31
at Endjin.Ais.Receiver.Program.<Main>(String[] args)

Consider refactoring the code to use TPL Dataflow and either using a BufferBlock to hold incoming messages while trying to re-establish a connection with Azure Storage, or write messages to a local repository, which are then transmitted, in order, when the connection is re-established.

Add support for Message Types 4 and 21

How about adding support for Message types 4 and 21? These fixed location messages may not be as interesting as mobile, but are important for shipboard navigation. Unclear to me if this app/library was mostly designed as a land-based application. Also I'm using them to study VHF propagation changes.

Currently I'm using python and pyais but my python expertise is not great, and python appears to have less mature support for threading, multiprocessing etc.

NetworkStreamNmeaReceiver not keeping up when parsing messages

We've been having issues where the network stream keeps stopping occasionally for a few seconds until it resumes. There are no logged error messages, and when it resumes the stream is as fast as before, so we're losing a lot of data. Eventually, after several minutes, it results in Corvus failing with the following error message:

System.ArgumentOutOfRangeException: The value needs to translate in milliseconds to -1 (signifying an infinite timeout), 0, or a positive integer less than or equal to the maximum allowed timer duration. (Parameter 'delay')
    at System.Threading.Tasks.Task.ValidateTimeout(TimeSpan timeout, ExceptionArgument argument)
    at System.Threading.Tasks.Task.Delay(TimeSpan delay)
    at Corvus.Retry.Async.SleepService.DefaultSleepService.SleepAsync(TimeSpan timeSpan)
    at Corvus.Retry.Retriable.RetryAsync(Func`1 asyncFunc, CancellationToken cancellationToken, IRetryStrategy strategy, IRetryPolicy policy, Boolean continueOnCapturedContext)
    at AisLog.Services.AisLogService.ExecuteAsync(CancellationToken cancellationToken) in /source/Services/AisLogService.cs:line 102
    at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)

This started happening a few weeks ago on our local development setups, and just recently it's also affecting our production server. CPU load is rarely high on either setup (<1%).

I've conducted some tests of running the receiver in an isolated environment, counting the number of messages received in a minute.

using Ais.Net.Receiver.Receiver;

var receiver = new NetworkStreamNmeaReceiver("153.44.253.27", 5631, TimeSpan.FromSeconds(0.5));
var receiverHost = new ReceiverHost(receiver);

var count = 0;

receiverHost.Sentences.Subscribe(_ => count++);
//receiverHost.Messages.Subscribe(_ => {});

await Task.WhenAny(receiverHost.StartAsync(), Task.Delay(TimeSpan.FromMinutes(1)));

Console.WriteLine(count);

Here are results of 5 runs from my laptop:

$ dotnet run -> 2473
$ dotnet run -> 2410
$ dotnet run -> 2402
$ dotnet run -> 2400
$ dotnet run -> 2401

Subscribing to ReceiverHost.Sentences seemingly works fine, as every test I've run has resulted in between 2400-2500 messages.

However, uncommenting receiverHost.Messages.Subscribe(_ => {}); thus also parsing the messages using a subscription, I've been getting mixed results on the sentence count. Another 5 runs of this new code on my laptop:

$ dotnet run -> 743
$ dotnet run -> 499
$ dotnet run -> 935
$ dotnet run -> 837
$ dotnet run -> 373

I've also tried only subscribing to messages and counting those which results in similar numbers. Running tests with something like receiverHost.Sentences.Subscribe(Console.WriteLine); I can also inspect that subscribing to Sentences results in a steady stream of messages, whereas when subscribing to Messages the stream pauses multiple times.

The reason I've been specifying laptop is that it seems to run fine (at the moment) on my desktop machine, which understandably is more powerful.

Laptop: Dell XPS 13 9310, Linux Ubuntu 22.04.1 LTS, Intel Core i7-1185G7, 16 GB RAM
Desktop: Windows 10, AMD Ryzen 5600X, 32 GB RAM
Production server: Virtual LXC Server, Linux Ubuntu 20.04 LTS, 2 vCPUs (unsure but might be Intel Core i7-3770), 2 GB RAM

What could be the cause of this? Consiering Ais.Net reportedly can handle more than a million sentences per minute, I'd assume this is a problem in the receiver.