Takes an AIS vessel position feed, and tries detecting suspicious or illegal fishing or other unauthorised activities. The feed will often be very large 1 to 100GB on average, hence performance and controlled memory usage are key. This program using state objects maintaining a minimal state per vessel instead of per position record.
This golang executable takes an AIS line delimited JSON file, containing AIS data messages. Some of them message types are positional messages containing a GPS position for a given vessel. The message types are :1, 2, 3, 18, 19, 27.
The program takes this AIS message list as an input, extracts the GPS position, calculates displacement using the Haversine formula, then computes the vessel speed. For each vessel keeps a state object up to date with the [i-1]-th and [i]-th position with i in [1-N] and the position set from [0-N]. It processes each message iteratively and updates the vessel state on every new position found. When calculating speed and time and meeting the following conditions: speed < 1 knot for 60 mins or more then a zero speed stopped geojson feature is created as an output. Whenever the vessel moves again the speed status is reset in the relevant vessel state object, until finding a new stopped postion with the above mentioned conditions, or until running out of input data, reaching the end of the data input file.
{"Message":{"MessageID":3,"RepeatIndicator":0,"UserID":12345,"Valid":true,"NavigationalStatus":3,"RateOfTurn":0,"Sog":1.7,"PositionAccuracy":false,"Longitude":166.90754833333332,"Latitude":-21.46249,"Cog":290.3,"TrueHeading":34,"Timestamp":57,"SpecialManoeuvreIndicator":0,"Spare":0,"Raim":false,"CommunicationState":4336},"UTCTimeStamp":1588636800} {"Message":{"MessageID":5,"RepeatIndicator":0,"UserID":56789,"Valid":true,"AisVersion":0,"ImoNumber":9344382,"CallSign":"V2CZ5 ","Name":"GRETA ","Type":79,"Dimension":{"A":86,"B":15,"C":11,"D":4},"FixType":1,"Eta":{"Month":4,"Day":18,"Hour":14,"Minute":0},"MaximumStaticDraught":3.9,"Destination":"CAPE TOWN ","Dte":false,"Spare":false},"UTCTimeStamp":1588636800} {"Message":{"MessageID":18,"RepeatIndicator":0,"UserID":89087,"Valid":true,"Spare1":0,"Sog":0,"PositionAccuracy":true,"Longitude":174.76563,"Latitude":-36.821688333333334,"Cog":360,"TrueHeading":511,"Timestamp":1,"Spare2":0,"ClassBUnit":true,"ClassBDisplay":false,"ClassBDsc":true,"ClassBBand":true,"ClassBMsg22":true,"AssignedMode":false,"Raim":true,"CommunicationStateIsItdma":true,"CommunicationState":393222},"UTCTimeStamp":1588636801} {"Message":{"MessageID":19,"RepeatIndicator":0,"UserID":654321,"Valid":true,"Spare1":0,"Sog":0,"PositionAccuracy":false,"Longitude":170.18316666666666,"Latitude":-15.192758333333334,"Cog":0,"TrueHeading":511,"Timestamp":60,"Spare2":0,"Name":"AABB 11V3","Type":0,"Dimension":{"A":3,"B":5,"C":1,"D":2},"FixType":1,"Raim":false,"Dte":true,"AssignedMode":false,"Spare3":0},"UTCTimeStamp":1588636801} {"Message":{"MessageID":27,"RepeatIndicator":3,"UserID":12345,"Valid":true,"PositionAccuracy":false,"Raim":false,"NavigationalStatus":0,"Longitude":10.520000000000001,"Latitude":-16.695,"Sog":9,"Cog":168,"PositionLatency":false,"Spare":false},"UTCTimeStamp":1588636801}
{"type":"FeatureCollection","features":[<array of features...>]}
A feature is of the following form: {"type":"Feature","geometry":{"type":"Point","coordinates":[<longitude_degrees>, <latitude_degrees>]},"properties":{"name":,"duration_sec":,"date_UTC":""}}
For instance: {"type":"Feature","geometry":{"type":"Point","coordinates":[173.54578, -41.67789]},"properties":{"name":12345,"duration_sec":3602,"date_UTC":"2021-06-05T01:02:21Z"}}
Fast iterative approach using the Golang Reader/ Json decoder interface + state objects that lazy load the required data. This allows fast processing and low RAM needs.
Steps:.
- Reading json input line by line using the golang Json decoder buffered interface.
- Creating a state object per found AIS vessel, updating the state at each relevant record found.
- Updating an ouput map of selected position for the final position report.
On a modest 1500 NZD 2020 laptop i5-10210U with 16GB of RAM, the program processed a 3GB input file counting 8.955.836 AIS messages (json records) in 1min20secs on my laptop and used max 700MB of RAM.
File sample available in the sample_data folder.
A linux x86_64 executable is provided govessel_x86_64_linux
- install golang version 1.15.8 or superior as per the go.mod file
- go build .
./govessel -in sample_data/part0000.json -out geojson_output_file.json
The generated geojson report can be uploaded on http://geojson.io in order to be visualised on the world map.