Comments (11)
The ability to handle lines with tag blocks is probably a missing feature and worth implementing on its own right.
Timestamps from tag blocks is another. As there is no time information in the normal NMEA0183 input the parser could just not produce timestamp, as Node Server will fill in missing timestamps for all input. This way the parser would produce timestamps only for data that actually has timestamps.
A third issue would be the ability to play back tag-timestamped data in real time (or speeded up by a factor for example). For N2K data from canboat actisense-serial with timestamps this is implemented in https://github.com/SignalK/signalk-server-node/blob/master/providers/timestamp-throttle.js - you add that to the provider pipeline like in https://github.com/SignalK/signalk-server-node/blob/master/settings/aava-file-settings.json#L27-L32
Node server has optional logging, with the ability to multiplex input from various sources in https://github.com/SignalK/signalk-server-node/blob/master/providers/log.js with the associated provider element that allows playback in real time: https://github.com/SignalK/signalk-server-node/blob/master/providers/multiplexedlog.js#L25-L38. You could take the same approach and handle tag blocks outside the parser: use the timestamp for throttling, hand just the normal NMEA0183 to parser, but you would have to fill in the correct timestamp to the parser output.
For batch oriented import for the use case "import this log file from a day last summer" I am going for still another solution. The system must support backpressure: the pipeline will have no throttling, but the end of the pipe, writing to InfluxDb, will not be able to write as fast as the beginning, reading from a file, will produce data. Node streams, that producer pipeline uses, does that nicely.
The current signalk-to-influxdb plugin does not handle the incoming timestamp and data will be timestamped in InfluxDb per insertion time. Easy to fix, but fyi.
I am not quite sure InfluxDb is the easiest solution to your problem. It will work I guess and signalk-to-influxdb has the code for calculating TWS and TWA. One way to go about this would be to just produce CSV-type files that contain STW, TWS, TWA triplets and feed them to your polar producing code. Then again having the data in a db would be easy for later analysis. A more traditional sql db might be a better fit than a time series database, if you are not using stuff like min/max/average/median heavily.
from nmea0183-signalk.
Thanks a lot for the pointers. I guess the first would be to strip the line for the tag block and parse the normal nmea as today.
Then I'm thinking of a boolean passed to the parser, for the summer trip import, use tag timestamp, else just ignore the tag timestamp and produce one as is done now.
The parser must of course be able to produce a current timestamp for normal operation, but can this be done globally and not for each sentence type?
Another useful info is the source in the tag. Say you use kplex with different inputs, and pass this to sk. It will be treated as one source in sk, even thought kplex uses several.
I follow you on influx for polars. But the first step is to get the data into influxdb to visualize. We are getting there one step at a time. It might end up with one influx for "log" and one mysql for polar data, dynamically updated.
from nmea0183-signalk.
Is it even possible to split a line with \ as delimiter in js?
Stackexchange is full of "you don't, you have to escape the backslash in the string".
from nmea0183-signalk.
Of course you can split a string on \
. You just need to escape it in the split function:
someString.split("\\");
from nmea0183-signalk.
I've been trying for a while now with this sentence: \s:compass,c:1438489697*13\$TIROT,0.1,A*3A
from nmea0183-signalk.
If you're trying in the browser console or a simple Node script like this:
let x = "\s:compass,c:1438489697*13\$TIROT,0.1,A*3A";
x.split("\\");
It won't work because the \
will get eaten; you need to escape it in string literals.
let x = "\\s:compass,c:1438489697*13\\$TIROT,0.1,A*3A";
x.split("\\");
will do what you expect (well, maybe not exactly what you expect), the resulting array will look like:
[ "", "s:compass,c:1438489697*13", "$TIROT,0.1,A*3A" ]
If you're reading the string from a file or from a network socket, stream, etc, it doesn't need to be escaped. It's no different than reading a newline or any other control character. Must be escaped in string literals, works as expected everywhere else.
from nmea0183-signalk.
At last an answer I could understand. Thanks @timmathews. Christmas saved!
from nmea0183-signalk.
This is implemented in the new version in the development
branch. Only downside is that, since NMEA chose to use escape characters (WHY?!?!??!) as the start and end characters of the tag block, it requires these to be escaped before parsing (which is not something that developers of most Providers will think of doing, I suspect).
from nmea0183-signalk.
That is ok for now. I can do a batch replace in my historic files, once the hooks are in place. Should we close the issue? Doesn't look like we will have a full solution
from nmea0183-signalk.
Well, technically they're supported - one could argue that the escaping is not our responsibility since a single \
in a javascript string is essentially not a backslash.
Edit: they're not supported entirely, as I still need to add one thing: verification of the timestamp using the checksum (why does the timestamp have a checksum?)
from nmea0183-signalk.
I'll close this for now, and add a new issue for the timestamp checksum.
from nmea0183-signalk.
Related Issues (20)
- VDM: Request of IMO number (/Registrations) from message 5
- Custom sentence procedure HOT 21
- Seatalk 0x84 fails for single digit hex values HOT 1
- ROT: navigation/rateOfTurn seems not to be parsed from NMEA AIVDM HOT 7
- Broken link in README.md HOT 1
- Tiny typo errors HOT 1
- Empty in certain nmea0183 strings prevents parsing HOT 3
- Datagram 25 trip calculation: factor for W should probably be 4096 HOT 1
- VDM, callsign and imo data are not pushed to SK HOT 4
- BWC not recognized HOT 5
- DSC Distress Alerts HOT 6
- callsignVhf not according to the spec HOT 8
- timestamp handling: use only data from NMEA sentences HOT 4
- COG value differently parsed from RMC and VTG sentences
- Depth sentences don't generate output with empty incoming data fields
- HDG sentence incorrectly parsed HOT 1
- APB and BOD sentences not decoded according to SignalK spec HOT 8
- RMB parser stores next waypoint position to incorrect SignalK field HOT 1
- Is error detection too aggressive? HOT 2
- AIS VDM to support meteorological data
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nmea0183-signalk.