52north / sos-importer Goto Github PK

View Code? Open in Web Editor NEW

10.0 14.0 8.0 8.01 MB

Application to import observation data into a running SOS instance using the T-interface.

Home Page: https://wiki.52north.org/SensorWeb/SosImporter

License: GNU General Public License v2.0

Java 100.00%

sos observation sensorweb csv

sos-importer's Introduction

52°North SOS Importer

For information on this project visit its homepage.

Branches

This project follows the Gitflow branching model. "master" reflects the latest stable release. Ongoing development is done in branch develop and dedicated feature branches (feature-*).

Credits

The development the 52°North Sensor Observation Service implementations was supported by several organizations and projects. Among other we would like to thank the following organisations and project

Project/Logo	Description
	The development of this version of the 52°North SOS Importer was supported by the German Federal Ministry of Education and Research research project COLABIS (co-funded by the German Federal Ministry of Education and Research, programme Geotechnologien, under grant agreement no. 03G0852A)
	The development of this version of the 52°North SOS Importer was supported by the German Federal Ministry of Education and Research research project MuDak-WRM (co-funded by the German Federal Ministry of Education and Research, programme FONA)
	The development of this version of the 52°North SOS Importer was supported by the European FP7 research project EO2HEAVEN (co-funded by the European Commission under the grant agreement n°244100)

sos-importer's People

Contributors

Stargazers

Watchers

Forkers

ridoo johnjohndoe geomatys gschwind mojioms doudino bengraeler geo4dam

sos-importer's Issues

Cannot open configuration file: UnsupportedOperationException

Description

When clicking on the latst wizard page on the button, nothing happens. The command line shows an exception.

Stackstrace

Exception in thread "AWT-EventQueue-0" java.lang.UnsupportedOperationException: The BROWSE action is not supported on the current platform!
        at java.awt.Desktop.checkActionSupport(Desktop.java:225)
        at java.awt.Desktop.browse(Desktop.java:381)
        at org.n52.sos.importer.view.Step8Panel$2.actionPerformed(Step8Panel.java:224)

OS

Distributor ID: neon
Description:    KDE neon User Edition 5.9
Release:        16.04
Codename:       xenial

Java

openjdk version "1.8.0_121"                                                                                                                                                        
OpenJDK Runtime Environment (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)                                                                                                       
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

Tutorial needs to be updated

The tutorial how to import the code into eclipse https://wiki.52north.org/bin/view/SensorWeb/SosImporter#Import_Current_Trunk_in_Eclipse needs to be updated.

Output specific number of non matching columns

The following message is output when the recognized number of columns from a .csv file does not match the expected number of columns:

Number of InsertObservations is not matching number of timeseries!

Please output the number of expected columns and the number of recognized columns.

TimeSeriesRepository.addObservations ➡️ ... ➡️ Configuration.getMeasureValueColumnIds

Always provide skip reason

I am running the importer on a CSV file. All lines are skipped. The importer does not state the reason. Here is the console output:

[org.n52.sos.importer.feeder.task.OneTimeFeeder] TRACE SensorObservationService \
-       Skip CSV line #2750: 124,2015-06-21,17.0,14.0,1.9,3.0,35.0,58.0,,,,,,,,
[org.n52.sos.importer.feeder.task.OneTimeFeeder] TRACE SensorObservationService \
-       Skip CSV line #2751: 143,2015-06-21,11.0,18.0,,,27.0,62.0,,,,,,,,

The relevant code parts can be found here and here. Please ensure a reason is always provided.

Add integration for ScraperWiki

ScraperWiki (https://scraperwiki.com/) is a nice little site that allows scraping of data from websites. We could add an integration that downloads a CSV file from scraperwiki at a regular interval (need to implement download and authentication support instead of local text file) and pushes the data to an SOS.

Could not parse ISO8601 timestamps with timezone "Z"

2015-05-18 10:20:33,437 ERROR [FeederTimer] [SensorObservationService.java:613] Could not retrieve all information required for insert observation because of parsing error: java.text.ParseException: Unparseable date: "2015-03-31T23:00:00.000Z". Skipped this one.

Feeder fails to read FoI position

Successfully created a configuration file from a .csv file of 4 humidity observations per feature of interest with lat-lon coordinates. Feeder fails, though, with a null position and then a null pointer exception reading the LongitudeUnit. Any suggestions? This happens with builds under both Java 7 and 8, both remotely and locally to the SOS service.
DATA_Obs_160907_1sensor_csv.txt
DATA_Obs_160907_1sensor.csv.52n-sos-import-config_xml.txt

java -jar 52n-sos-importer-feeder-bin.jar -c ~/tmp/FCP1_-RBG_DATA_Obs_160907_1sensor.csv.52n-sos-import-config.xml
10:56:45.177 [main] INFO Feeder - Application started
Implementation-Vendor: N/A
Implementation-Title: Java Runtime Environment
Implementation-Version: 1.7.0_111
Implementation-Vendor-Id: com.sun
Specification-Vendor: Oracle Corporation
Manifest-Version: 1.0
Specification-Title: Java Platform API Specification
Created-By: 1.7.0_0 (Oracle Corporation)
Specification-Version: 1.7
Extension-Name: javax.crypto
HeapSize Information: max: 878MB; total now: 59MB; free now: 47MB; used now: 11MB
os.name: Linux; os.arch: amd64; os.version: 4.5.5-x86_64-linode69
10:56:45.715 [OneTimeFeeder] INFO OneTimeFeeder - Starting feeding data from file via configuration 'FCP1-RBG_DATA_Obs_160907_1sensor.csv.52n-sos-import-config.xml' to SOS instance
10:56:45.745 [OneTimeFeeder] TRACE SensorObservationService - SensorObservationService(Configuration [file=/home/server/tmp/FCP1-_RBG_DATA_Obs_160907_1sensor.csv.52n-sos-import-config.xml])
10:56:46.025 [OneTimeFeeder] DEBUG SimpleHttpClient - executing POST method to 'http://serv.tumblingwalls.com/52n-sos-webapp/service'.
10:56:47.443 [OneTimeFeeder] DEBUG SensorObservationService - OperationsMetadata found: org.n52.oxf.ows.capabilities.OperationsMetadata@5ed7443d
10:56:47.444 [OneTimeFeeder] DEBUG SensorObservationService - Found all required operations: (RegisterSensor|InsertSensor), InsertObservation
10:56:47.448 [OneTimeFeeder] DEBUG OneTimeFeeder - Check counter file '/home/server/.SOSImporter/b29600c2497f5fd23ef620bf97fd7819'.
10:56:47.449 [OneTimeFeeder] DEBUG OneTimeFeeder - Counter file does not exist.
10:56:47.449 [OneTimeFeeder] TRACE SensorObservationService - importData()
10:56:47.469 [OneTimeFeeder] DEBUG SensorObservationService - Handling CSV line #2: [SE10_0AB_1, Sensor1, 51.487902, 0.004408, 0.71, Rel_Humidity, Percent, 3/14/2016 1.30 PM, , , �]
10:56:47.470 [OneTimeFeeder] DEBUG SensorObservationService - Parsing measured value column 4
10:56:47.474 [OneTimeFeeder] DEBUG SensorObservationService - Sensor: Sensor [name=Sensor1, uri=Sensor1]
10:56:47.478 [OneTimeFeeder] DEBUG SensorObservationService - Feature of Interest: FeatureOfInterest [p=null, name=SE10_0AB_1, uri=SE10_0AB_1]
10:56:47.499 [OneTimeFeeder] DEBUG SensorObservationService - Value: 0.71
10:56:47.516 [OneTimeFeeder] DEBUG SensorObservationService - Timestamp: 2016-03-14T13:30:00+01:00
10:56:47.519 [OneTimeFeeder] DEBUG SensorObservationService - UomCode: 'UnitOfMeasurement [name=Percent, uri=Percent]'
10:56:47.524 [OneTimeFeeder] DEBUG SensorObservationService - ObservedProperty: ObservedProperty [name=Rel_Humidity, uri=Rel_Humidity]
10:56:47.527 [OneTimeFeeder] DEBUG SensorObservationService - Offering: Offering [name=Sensor1, uri=Sensor1]
10:56:47.529 [OneTimeFeeder] DEBUG SensorObservationService - Found '1' Observed Properties for Sensor 'Sensor1': '[ObservedProperty [name=Rel_Humidity, uri=Rel_Humidity]]'
10:56:47.530 [OneTimeFeeder] DEBUG SensorObservationService - Found '1' Measured value types for observed properties of sensor 'Sensor1': '{ObservedProperty [name=Rel_Humidity, uri=Rel_Humidity]=NUMERIC}'.
10:56:47.530 [OneTimeFeeder] DEBUG SensorObservationService - Found '1' units of measurement for observed properties of sensor 'Sensor1': '{ObservedProperty [name=Rel_Humidity, uri=Rel_Humidity]=Percent}'.
Exception in thread "OneTimeFeeder" java.lang.NullPointerException
at org.n52.sos.importer.feeder.model.requests.RegisterSensor.getLongitudeUnit(RegisterSensor.java:107)
at org.n52.sos.importer.feeder.util.DescriptionBuilder.createSML(DescriptionBuilder.java:79)
at org.n52.sos.importer.feeder.SensorObservationService.createInsertSensorParametersFromRS(SensorObservationService.java:1018)
at org.n52.sos.importer.feeder.SensorObservationService.registerSensor(SensorObservationService.java:940)
at org.n52.sos.importer.feeder.SensorObservationService.insertObservationsForOneLine(SensorObservationService.java:667)
at org.n52.sos.importer.feeder.SensorObservationService.importData(SensorObservationService.java:304)
at org.n52.sos.importer.feeder.task.OneTimeFeeder.run(OneTimeFeeder.java:203)
at java.lang.Thread.run(Thread.java:745)

org.n52.oxf.OXFException: Could not parse response to XML.

Hi all,
I wonder if sos-importer work on only 52NorthSos server? Because when i trying to execute sos-impoerter, it gives me this error : https://gist.github.com/agtokty/78644222c6720d8e8ada

is there something worng with my service or with OX-Framework ?

Regards.

Current GUI is broken when using sample based files with minor inconsistencies

The current implementation is not able to deal with sample based files which contain BAD lines like this

Sample File,C:\datafolder_\something\we\dont\care.C83
Model,0815

Sample #,1,,,,,,,,,
Start Date,04/11/14,,,,,,,,,
Start Time,12:00:00,,,,,,,,,
Sample Length,43140,,,,,,,,,
Averaging Interval (s),60.0,,,,,,,,,
Title,,,,,,,,,,
Instrument ID,Model 0815 SN  520042 FW Version 1.23,,,,,,,,,
Instrument Errors,Vacuum Level Error;Nozzle Flow Error;,,,,,,,,,

Conc Mean,12160.5,,,,,,,,,
Conc Min,1111,,,,,,,,,
Conc Max,38310.2,,,,,,,,,
Conc Std Dev,5234.36,,,,,,,,,

Time,Conc (#/cm³),Count (#),LiveTime (s),DTC,Abs Pressure (mbars),Analog In,Pulse Mean (mV),Pulse Std Dev (mV),Instr Status,
12:01:00,11646.8,1454570,0.0962167,1.1,228,0,0,0,None,

The problem here is that the first lines differ in the number of columns from the real data. The first three lines miss the empty columns. This seems to break the CSV lib we are using.

Wizard module ClassCastException

Hey there,

I just ran into a CCE when trying a fresh build from 95aa1f4 on develop.

The CCE occured when hitting the next button of the Feature of Interest page, after manually setting the URI and name for the first Measured Value column.
I was trying to parse following data using this (yyyy-MM-dd'T'HH:mm:ss.SSSSSS) timestamp format.

utc;temp [°C];humidity[%]
2017-04-13T08:27:28.154205;20.80;39.20
2017-04-13T08:28:05.609667;20.70;39.10
2017-04-13T08:28:36.165725;20.70;39.10
2017-04-13T08:29:06.721869;20.70;39.10
2017-04-13T08:29:37.277954;20.70;39.10
2017-04-13T08:30:07.834017;20.70;39.10
2017-04-13T08:30:38.390001;20.70;39.00
2017-04-13T08:31:08.938793;20.70;39.10

Here is the full log since startup of the wizard.

$ java -Xmx4g -jar 52n-sos-importer-wizard-bin.jar
16:21:38,163 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
16:21:38,163 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
16:21:38,163 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/C:/Users/bruno/Desktop/Neuer%20Ordner/52n-sos-importer-wizard-bin.jar!/logback.xml]
16:21:38,180 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@71be98f5 - URL [jar:file:/C:/Users/bruno/Desktop/Neuer%20Ordner/52n-sos-importer-wizard-bin.jar!/logback.xml] is not of type file
16:21:38,231 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
16:21:38,244 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Will scan for changes in [jar:file:/C:/Users/bruno/Desktop/Neuer%20Ordner/52n-sos-importer-wizard-bin.jar!/logback.xml]
16:21:38,244 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Setting ReconfigureOnChangeTask scanning period to 30 seconds
16:21:38,248 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
16:21:38,252 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [FILE]
16:21:38,325 |-INFO in c.q.l.core.rolling.TimeBasedRollingPolicy@1873653341 - No compression will be used
16:21:38,326 |-INFO in c.q.l.core.rolling.TimeBasedRollingPolicy@1873653341 - Will use the pattern logs/sos-importer-core/%d{yyyy-MM-dd}-%i.log for the active file
16:21:38,329 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@17f6480 - The date pattern is 'yyyy-MM-dd' from file name pattern 'logs/sos-importer-core/%d{yyyy-MM-dd}-%i.log'.
16:21:38,329 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@17f6480 - Roll-over at midnight.
16:21:38,333 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@17f6480 - Setting initial period to Tue Jul 18 16:21:38 CEST 2017
16:21:38,335 |-WARN in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@17f6480 - SizeAndTimeBasedFNATP is deprecated. Use SizeAndTimeBasedRollingPolicy instead
16:21:38,335 |-WARN in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@17f6480 - For more information see http://logback.qos.ch/manual/appenders.html#SizeAndTimeBasedRollingPolicy
16:21:38,336 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
16:21:38,353 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log file name: logs/sos-importer-core.log
16:21:38,353 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property is set to [logs/sos-importer-core.log]
16:21:38,356 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
16:21:38,358 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
16:21:38,359 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
16:21:38,363 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.n52] to TRACE
16:21:38,363 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
16:21:38,363 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [FILE] to Logger[ROOT]
16:21:38,364 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
16:21:38,365 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
16:21:38,366 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@2d6e8792 - Registering current configuration as safe fallback point

16:21:38.531 [main] TRACE Lang - setCurrentLocale(en)
16:21:38.655 [main] INFO  ComboBoxItems - Load settings from C:\Users\bruno\.SOSImporter\52n-sensorweb-sos-importer.properties
16:21:39.561 [main] TRACE Model - registerProvider(Step1Model)
16:21:39.562 [main] TRACE Model -       createArrayListFromArray()
16:21:39.562 [main] TRACE Model -       saveProvidersInArray()
16:21:39.562 [main] TRACE BackNextModel - setCurrentStepController(Step1Controller[131206411])
16:21:52.235 [AWT-EventQueue-0] TRACE BackNextController - nextButtonClicked()


16:21:52.237 [AWT-EventQueue-0] TRACE BackNextModel - getCurrentStepController(): result:Step1Controller[131206411]
16:21:52.237 [AWT-EventQueue-0] INFO  Step1Controller - Read CSV file D:\pi\importer\dht22-small.csv
16:21:52.240 [AWT-EventQueue-0] TRACE Model - updateModel()
16:21:52.241 [AWT-EventQueue-0] TRACE Step1ModelHandler - handleModel()
16:21:52.256 [AWT-EventQueue-0] TRACE BackNextModel - addPreviousStepController(Step1Controller[131206411])
16:21:52.256 [AWT-EventQueue-0] TRACE BackNextModel - getFollowingStepController()
16:21:52.257 [AWT-EventQueue-0] TRACE BackNextModel - followingSC: null
16:21:52.260 [AWT-EventQueue-0] TRACE Step2Controller - loadSettings()
16:21:52.332 [AWT-EventQueue-0] TRACE Step2Panel - setCSVFileHighlight()
16:21:52.356 [AWT-EventQueue-0] TRACE Model - registerProvider(Step2Model)
16:21:52.356 [AWT-EventQueue-0] TRACE Model -   createArrayListFromArray()
16:21:52.357 [AWT-EventQueue-0] TRACE Model -   saveProvidersInArray()
16:21:52.357 [AWT-EventQueue-0] TRACE BackNextModel - setCurrentStepController(Step2Controller[36624701])
16:21:52.357 [AWT-EventQueue-0] TRACE Model - removeProvider(Step1Model)
16:21:52.357 [AWT-EventQueue-0] TRACE Model -   createArrayListFromArray()
16:21:52.358 [AWT-EventQueue-0] TRACE Model -   saveProvidersInArray()
16:21:55.829 [AWT-EventQueue-0] TRACE Step2Panel - setCSVFileHighlight()
16:21:57.724 [AWT-EventQueue-0] TRACE BackNextController - nextButtonClicked()


16:21:57.725 [AWT-EventQueue-0] TRACE BackNextModel - getCurrentStepController(): result:Step2Controller[36624701]
16:21:57.725 [AWT-EventQueue-0] TRACE Step2Controller - saveSettings()
16:21:57.726 [AWT-EventQueue-0] TRACE Model - updateModel()
16:21:57.728 [AWT-EventQueue-0] TRACE Step2ModelHandler - handleModel()
16:21:57.739 [AWT-EventQueue-0] TRACE BackNextModel - addPreviousStepController(Step2Controller[36624701])
16:21:57.739 [AWT-EventQueue-0] TRACE BackNextModel - getFollowingStepController()
16:21:57.739 [AWT-EventQueue-0] TRACE BackNextModel - followingSC: null
16:21:57.740 [AWT-EventQueue-0] INFO  Step2Controller - Parse CSV file: column separator: ';', comment indicator: '#', text qualifier: '"', first line with data: 1; use header? false
16:21:57.780 [AWT-EventQueue-0] TRACE ParseTestLabel - ParseTestLabel()[1633838190]
16:21:57.781 [AWT-EventQueue-0] TRACE ParseTestLabel - ParseTestLabel()[367364578]
16:21:57.782 [AWT-EventQueue-0] TRACE ParseTestLabel - ParseTestLabel()[169695775]
16:21:57.782 [AWT-EventQueue-0] TRACE ParseTestLabel - ParseTestLabel()[479264955]
16:21:57.791 [AWT-EventQueue-0] TRACE ParseTestLabel - ParseTestLabel()[957600186]
16:21:57.799 [AWT-EventQueue-0] TRACE ExampleFormatLabel - ExampleFormatLabel(formatter: DateAndTime)
16:21:57.813 [AWT-EventQueue-0] TRACE ParseTestLabel - ParseTestLabel()[1818711122]
16:21:57.819 [AWT-EventQueue-0] TRACE ExampleFormatLabel - ExampleFormatLabel(formatter: Position)
16:21:57.822 [AWT-EventQueue-0] TRACE Step3Controller - loadSettings()
16:21:57.822 [AWT-EventQueue-0] DEBUG Step3Controller - Step3Model: Step3Model[2004059201]: fLWD: 1, uH: false, mC: 0, cA: {}
16:21:57.823 [AWT-EventQueue-0] DEBUG Step3Controller - Step3Panel: [472977163]
16:21:57.823 [AWT-EventQueue-0] DEBUG Step3Controller - Loading settings for column# 0
16:21:57.823 [AWT-EventQueue-0] TRACE Step3Model - getSelectionForColumn(colIndex:=0)
16:21:57.823 [AWT-EventQueue-0] DEBUG Step3Model - found selection: null
16:21:57.838 [AWT-EventQueue-0] TRACE Model - registerProvider(Step3Model)
16:21:57.839 [AWT-EventQueue-0] TRACE Model -   createArrayListFromArray()
16:21:57.839 [AWT-EventQueue-0] TRACE Model -   saveProvidersInArray()
16:21:57.839 [AWT-EventQueue-0] TRACE BackNextModel - setCurrentStepController(Step3Controller[1289360480])
16:21:57.840 [AWT-EventQueue-0] TRACE Model - removeProvider(Step2Model)
16:21:57.840 [AWT-EventQueue-0] TRACE Model -   createArrayListFromArray()
16:21:57.840 [AWT-EventQueue-0] TRACE Model -   saveProvidersInArray()
16:21:59.587 [AWT-EventQueue-0] TRACE RadioButtonPanel - actionPerformed(cmd: Date & Time)
16:21:59.588 [AWT-EventQueue-0] TRACE CombinationPanel - reInit()
16:21:59.589 [AWT-EventQueue-0] TRACE CombinationPanel - patternChanged(yyyy-MM-dd'T'HH:mm:ss.SSSSSS)
16:21:59.590 [AWT-EventQueue-0] TRACE ParseTestLabel - [957600186].parseValues()
16:21:59.611 [AWT-EventQueue-0] TRACE ParserThread - [2015815920].run() <- parsing values ###########################################################
16:21:59.611 [AWT-EventQueue-0] DEBUG ParserThread - skipping line to parse #0
16:22:03.515 [AWT-EventQueue-0] TRACE BackNextController - nextButtonClicked()


16:22:03.516 [AWT-EventQueue-0] TRACE BackNextModel - getCurrentStepController(): result:Step3Controller[1289360480]
16:22:03.516 [AWT-EventQueue-0] TRACE Step3Controller - saveSettings()
16:22:03.516 [AWT-EventQueue-0] DEBUG Step3Controller - Start:
16:22:03.517 [AWT-EventQueue-0] DEBUG Step3Controller - Step3Model: Step3Model[2004059201]: fLWD: 1, uH: false, mC: 0, cA: {}
16:22:03.517 [AWT-EventQueue-0] DEBUG Step3Controller - Step3Panel: [472977163]
16:22:03.517 [AWT-EventQueue-0] TRACE Step3Model - addSelection()
16:22:03.518 [AWT-EventQueue-0] DEBUG Step3Model - Next two values should be equal: addedValue: "[Date & Time, Combination, yyyy-MM-dd'T'HH:mm:ss.SSSSSSSEP1]"; selection : "[Date & Time, Combination, yyyy-MM-dd'T'HH:mm:ss.SSSSSSSEP1]"; aV==sel? true; aV.equals(sel)? true
16:22:03.519 [AWT-EventQueue-0] INFO  DateAndTimeController - Assign pattern yyyy-MM-dd'T'HH:mm:ss.SSSSSS to Date&Time group null in column[#:0; fLWD:1]
16:22:03.521 [AWT-EventQueue-0] INFO  DateAndTime - Add Year column[#:0; fLWD:1] to Date&Time group null
16:22:03.522 [AWT-EventQueue-0] INFO  DateAndTime - Add Month column[#:0; fLWD:1] to Date&Time group null
16:22:03.523 [AWT-EventQueue-0] INFO  DateAndTime - Add Day column[#:0; fLWD:1] to Date&Time group null
16:22:03.524 [AWT-EventQueue-0] INFO  DateAndTime - Add Hour column[#:0; fLWD:1] to Date&Time group null
16:22:03.525 [AWT-EventQueue-0] INFO  DateAndTime - Add Minute column[#:0; fLWD:1] to Date&Time group null
16:22:03.526 [AWT-EventQueue-0] INFO  DateAndTime - Add Second column[#:0; fLWD:1] to Date&Time group null
16:22:03.527 [AWT-EventQueue-0] DEBUG Step3Controller - End:
16:22:03.527 [AWT-EventQueue-0] DEBUG Step3Controller - Step3Model: Step3Model[2004059201]: fLWD: 1, uH: false, mC: 0, cA: {0=[Date & Time, Combination, yyyy-MM-dd'T'HH:mm:ss.SSSSSSSEP1]}
16:22:03.528 [AWT-EventQueue-0] DEBUG Step3Controller - Step3Panel: null
16:22:03.528 [AWT-EventQueue-0] TRACE Model - updateModel()
16:22:03.529 [AWT-EventQueue-0] TRACE Step3ModelHandler - handleModel()
16:22:03.529 [AWT-EventQueue-0] TRACE Step3Model - getAllSelections()
16:22:03.531 [AWT-EventQueue-0] DEBUG Step3ModelHandler - Added new ColumnAssignments element
16:22:03.532 [AWT-EventQueue-0] TRACE Step3ModelHandler -               getColumnForKey()
16:22:03.537 [AWT-EventQueue-0] TRACE Step3ModelHandler -               setComplexTypeDateAndTime()
16:22:03.543 [AWT-EventQueue-0] TRACE Helper -          addOrUpdateColumnMetadata()
16:22:03.548 [AWT-EventQueue-0] DEBUG Helper - Added column metadata. Key: 'GROUP'; Value: '1' in column '0'.
16:22:03.549 [AWT-EventQueue-0] TRACE Helper -          addOrUpdateColumnMetadata()
16:22:03.549 [AWT-EventQueue-0] DEBUG Helper - Added column metadata. Key: 'PARSE_PATTERN'; Value: 'yyyy-MM-dd'T'HH:mm:ss.SSSSSS' in column '0'.
16:22:03.550 [AWT-EventQueue-0] TRACE Helper -          addOrUpdateColumnMetadata()
16:22:03.550 [AWT-EventQueue-0] DEBUG Helper - Added column metadata. Key: 'TYPE'; Value: 'COMBINATION' in column '0'.
16:22:03.550 [AWT-EventQueue-0] DEBUG Step3ModelHandler - handling of Step3Model finished
16:22:03.550 [AWT-EventQueue-0] TRACE BackNextModel - addPreviousStepController(Step3Controller[1289360480])
16:22:03.551 [AWT-EventQueue-0] TRACE BackNextModel - getFollowingStepController()
16:22:03.551 [AWT-EventQueue-0] TRACE BackNextModel - followingSC: null
16:22:03.552 [AWT-EventQueue-0] TRACE ParseTestLabel - ParseTestLabel()[995488773]
16:22:03.553 [AWT-EventQueue-0] TRACE ParseTestLabel - ParseTestLabel()[441949892]
16:22:03.553 [AWT-EventQueue-0] TRACE ParseTestLabel - ParseTestLabel()[1808224053]
16:22:03.554 [AWT-EventQueue-0] TRACE ParseTestLabel - ParseTestLabel()[1246357988]
16:22:03.558 [AWT-EventQueue-0] TRACE ParseTestLabel - ParseTestLabel()[1579735412]
16:22:03.561 [AWT-EventQueue-0] TRACE ExampleFormatLabel - ExampleFormatLabel(formatter: DateAndTime)
16:22:03.566 [AWT-EventQueue-0] TRACE ParseTestLabel - ParseTestLabel()[876138124]
16:22:03.568 [AWT-EventQueue-0] TRACE ExampleFormatLabel - ExampleFormatLabel(formatter: Position)
16:22:03.570 [AWT-EventQueue-0] TRACE Step3Controller - loadSettings()
16:22:03.570 [AWT-EventQueue-0] DEBUG Step3Controller - Step3Model: Step3Model[855575637]: fLWD: 1, uH: false, mC: 1, cA: {}
16:22:03.571 [AWT-EventQueue-0] DEBUG Step3Controller - Step3Panel: [533363069]
16:22:03.571 [AWT-EventQueue-0] DEBUG Step3Controller - Loading settings for column# 1
16:22:03.571 [AWT-EventQueue-0] TRACE Step3Model - getSelectionForColumn(colIndex:=1)
16:22:03.571 [AWT-EventQueue-0] DEBUG Step3Model - found selection: null
16:22:03.579 [AWT-EventQueue-0] TRACE Model - registerProvider(Step3Model)
16:22:03.579 [AWT-EventQueue-0] TRACE Model -   createArrayListFromArray()
16:22:03.580 [AWT-EventQueue-0] TRACE Model -   saveProvidersInArray()
16:22:03.580 [AWT-EventQueue-0] TRACE BackNextModel - setCurrentStepController(Step3Controller[849401530])
16:22:03.580 [AWT-EventQueue-0] TRACE Model - removeProvider(Step3Model)
16:22:03.580 [AWT-EventQueue-0] TRACE Model -   createArrayListFromArray()
16:22:03.580 [AWT-EventQueue-0] TRACE Model -   saveProvidersInArray()
16:22:05.117 [AWT-EventQueue-0] TRACE RadioButtonPanel - actionPerformed(cmd: Measured Value)
16:22:06.195 [AWT-EventQueue-0] TRACE BackNextController - nextButtonClicked()


16:22:06.195 [AWT-EventQueue-0] TRACE BackNextModel - getCurrentStepController(): result:Step3Controller[849401530]
16:22:06.195 [AWT-EventQueue-0] TRACE Step3Controller - saveSettings()
16:22:06.195 [AWT-EventQueue-0] DEBUG Step3Controller - Start:
16:22:06.195 [AWT-EventQueue-0] DEBUG Step3Controller - Step3Model: Step3Model[855575637]: fLWD: 1, uH: false, mC: 1, cA: {}
16:22:06.195 [AWT-EventQueue-0] DEBUG Step3Controller - Step3Panel: [533363069]
16:22:06.195 [AWT-EventQueue-0] TRACE Step3Model - addSelection()
16:22:06.195 [AWT-EventQueue-0] DEBUG Step3Model - Next two values should be equal: addedValue: "[Measured Value, Numeric Value, 0]"; selection : "[Measured Value, Numeric Value, 0]"; aV==sel? true; aV.equals(sel)? true
16:22:06.195 [AWT-EventQueue-0] INFO  MeasuredValue - In column[#:1; fLWD:1] are Numeric Values
16:22:06.195 [AWT-EventQueue-0] DEBUG Step3Controller - End:
16:22:06.195 [AWT-EventQueue-0] DEBUG Step3Controller - Step3Model: Step3Model[855575637]: fLWD: 1, uH: false, mC: 1, cA: {1=[Measured Value, Numeric Value, 0]}
16:22:06.195 [AWT-EventQueue-0] DEBUG Step3Controller - Step3Panel: null
16:22:06.195 [AWT-EventQueue-0] TRACE Model - updateModel()
16:22:06.195 [AWT-EventQueue-0] TRACE Step3ModelHandler - handleModel()
16:22:06.195 [AWT-EventQueue-0] TRACE Step3Model - getAllSelections()
16:22:06.195 [AWT-EventQueue-0] TRACE Step3ModelHandler -               getColumnForKey()
16:22:06.195 [AWT-EventQueue-0] TRACE Step3ModelHandler -               setComplexColumnTypeMeasuredValue()
16:22:06.195 [AWT-EventQueue-0] TRACE Helper -          addOrUpdateColumnMetadata()
16:22:06.195 [AWT-EventQueue-0] DEBUG Helper - Added column metadata. Key: 'TYPE'; Value: 'NUMERIC' in column '1'.
16:22:06.195 [AWT-EventQueue-0] DEBUG Step3ModelHandler - handling of Step3Model finished
16:22:06.195 [AWT-EventQueue-0] TRACE BackNextModel - addPreviousStepController(Step3Controller[849401530])
16:22:06.195 [AWT-EventQueue-0] TRACE BackNextModel - getFollowingStepController()
16:22:06.195 [AWT-EventQueue-0] TRACE BackNextModel - followingSC: null
16:22:06.195 [AWT-EventQueue-0] TRACE ParseTestLabel - ParseTestLabel()[1996427563]
16:22:06.195 [AWT-EventQueue-0] TRACE ParseTestLabel - ParseTestLabel()[83742738]
16:22:06.195 [AWT-EventQueue-0] TRACE ParseTestLabel - ParseTestLabel()[1194213734]
16:22:06.195 [AWT-EventQueue-0] TRACE ParseTestLabel - ParseTestLabel()[116989566]
16:22:06.195 [AWT-EventQueue-0] TRACE ParseTestLabel - ParseTestLabel()[142684611]
16:22:06.211 [AWT-EventQueue-0] TRACE ExampleFormatLabel - ExampleFormatLabel(formatter: DateAndTime)
16:22:06.211 [AWT-EventQueue-0] TRACE ParseTestLabel - ParseTestLabel()[1645616513]
16:22:06.211 [AWT-EventQueue-0] TRACE ExampleFormatLabel - ExampleFormatLabel(formatter: Position)
16:22:06.211 [AWT-EventQueue-0] TRACE Step3Controller - loadSettings()
16:22:06.211 [AWT-EventQueue-0] DEBUG Step3Controller - Step3Model: Step3Model[1460028884]: fLWD: 1, uH: false, mC: 2, cA: {}
16:22:06.211 [AWT-EventQueue-0] DEBUG Step3Controller - Step3Panel: [423799716]
16:22:06.211 [AWT-EventQueue-0] DEBUG Step3Controller - Loading settings for column# 2
16:22:06.211 [AWT-EventQueue-0] TRACE Step3Model - getSelectionForColumn(colIndex:=2)
16:22:06.211 [AWT-EventQueue-0] DEBUG Step3Model - found selection: null
16:22:06.211 [AWT-EventQueue-0] TRACE Model - registerProvider(Step3Model)
16:22:06.211 [AWT-EventQueue-0] TRACE Model -   createArrayListFromArray()
16:22:06.211 [AWT-EventQueue-0] TRACE Model -   saveProvidersInArray()
16:22:06.211 [AWT-EventQueue-0] TRACE BackNextModel - setCurrentStepController(Step3Controller[751034550])
16:22:06.226 [AWT-EventQueue-0] TRACE Model - removeProvider(Step3Model)
16:22:06.226 [AWT-EventQueue-0] TRACE Model -   createArrayListFromArray()
16:22:06.226 [AWT-EventQueue-0] TRACE Model -   saveProvidersInArray()
16:22:07.164 [AWT-EventQueue-0] TRACE RadioButtonPanel - actionPerformed(cmd: Measured Value)
16:22:09.617 [AWT-EventQueue-0] TRACE BackNextController - nextButtonClicked()


16:22:09.617 [AWT-EventQueue-0] TRACE BackNextModel - getCurrentStepController(): result:Step3Controller[751034550]
16:22:09.617 [AWT-EventQueue-0] TRACE Step3Controller - saveSettings()
16:22:09.617 [AWT-EventQueue-0] DEBUG Step3Controller - Start:
16:22:09.617 [AWT-EventQueue-0] DEBUG Step3Controller - Step3Model: Step3Model[1460028884]: fLWD: 1, uH: false, mC: 2, cA: {}
16:22:09.617 [AWT-EventQueue-0] DEBUG Step3Controller - Step3Panel: [423799716]
16:22:09.617 [AWT-EventQueue-0] TRACE Step3Model - addSelection()
16:22:09.617 [AWT-EventQueue-0] DEBUG Step3Model - Next two values should be equal: addedValue: "[Measured Value, Numeric Value, 0]"; selection : "[Measured Value, Numeric Value, 0]"; aV==sel? true; aV.equals(sel)? true
16:22:09.617 [AWT-EventQueue-0] INFO  MeasuredValue - In column[#:2; fLWD:1] are Numeric Values
16:22:09.617 [AWT-EventQueue-0] TRACE DateAndTimeController - mergeDateAndTimes()
16:22:09.617 [AWT-EventQueue-0] INFO  DateAndTimeController - Merge Date & Times
16:22:09.617 [AWT-EventQueue-0] INFO  PositionController - Merge Positions
16:22:09.617 [AWT-EventQueue-0] DEBUG Step3Controller - End:
16:22:09.617 [AWT-EventQueue-0] DEBUG Step3Controller - Step3Model: Step3Model[1460028884]: fLWD: 1, uH: false, mC: 2, cA: {2=[Measured Value, Numeric Value, 0]}
16:22:09.617 [AWT-EventQueue-0] DEBUG Step3Controller - Step3Panel: null
16:22:09.617 [AWT-EventQueue-0] TRACE Model - updateModel()
16:22:09.617 [AWT-EventQueue-0] TRACE Step3ModelHandler - handleModel()
16:22:09.617 [AWT-EventQueue-0] TRACE Step3Model - getAllSelections()
16:22:09.617 [AWT-EventQueue-0] TRACE Step3ModelHandler -               getColumnForKey()
16:22:09.617 [AWT-EventQueue-0] TRACE Step3ModelHandler -               setComplexColumnTypeMeasuredValue()
16:22:09.617 [AWT-EventQueue-0] TRACE Helper -          addOrUpdateColumnMetadata()
16:22:09.617 [AWT-EventQueue-0] DEBUG Helper - Added column metadata. Key: 'TYPE'; Value: 'NUMERIC' in column '2'.
16:22:09.617 [AWT-EventQueue-0] DEBUG Step3ModelHandler - handling of Step3Model finished
16:22:09.617 [AWT-EventQueue-0] TRACE BackNextModel - addPreviousStepController(Step3Controller[751034550])
16:22:09.617 [AWT-EventQueue-0] TRACE BackNextModel - getFollowingStepController()
16:22:09.617 [AWT-EventQueue-0] TRACE BackNextModel - followingSC: null
16:22:09.617 [AWT-EventQueue-0] INFO  Step4aController - Skip Step 4a since there is just Date&Time group null
16:22:09.617 [AWT-EventQueue-0] INFO  MeasuredValue - Assign Date&Time group null to Numeric Value column[#:1; fLWD:1]
16:22:09.617 [AWT-EventQueue-0] INFO  MeasuredValue - Assign Date&Time group null to Numeric Value column[#:2; fLWD:1]
16:22:09.617 [AWT-EventQueue-0] INFO  Step4bController - Skip Step 4b for Feature Of Interests since there are not any Feature Of Interests
16:22:09.617 [AWT-EventQueue-0] INFO  Step4bController - Skip Step 4b for Observed Propertys since there are not any Observed Propertys
16:22:09.617 [AWT-EventQueue-0] INFO  Step4bController - Skip Step 4b for Unit Of Measurements since there are not any Unit Of Measurements
16:22:09.617 [AWT-EventQueue-0] INFO  Step4bController - Skip Step 4b for Sensors since there are not any Sensors
16:22:09.617 [AWT-EventQueue-0] INFO  Step4cController - Skip Step 4c since there are not any Positions
16:22:09.633 [AWT-EventQueue-0] TRACE Model - registerProvider(Step5aModel)
16:22:09.633 [AWT-EventQueue-0] TRACE Model -   createArrayListFromArray()
16:22:09.633 [AWT-EventQueue-0] TRACE Model -   saveProvidersInArray()
16:22:09.633 [AWT-EventQueue-0] TRACE BackNextModel - setCurrentStepController(Step5aController[952651933])
16:22:09.633 [AWT-EventQueue-0] TRACE Model - removeProvider(Step3Model)
16:22:09.633 [AWT-EventQueue-0] TRACE Model -   createArrayListFromArray()
16:22:09.633 [AWT-EventQueue-0] TRACE Model -   saveProvidersInArray()
16:22:12.023 [AWT-EventQueue-0] TRACE BackNextController - nextButtonClicked()


16:22:12.023 [AWT-EventQueue-0] TRACE BackNextModel - getCurrentStepController(): result:Step5aController[952651933]
16:22:12.023 [AWT-EventQueue-0] INFO  DateAndTime - Add Timezone '0' to Date&Time group null
16:22:12.023 [AWT-EventQueue-0] TRACE Model - updateModel()
16:22:12.023 [AWT-EventQueue-0] TRACE Step5aModelHandler - handleModel()
16:22:12.023 [AWT-EventQueue-0] TRACE Step5aModelHandler - getTableElementFromDateTime()
16:22:12.023 [AWT-EventQueue-0] TRACE Helper - getColumnIdFromTableElement()
16:22:12.023 [AWT-EventQueue-0] TRACE Helper - getColumnById()
16:22:12.023 [AWT-EventQueue-0] TRACE Helper -          addOrUpdateColumnMetadata()
16:22:12.039 [AWT-EventQueue-0] DEBUG Helper - Updated column metadata. Key: 'GROUP'; Value: 'null' in column '0'.
16:22:12.039 [AWT-EventQueue-0] TRACE Helper -          addOrUpdateColumnMetadata()
16:22:12.039 [AWT-EventQueue-0] DEBUG Helper - Added column metadata. Key: 'TIME_ZONE'; Value: '0' in column '0'.
16:22:12.039 [AWT-EventQueue-0] TRACE BackNextModel - addPreviousStepController(Step5aController[952651933])
16:22:12.039 [AWT-EventQueue-0] TRACE BackNextModel - getFollowingStepController()
16:22:12.039 [AWT-EventQueue-0] TRACE BackNextModel - followingSC: null
16:22:12.039 [AWT-EventQueue-0] INFO  Step5cController - Skip Step 5c since there are not any Positions with missing values
16:22:12.039 [AWT-EventQueue-0] INFO  Step6aController - Skip Step 6a since there is at least one Date&Time
16:22:12.039 [AWT-EventQueue-0] INFO  MeasuredValue - Unassign Feature Of Interest from Numeric Value column[#:1; fLWD:1]
16:22:12.039 [AWT-EventQueue-0] TRACE MissingResourcePanel - toListModel()
16:22:12.102 [AWT-EventQueue-0] TRACE Model - registerProvider(Step6bModel)
16:22:12.102 [AWT-EventQueue-0] TRACE Model -   createArrayListFromArray()
16:22:12.102 [AWT-EventQueue-0] TRACE Model -   saveProvidersInArray()
16:22:12.102 [AWT-EventQueue-0] TRACE BackNextModel - setCurrentStepController(Step6bController[1899239827])
16:22:12.102 [AWT-EventQueue-0] TRACE Model - removeProvider(Step5aModel)
16:22:12.102 [AWT-EventQueue-0] TRACE Model -   createArrayListFromArray()
16:22:12.102 [AWT-EventQueue-0] TRACE Model -   saveProvidersInArray()
16:22:15.437 [AWT-EventQueue-0] TRACE BackNextController - nextButtonClicked()


16:22:15.437 [AWT-EventQueue-0] TRACE BackNextModel - getCurrentStepController(): result:Step6bController[1899239827]
16:22:15.437 [AWT-EventQueue-0] TRACE MissingResourcePanel - isUriValid()
16:22:15.437 [AWT-EventQueue-0] TRACE MissingResourcePanel - toURI()
16:22:15.437 [AWT-EventQueue-0] INFO  MeasuredValue - Assign Feature Of Interest "foi-group99" to Numeric Value column[#:1; fLWD:1]
16:22:15.437 [AWT-EventQueue-0] TRACE Model - updateModel()
16:22:15.437 [AWT-EventQueue-0] TRACE Step6bModelHandler - handleModel()
16:22:15.437 [AWT-EventQueue-0] DEBUG Step6bModelHandler - Measured value: "Numeric Value column[#:1; fLWD:1]"; Resource: "Feature Of Interest "foi-group99""
16:22:15.437 [AWT-EventQueue-0] TRACE Helper - getColumnIdFromTableElement()
16:22:15.437 [AWT-EventQueue-0] DEBUG Step6bModelHandler - Column ID of measured value: 1
16:22:15.437 [AWT-EventQueue-0] DEBUG Step6bModelHandler - added new AddtionalMetadata element
16:22:15.437 [AWT-EventQueue-0] TRACE Helper - getColumnById()
16:22:15.437 [AWT-EventQueue-0] TRACE Step6bModelHandler -              addRelatedResource()
16:22:15.437 [AWT-EventQueue-0] TRACE Step6bModelHandler -              addRelatedFOI()
Exception in thread "AWT-EventQueue-0" java.lang.ClassCastException: org.x52North.sensorweb.sos.importer.x05.impl.ResourceTypeImpl cannot be cast to org.x52North.sensorweb.sos.importer.x05.SpatialResourceType
        at org.n52.sos.importer.model.xml.Step6bModelHandler.addRelatedFOI(Step6bModelHandler.java:241)
        at org.n52.sos.importer.model.xml.Step6bModelHandler.addRelatedResource(Step6bModelHandler.java:139)
        at org.n52.sos.importer.model.xml.Step6bModelHandler.handleModel(Step6bModelHandler.java:116)
        at org.n52.sos.importer.model.xml.Model.updateModel(Model.java:312)
        at org.n52.sos.importer.controller.MainController.updateModel(MainController.java:103)
        at org.n52.sos.importer.controller.BackNextController.nextButtonClicked(BackNextController.java:151)
        at org.n52.sos.importer.view.BackNextPanel$NextButtonClicked.actionPerformed(BackNextPanel.java:162)
        at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
        at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
        at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
        at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
        at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
        at java.awt.Component.processMouseEvent(Unknown Source)
        at javax.swing.JComponent.processMouseEvent(Unknown Source)
        at java.awt.Component.processEvent(Unknown Source)
        at java.awt.Container.processEvent(Unknown Source)
        at java.awt.Component.dispatchEventImpl(Unknown Source)
        at java.awt.Container.dispatchEventImpl(Unknown Source)
        at java.awt.Component.dispatchEvent(Unknown Source)
        at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
        at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
        at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
        at java.awt.Container.dispatchEventImpl(Unknown Source)
        at java.awt.Window.dispatchEventImpl(Unknown Source)
        at java.awt.Component.dispatchEvent(Unknown Source)
        at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
        at java.awt.EventQueue.access$500(Unknown Source)
        at java.awt.EventQueue$3.run(Unknown Source)
        at java.awt.EventQueue$3.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
        at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
        at java.awt.EventQueue$4.run(Unknown Source)
        at java.awt.EventQueue$4.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
        at java.awt.EventQueue.dispatchEvent(Unknown Source)
        at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
        at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
        at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
        at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
        at java.awt.EventDispatchThread.run(Unknown Source)

Cannot build importer when host in timezone MST (-07:00)

[...]
-------------------------------------------------------------------------------
Test set: org.n52.sos.importer.feeder.model.TimestampTest
-------------------------------------------------------------------------------
Tests run: 10, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec
<<< FAILURE!
shouldSetAllValuesViaSetLong(org.n52.sos.importer.feeder.model.TimestampTest) 
Time elapsed: 0.005 sec  <<< FAILURE!
java.lang.AssertionError: 
Expected: is "1970-01-02T03:00:31-03:30"
     got: "1970-01-01T20:30:01-03:00"
[...]
shouldCreateDateFromTimestamp(org.n52.sos.importer.feeder.model.TimestampTest) 
Time elapsed: 0 sec  <<< FAILURE!
java.lang.AssertionError: 
Expected: is <0>
     got: <-1>
[...]
shouldAddDayDelta(org.n52.sos.importer.feeder.model.TimestampTest)  Time elapsed: 0.001 sec  <<< FAILURE! 
java.lang.AssertionError: 
Expected: is <3> 
     got: <2>

Source

Null Pointer Exception (NPE) when feeder tries to register a sensor

Setup

Linux RaspberryPi 4.4.16-v7+
Apache Tomcat 7.0.56 (Debian)
Java JDK 7.0
SOS 4.3.7
SOS importer 0.4.0

pi@raspberrypi:~/Desktop/SOS/sosimporter/Feeder $ java -jar
52n-sos-importer-feeder-0.4.0-bin.jar -c
/home/pi/Desktop/SOS/test/temperature.csv.52n-sos-import-config.xml 			
15:17:45.344 [main] INFO  Feeder - Application started			
		Manifest-Version: 1.0"			
		Created-By: 1.7.0_65 (Oracle Corporation)"			
		HeapSize Information: max: 448MB"	 total now: 14MB	 free now: 9MB	 used
now: 4MB
		os.name: Linux"	 os.arch: arm	 os.version: 4.4.16-v7+	
15:17:47.380 [OneTimeFeeder] INFO  OneTimeFeeder - Starting feeding data
from file via configuration 'temperature.csv.52n-sos-import-config.xml' to
SOS instance			
15:17:47.493 [OneTimeFeeder] TRACE SensorObservationService -
SensorObservationService(Configuration
[file=/home/pi/Desktop/SOS/test/temperature.csv.52n-sos-import-config.xml])			
15:17:48.640 [OneTimeFeeder] DEBUG SimpleHttpClient - executing POST method
to 'http://localhost:8080/52n-sos-webapp/service'.			
15:17:57.408 [OneTimeFeeder] DEBUG SensorObservationService -
OperationsMetadata found:
org.n52.oxf.ows.capabilities.OperationsMetadata@1bbe38d			
15:17:57.411 [OneTimeFeeder] DEBUG SensorObservationService - Found all
required operations: (RegisterSensor|InsertSensor), InsertObservation			
15:17:57.429 [OneTimeFeeder] DEBUG OneTimeFeeder - Check counter file
'/home/pi/.SOSImporter/6836f288bdd5b943eaefaaa3151f9dbf'.			
15:17:57.431 [OneTimeFeeder] DEBUG OneTimeFeeder - Counter file does not
exist.			
15:17:57.433 [OneTimeFeeder] TRACE SensorObservationService - importData()			
15:17:57.522 [OneTimeFeeder] DEBUG SensorObservationService - Handling CSV
line #1: [TemperaturesensorAdrian, TemperaturesensorAdrian, 20.10.2016
11:50, 48.14935 11.567826, Temperature, CEL, 24]			
15:17:57.524 [OneTimeFeeder] DEBUG SensorObservationService - Parsing
measured value column 6			
15:17:57.539 [OneTimeFeeder] DEBUG SensorObservationService - Sensor: Sensor
[name=TemperaturesensorAdrian, uri=TemperaturesensorAdrian]			
15:17:57.557 [OneTimeFeeder] DEBUG SensorObservationService - Feature of
Interest: FeatureOfInterest [p=null, name=TemperaturesensorAdrian,
uri=TemperaturesensorAdrian]			
15:17:57.576 [OneTimeFeeder] DEBUG SensorObservationService - Value: 24.0			
15:17:57.634 [OneTimeFeeder] DEBUG SensorObservationService - Timestamp:
2016-10-20T11:50:00+02:00			
15:17:57.645 [OneTimeFeeder] DEBUG SensorObservationService - UomCode:
'UnitOfMeasurement [name=CEL, uri=CEL]'			
15:17:57.656 [OneTimeFeeder] DEBUG SensorObservationService -
ObservedProperty: ObservedProperty [name=Temperature, uri=Temperature]			
15:17:57.679 [OneTimeFeeder] DEBUG SensorObservationService - Offering:
Offering [name=, uri=]			
15:17:57.694 [OneTimeFeeder] DEBUG SensorObservationService - Found '1'
Observed Properties for Sensor 'TemperaturesensorAdrian': '[ObservedProperty
[name=Temperature, uri=Temperature]]'			
15:17:57.698 [OneTimeFeeder] DEBUG SensorObservationService - Found '1'
Measured value types for observed properties of sensor
'TemperaturesensorAdrian': '{ObservedProperty [name=Temperature,
uri=Temperature]=NUMERIC}'.			
15:17:57.703 [OneTimeFeeder] DEBUG SensorObservationService - Found '1'
units of measurement for observed properties of sensor
'TemperaturesensorAdrian': '{ObservedProperty [name=Temperature,
uri=Temperature]=CEL}'.			
Exception in thread "OneTimeFeeder" java.lang.NullPointerException			
	at org.n52.sos.importer.feeder.model.requests.RegisterSensor.getLongitudeUnit(RegisterSensor.java:107)			
	at org.n52.sos.importer.feeder.util.DescriptionBuilder.createSML(DescriptionBuilder.java:79)"			
	at org.n52.sos.importer.feeder.SensorObservationService.createInsertSensorParametersFromRS(SensorObservationService.java:1018)
        at org.n52.sos.importer.feeder.SensorObservationService.registerSensor(SensorObservationService.java:940)			
	at org.n52.sos.importer.feeder.SensorObservationService.insertObservationsForOneLine(SensorObservationService.java:667)			
	at org.n52.sos.importer.feeder.SensorObservationService.importData(SensorObservationService.java:304)"			
	at org.n52.sos.importer.feeder.task.OneTimeFeeder.run(OneTimeFeeder.java:203)"			
	at java.lang.Thread.run(Thread.java:745)"

Files:

BadLocationException when clicking BACK after setting CSV header

I noticed a BadLocationException while using the importer. It is already catched here.

Steps to reproduce

Start the importer from the shell via java -jar ./wizard/target/52n-sos-importer-wizard-bin.jar
Select a .csv file, press Next
Configure the .csv header: set Ignore data until line to 1, press Next
Press Back

Find the following stacktrace in the shell

[AWT-EventQueue-0] ERROR Step2Panel - Exception thrown: No such line
javax.swing.text.BadLocationException: No such line

    at javax.swing.JTextArea.getLineStartOffset(JTextArea.java:409) ~[na:1.7.0_79]
    at org.n52.sos.importer.view.Step2Panel.setCSVFileHighlight(Step2Panel.java:543) [52n-sos-importer-wizard-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.sos.importer.view.Step2Panel$5.stateChanged(Step2Panel.java:362) [52n-sos-importer-wizard-bin.jar:0.5.0-SNAPSHOT]
    at javax.swing.JSpinner.fireStateChanged(JSpinner.java:457) [na:1.7.0_79]
    at javax.swing.JSpinner$ModelListener.stateChanged(JSpinner.java:385) [na:1.7.0_79]
    at javax.swing.AbstractSpinnerModel.fireStateChanged(AbstractSpinnerModel.java:119) [na:1.7.0_79]
    at javax.swing.SpinnerNumberModel.setValue(SpinnerNumberModel.java:447) [na:1.7.0_79]
    at org.n52.sos.importer.view.Step2Panel.setFirstLineWithData(Step2Panel.java:563) [52n-sos-importer-wizard-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.sos.importer.controller.Step2Controller.loadSettings(Step2Controller.java:141) [52n-sos-importer-wizard-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.sos.importer.controller.MainController.setStepController(MainController.java:86) [52n-sos-importer-wizard-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.sos.importer.controller.BackNextController.backButtonClicked(BackNextController.java:96) [52n-sos-importer-wizard-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.sos.importer.view.BackNextPanel$BackButtonClicked.actionPerformed(BackNextPanel.java:129) [52n-sos-importer-wizard-bin.jar:0.5.0-SNAPSHOT]
    at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2018) [na:1.7.0_79]
...

Number of read lines not stored

Hi,

Just few questions because i used the SOSImporter tools. When i run
repettitive Feed from a folder, the number of line isn't stored in the file
in .SOSimporter. The number is already 0 ... do you have a solution please ?

Also a don't have in the folder SOSimporter, the file 52.north< ...

.properties do you know how this file is create ?
Many thanks
Regards

The interesting sections in the log file are:

2016-11-18 14:55:22,005 INFO [main] [Feeder.java:218] Application started
		Implementation-Title: Feeding Module
		Implementation-Version: 0.4.0
		Archiver-Version: Plexus Archiver
		Built-By: debard
		Implementation-Vendor-Id: org.n52.sensorweb
		geotools-version: 9.5
		n52-oxf-version: 2.0.0-alpha.4
		Manifest-Version: 1.0
		Implementation-Vendor: 52°North
		Main-Class: org.n52.sos.importer.feeder.Feeder
		build-time: 2016-10-06_13-08
		Created-By: Apache Maven
		Build-Jdk: 1.8.0_101
		n52-sos-importer-xmlbeans-version: 0.2
		HeapSize Information: max: 1796MB; total now: 123MB; free now: 110MB; used now: 12MB
		os.name: Windows 10; os.arch: amd64; os.version: 10.0
2016-11-18 14:55:22,292 DEBUG [FeederTimer] [RepeatedFeeder.java:187] Last feed file properties not found: C:\Users\USER\.SOSImporter\71ee833fcaee6f6356b2a9810cceb18d.properties
2016-11-18 14:55:22,298 INFO [FeederTimer] [RepeatedFeeder.java:96] Start feeding file GALION_20161118060100798699639660532436.csv
2016-11-18 14:55:22,311 INFO [FeederTimer] [OneTimeFeeder.java:151] Starting feeding data from file via configuration 'config.xml' to SOS instance
2016-11-18 14:55:22,326 TRACE [FeederTimer] [SensorObservationService.java:177] SensorObservationService(Configuration [file=config.xml])
2016-11-18 14:55:22,492 DEBUG [FeederTimer] [SimpleHttpClient.java:158] executing POST method to 'http://example.com/52n-sos-webapp/sos'.
2016-11-18 14:55:24,231 DEBUG [FeederTimer] [SensorObservationService.java:258] OperationsMetadata found: org.n52.oxf.ows.capabilities.OperationsMetadata@24e54449
2016-11-18 14:55:24,237 DEBUG [FeederTimer] [SensorObservationService.java:264] Found all required operations: (RegisterSensor|InsertSensor), InsertObservation
2016-11-18 14:55:24,243 DEBUG [FeederTimer] [OneTimeFeeder.java:190] Check counter file 'C:\Users\USER\.SOSImporter\ee6327faf2a05d5aa19f8812f7bdca84'.
2016-11-18 14:55:24,249 DEBUG [FeederTimer] [OneTimeFeeder.java:199] Counter file does not exist.
2016-11-18 14:55:24,251 TRACE [FeederTimer] [SensorObservationService.java:274] importData()
[...]
2016-11-18 14:55:30,104 DEBUG [FeederTimer] [SensorObservationService.java:319] Timing:
Start File: Fri Nov 18 14:55:24 CET 2016
Finished importing: Fri Nov 18 14:55:30 CET 2016
2016-11-18 14:55:30,108 INFO [FeederTimer] [SensorObservationService.java:387] New observations in SOS: 20. Failed observations: 0.
2016-11-18 14:55:30,124 INFO [FeederTimer] [OneTimeFeeder.java:205] OneTimeFeeder: save read lines count: 0 to 'C:\Users\USER\.SOSImporter\ee6327faf2a05d5aa19f8812f7bdca84'
2016-11-18 14:55:30,131 INFO [FeederTimer] [OneTimeFeeder.java:226] Feeding data from file DATA_20161118060100798699639660532436.csv to SOS instance finished.
2016-11-18 14:55:30,146 INFO [FeederTimer] [RepeatedFeeder.java:169] Saved last used data file: DATA_20161118060100798699639660532436.csv
2016-11-18 14:55:30,150 INFO [FeederTimer] [RepeatedFeeder.java:101] Finished feeding file DATA_20161118060100798699639660532436.csv.
2016-11-18 14:55:30,154 INFO [FeederTimer] [RepeatedFeeder.java:96] Start feeding file DATA_20161118060100995276705863185179.csv
2016-11-18 14:55:30,159 INFO [FeederTimer] [OneTimeFeeder.java:151] Starting feeding data from file via configuration 'config-test-all-sensor-BDL.xml' to SOS instance
2016-11-18 14:55:30,179 TRACE [FeederTimer] [SensorObservationService.java:177] SensorObservationService(Configuration [file=config-test-all-sensor-BDL.xml])
2016-11-18 14:55:30,184 DEBUG [FeederTimer] [SimpleHttpClient.java:158] executing POST method to 'http://example.com/52n-sos-webapp/sos'.
2016-11-18 14:55:30,555 DEBUG [FeederTimer] [SensorObservationService.java:258] OperationsMetadata found: org.n52.oxf.ows.capabilities.OperationsMetadata@3194b65d
2016-11-18 14:55:30,560 DEBUG [FeederTimer] [SensorObservationService.java:264] Found all required operations: (RegisterSensor|InsertSensor), InsertObservation
2016-11-18 14:55:30,566 DEBUG [FeederTimer] [OneTimeFeeder.java:190] Check counter file 'C:\Users\USER\.SOSImporter\c64f3267f43cc549dc81dc6b57c69b4d'.
2016-11-18 14:55:30,569 DEBUG [FeederTimer] [OneTimeFeeder.java:199] Counter file does not exist.
[...]

Why do we have two different counter files for the same datafile and configuration?

C:\Users\USER\.SOSImporter\ee6327faf2a05d5aa19f8812f7bdca84
C:\Users\USER\.SOSImporter\c64f3267f43cc549dc81dc6b57c69b4d

Document usage of folder ~/.SOSImporter/

While using the importer I found out that some of the configuration data is stored in ~/.SOSImporter/. I noticed that strings entered as identifier and URI names as shown in the screenshot of the wizard are stored in the file 52n-sensorweb-sos-importer.properties.

I could however not find any documentation concerning the files stored in the folder.

Documentation request

Please describe what the files in this folder are used for.
Please describe what values are stored in the individual files.

/cc @CarstenHollmann

NPE when optional parameter -d is not used

i can't launch the jar without specifying the optional -d datafile anymore. It cause a NPE because of a log added recently printing the datafile before extracting it from configuration.
Exception in thread "OneTimeFeeder" java.lang.NullPointerException at org.n52.sos.importer.feeder.task.OneTimeFeeder.run(OneTimeFeeder.java:176)

Add working configuration file for demo-data section in documentation

#55 contains the request to add a working configuration example:

For convenience, you could serve a ready-to-use config-file for the feeder in section Demo data.

Update user guide

I am trying to follow the user guide available in the example tutorial which seems to be outdated:

SOS Importer 0.1.0, document revision 0.1, 2011-09-13

Therefore, I am not able to answer these questions showing up in the configuration file wizard:

What is the Feature Of Interest for the marked measured value column?
What is the Observed Property for the marked measured value column?
What is the Unit Of Measurement for the marked measured value column?

If I accept the defaults and start the actual import process with the generated configuration file I end up with a program failure as reported in this issue.

It would probably a good idea to provide a configuration file with the demo data. This will enable beginners to run the demo import independent from understanding the importer.

Strategy "SweArrayObservationWithSplitExtension" doesn't work with TextObservations

Current implementation results in HTTP 500 error while importing TextObservations using import strategy "SweArrayObservationWithSplitExtension".

Log output:

2015-05-18 11:05:15,953 DEBUG [FeederTimer] [SimpleHttpClient.java:158] executing POST method to '[...]'.
2015-05-18 11:05:16,046 ERROR [FeederTimer] [SensorObservationService.java:794] Problem with OXF. Exception thrown: Could not parse response to XML.
org.n52.oxf.OXFException: Could not parse response to XML.
    at org.n52.oxf.sos.adapter.SOSAdapter.doOperation(SOSAdapter.java:387) ~[52n-sos-importer-feeder-0.5.0-SNAPSHOT-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.oxf.sos.adapter.SOSAdapter.doOperation(SOSAdapter.java:437) ~[52n-sos-importer-feeder-0.5.0-SNAPSHOT-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.oxf.sos.adapter.wrapper.SOSWrapper.doInsertObservation(SOSWrapper.java:319) ~[52n-sos-importer-feeder-0.5.0-SNAPSHOT-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.sos.importer.feeder.SensorObservationService.insertSweArrayObservation(SensorObservationService.java:746) [52n-sos-importer-feeder-0.5.0-SNAPSHOT-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.sos.importer.feeder.SensorObservationService.insertTimeSeries(SensorObservationService.java:643) [52n-sos-importer-feeder-0.5.0-SNAPSHOT-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.sos.importer.feeder.SensorObservationService.importData(SensorObservationService.java:376) [52n-sos-importer-feeder-0.5.0-SNAPSHOT-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.sos.importer.feeder.task.OneTimeFeeder.run(OneTimeFeeder.java:203) [52n-sos-importer-feeder-0.5.0-SNAPSHOT-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.sos.importer.feeder.task.RepeatedFeeder.run(RepeatedFeeder.java:98) [52n-sos-importer-feeder-0.5.0-SNAPSHOT-bin.jar:0.5.0-SNAPSHOT]
    at java.util.TimerThread.mainLoop(Unknown Source) [na:1.7.0_67]
    at java.util.TimerThread.run(Unknown Source) [na:1.7.0_67]
Caused by: org.apache.xmlbeans.XmlException: error: </body> does not close tag <HR>.
    at org.apache.xmlbeans.impl.store.Locale$SaxLoader.load(Locale.java:3486) ~[52n-sos-importer-feeder-0.5.0-SNAPSHOT-bin.jar:0.5.0-SNAPSHOT]
    at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:1276) ~[52n-sos-importer-feeder-0.5.0-SNAPSHOT-bin.jar:0.5.0-SNAPSHOT]
    at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:1263) ~[52n-sos-importer-feeder-0.5.0-SNAPSHOT-bin.jar:0.5.0-SNAPSHOT]
    at org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:345) ~[52n-sos-importer-feeder-0.5.0-SNAPSHOT-bin.jar:0.5.0-SNAPSHOT]
    at org.apache.xmlbeans.XmlObject$Factory.parse(XmlObject.java:663) ~[52n-sos-importer-feeder-0.5.0-SNAPSHOT-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.oxf.sos.adapter.SOSAdapter.doOperation(SOSAdapter.java:381) ~[52n-sos-importer-feeder-0.5.0-SNAPSHOT-bin.jar:0.5.0-SNAPSHOT]
    ... 9 common frames omitted
Caused by: org.xml.sax.SAXParseException: </body> does not close tag <HR>.
    at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.reportFatalError(Piccolo.java:1038) ~[52n-sos-importer-feeder-0.5.0-SNAPSHOT-bin.jar:0.5.0-SNAPSHOT]
    at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.parse(Piccolo.java:723) ~[52n-sos-importer-feeder-0.5.0-SNAPSHOT-bin.jar:0.5.0-SNAPSHOT]
    at org.apache.xmlbeans.impl.store.Locale$SaxLoader.load(Locale.java:3454) ~[52n-sos-importer-feeder-0.5.0-SNAPSHOT-bin.jar:0.5.0-SNAPSHOT]
    ... 14 common frames omitted
2015-05-18 11:05:16,046 ERROR [FeederTimer] [SensorObservationService.java:645] Insert observation failed for sensor '[...]. Store: TimeSeries [sensor=wwu-ws-kli-hsb, observedProperty=ObservedProperty [name=[...], uri=[...]], feature=[...]]

Allow to resize application window

Very long identifier and URI names push some of the controls out of the window. They cannot be reached any more as indicated in the screenshot.

Please allow to resize the application window.

Count property prompts for unit of measurement

One of the properties in my .csv file contains a count value (column "partikelPM10Ueberschreitungen"):

So I have chosen "Measured Value" and "Count".
Then later when the wizard hits step 6b "Unit Of Measurement" I am forced to choose a unit for the mentioned column as shown in the screenshot:

Do I misunderstand what "Count" should be used for? Or is this a bug?

Skip empty cells, do not skip whole row

I am working with a data set which has multiple phenomena per station. Some station do not provide any data (empty cells, red in the screenshot) for certain phenomenas. Other stations do but lack data for other phenomena. This can be seen from the screenshot of the tabular data.

It also happens that a station misses data for a phenomena for a certain day while data is available for the same phenomena on all the other days. This can been seen in the following screenshot:

Status quo

At the current state the importer skips the whole row if it lacks data for at least one of the columns.

Skip CSV line (2) #2751; Reason: Line is empty.; Raw data: '[32, 2015-06-19, , , , , 13.0, 18.0, , , , , , , 2.0, 6.0]'

Workaround

I could enter 0 for these cells but that would actually be an incorrect statement.

Change request

Please allow to import all available values from a row. Skip empty cells, continue processing the next cell.

Issue #43: Minimum requirements for a successful import

Minimum requirements for a successful import

I did several attempts to import data from a CSV file. I studied the source code of the importer to follow the various checks. Finally, I have to admit that it is very hard to guess as a random user what are the minimum requirements to successfully import data. I ask myself question such as:

Can there be empty cells in a data row?
Can there be 0 values in cells within a data row?
Can there be data rows from multiple sensors in one CSV file?
Why is a data row skipped on a second try if has not been imported successfully before?

Please clearly write down the minimum requirements for the input data. I appreciate the demo data you provide - please provide another example with regards to my questions.

/cc @nuest

Add GUI for new features

Toogle to activate import strategy "SweArrayObservationWithSplitExtension"
Input field to define "hunkSize"
Input field to define "headerLine"
Toogle to activate "useDateFromLastModifiedDate"
Input field to define "lastModifiedDelta"
Input field to define "regExDateInfoInFileName" and "dateInfoPattern"
Input field for "sweArrayObservationTimeoutBuffer"
Elements for Sample based data file attributes

Implement support for sample based files in the GUI

Subtask of #16

Write blog post about release 0.4.0

Write a blog post which informs about the release and the new features.

Hardcoded time zone in test

https://github.com/52North/sos-importer/blob/master/52n-sos-importer-feeder/src/test/java/org/n52/sos/importer/feeder/model/TimestampTest.java#L48

This breaks on a computer in a different time zone.

NoSuchMethodException SOS Importer Wizard Module

I am new to 52n SOS and I have installed SOS version 4.3.3. Java version is 8. I am trying to import data from example.csv file using the wizard jar file and getting the following exception:

14:09:22.076 [main] INFO Feeder - Application started
Implementation-Title: Feeding Module
Implementation-Version: 0.4.1
Archiver-Version: Plexus Archiver
Built-By: eike
Implementation-Vendor-Id: org.n52.sensorweb
geotools-version: 9.5
n52-oxf-version: 2.0.0-alpha.4
Manifest-Version: 1.0
Implementation-Vendor: 52°North
Main-Class: org.n52.sos.importer.feeder.Feeder
build-time: 2018-11-14_16-10
Created-By: Apache Maven
Build-Jdk: 1.8.0_181
n52-sos-importer-xmlbeans-version: 0.2
HeapSize Information: max: 3540MB; total now: 240MB; free now: 227MB; used now: 12MB
os.name: Linux; os.arch: amd64; os.version: 4.15.0-39-generic
14:09:22.285 [OneTimeFeeder] INFO OneTimeFeeder - Starting feeding data from file via configuration 'example-data.csv.52n-sos-import-config.xml' to SOS instance
14:09:22.293 [OneTimeFeeder] TRACE SensorObservationService - SensorObservationService(Configuration [file=/opt/sos-importer-master/example-data.csv.52n-sos-import-config.xml])
Exception in thread "OneTimeFeeder" java.lang.NoSuchMethodError: org.apache.http.util.VersionInfo.getUserAgent(Ljava/lang/String;Ljava/lang/String;Ljava/lang/Class;)Ljava/lang/String;
at org.apache.http.impl.client.DefaultHttpClient.setDefaultHttpParams(DefaultHttpClient.java:185)
at org.apache.http.impl.client.DefaultHttpClient.createHttpParams(DefaultHttpClient.java:160)
at org.apache.http.impl.client.AbstractHttpClient.getParams(AbstractHttpClient.java:418)
at org.n52.oxf.util.web.SimpleHttpClient.(SimpleHttpClient.java:91)
at org.n52.oxf.util.web.SimpleHttpClient.(SimpleHttpClient.java:77)
at org.n52.oxf.util.web.SimpleHttpClient.(SimpleHttpClient.java:67)
at org.n52.oxf.sos.adapter.SOSAdapter.(SOSAdapter.java:163)
at org.n52.oxf.sos.adapter.SOSAdapter.(SOSAdapter.java:134)
at org.n52.oxf.sos.adapter.wrapper.SosWrapperFactory.doGetCapabilities(SosWrapperFactory.java:83)
at org.n52.oxf.sos.adapter.wrapper.SosWrapperFactory.newInstance(SosWrapperFactory.java:65)
at org.n52.sos.importer.feeder.SensorObservationService.(SensorObservationService.java:196)
at org.n52.sos.importer.feeder.task.OneTimeFeeder.run(OneTimeFeeder.java:167)
at java.lang.Thread.run(Thread.java:748)

Please help me insert data into SOS using a csv file.

NullPointerException when parsing Date/Time

I am facing some issues when trying to import data from a csv file related to parsing different datetime formats. I tried both release 0.4.0 and a fresh build I created from develop/abdbd0ad.

The SOS I am trying to import to is v4.4.0.

The import works fine with both versions for the following file and configuration:

CSV

datetime;temp;humidity
2017-04-13T08:27:28;20.80;39.20
2017-04-13T08:28:05;20.70;39.10
2017-04-13T08:28:36;20.70;39.10
2017-04-13T08:29:06;20.70;39.10
2017-04-13T08:29:37;20.70;39.10

Feeder config

<!-- ... -->
<CsvMetadata>
    <ColumnAssignments>
        <Column>
            <Number>0</Number>
            <Type>DATE_TIME</Type>
            <Metadata>
                <Key>GROUP</Key>
                <Value>1</Value>
            </Metadata>
            <Metadata>
                <Key>PARSE_PATTERN</Key>
                <Value>yyyy-MM-dd'T'HH:mm:ss</Value>
            </Metadata>
            <Metadata>
                <Key>TYPE</Key>
                <Value>COMBINATION</Value>
            </Metadata>
            <Metadata>
                <Key>TIME_ZONE</Key>
                <Value>0</Value>
            </Metadata>
        </Column>
<!-- ... -->

For the following combination, however, both versions produce an exception.
For the given example, the wizard of both versions tells me that all values could be interpreted, but, the feeder module produces an exception, when trying to import the data.

Am I missing something here, or is this a bug?

CSV

datetime;temp;humidity
2017-04-13T08:27:28Z;20.80;39.20
2017-04-13T08:28:05Z;20.70;39.10
2017-04-13T08:28:36Z;20.70;39.10
2017-04-13T08:29:06Z;20.70;39.10
2017-04-13T08:29:37Z;20.70;39.10
2017-04-13T08:30:07Z;20.70;39.10

Feeder config

<!-- ... -->
<CsvMetadata>
    <ColumnAssignments>
        <Column>
            <Number>0</Number>
            <Type>DATE_TIME</Type>
            <Metadata>
                <Key>GROUP</Key>
                <Value>1</Value>
            </Metadata>
            <Metadata>
                <Key>PARSE_PATTERN</Key>
                <Value>yyyy-MM-dd'T'HH:mm:ss'Z'</Value>
            </Metadata>
            <Metadata>
                <Key>TYPE</Key>
                <Value>COMBINATION</Value>
            </Metadata>
        </Column>
<!-- ... -->

Exception version 0.4.0

18:23:50.832 [OneTimeFeeder] TRACE SensorObservationService - importData()
18:23:50.848 [OneTimeFeeder] DEBUG SensorObservationService - Using hunkSize '5000'
18:23:50.848 [OneTimeFeeder] DEBUG SensorObservationService - Handling CSV line #2: [2017-04-13T08:27:28Z, 20.80, 39.20]
18:23:50.848 [OneTimeFeeder] DEBUG SensorObservationService - Parsing measured value column 1
18:23:50.848 [OneTimeFeeder] DEBUG SensorObservationService - Sensor: Sensor [name=dht-22, uri=http://wiki.seeed.cc/Grove-Temperature_and_Humidity_Sensor_Pro/]
18:23:50.864 [OneTimeFeeder] DEBUG SensorObservationService - Feature of Interest: FeatureOfInterest [p=Position [values=[11.549077033996582, 48.100852966308594, 525.0], units=[deg, deg, meters], epsgCode=4326], name=foi-group99, uri=urn:foi:group99]
18:23:50.864 [OneTimeFeeder] DEBUG SensorObservationService - Value: 20.8
18:23:50.864 [OneTimeFeeder] DEBUG SensorObservationService - Timestamp: 2017-04-13T08:27:28
18:23:50.864 [OneTimeFeeder] DEBUG SensorObservationService - UomCode: 'UnitOfMeasurement [name=degC, uri=urn:ucum:temperature:celsius]'
18:23:50.879 [OneTimeFeeder] DEBUG SensorObservationService - ObservedProperty: ObservedProperty [name=temperature, uri=temperature]
18:23:50.879 [OneTimeFeeder] DEBUG SensorObservationService - Offering: Offering [name=dht-22, uri=http://wiki.seeed.cc/Grove-Temperature_and_Humidity_Sensor_Pro/]
18:23:50.879 [OneTimeFeeder] DEBUG SensorObservationService - Parsing measured value column 2
18:23:50.879 [OneTimeFeeder] DEBUG SensorObservationService - Sensor: Sensor [name=dht-22, uri=http://wiki.seeed.cc/Grove-Temperature_and_Humidity_Sensor_Pro/]
18:23:50.879 [OneTimeFeeder] DEBUG SensorObservationService - Feature of Interest: FeatureOfInterest [p=Position [values=[11.549077033996582, 48.100852966308594, 525.0], units=[deg, deg, meters], epsgCode=4326], name=foi-group99, uri=urn:foi:group99]
18:23:50.879 [OneTimeFeeder] DEBUG SensorObservationService - Value: 39.2
18:23:50.879 [OneTimeFeeder] DEBUG SensorObservationService - Timestamp: 2017-04-13T08:27:28

...
...

18:23:51.176 [OneTimeFeeder] DEBUG SensorObservationService - Timestamp: 2017-04-13T08:45:27
18:23:51.176 [OneTimeFeeder] DEBUG SensorObservationService - UomCode: 'UnitOfMeasurement [name=degC, uri=urn:ucum:temperature:celsius]'
18:23:51.176 [OneTimeFeeder] DEBUG SensorObservationService - ObservedProperty: ObservedProperty [name=temperature, uri=temperature]
18:23:51.176 [OneTimeFeeder] DEBUG SensorObservationService - Offering: Offering [name=dht-22, uri=http://wiki.seeed.cc/Grove-Temperature_and_Humidity_Sensor_Pro/]
18:23:51.176 [OneTimeFeeder] DEBUG SensorObservationService - Parsing measured value column 2
18:23:51.176 [OneTimeFeeder] DEBUG SensorObservationService - Sensor: Sensor [name=dht-22, uri=http://wiki.seeed.cc/Grove-Temperature_and_Humidity_Sensor_Pro/]
18:23:51.176 [OneTimeFeeder] DEBUG SensorObservationService - Feature of Interest: FeatureOfInterest [p=Position [values=[11.549077033996582, 48.100852966308594, 525.0], units=[deg, deg, meters], epsgCode=4326], name=foi-group99, uri=urn:foi:group99]
18:23:51.176 [OneTimeFeeder] DEBUG SensorObservationService - Value: 39.1
18:23:51.176 [OneTimeFeeder] DEBUG SensorObservationService - Timestamp: 2017-04-13T08:45:27
18:23:51.176 [OneTimeFeeder] DEBUG SensorObservationService - UomCode: 'UnitOfMeasurement [name=%, uri=urn:ucum:percent]'
18:23:51.176 [OneTimeFeeder] DEBUG SensorObservationService - ObservedProperty: ObservedProperty [name=humidity, uri=urn:humidity]
18:23:51.176 [OneTimeFeeder] DEBUG SensorObservationService - Offering: Offering [name=dht-22, uri=http://wiki.seeed.cc/Grove-Temperature_and_Humidity_Sensor_Pro/]
18:23:51.176 [OneTimeFeeder] DEBUG SensorObservationService - HeapSize Information: max: 7282MB; total now: 491MB; free now: 426MB; used now: 64MB
18:23:51.176 [OneTimeFeeder] TRACE SensorObservationService - insertTimeSeries()
18:23:51.442 [OneTimeFeeder] DEBUG SimpleHttpClient - executing POST method to 'http://xxx/service/pox'.
18:23:51.740 [OneTimeFeeder] DEBUG SensorObservationService - InsertSensorResponse parsed
18:23:51.743 [OneTimeFeeder] INFO  SensorObservationService - Sensor registered at SOS  'http://xxx/service' with assigned id 'http://wiki.seeed.cc/Grove-Temperature_and_Humidity_Sensor_Pro/'
Exception in thread "OneTimeFeeder" java.lang.NullPointerException
	at org.n52.oxf.valueDomains.time.TimePosition.toISO8601Format(TimePosition.java:447)
	at org.n52.oxf.sos.adapter.v200.SOSRequestBuilder200POX.addResultTimeObject(SOSRequestBuilder200POX.java:608)
	at org.n52.oxf.sos.adapter.v200.SOSRequestBuilder200POX.addResultTime(SOSRequestBuilder200POX.java:588)
	at org.n52.oxf.sos.adapter.v200.SOSRequestBuilder200POX.addObservations(SOSRequestBuilder200POX.java:334)
	at org.n52.oxf.sos.adapter.v200.SOSRequestBuilder200POX.buildInsertObservationRequest(SOSRequestBuilder200POX.java:290)
	at org.n52.oxf.sos.adapter.v200.SOSRequestBuilder_200.buildInsertObservationRequest(SOSRequestBuilder_200.java:111)
	at org.n52.oxf.sos.adapter.SOSAdapter.buildRequest(SOSAdapter.java:499)
	at org.n52.oxf.sos.adapter.SOSAdapter.doOperation(SOSAdapter.java:313)
	at org.n52.oxf.sos.adapter.SOSAdapter.doOperation(SOSAdapter.java:437)
	at org.n52.oxf.sos.adapter.wrapper.SOSWrapper.doInsertObservation(SOSWrapper.java:319)
	at org.n52.sos.importer.feeder.SensorObservationService.insertSweArrayObservation(SensorObservationService.java:746)
	at org.n52.sos.importer.feeder.SensorObservationService.insertTimeSeries(SensorObservationService.java:643)
	at org.n52.sos.importer.feeder.SensorObservationService.importData(SensorObservationService.java:376)
	at org.n52.sos.importer.feeder.task.OneTimeFeeder.run(OneTimeFeeder.java:203)
	at java.lang.Thread.run(Unknown Source)

Exception develop/abdbd0ad

18:19:57.847 [OneTimeFeeder] DEBUG FileHelper - Shortened String 'D_pi_importer_config_Z_pi01-dht22-Z.xml_D_pi_importer_config_Z_dht22-small-Z.csv_counter' to '0eec5bc0932069768bca7cdcf9461d86'
18:19:57.847 [OneTimeFeeder] DEBUG OneTimeFeeder - Check counter file 'C:\Users\bruno\.SOSImporter\0eec5bc0932069768bca7cdcf9461d86'.
18:19:57.847 [OneTimeFeeder] DEBUG OneTimeFeeder - Counter file does not exist.
18:19:57.847 [OneTimeFeeder] TRACE SensorObservationService - importData()
18:19:57.847 [OneTimeFeeder] DEBUG SensorObservationService - Using hunkSize '5000'
18:19:57.863 [OneTimeFeeder] DEBUG SensorObservationService - Handling CSV line #2: [2017-04-13T08:27:28Z, 20.80, 39.20]
18:19:57.863 [OneTimeFeeder] DEBUG SensorObservationService - Parsing measured value column 1
18:19:57.863 [OneTimeFeeder] DEBUG SensorObservationService - Sensor: Sensor [name=dht-22, uri=http://wiki.seeed.cc/Grove-Temperature_and_Humidity_Sensor_Pro/]
18:19:57.894 [OneTimeFeeder] DEBUG SensorObservationService - Feature of Interest: FeatureOfInterest [p=Position [values=[11.549077033996582, 48.100852966308594, 525.0], units=[deg, deg, meters], epsgCode=4326], name=foi-group99, uri=urn:foi:group99]
18:19:57.894 [OneTimeFeeder] DEBUG SensorObservationService - Value: 20.8
18:19:57.894 [OneTimeFeeder] ERROR SensorObservationService - Could not retrieve all information required for insert observation because of parsing error: java.text.ParseException: Unparseable date: "2017-04-13T08:27:28+0100". Skipped this one.
18:19:57.894 [OneTimeFeeder] DEBUG SensorObservationService - Exception stack trace:
java.text.ParseException: Unparseable date: "2017-04-13T08:27:28+0100"
	at java.text.DateFormat.parse(Unknown Source)
	at org.n52.sos.importer.feeder.DataFile.parseTimestampComponent(DataFile.java:976)
	at org.n52.sos.importer.feeder.DataFile.handleDateTimeCombination(DataFile.java:508)
	at org.n52.sos.importer.feeder.DataFile.getTimeStamp(DataFile.java:481)
	at org.n52.sos.importer.feeder.SensorObservationService.getInsertObservationForColumnIdFromValues(SensorObservationService.java:688)
	at org.n52.sos.importer.feeder.SensorObservationService.getInsertObservations(SensorObservationService.java:659)
	at org.n52.sos.importer.feeder.SensorObservationService.importData(SensorObservationService.java:368)
	at org.n52.sos.importer.feeder.task.OneTimeFeeder.run(OneTimeFeeder.java:228)
	at java.lang.Thread.run(Unknown Source)
18:19:57.894 [OneTimeFeeder] DEBUG SensorObservationService - Parsing measured value column 2

Bad srsName forging in insertObservation request

When the sos-importer send an InsertObservation request, the srsName of the feature of interest is malformed :
http://www.opengis.net/def/crs/EPSG/0/http://www.opengis.net/def/crs/EPSG/0/4326

The prefix is added twice.
I join the datafile, configuration file and an xml file containing the insertObservation request
config-files.zip

MissingPositionPanel does not show map / HTTP 503 for WMS

When I run the importer (built from source 99469b6) there is no map shown in the "Add missing position" step as shown in the screenshot.

The console output reveals that the WMS backend is not reachable:

15:26:03.881 [AWT-EventQueue-0] DEBUG Constants - WMS url: 
    'http://129.206.228.72/cached/osm'
15:26:35.207 [AWT-EventQueue-0] ERROR MissingPositionPanel - Exception thrown: 
    Server returned HTTP response code: 503 for URL: 
    http://129.206.228.72/cached/osm?REQUEST=GetCapabilities&VERSION=1.1.0&SERVICE=WMS
java.io.IOException: Server returned HTTP response code: 503 for URL: 
    http://129.206.228.72/cached/osm?REQUEST=GetCapabilities&VERSION=1.1.0&SERVICE=WMS

    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1838) ~[na:1.8.0_40-internal]
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1439) ~[na:1.8.0_40-internal]
    at org.geotools.data.ows.SimpleHttpClient$SimpleHTTPResponse.<init>(SimpleHttpClient.java:170) ~[52n-sos-importer-core-0.4.0-SNAPSHOT-bin.jar:0.4.0-SNAPSHOT]
    at org.geotools.data.ows.SimpleHttpClient.get(SimpleHttpClient.java:102) ~[52n-sos-importer-core-0.4.0-SNAPSHOT-bin.jar:0.4.0-SNAPSHOT]
    at org.geotools.data.ows.AbstractOpenWebService.internalIssueRequest(AbstractOpenWebService.java:424) ~[52n-sos-importer-core-0.4.0-SNAPSHOT-bin.jar:0.4.0-SNAPSHOT]
    at org.geotools.data.wms.WebMapServer.issueRequest(WebMapServer.java:419) ~[52n-sos-importer-core-0.4.0-SNAPSHOT-bin.jar:0.4.0-SNAPSHOT]
    at org.geotools.data.ows.AbstractOpenWebService.negotiateVersion(AbstractOpenWebService.java:260) ~[52n-sos-importer-core-0.4.0-SNAPSHOT-bin.jar:0.4.0-SNAPSHOT]
    at org.geotools.data.ows.AbstractOpenWebService.<init>(AbstractOpenWebService.java:110) ~[52n-sos-importer-core-0.4.0-SNAPSHOT-bin.jar:0.4.0-SNAPSHOT]
    at org.geotools.data.ows.AbstractOpenWebService.<init>(AbstractOpenWebService.java:75) ~[52n-sos-importer-core-0.4.0-SNAPSHOT-bin.jar:0.4.0-SNAPSHOT]
    at org.geotools.data.wms.WebMapServer.<init>(WebMapServer.java:370) ~[52n-sos-importer-core-0.4.0-SNAPSHOT-bin.jar:0.4.0-SNAPSHOT]
    at org.n52.sos.importer.view.position.MissingPositionPanel.initMap(MissingPositionPanel.java:246) [52n-sos-importer-core-0.4.0-SNAPSHOT-bin.jar:0.4.0-SNAPSHOT]
    at org.n52.sos.importer.view.position.MissingPositionPanel.initMapPanel(MissingPositionPanel.java:139) [52n-sos-importer-core-0.4.0-SNAPSHOT-bin.jar:0.4.0-SNAPSHOT]
    at org.n52.sos.importer.view.position.MissingPositionPanel.<init>(MissingPositionPanel.java:109) [52n-sos-importer-core-0.4.0-SNAPSHOT-bin.jar:0.4.0-SNAPSHOT]
    at org.n52.sos.importer.view.Step6cPanel.<init>(Step6cPanel.java:63) [52n-sos-importer-core-0.4.0-SNAPSHOT-bin.jar:0.4.0-SNAPSHOT]
    at org.n52.sos.importer.controller.Step6cController.loadSettings(Step6cController.java:76) [52n-sos-importer-core-0.4.0-SNAPSHOT-bin.jar:0.4.0-SNAPSHOT]
    at org.n52.sos.importer.controller.MainController.setStepController(MainController.java:81) [52n-sos-importer-core-0.4.0-SNAPSHOT-bin.jar:0.4.0-SNAPSHOT]
    at org.n52.sos.importer.controller.BackNextController.nextButtonClicked(BackNextController.java:131) [52n-sos-importer-core-0.4.0-SNAPSHOT-bin.jar:0.4.0-SNAPSHOT]
    at org.n52.sos.importer.view.BackNextPanel$NextButtonClicked.actionPerformed(BackNextPanel.java:130) [52n-sos-importer-core-0.4.0-SNAPSHOT-bin.jar:0.4.0-SNAPSHOT]
        ...

Failed import of new sensor

Hello,

I'm trying to upload a very simple csv but I'm running into issues, specifically with importing a new sensor. The error states that the sensor could not be registered at SOS but I can't find any reason why.

The csv is very simple. It consists of a single observation made for a single sensor at a single point in time. Attached are my configuration, log and csv files that I have been using .

SOSimporttest.tar.gz

any help would be very much appreciated.

Parsing Unix time fails

I am using SOS Importer (https://wiki.52north.org/bin/view/SensorWeb/SosImporter) to import observations from a CSV file. The Date & Time format in my CSV file is in UNIX Time. However, it fails to import and gives the error message, stating unparseable date. I selected appropriate option 'UNIX Time' in step 3 of SOS Import process.
Could you please suggest, how can I import UNIX Time using SOS Importer. I have pasted complete error message below.
Error Log:

15:32:42.138 [OneTimeFeeder] DEBUG SensorObservationService - HeapSize
Information: max: 247MB; total now: 15MB; free now: 2MB; used now: 13MB
15:32:42.138 [OneTimeFeeder] DEBUG SensorObservationService - Handling CSV line #2: [Wind Speed, 1465468240, 1.610000, Kph]
15:32:42.138 [OneTimeFeeder] DEBUG SensorObservationService - Parsing measured value column 2
15:32:42.139 [OneTimeFeeder] DEBUG SensorObservationService - Sensor: Sensor [name=Offering_mySensor, uri=Offering_mySensor]
15:32:42.141 [OneTimeFeeder] DEBUG SensorObservationService - Feature of Interest: FeatureOfInterest [p=Position [values=[-0.011892000213265419, 51.53843688964844, 0.0], units=[deg, deg, meters], epsgCode=4326], name=mySensor_London, uri=mySensor_London]
15:32:42.141 [OneTimeFeeder] DEBUG SensorObservationService - Value: 1.61
15:32:42.142 [OneTimeFeeder] ERROR SensorObservationService - Could not retrieve all information required for insert observation because of parsing error: java.text.ParseException: Unparseable date: "1465468240". Skipped this one.
15:32:42.143 [OneTimeFeeder] DEBUG SensorObservationService - Exception stack trace:
java.text.ParseException: Unparseable date: "1465468240"
	at java.text.DateFormat.parse(Unknown Source) ~[na:1.8.0_91]
	at org.n52.sos.importer.feeder.DataFile.parseTimestampComponent(DataFile.java:846) ~[52n-sos-importer-feeder-0.5.0-20150807.134409-5-bin.jar:0.5.0-SNAPSHOT]
	at org.n52.sos.importer.feeder.DataFile.getTimeStamp(DataFile.java:418) ~[52n-sos-importer-feeder-0.5.0-20150807.134409-5-bin.jar:0.5.0-SNAPSHOT]
	at org.n52.sos.importer.feeder.SensorObservationService.getInsertObservationForColumnIdFromValues(SensorObservationService.java:585) [52n-sos-importer-feeder-0.5.0-20150807.134409-5-bin.jar:0.5.0-SNAPSHOT]
	at org.n52.sos.importer.feeder.SensorObservationService.getInsertObservations(SensorObservationService.java:557) [52n-sos-importer-feeder-0.5.0-20150807.134409-5-bin.jar:0.5.0-SNAPSHOT]
	at org.n52.sos.importer.feeder.SensorObservationService.importData(SensorObservationService.java:302) [52n-sos-importer-feeder-0.5.0-20150807.134409-5-bin.jar:0.5.0-SNAPSHOT]
	at org.n52.sos.importer.feeder.task.OneTimeFeeder.run(OneTimeFeeder.java:203) [52n-sos-importer-feeder-0.5.0-20150807.134409-5-bin.jar:0.5.0-SNAPSHOT]
	at java.lang.Thread.run(Unknown Source) [na:1.8.0_91]

[...]

thank you for your response. The CSV lines are as follows:
Wind Speed,1465468180.561000,4.830000,Kph
Wind Speed,1465468240.993000,1.610000,Kph
which are in the order
ObservationPropertyName, UnixTime, Value, Unit
My query is, SOS Importer is unable to parse Unix Time (even if I remove milliseconds). When I change this timestamp to normal ISO one (e.g. YYYY-MM-DD HH:MM:SS), it works perfectly fine.
Could you just let me know, if it is possible to import Unix Timestamps using SOS Importer.

Import fails with SocketTimeoutException (through duplicate key value) or ExceptionReport: null

I am using the latest development version at commit d678a8b. I am trying to import a single row. This process fails in different scenarios. I boiled it down to the following.

Test files

These files can be used to reproduce the errors:

Shell command

I use the following command to execute the importer and write the console output to a log file.

$ java -jar ./feeder/target/52n-sos-importer-feeder-bin.jar \
    -c sensordata.csv.52n-sos-import-config-0.5.xml \
    -d sensordata-dummy-1.csv > /tmp/sensordata-dummy-1a.log

The names of the data file and log files change within the scenarios and subsequent runs.

Scenario 1

The SOS instance has enabled the following insert operations: InsertObservation, InsertSensor. InsertResult and InsertResultTemplate are disabled as shown in the screenshot.

At the 1st run the import fails with a SocketTimeoutException: Read timed out for each sensor value. When I run the same import a 2nd time the line is skipped.

The failure of the 1st run can be reproduced by simply renaming the data input file.

Scenario 2

In the scenario I renamed the data file to avoid that the row is skipped.

The SOS instance has enabled the following insert operations: InsertObservation, InsertSensor, InsertResult and InsertResultTemplate as shown in the screenshot.

At the 1st run the import fails with org.n52.oxf.ows.ExceptionReport: null. When I run the same import a 2nd time the line is skipped.

NPE during feeding if binding value is not set

Report on mailing list:

got some problem when I imported the data.
After I import the data,
It said "Configuration - Optional SosMetadata.Binding not set!"
Is it any idea to solve it?
Thank you.
[...]

10:46:23.464 [main] TRACE Feeder - main()
10:46:23.468 [main] TRACE Feeder - logApplicationMetadata()
10:46:23.472 [main] INFO  Feeder - Application started
        build-time: 2014-03-18_10-39
        n52-sos-importer-xmlbeans-version: 0.2
        Implementation-Vendor: 52°North
        geotools-version: 9.5
        Implementation-Title: Feeding Module
        Implementation-Version: 0.4.0-SNAPSHOT
        Implementation-Vendor-Id: org.n52.sensorweb
        Build-Jdk: 1.7.0_45
        Built-By: [...]
        Manifest-Version: 1.0
        n52-oxf-version: 2.0.0-alpha.4-SNAPSHOT
        Created-By: Apache Maven
        Main-Class: org.n52.sos.importer.feeder.Feeder
        Archiver-Version: Plexus Archiver
        HeapSize Information: max: 1323MB; total now: 89MB; free now: 78MB; used
now: 11MB
10:46:23.473 [main] TRACE Feeder - checkArgs([-c,
E:\car.csv.52n-sos-import-config.xml])
10:46:23.775 [OneTimeFeeder] TRACE OneTimeFeeder - run()
10:46:23.775 [OneTimeFeeder] INFO  OneTimeFeeder - Starting feeding data
from file via configuration '[...].csv.52n-sos-import-config.xml' to SOS
instance
10:46:23.783 [OneTimeFeeder] INFO  Configuration - Optional
SosMetadata.Binding not set!
10:46:23.788 [OneTimeFeeder] TRACE SensorObservationService -
SensorObservationService(http://localhost:8080/52n-sos-webapp/sos)
Exception in thread "OneTimeFeeder" java.lang.NullPointerException
    at
org.n52.sos.importer.feeder.SensorObservationService.getBinding(SensorObservationService.java:134)
    at
org.n52.sos.importer.feeder.SensorObservationService.<init>(SensorObservationService.java:117)
    at
org.n52.sos.importer.feeder.task.OneTimeFeeder.run(OneTimeFeeder.java:164)
    at java.lang.Thread.run(Unknown Source)

Multi-Phenomenon InsertObservation

I used the importer with the example data and observed that the importer send separated insertObservation for each phenomenon.

I'd like to add a new insertion mode where the N phenomenon are send in a single InsertObservation array.

Instead of :

<ns25:DataRecord>
          <ns25:field name="phenomenonTime">
              <ns25:Time definition="http://www.opengis.net/def/property/OGC/0/PhenomenonTime">
                   <ns25:uom xlink:href="http://www.opengis.net/def/uom/ISO-8601/0/Gregorian"/>
               </ns25:Time>
            </ns25:field>
             <ns25:field name="Water">
                  <ns25:Quantity definition="urn:phenomenon:Water">
                        <ns25:uom code="kg"/>
                    </ns25:Quantity>
            </ns25:field>
  </ns25:DataRecord>

and

<ns25:DataRecord>
          <ns25:field name="phenomenonTime">
              <ns25:Time definition="http://www.opengis.net/def/property/OGC/0/PhenomenonTime">
                   <ns25:uom xlink:href="http://www.opengis.net/def/uom/ISO-8601/0/Gregorian"/>
               </ns25:Time>
            </ns25:field>
             <ns25:field name="Krypton">
                  <ns25:Quantity definition="urn:phenomenon:Krypton">
                         <ns25:uom code="kg"/>
                   </ns25:Quantity>
               </ns25:field>
  </ns25:DataRecord>

The dataRecord will embed each phenomenon in a field :

<ns25:DataRecord>
          <ns25:field name="phenomenonTime">
              <ns25:Time definition="http://www.opengis.net/def/property/OGC/0/PhenomenonTime">
                   <ns25:uom xlink:href="http://www.opengis.net/def/uom/ISO-8601/0/Gregorian"/>
               </ns25:Time>
            </ns25:field>
             <ns25:field name="Water">
                  <ns25:Quantity definition="urn:phenomenon:Water">
                        <ns25:uom code="kg"/>
                    </ns25:Quantity>
            </ns25:field>
             <ns25:field name="Krypton">
                  <ns25:Quantity definition="urn:phenomenon:Krypton">
                         <ns25:uom code="kg"/>
                   </ns25:Quantity>
               </ns25:field>
  </ns25:DataRecord>

This mode can be activated by adding a new IMPORT_STRATEGY.

I join some example files of the insertObservation requests.
config-files.zip

Transactional authorization token support

Hello everybody,

is there currently any way to use the SOS Importer with a SOS server, that is using a transactional authorization token to limit access to transactional operations? I could not find any documentation on this on Github or here.

For instance, sending an InsertSensor request from an XML file to a token secured server from the shell using CURL works fine:

curl -v \
       -H "Content-Type: application/soap+xml;charset=UTF-8" \
       -H "Authorization: IamTheSecretToken-c1db8eb9f3f7" \
       -X POST \
       -d @insertSensor-SOAP.xml \
     http://my.sos.server:8080/52n-sos-webapp/service > response.xml

However, is there any way (I would not mind a hacky solution) to get this working with the SOS Importer at this time?

If not, this would be a very handy feature I would love to seen in a future release! One option could be a new element for a <Token> in <SOSMetadata>.

ERROR OneTimeFeeder ... does not support required transactional operations!

I successfully created a config file for a .csv file using the 52n-sos-importer-core module (d678a8b). I have an instance of SOS server v.4.3.0 running on Tomcat 7 to store the data. I temporarily enabled transactional operations as shown in the screenshot:

I use the 52n-sos-importer-feeder module to run the import. However it fails with the error:

ERROR OneTimeFeeder ... does not support required transactional operations!

Here is the console output:

$ java -jar ./52n-sos-importer-feeder/target/52n-sos-importer-feeder-bin.jar 
    -c ~/Desktop/sensordata.config.xml -d ~/Desktop/sensordata.csv
20:19:01.509 [main] INFO  Feeder - Application started
        Implementation-Title: Feeding Module
        Implementation-Version: 0.4.0-SNAPSHOT
        Archiver-Version: Plexus Archiver
        Built-By: john
        Implementation-Vendor-Id: org.n52.sensorweb
        geotools-version: 9.5
        n52-oxf-version: 2.0.0-alpha.4-SNAPSHOT
        Manifest-Version: 1.0
        Implementation-Vendor: 52°North
        Main-Class: org.n52.sos.importer.feeder.Feeder
        build-time: 2015-04-13_21-27
        Created-By: Apache Maven
        Build-Jdk: 1.8.0_40
        n52-sos-importer-xmlbeans-version: 0.2
        HeapSize Information: max: 1303MB; total now: 88MB; free now: 76MB; used now: 11MB
        os.name: Linux; os.arch: amd64; os.version: 3.13.0-55-generic

20:19:02.024 [org.n52.sos.importer.feeder.task.OneTimeFeeder] 
    INFO  OneTimeFeeder - Starting feeding data from file via configuration 
    'sensordata2.config.xml' to SOS instance
20:19:02.039 [org.n52.sos.importer.feeder.task.OneTimeFeeder] 
    TRACE SensorObservationService - SensorObservationService(Configuration 
    [file=/home/john/Desktop/sensordata.config.xml])
20:19:02.196 [org.n52.sos.importer.feeder.task.OneTimeFeeder] 
    DEBUG SimpleHttpClient - executing POST method to 
    'http://localhost:8080/52n-sos-webapp/service'.
20:19:02.901 [org.n52.sos.importer.feeder.task.OneTimeFeeder] 
    INFO  SensorObservationService - Using 25ms timeout buffer during insert 
    observation requests. Change <SosImportConfiguration><SosMetadata 
    insertSweArrayObservationTimeoutBuffer> if required.
20:19:02.903 [org.n52.sos.importer.feeder.task.OneTimeFeeder] 
    DEBUG SensorObservationService - OperationsMetadata found: 
    org.n52.oxf.ows.capabilities.OperationsMetadata@3e0d0dba
20:19:02.918 [org.n52.sos.importer.feeder.task.OneTimeFeeder] 
    ERROR OneTimeFeeder - SOS 'http://localhost:8080/52n-sos-webapp/service' 
    does not support required transactional operations!

NoSuchElementException when counter file is empty => import stopped

The disk where the counter files are stored was full. This caused a problem because now the counter file is empty. This causes a NoSuchElementException which breaks the current import run.

Exception log:

2015-07-09 08:20:15,157 DEBUG [FeederTimer] [OneTimeFeeder.java:185] Check counter file '/home/user/.SOSImporter/348d8ad8a126fcd2e5a8878a13be
a0de'.
2015-07-09 08:20:15,157 DEBUG [FeederTimer] [OneTimeFeeder.java:188] Read already read lines from file
2015-07-09 08:20:15,158 ERROR [FeederTimer] [RepeatedFeeder.java:121] Exception catched. Switch logging to debug for more details: null
2015-07-09 08:20:15,159 DEBUG [FeederTimer] [RepeatedFeeder.java:122] StackTrace:
java.util.NoSuchElementException: null
        at java.util.Scanner.throwFor(Scanner.java:907) ~[na:1.7.0_45]
        at java.util.Scanner.next(Scanner.java:1530) ~[na:1.7.0_45]
        at java.util.Scanner.nextInt(Scanner.java:2160) ~[na:1.7.0_45]
        at java.util.Scanner.nextInt(Scanner.java:2119) ~[na:1.7.0_45]
        at org.n52.sos.importer.feeder.task.OneTimeFeeder.run(OneTimeFeeder.java:190) ~[52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.sos.importer.feeder.task.RepeatedFeeder.run(RepeatedFeeder.java:93) ~[52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at java.util.TimerThread.mainLoop(Timer.java:555) [na:1.7.0_45]
        at java.util.TimerThread.run(Timer.java:505) [na:1.7.0_45]

Solution:

Catch this exception, log it and assume "0" or stop this run?

Error while building current version

Failed to execute goal on project 52n-sos-importer-core: Could not resolve dependencies for project org.n52.sensorweb:52n-sos-importer-core:jar:0.4.0-SNAPSHOT: The following artifacts could not be resolved: org.geotools:gt-swing:jar:9.5, org.geotools:gt-wms:jar:9.5, org.geotools:gt-epsg-hsql:jar:9.5

Import logs: Sending request failed caused by SocketTimeoutException

Each time I run the importer the following is output to the console:

[org.n52.sos.importer.feeder.task.OneTimeFeeder] ERROR SensorObservationService - 
  Insert observation failed for sensor '10'[10]. Store: TimeSeries [sensor=10, observedProperty=ObservedProperty 
  [name=Partikel, PM10, Tagesmittel, uri=Partikel_PM10_Tagesmittel], 
  feature=http://www.stadtentwicklung.berlin.de/umwelt/umweltatlas/extra/3_12/d312_MC010.htm]
[org.n52.sos.importer.feeder.task.OneTimeFeeder] DEBUG SimpleHttpClient - 
  executing POST method to 'http://localhost:8080/52n-sos-webapp/service/pox'.
[org.n52.sos.importer.feeder.task.OneTimeFeeder] ERROR SensorObservationService - 
  Problem with OXF. Exception thrown: Sending request failed.
org.n52.oxf.OXFException: Sending request failed.

    at org.n52.oxf.sos.adapter.SOSAdapter.doOperation(SOSAdapter.java:392) ~[52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.oxf.sos.adapter.SOSAdapter.doOperation(SOSAdapter.java:437) ~[52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.oxf.sos.adapter.wrapper.SOSWrapper.doInsertObservation(SOSWrapper.java:319) ~[52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.sos.importer.feeder.SensorObservationService.insertSweArrayObservation(SensorObservationService.java:755) [52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.sos.importer.feeder.SensorObservationService.insertTimeSeries(SensorObservationService.java:652) [52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.sos.importer.feeder.SensorObservationService.importData(SensorObservationService.java:376) [52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.sos.importer.feeder.task.OneTimeFeeder.run(OneTimeFeeder.java:203) [52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]

Caused by: org.n52.oxf.util.web.HttpClientException: Could not execute request.
    at org.n52.oxf.util.web.SimpleHttpClient.executeMethod(SimpleHttpClient.java:169) ~[52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.oxf.util.web.SimpleHttpClient.executePost(SimpleHttpClient.java:161) ~[52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.oxf.util.web.SimpleHttpClient.executePost(SimpleHttpClient.java:154) ~[52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.oxf.util.web.HttpClientDecorator.executePost(HttpClientDecorator.java:68) ~[52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.oxf.util.web.HttpClientDecorator.executePost(HttpClientDecorator.java:68) ~[52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.oxf.sos.adapter.SOSAdapter.doOperation(SOSAdapter.java:366) ~[52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    ... 7 common frames omitted

Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.7.0_79]
    at java.net.SocketInputStream.read(SocketInputStream.java:152) ~[na:1.7.0_79]
    at java.net.SocketInputStream.read(SocketInputStream.java:122) ~[na:1.7.0_79]
    at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166) ~[52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    ....
    at org.n52.oxf.util.web.SimpleHttpClient.executeMethod(SimpleHttpClient.java:166) ~[52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    ... 12 common frames omitted

This messages are very misleading until I realized that all data is imported successfully nonetheless.

Steps to reproduce

Please use the following files to reproduce the case:

Here you find the server log and the console output of the importer.

Issue #44: Import fails with SocketTimeoutException (through duplicate key value) or ExceptionReport: null

/cc @CarstenHollmann

Allow double values for count fields

The importer allows to import Measured values of type Count as shown in the screenshot.

The values for this column are processed here:

// COUNT
else if (m.getValue().equals("COUNT")) {
    return Integer.parseInt(value);
}

Integer values are expected here.

Error case

If the values is provided as a double value such as 23.0 the following error occurs:

[org.n52.sos.importer.feeder.task.OneTimeFeeder] ERROR SensorObservationService - 
    Could not retrieve all information required for insert observation because of 
    parsing error: java.lang.NumberFormatException: For input string: "23.0". 
    Skipped this one.
[org.n52.sos.importer.feeder.task.OneTimeFeeder] DEBUG SensorObservationService - Exception stack trace:
java.lang.NumberFormatException: For input string: "23.0"
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[na:1.7.0_79]
    at java.lang.Integer.parseInt(Integer.java:492) ~[na:1.7.0_79]
    at java.lang.Integer.parseInt(Integer.java:527) ~[na:1.7.0_79]
    at org.n52.sos.importer.feeder.DataFile.getValue(DataFile.java:372) ~[52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.sos.importer.feeder.SensorObservationService.getInsertObservationForColumnIdFromValues(SensorObservationService.java:590) [52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.sos.importer.feeder.SensorObservationService.getInsertObservations(SensorObservationService.java:566) [52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.sos.importer.feeder.SensorObservationService.importData(SensorObservationService.java:343) [52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    at org.n52.sos.importer.feeder.task.OneTimeFeeder.run(OneTimeFeeder.java:203) [52n-sos-importer-feeder-bin.jar:0.5.0-SNAPSHOT]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]

Feature request

Please consider processing double values such as 23.0 as 23 or 0.0 as 0 instead of skipping the value.

Fix failing travis build

Currently, travis fails to build the importer:
https://travis-ci.org/52North/sos-importer
Our jenkins is able to build it:
http://build.dev.52north.org/jenkins/view/Sensor%20Web/job/sos-importer/

Feedback for improvement (unfiltered)

From user Fabien:
I wanted to give you some feedback by importance order for the tool :

could it be possible to add a step to request the sensor's type (thermometer...)
adding observations for several observed property (different sensor) doesn't work yet (but I think you know it as I saw it in code comment). It will only need to repeat some steps for each sensor/observed property like requesting sensor name, observed property and uom
for the unit of measure (step 6b), using '²' character causes an UTF8-error (added SOS log in attachment)
could the configuration file be applied to a fileset, for example I got 30 files for a mounth which only vary on the date. By setting a shortcut to the last step and only request a csv file and a config-file.xml or setting at first step a multiple target

Less important feedbacks :

append an other step for asking the user if he wants to add metadata which could be used in the client (like quality of the sensor, owner ...) with

<sml:characteristics>
    <swe:SimpleDataRecord definition="urn:x-ogc:def:property:stationProperties">
        <swe:field name="sensorquality">
            <swe:Text definition="urn:x-ogc:def:property:sensorquality">
                <swe:value>0.75</swe:value>
            </swe:Text>
        </swe:field>
        ...
    </swe:SimpleDataRecord>
</sml:characteristics>

for having a complete client as on http://sensorweb.demo.52north.org/sensorwebclient-webapp-stable/, when you click on a station provided by Pegelonline you got the river name for example which is set on the sensorML.

could the tool be a little more connected by requesting first url of the SOS, for parsing throught the GetCapabilities, which FOI, procedures and unit of measure already exist to avoid misspelling
could the policy when a file already been read be less strict, because it needs to rename the file if you want to use it again when you got an error while feeding (skip lines but data haven't been yet inserted in the database)"

2013-07-30 15:56:57,209 ERROR [http-apr-8080-exec-8] [SOS.java:234] Error while parsing request!
org.xml.sax.SAXParseException: Invalid byte 1 of 1-byte UTF-8 sequence.
    at org.apache.xerces.parsers.DOMParser.parse(Unknown Source) ~[xercesImpl-2.9.1.jar:na]
    at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source) ~[xercesImpl-2.9.1.jar:na]
    at javax.xml.parsers.DocumentBuilder.parse(Unknown Source) ~[na:1.7.0_17]
    at org.n52.sos.SOS.doPost(SOS.java:224) ~[52n-sos-core-3.5.0.jar:na]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) [servlet-api.jar:na]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) [servlet-api.jar:na]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) [catalina.jar:7.0.42]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) [catalina.jar:7.0.42]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) [catalina.jar:7.0.42]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) [catalina.jar:7.0.42]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) [catalina.jar:7.0.42]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) [catalina.jar:7.0.42]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) [catalina.jar:7.0.42]
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) [catalina.jar:7.0.42]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) [catalina.jar:7.0.42]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) [catalina.jar:7.0.42]
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023) [tomcat-coyote.jar:7.0.42]
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) [tomcat-coyote.jar:7.0.42]
    at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1852) [tomcat-coyote.jar:7.0.42]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.7.0_17]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.7.0_17]
    at java.lang.Thread.run(Unknown Source) [na:1.7.0_17]
Caused by: org.apache.xerces.impl.io.MalformedByteSequenceException: Invalid byte 1 of 1-byte UTF-8 sequence.
    at org.apache.xerces.impl.io.UTF8Reader.invalidByte(Unknown Source) ~[xercesImpl-2.9.1.jar:na]
    at org.apache.xerces.impl.io.UTF8Reader.read(Unknown Source) ~[xercesImpl-2.9.1.jar:na]
    at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source) ~[xercesImpl-2.9.1.jar:na]
    at org.apache.xerces.impl.XMLEntityScanner.scanLiteral(Unknown Source) ~[xercesImpl-2.9.1.jar:na]
    at org.apache.xerces.impl.XMLScanner.scanAttributeValue(Unknown Source) ~[xercesImpl-2.9.1.jar:na]
    at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanAttribute(Unknown Source) ~[xercesImpl-2.9.1.jar:na]
    at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source) ~[xercesImpl-2.9.1.jar:na]
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) ~[xercesImpl-2.9.1.jar:na]
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) ~[xercesImpl-2.9.1.jar:na]
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) ~[xercesImpl-2.9.1.jar:na]
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) ~[xercesImpl-2.9.1.jar:na]
    at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) ~[xercesImpl-2.9.1.jar:na]
    ... 22 common frames omitted

BadLocationException in the case of having empty lines in csv file

When trying to create an configuration for an CSV file with empty lines between each line, the importer fails with a BadLocationException:

2014-03-24 10:50:48,045 TRACE [AWT-EventQueue-0] [Step2Panel.java:299] setCSVFileHighlight()
2014-03-24 10:50:48,052 ERROR [AWT-EventQueue-0] [Step2Panel.java:312] Exception thrown: No such line
javax.swing.text.BadLocationException: No such line
    at javax.swing.JTextArea.getLineEndOffset(JTextArea.java:432) ~[na:1.7.0_40]
    at org.n52.sos.importer.view.Step2Panel.setCSVFileHighlight(Step2Panel.java:305) ~[classes/:na]
    at org.n52.sos.importer.controller.Step2Controller.loadSettings(Step2Controller.java:123) [classes/:na]
    at org.n52.sos.importer.controller.MainController.setStepController(MainController.java:81) [classes/:na]
    at org.n52.sos.importer.controller.BackNextController.nextButtonClicked(BackNextController.java:143) [classes/:na]
[...]

Provide basic authentication for secure SOS

I'm trying to use the SOS importer with a SOS server which has the transactional operation secure by basic authentication.

I'd like to add the support of this feature by adding credentials to the configuration file and use PreemptiveBasicAuthenticationHttpClient when the credentials are available.

How many "Feature Of Interest" and "Observed Property" to create?

Given the following property columns from a .csv file:

sensor_id
date
partikelPM10Mittel
partikelPM10Ueberschreitungen
...

Do I have to create one or two "Feature Of Interest" for "partikelPM10Mittel" and "partikelPM10Ueberschreitungen"?

Do I have to create one or two "Observed Property" for "partikelPM10Mittel" and "partikelPM10Ueberschreitungen"?

At the end of the wizard coordinates can be specified for each "Feature Of Interest" which tells me that I probably should use the same "Feature Of Interest" for "partikelPM10Mittel" and "partikelPM10Ueberschreitungen" since both refer to "partikel".

What I do not understand then: What is a "Sensor" in the meaning of the importer? I expected that it is the physical station which has multiple sensors and therefore latitude/longitude are defined once.

IllegalArgumentException: year does not match pattern. => breaks import for sample based files

I have a problem while importing data from a sample based file. The importer seems to break the import while processing the last lines of the sample.

2015-07-09 10:45:22,381 DEBUG [FeederTimer] [SensorObservationService.java:351] SampleFile: true; isInSample: false; lineCounter: 31775; sampleStartLine: 31733; sampleSize: 0; sampleDataOffset: 18
2015-07-09 10:45:22,382 DEBUG [FeederTimer] [SensorObservationService.java:505] Current line '[]' contains empty values . Skipping this line!
2015-07-09 10:45:22,382 DEBUG [FeederTimer] [SensorObservationService.java:351] SampleFile: true; isInSample: false; lineCounter: 31776; sampleStartLine: 31733; sampleSize: 0; sampleDataOffset: 18
2015-07-09 10:45:22,383 DEBUG [FeederTimer] [SensorObservationService.java:351] SampleFile: true; isInSample: false; lineCounter: 31777; sampleStartLine: 31733; sampleSize: 0; sampleDataOffset: 18
2015-07-09 10:45:22,621 DEBUG [FeederTimer] [SOSRequestBuilder200POX.java:459] Could not create ITime from String '23:59:00+00:00' Exception thrown: 'year does not match pattern: applied Pattern: -?\d+'
java.lang.IllegalArgumentException: year does not match pattern: applied Pattern: -?\d+
        at org.n52.oxf.valueDomains.time.TimePosition.setYear(TimePosition.java:219) ~[52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.valueDomains.time.TimePosition.initDate(TimePosition.java:117) ~[52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.valueDomains.time.TimePosition.<init>(TimePosition.java:97) ~[52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.sos.adapter.v200.SOSRequestBuilder200POX.getITimeFromString(SOSRequestBuilder200POX.java:457) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.sos.adapter.v200.SOSRequestBuilder200POX.addResultTime(SOSRequestBuilder200POX.java:573) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.sos.adapter.v200.SOSRequestBuilder200POX.addObservations(SOSRequestBuilder200POX.java:321) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.sos.adapter.v200.SOSRequestBuilder200POX.buildInsertObservationRequest(SOSRequestBuilder200POX.java:277) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.sos.adapter.v200.SOSRequestBuilder_200.buildInsertObservationRequest(SOSRequestBuilder_200.java:111) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.sos.adapter.SOSAdapter.buildRequest(SOSAdapter.java:488) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.sos.adapter.SOSAdapter.doOperation(SOSAdapter.java:302) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.sos.adapter.SOSAdapter.doOperation(SOSAdapter.java:426) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.sos.adapter.wrapper.SOSWrapper.doInsertObservation(SOSWrapper.java:319) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.sos.importer.feeder.SensorObservationService.insertSweArrayObservation(SensorObservationService.java:725) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.sos.importer.feeder.SensorObservationService.insertTimeSeries(SensorObservationService.java:622) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.sos.importer.feeder.SensorObservationService.importData(SensorObservationService.java:361) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.sos.importer.feeder.task.OneTimeFeeder.run(OneTimeFeeder.java:198) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.sos.importer.feeder.task.RepeatedFeeder.run(RepeatedFeeder.java:93) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at java.util.TimerThread.mainLoop(Timer.java:555) [na:1.7.0_45]
        at java.util.TimerThread.run(Timer.java:505) [na:1.7.0_45]
2015-07-09 10:45:22,625 ERROR [FeederTimer] [SensorObservationService.java:773] Problem with OXF. Exception thrown: Unsupported type for phenTime not yet implemented. Received type: java.lang.String
org.n52.oxf.OXFException: Unsupported type for phenTime not yet implemented. Received type: java.lang.String
        at org.n52.oxf.sos.adapter.v200.SOSRequestBuilder200POX.wrongResultTimeType(SOSRequestBuilder200POX.java:587) ~[52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.sos.adapter.v200.SOSRequestBuilder200POX.addResultTime(SOSRequestBuilder200POX.java:578) ~[52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.sos.adapter.v200.SOSRequestBuilder200POX.addObservations(SOSRequestBuilder200POX.java:321) ~[52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.sos.adapter.v200.SOSRequestBuilder200POX.buildInsertObservationRequest(SOSRequestBuilder200POX.java:277) ~[52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.sos.adapter.v200.SOSRequestBuilder_200.buildInsertObservationRequest(SOSRequestBuilder_200.java:111) ~[52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.sos.adapter.SOSAdapter.buildRequest(SOSAdapter.java:488) ~[52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.sos.adapter.SOSAdapter.doOperation(SOSAdapter.java:302) ~[52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.sos.adapter.SOSAdapter.doOperation(SOSAdapter.java:426) ~[52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.oxf.sos.adapter.wrapper.SOSWrapper.doInsertObservation(SOSWrapper.java:319) ~[52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.sos.importer.feeder.SensorObservationService.insertSweArrayObservation(SensorObservationService.java:725) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.sos.importer.feeder.SensorObservationService.insertTimeSeries(SensorObservationService.java:622) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.sos.importer.feeder.SensorObservationService.importData(SensorObservationService.java:361) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.sos.importer.feeder.task.OneTimeFeeder.run(OneTimeFeeder.java:198) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at org.n52.sos.importer.feeder.task.RepeatedFeeder.run(RepeatedFeeder.java:93) [52n-sos-importer-feeder-bin_DEBUG.jar:0.4.0-SNAPSHOT]
        at java.util.TimerThread.mainLoop(Timer.java:555) [na:1.7.0_45]
        at java.util.TimerThread.run(Timer.java:505) [na:1.7.0_45]
2015-07-09 10:45:22,626 ERROR [FeederTimer] [SensorObservationService.java:624] Insert observation failed for sensor '***'[***]. Store: TimeSeries [sensor=***, observedProperty=Observed
Property [name=***, uri=***], feature=***]

Missleading Typo in Wiki-documentation (Example Tutorial)

in the Example Tutorial, Step 5, the pattern is given as 'dd.mm.yyyy HH:mm', while it has to be 'dd.MM.yyyy HH:mm ' (compare the month-entry). With the wrong pattern, the feeder is not able to properly insert any observation.

By the way, you could add a hint in How to build, Step 1, that a JDK is needed. For convenience, you could serve a ready-to-use config-file for the feeder in section Demo data

Best wishes
Daniel

Server generated offering identifier

I try to use the sos importer with the automtic offering generation mode

<SosMetadata>
............................................
   <Offering generate="true"/>
............................................
</SosMetadata>

After the InsertSensor request sent, my SOS server respond the following :

<swes:InsertSensorResponse>
  <swes:assignedProcedure>urn:sensor:csv-sensor</swes:assignedProcedure>
  <swes:assignedOffering>offering-urn:sensor:csv-sensor</swes:assignedOffering>
</swes:InsertSensorResponse>

But the value of "assignedOffering" is never used.

I see that the InsertSensor request contains a block :

<sml:capabilities name="offerings">
   <swe:SimpleDataRecord>
      <swe:field name="csv-sensor">
         <swe:Text definition="urn:ogc:def:identifier:OGC:1.0:offeringID">
            <swe:value>urn:sensor:csv-sensor</swe:value>
         </swe:Text>
      </swe:field>
   </swe:SimpleDataRecord>
</sml:capabilities>

This value is used after in InsertObservation request and my server launch an error as the offering is not registered.

Is this somehow normative ? the importer should be able to use the value assigned by the server.

I join the configuration file used and the insertSensor request/response
config-files.zip

52north / sos-importer Goto Github PK

sos-importer's Introduction

52°North SOS Importer

Branches

Credits

sos-importer's People

Contributors

Stargazers

Watchers

Forkers

sos-importer's Issues

Description

Stackstrace

OS

Java

Related

Steps to reproduce

Related

Documentation request

Status quo

Workaround

Change request

Related

CSV

Feeder config

CSV

Feeder config

Exception version 0.4.0

Exception develop/abdbd0ad

Test files

Shell command

Scenario 1

Scenario 2

Steps to reproduce

Related

Error case

Feature request

Recommend Projects

Recommend Topics

Recommend Org

Jobs