aipescience / asciiingest Goto Github PK

Data file contains a row with empty last column, e.g.
1,2,3,,5,
2,2,3,4,5,6

Format file: ',' as delimiter, '\n' as final delimiter.

When trying to ingest, I get following error:

ERROR IN AsciiIngest:
AsciiReader: Error in getItemInRow. You want to read more than there is available.

When ingesting into a MySQL database with MyISAM engine, I get following error message:

Ingest DONE
Releasing savepoint...
ErrNr 1305: SAVEPOINT dbIngst_mysql_sp does not exist
DBMySQL: savepoint not realeased successfully.
Releasing savepoint DONE

The ingest works perfectly, all the data arrives in the database. It's just confusing to see an error when everything seems to be OK.

I've create a table like this:

CREATE TABLE test (
snapnum smallint NULL,
redshift float NULL,
BDM int NULL
) Engine=MyISAM;

and tried inserting NULL values by leaving the corresponding column empty, i.e. I have a line
1 ' ' 35123
in an ascii file (delimiter =' ' in format file) or
1,'',35123
in csv file (delimiter = ',').

Both result in:

Starting ingest...
Error in reading line 0
Problem with schema item number: 1
ERROR IN DBIngestor:
DBIngestor: Ingesting NULL in column that is set IS NOT NULL!```

Ideally, for csv it would be great, if NULL values could be written just like this:
1,,35123
but this gives a parse error:
ERROR IN AsciiIngest:
mdelimGetfield: I could not parse the data file since a given field delimiter could not be found!
Please check the format file for errors...

E.g. columns like "[0,1,2]" for csv-file or "This is an example" in a space-separated file.

AsciiIngest does not respect quotes in columns, i.e. if there is a column "[0,1,2]" in a csv-file, then it will be read as 3 columns, not 1, if the delimiter is specified as comma ',' in the structure file.
Luckily it is possible to define a different delimiter for each column, so the correct delimiter could be '",' (double quotes+comma) for finding the end of the quoted columns, but AsciiIngest seems to enter an infinite loop if there is more than one delimiting character.

So currently, the work-around is:
Use delimiter ',' everywhere, then just before the quoted column: use double quote '"' as delimiter and skip the column with SKIP_THIS_COL, the part until the next double quote is the stuff that can be mapped to a field in the database, then one needs to skip again until the next comma and then one can continue as usual.
E.g. for the following data row:

1,54.6,"[0,1,2]",2.8,5

the delimiter should be:

','
','
'"' SKIP_THIS_COL
'"'
',' SKIP_THIS_COL
','
'\n'

An example is also given in this post.
Of course, it would be possible to edit the data file and use e.g. semi-colons to be on the save side. Anyway, this is a feature request, not a bug report, to:

• allow usage of multiple different characters as delimiter
• introduce a quote-character in addition to the delimiter character for each column

When I provide the wrong delimiter in my format file, AsciiIngestor complains like this:

ERROR IN AsciiIngest:
mdelimGetfield: I could not parse the data file since a given field delimiter could not be found!
Please check the format file for errors...

That's great already, but it would be really nice and very helpful if AsciiIngestor could also tell explicitely which delimiter it cannot find.

I am generating a line number using the feature of creating a "saved variable" (S-line in format file) and adding +1 for each line using CONV_ADD.
This works fine so far, however, if I reuse the S- variable more than once to ingest something into the database, then it is incremented again, i.e. CONV_ADD is applied again, so the line numbers don't fit anymore.
Is there a way to limit this to just one application of the converters?