v1.23, perl 5.22.1

Test case as follows, note the third line of the __DATA__ section with badly balanced quotes.

#!perl

use strict;
use warnings;

use Test::More;
use Text::CSV_XS;

my $csv = Text::CSV_XS->new(
    {
        binary             => 1,
        sep_char           => '|',
        allow_loose_quotes => 1,
    }
);

my $fh = \*DATA;
$csv->column_names( @{ $csv->getline( $fh ) } );

my $nbr_lines = 0;
$nbr_lines++ while ( my $row = $csv->getline( $fh ) );

is( $nbr_lines,3,'processed expected number of lines' );

done_testing();

__DATA__
first|second|third|fourth|fifth|sixth|seventh|eigth|ninth|tenth|eleventh|twelth|thriteenth|fourteenth|fifteenth|sixteenth|seventeenth|eighteenth|nineteenth
1|||||||156999||12 Valley||D||N|3610|||68 V D|EA MATCH
2|||||||195658|"""The Cottage"" 54"|"""The "|K|||||307652|R, M|"""The ", K, |EA MATCH
3|||||||216058|117 The K|||||||||117 The K, |EA MATCH

If you use Text::CSV_PP to process the above file then the test passes, so there is a difference in behaviour. Possibly unexpected for those using Text::CSV as they will get different behaviour on different machines if they haven't watched their installation/dependency tree.

I don't know if the behaviour of Text::CSV_XS is supposed to map to Text::CSV_PP exactly, hence this issue.

I have created a csv object with a no-arg constructor call. I have set the quote character and the escape character to the undef value

my $csv = Text::CSV_XS->new;
$csv->quote_char (undef);
$csv->escape_char(undef);

Instead of undef the quote_char and escape_char getters return 0. That means I am not able to use the getters to pass this information to a DBI:CSV connect() call?!

$dbh = DBI->connect ("DBI:CSV:", undef, undef, {
  ...
  csv_quote_char  => $csv->quote_char,
  csv_escape_char => $csv->escape_char,
  ...

Please have a look at the Text-CSV-PR-68.

Both Text::CSV and Text::CSV_XS have the same bug.

This bug is described in:
makamaka/Text-CSV#64

I will open a pull request for this.

Thanks,
Corey

Serialising and deserialising objects from previous script runs leads to segmentation faults.

See attached scripts for reproduction code using Storable and Sereal formats.

Tested using Strawberry Perl 5.38.0.1 on Windows and a perlbrew 5.36.0 on Ubuntu via WSL2. Text::CSV_XS version is latest (1.51).

Encoding and decoding within the same process works.

perl same_process.pl

However, encoding in one process and decoding in a later process results in a segmentation fault.

perl create_and_serialise.pl
perl decode.pl

gdb backtrace:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff777b5b9 in cx_c_xsParse ()
   from /home/user/perl5/perlbrew/perls/perl-5.36.0/lib/site_perl/5.36.0/x86_64-linux/auto/Text/CSV_XS/CSV_XS.so
(gdb) bt
#0  0x00007ffff777b5b9 in cx_c_xsParse ()
   from /home/user/perl5/perlbrew/perls/perl-5.36.0/lib/site_perl/5.36.0/x86_64-linux/auto/Text/CSV_XS/CSV_XS.so
#1  0x00007ffff777f7fb in cx_xsParse ()
   from /home/user/perl5/perlbrew/perls/perl-5.36.0/lib/site_perl/5.36.0/x86_64-linux/auto/Text/CSV_XS/CSV_XS.so
#2  0x00007ffff777fc6d in XS_Text__CSV_XS_Parse ()
   from /home/user/perl5/perlbrew/perls/perl-5.36.0/lib/site_perl/5.36.0/x86_64-linux/auto/Text/CSV_XS/CSV_XS.so
#3  0x000055555566ca4e in Perl_pp_entersub ()
#4  0x0000555555662ea3 in Perl_runops_standard ()
#5  0x00005555555cbf07 in perl_run ()
#6  0x00005555555a2292 in main ()

create_and_serialise.pl.txt
decode.pl.txt
same_process.pl.txt

This is a clone of makamaka/Text-CSV#66. The workaround that is based on $csv->{ _CACHE }->{ tmp } does not work for the XS backend Text-CSV_XS. Could you offer something official/public?!

As the error codes are not very end-user friendly.

Either as a method (to be used instead of $csv->error_diag) or as a procedure wrapper (e.g. textual_error($csv->error_diag).

Previously when strict was used with bind_columns, an error would be raised if the number of fields for a row didn't match the number of fields for the previous rows:

# Text::CSV_XS 1.53

$ perl test.pl
Line parsed: foo = a, bar = b, baz = c
Error parsing CSV: 2014ENF - Inconsistent number of fields522

As of the 1.54 release this behavior has changed:

# Text::CSV_XS 1.54

$ perl test.pl
Line parsed: foo = a, bar = b, baz = c
Line parsed: foo = d, bar = e, baz = c
Line parsed: foo = f, bar = g, baz = h

# test.pl

use strict;
use warnings;
use Text::CSV;
use IO::String;

my $csv_data = "a,b,c\nd,e\nf,g,h\n";
my $io = IO::String->new($csv_data);
my $csv = Text::CSV->new();
$csv->strict(1);

my ($foo, $bar, $baz);
$csv->bind_columns(\$foo, \$bar, \$baz);

while (my $status = $csv->getline($io)) {
    print "Line parsed: foo = $foo, bar = $bar, baz = $baz\n";
}

if ($csv->error_input()) {
    warn "Error parsing CSV: ", $csv->error_diag(), "\n";
}

fany@homer:/tmp> perl -MText::CSV_XS=csv -E 'csv in => [[1,2],[3,4]], out => "test.csv"'; xxd test.csv
00000000: 312c 320d 0a33 2c34 0d0a                 1,2..3,4..
fany@homer:/tmp> perl -MText::CSV_XS=csv -E 'csv in => [[1,2]], out => "test.csv"'; xxd test.csv
00000000: 312c 320d 0a                             1,2..
fany@homer:/tmp> perl -MText::CSV_XS=csv -E 'csv in => [], out => "test.csv"'; xxd test.csv
-rw------- 1 fany users 4 11. Mai 04:07 test.csv
00000000: 0d0a 0d0a                                ....

So two records in $aoa give two lines of CSV, one gives one line, but none give two empty lines.
I think it should just output an empty file in this case.

Is this possible? (e.g. to parse output of ps shell command.) sep_char=>' ', allow_whitespace=>1 parse to multiple empty columns.

Hello!

I am using:
-> Text-CSV_XS-1.50
-> Strawberry perl, Perl version: 5.36.0 / MSWin32-x64-multi-thread

I try to compare two CSV files "file1.csv" and "file2.csv" with the script "csvdiff". I do not apply any changes inside this script. My goal is to use this approach for bigger csv data files. In all the cases I use the command:
perl csvdiff file1.csv file2.csv

file1.csv
a,b,c,d
1,2,3,4
5,6,7,8

file2.csv (here the change is the "X" in the last row)
a,b,c,d
1,2,3,4
5,6,X,8

Here as expected is presented that 7 is replaced by X:
"# csvdiff < file1.csv > file2.csv"
"< 5,6,"[31m7[0m",8[0m"
"> 5,6,"[32mX[0m",8[0m"

But when compare these files with the same content:
file1.csv
a,b,c,d
1,2,3,4
5,6,7,8
9,10,11,12
13,14,15,16
17,18,19,20

file2.csv
a,b,c,d
1,2,3,4
5,6,7,8
9,10,11,12
13,14,15,16
17,18,19,20

Why the result is:
"# csvdiff < file1.csv > file2.csv"
"< 9,10,11,12[0m"
"> 9,10,11,12[0m"
"< 17,18,19,20[0m"
"> 17,18,19,20[0m"

By the way in all these cases when I use the option "-h", with the command:

perl csvdiff -h file1.csv file2.csv

the result is:
The getpwuid function is unimplemented at csvdiff line 61.

How to apply a html support?

Thanks!

When run on an empty file or one consisting only of whitespace, csv2xls throws some errors:

Use of uninitialized value $sep in string eq at csv2xls line 132.
Use of uninitialized value $sep in string ne at .../perl-5.16.2/lib/perl5/site_perl/5.16.2/x86_64-linux/Text/CSV_XS.pm line 135.
# CSV_XS ERROR: 1008 - INI - SEP undefined @ rec 0 pos 0
Can't call method "getline" on an undefined value at /grasp_mob_c/local/bin/csv2xls line 163.

It would be nice if it would just output an empty file, or one with a cell containing the whitespace.

A minor typo causes header() not to use the set of sep chars defined.

Line 1363 sets the key 'set_set'

defined $c->{'hd_s'} and $harg{'set_set'} = $c->{'hd_s'};

But it should be 'sep_set' :

defined $c->{'hd_s'} and $harg{'sep_set'} = $c->{'hd_s'};

Current NULL encoding options are limited. It works for some cases - where upstream can handle what we produce. Other cases - eg MySQL 'load data infile' - is unable to correctly identify NULLs using our encoding method ( eg ,, ). The docs here:
http://search.cpan.org/~hmbrand/Text-CSV_XS-1.35/CSV_XS.pm#csv
... suggest you can produce output that databases can parse by doing:

while (my $row = $sth->fetch) {
  $csv->print ($fh, [ map { $_ // "\\N" } @$row ]);
  }

... but this is absolutely not the case. Given the data:

[ "blah", undef, 3 ]

... the required output for importing into MySQL or other DBs would be:

"blah",\N,3

... but the above hack instead gives us:

"blah","\\N",3

There are 2 problems with this:

1. Text::CSV_XS is escaping the \N, giving us \\N. DBs won't parse this correctly.
2. Text::CSV_XS is quoting the \\N. DBs won't parse this correctly either.

What we really need is a way to pass in any string sequence that can be used to encode a NULL value. Additionally, this string sequence should not be quoted.

You may have seen this blog post, about potential security problems with CSV files that have formulae in cells:

http://georgemauer.net/2017/10/07/csv-injection.html

In a perfect world, all of my existing code that uses Text::CSV_XS would start throwing exceptions if they got CSVs with formulae in them. If users have to turn on some "don't allow formulae in cells" feature, then it's not going to help most people, because most people won't (a) know about the potential problem, or (b) the module's support for protecting you.

I'm guessing that for backwards compatibility reasons you might not want to add this as a feature that's enabled by default, but I think you should at least consider it.

That said, following on from email, I might want to write:

use Text::CSV_XS qw/ csv /;
csv(in => $fh, headers => 'auto', formula => 'croak');

The formula parameter could be croak, allow, diag, empty (blank out all such cells), or undef (return cell as undef).

Personally i'd make the default be croak, but I realise you don't want code to suddenly start breaking, so maybe diag could be the default?

Currently, if both of these are file names, you get an error like:

Can't use string ("infile.csv") as an ARRAY ref while "strict refs" in use at /home/greg/perl5/perlbrew/perls/perl-5.26.0/lib/site_perl/5.26.0/x86_64-linux/Text/CSV_XS.pm line 1169.

I was trying to do something like:

 csv (
    in      => 'infile.csv',
    headers => "auto",
    on_in   => sub { $_{'domain'} =~ s/^(.+)$/${1}.com/ },
    out => 'outfile.csv',
    );

In this short test script, I expect a "CSV_XS ERROR: 2023 - EIQ - QUO character not allowed @ rec 2 pos 6 field 2" exception in the loop, and that happens if the line calling $csv->header is commented out. However, after calling ->header it seems Text::CSV_XS::error_diag is getting called with no arguments and so $self is falsy and so the croak is never triggered. Instead we fall through to the "why no auto_diag" exception.

#!/usr/bin/perl
use strict;
use warnings;
use Text::CSV_XS;

my $csv = Text::CSV_XS->new( { auto_diag => 2 } );
$csv->header(*DATA); # comment out this line to hide the bug
while ( my $row = $csv->getline(*DATA) ) { print "$row->[0]\n" }
die "why no auto_diag for: " . $csv->error_diag
    if $csv->error_diag and $csv->error_diag != 2012;

__DATA__
Foo,Bar,Baz
a,xxx,1
b,"xx"xx", 2
c, foo , 3

makamaka/Text-CSV#27

Maybe this could be part of the planned attribute strict

Version: 1.54

When I create a csv instance with strict => enabled, and the quote_char => '"' and escape_char => '"', for certain inputs I get a misleading error response on parsed strings. Consider the following example:

my $csv = Text::CSV_XS->new(
  {
    binary      => 1,
    comment_str => '#',
    eol         => "\n",
    escape_char => '"',
    quote_char  => '"',
    sep_char    => '|',
    strict      => 1,
  }
);

my $status = $csv->parse( 'a|b|"d"' );
note( ( 0 + $csv->error_diag ) . ' - ' . ( '"' . $csv->error_diag . '"' ) . "\n" ); # prints '0 - ', i.e. works fine
$status = $csv->parse( 'a|b|c"d"e' );
note( ( 0 + $csv->error_diag ) . ' - ' . ( '"' . $csv->error_diag . '"' ) . "\n" ); # prints '2014 - "ENF - Inconsistent number of fields"'

From my understanding Text::CSV_XS treats the occurrence of a quote character as the beginning of an entirely new field, ignoring that the previous character here was not the sep_char '|'.

This in my opinion is at least a very misleading error to print here, because on first glance the string 'a|b|c"d"e' has the exact right number of fields: 3.
If, with strict => 1 and a quote_char set, quotes are only allowed at the beginning/end of a field but not in the middle, there should be a different error code/message signaling that an irregular quoting was detected.

Let's say I just want to populate a structure in after_parse or on_in, but I don't want to get the whole structure back, and I don't want to print it. E.g. I might want to accumulate information by id:

csv (in      => shift,
     headers => "skip",
     on_in   => sub {
         my ($id, $email) = @{ $_[1] };
         push @{ $by_id{$id}{email} }, $email;
         },
    );

I'm only interested in %by_id. Something like out => undef or similar.

Hello

Consider input file with 1 billion records:

my $batch = $csv->getline_all ($fh, 0, 1e6) will return all data
my $batch = $csv->getline_all ($fh, 0, 1000000) will return 1000000 recs

re: makamaka/Text-CSV#62

Occasionally I have to process files that contain multiple CSVs. Each of these CSVs is stored in the file as a heading, followed by the data lines, followed by an empty line (or eof).

It would be nice if Text::CSV had an option that basically says: stop reading after an empty line. This would make it possible to write something similar to:

csv (in => $fh, out => @aoh1, stop_at_empty => 1);
csv (in => $fh, out => @Aoh2, stop_at_empty => 1);
csv (in => $fh, out => @aoh3, stop_at_empty => 1);

I wanted to turn off column name munging, and ended up with:

$csv->header( $fh, { munge_column_names => sub { $_[0] } } );

Aside from noting that lc is the default, I'd like another special value, perhaps undef or any false, to turn off any processing:

$csv->header( $fh, { munge_column_names => undef } );

Not a big deal and there's no urgency.

re: makamaka/Text-CSV#63

Using the module for a command-line tool, I'm consistently setting headers => 'auto' and give the possibility to specify a detect_bom based on a command-line option.

Tests show that when both options are active, headers are read twice, which ends up skipping the first line (where real headers live) and trying to set headers from the second line.

I'd say that this behaviour is surprising and sub-optimal, although admittedly there's no indication that this should not behave this way in the docs. I'm currently working this around like this:

my $has_bom = ...;
my @csv_args = (
   ($has_bom ? (detect_bom => 1) : (headers => 'auto')),
    ...
);

If you're interested, this is a full test for this:

(snip)

The test also fails for Text::CSV_XS. Could you take a look at it when you have some time and tuits?

See implementation: https://github.com/Tux/Text-CSV_XS/blob/master/CSV_XS.xs#L60:

#define _is_arrayref(f) ( f && \
     (SvROK (f) || (SvRMAGICAL (f) && (mg_get (f), 1) && SvROK (f))) && \
      SvOK (f) && SvTYPE (SvRV (f)) == SVt_PVAV )
#define _is_hashref(f) ( f && \
     (SvROK (f) || (SvRMAGICAL (f) && (mg_get (f), 1) && SvROK (f))) && \
      SvOK (f) && SvTYPE (SvRV (f)) == SVt_PVHV )
#define _is_coderef(f) ( f && \
     (SvROK (f) || (SvRMAGICAL (f) && (mg_get (f), 1) && SvROK (f))) && \
      SvOK (f) && SvTYPE (SvRV (f)) == SVt_PVCV )

Code first checks for SvRMAGICAL() and then calls mg_get(). But SvRMAGICAL checks for SVs_RMG -- magic different from get/set, in most cases uses for clear function. So it does not make sense to call mg_get() method based on SVs_RMG result.

Instead SvRMAGICAL() there should be used SvGMAGICAL(), check for SVs_GMG that scalar has get magic which means that mg_get needs to be called.

To simplify code I would propose to use SvGETMAGIC() macro which calls mg_get() when it is needed. E.g. _is_arrayref(f) could looks like this:

static inline bool _is_arrayref(SV *sv) {
    if (!sv) return false;
    SvGETMAGICAL(sv);
    if (!SvROK(sv)) return false;
    if (SvTYPE(SvRV(sv)) != SVt_PVAV) return false;
    return true;
}

re: makamaka/Text-CSV#61

yewtc commented 19 hours ago
The option detect_bom is only available in the header method, but not all CSVs have header lines. The detect_bom functionality should be used on the first line of the file, not just the header line.

If a field is empty or blank, it would be nice if if could be skipped completely not made undef. So how about skip_empty and skip_blank?

(I was going to try to see if I could write this based on blank_is_undef and empty_is_undef, but I could not figure out how those interact with the method or function.)

Please see makamaka/Text-CSV#48 .
I don't know how to reassign this bug to here.

In the POD, there are many usage examples. However, they flip between $fh and $io. $fh is used more often, so I would suggest turning all $io uses into $fh. Also, $fh is more widely used by the community in general when making a common variable for a file handle.

I could copy the POD and make the changes, but I don't know how to do a pull request. I could paste the edit here or elsewhere if you want me to do it.

Lady Aleena

Got below warning while installing related modules for GNU parallel:

"/opt/homebrew/Cellar/perl/5.38.0/bin/perl" "/opt/homebrew/opt/perl/lib/perl5/5.38/ExtUtils/xsubpp"  -typemap '/opt/homebrew/opt/perl/lib/perl5/5.38/ExtUtils/typemap'  CSV_XS.xs > CSV_XS.xsc
mv CSV_XS.xsc CSV_XS.c
cc -c   -fno-common -DPERL_DARWIN -mmacosx-version-min=13.5 -DNO_POSIX_2008_LOCALE -fno-strict-aliasing -pipe -fstack-protector-strong -Wno-error=implicit-function-declaration -O3   -DVERSION=\"1.52\" -DXS_VERSION=\"1.52\"  "-I/opt/homebrew/opt/perl/lib/perl5/5.38/darwin-thread-multi-2level/CORE"   CSV_XS.c
CSV_XS.xs:617:41: warning: format specifies type 'int' but the argument has type 'unsigned long' [-Wformat]
        warn ("  %-20s %4d:0x%08x\n", "cache", sizeof (csv_t), csv->cache);
                       ~~~                     ^~~~~~~~~~~~~~
                       %4lu
CSV_XS.xs:617:57: warning: format specifies type 'unsigned int' but the argument has type 'unsigned char *' [-Wformat]
        warn ("  %-20s %4d:0x%08x\n", "cache", sizeof (csv_t), csv->cache);
                             ~~~~                              ^~~~~~~~~~
                             %8s
2 warnings generated.

Related source

Hi,

I'm attempting to use the 'csv' function of Text::CSV_XS with munge_column_names set to "db". This fails when the .csv file passed to the script starts with the sep=; line to indicate the field separator, with the following error message:

CSV_XS ERROR: 1012 - INI - the header contains an empty field @ rec 1 pos 0

Setting sep_set to 1 doesn't seem to make a difference but it all works correctly when I manually remove the sep=; line from the .csv file. Reading the documentation, I was under the impression that the 'sep=' feature was implemented in version 1.17. I am missing a detail to get this to work?

I am running Text::CSV_XS version 1.41 built from MacPorts on Perl v5.26.3 (Mac OS X 10.13.6)

I have the following line in my CSV file:
nj23h32n,"By using "pseudoenergies", we were able to design multiple peptide sequences that showed low micromolar viral entry inhibitory activity.",2010-06-22,

The 2nd field is the "abstract" field which could contain double quotes and commas as in the example above. I use the following one-liner to parse the file:
perl -MText::CSV_XS=csv -wE'csv(in=>csv(in=>"test.csv",sep=>",",allow_loose_quotes=>1,allow_loose_escapes=>1))'

But I get:
nj23h32n,"By using ""pseudoenergies"," we were able to design multiple peptide sequences that showed low micromolar viral entry inhibitory activity.""",2010-06-22,

Any idea how to correctly parse this example?

Thanks in advance,
Andrej