mithunkr / parsecsv-for-php Goto Github PK

What steps will reproduce the problem?
1. just try to parse a file containing plain wrong data such as a field
"some data" get replaced by "some" data" which cannot by valid

What is the expected output? What do you see instead?
I expected an error, instead line get dropped silently (parsing seems to
stop but valid rows are returned, there's no error message).

What version of the product are you using? On what operating system?
0.3.1

Please provide any additional information below.
Parsing valid file is really easy, the problem always comes when parsing
broken files. In such case at least would be good to know that there's a
problem you can enforce valid file to be sent from user.

What steps will reproduce the problem?
1.Saving the csv 
2.
3.

What is the expected output? What do you see instead?
file is not saving. How can i give the path for saving the csv.

What version of the product are you using? On what operating system?

Csv v0.4.3 beta in linux server

Please provide any additional information below.

I dont know how can i give the path for saving the CSV . Please help me, Its an 
urgent work. 

Step to produce the problem:

1. $filename = "res/autors.csv"; // Latge files with 19 columns and 6900 
records, separated by semicolon.
2. $csv = new parseCSV();
3. $csv->auto($filename);
4. print_r($csv);

Those steps will produce: 

Fatal error: Allowed memory size of 8388608 bytes exhausted (tried to allocate 
780 bytes) in /Applications/xampp/xamppfiles/htdocs/user143/Cactus/parseCSV.php 
on line 454

I understand that I have limitation on memeory, but I am not going to increase 
the memory because it will impact to my policy and also for security reason.

Expected output:
The data will appear.


Version: parseCSV v0.4.3 beta
Operating system: Windows 7 64 bit, with Firefox 3.6.15

Additional data:
You could try it using your dummy data

I think it would be advantageous to include the ability to read CSV values 
from a string, if passed to the class. Thanks!

What steps will reproduce the problem?
1. Loading a CSV file with more than 517 lines

What is the expected output? What do you see instead?
When I load my CSV file, which has over 600 lines, it only shows up to 
516+heading

What version of the product are you using? On what operating system?
I tried both 0.3.2 and 0.4.3

Please provide any additional information below.
I tried playing around with the limit option, but that didn't help. Although I 
could set the limit to any number under 516(for example setting it to 10 it 
worked), but when i set the limit to, lets say 600, 600 lines don't appear, 
even though I do have all those lines.

Currently, parsecsv-for-php has a $heading parameter that will consume the
first line and use it as headers for each respective column. Unfortunately,
it seems to have lumped two distinct features together.

That is, not only is parseCSV::$titles populated with the first row, but
parseCSV::$data becomes an associative array. I believe that
parseCSV::$data should stay as an indexed array and a new parameter,
$associative_data, should be introduced to cover that respective functionality.

Of course, there's a workaround to this issue at present, but it means
having to manipulate both parseCSV::$heading as well as parseCSV::$offset.
I'd prefer to fix problems rather than fix symptoms, though. :)

I have little proposal for unparse method.
When I am trying send CSV to MS Excel I have to convert encoding from UTF8
to CP1250, here is the code:

$csv = new parseCSV();
$csv->encoding('utf-8', 'cp1250');
$result = $csv->unparse($csv_data, $headers, false, false, ';');

Unfortunately you don't perform encoding change in uparse() method as you
do it in load_data(). Here is my patch:

// create heading
        if ( $this->heading && !$append ) {
            foreach( $fields as $key => $value ) {
                if ( $this->convert_encoding ) $value = iconv($this->input_encoding,
$this->output_encoding, $value);
                $entry[] = $this->_enclose_value($value);
            }
            $string .= implode($delimiter, $entry).$this->linefeed;
            $entry = array();
        }

        // create data
        foreach( $data as $key => $row ) {
            foreach( $row as $field => $value ) {
                if ( $this->convert_encoding ) $value = iconv($this->input_encoding,
$this->output_encoding, $value);
                $entry[] = $this->_enclose_value($value);
            }
            $string .= implode($delimiter, $entry).$this->linefeed;
            $entry = array();
        }

First off, I'm not sure if this a problem with parseCSV - or if I have csv 
files that are violating the standard.

Basically, the problem is having columns with the same name.

For example:

"id","name","column_A","numbers","column_A"
0,"test","abcd",12,""
0,"hello","defg",14,""

Etc...

The last column (column_A) has the same name as the third. When I try and auto 
parse a file like this, the last column gets ignored.
In addition, when it parses this file, it looks like there is only 4 columns - 
when there is actually 5.


What steps will reproduce the problem?
1. Create a CSV file that has multiple columns with the exact same name in the 
heading.
2. Have parseCSV try to auto parse the file.
3. print_r($csv->data) the result and notice that the fields with duplicate 
names are missing / have been ignored.

What version of the product are you using? On what operating system?
This issue exists with both v0.4.3 beta and the stable v0.3.2.

I have formulated a script that parses each row one by one. At the end of the 
script I would like to delete the row I execute with my function.

Steps
========
$csv = new parseCSV();
$csv->sort_by = 'Submitted';
$csv->conditions = 'A contains X';
$csv->offset = 1;
$csv->limit 1;
$csv->auto('file.csv');

$i=0;$i<999999;$i++;
$contact = $csv->data;

//Set various variable
$var = $contact[$i]["value"];

//Create Array
$arr = array('things' => $contact[$i]);

//Encode Array JSON
$string = json_encode($arr);

/cURL String

// cURL funtion here, does something Awesom!!!

echo 'Results rather Good or Bad';

// Here I would like to delete the row I just submitted
// So I can sleep the script and continue to process through the entire 
// CSV file

//Sleep for a few seconds and get the next row
sleep(4);

What steps will reproduce the problem?
1. In the example "limit.php", comment out $csv->sort_by = "title" (line 22)
2. Execute limit.php

What is the expected output? What do you see instead?
Expect to see the header names in the first column, then the data rows

What version of the product are you using? On what operating system?
0.4.3, Linux

Please provide any additional information below.
The notes in limit.php describe how the data will be fetched but it's not clear 
that the header row will not be returned.

Suggestion:
Have an option to include/not include the header row.

>>What steps will reproduce the problem?
1. Obtain the basic.php file from the download and the attached _books.csv.
2. The attached _books.csv file has a single column, and has some 0's
peppered in the text of some of the rows.
3. Run basic.php.

>>What is the expected output? What do you see instead?
The file should parse as an array with a single member in each row, instead
the auto() function is evaluating the 0's as delimeters. Since there are no
delimeters used in the file, the $this->delimiter variable in
parsecsv.lib.php is empty. The subsequent evaluation on line 379 in
parsecsv.lib.php, since it only uses ==, is evaluating the 0's in the file
as equal to the blank delimeter.

>>What version of the product are you using? On what operating system?
Tested in 0.3.1 and 0.3.2 on RedHat


>>Please provide any additional information below.
The solution would seem to be to change the line below (line 379) to use
=== instead of == for string comparisons:

} elseif ( ($ch == $this->delimiter || ($ch == "\n" && $pch != "\r") || $ch
== "\r") && !$enclosed ) {

                         Becomes

} elseif ( ($ch === $this->delimiter || ($ch === "\n" && $pch !== "\r") ||
$ch === "\r") && !$enclosed ) {

I realise an amelioration of the function "_validate_row_condition"  for
add operator and add error of each field. I attach my version.


What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?
0.4.3

Please provide any additional information below.

If you want to contact me : [email protected].  Thanks

> What steps will reproduce the problem?
1. Obtain basic.php and _books.csv from ./examples.
2. Add a blank line anywhere within the first few lines, but no deeper than
$auto_depth (15 by default).
3. Execute basic.php.

> What is the expected output? What do you see instead?
The parser should return the same delimiter as the unaltered _books.csv,
but it does not; instead, it uses "0" as a delimiter.


> What version of the product are you using? On what operating system?
Reproducible in both 0.3.2 and 0.4.2b.


> Please provide any additional information below.
Because the parser never expects blank lines, the $filtered array always
ends up being simply array() for some reason.

What steps will reproduce the problem?
1. Use parseCSV::output()

What is the expected output? What do you see instead?
I expect this function (actually parseCSV::unparse() ) to take into 
consideration a value of parseCSV::convert_encoding, so if it is set to true it 
would convert the data.

What version of the product are you using? On what operating system?
not relevant

Please provide any additional information below.
In method unparse(),after line 446 and after 454 add:
if ( $this->convert_encoding ) $value = iconv($this->output_encoding, 
$this->input_encoding, $value);

What steps will reproduce the problem?
1. Read a file having columns that use an enclosure character such as a 
double-quote.
2. Include a field value that uses an escape character (backslash) then a 
double-quote as part of the data such as "Tom's height was 5'9\" when he was 12 
years old". 

What is the expected output? 
A value containing a double-quote within the data.
Tom's height was 5'9" when he was 12 years old

What do you see instead?
Data that ends due to the presence of the escaped double-quote.
Tom's height was 5'9\

What version of the product are you using? 
v0.3.2 and v0.4.3 beta

On what operating system?
Any

Please provide any additional information below.
The Wikipedia "Comma-separated values" article was provided as the 
specification reference. The article states "Embedded double quote characters 
may then be represented by a pair of consecutive double quotes (Creativyst 
2010), or by prefixing an escape character such as a backslash (for example in 
Sybase Central)."

I do not have control over the input file so I added the following to v0.4.3 
beta before line 413
} elseif ($pch == "\\") {
//do nothing; enclosure character may be escaped with a backslash

This needs more code but got me through my problem.

What steps will reproduce the problem?
1. Parse the attached file, which is UTF-8 with BOM:

$ file utf8-bom.csv 
utf8-bom.csv: UTF-8 Unicode (with BOM) text, with CRLF line terminators

<?php
$csv = new parseCSV();
$csv->parse(utf8-bom.csv);

2. The first heading, 'Employee_ID', will have the BOM on it, making it very 
difficult to access:

<?php
print_r($csv->data[0]); // you will see [Employee_ID] => E001689
echo $csv->data[0]['Employee_ID'];  // this will return nothing 

3. If the BOM is first stripped off of the file, then there is no issue

http://www.dotvoid.com/2010/04/detecting-utf-bom-byte-order-mark/

What steps will reproduce the problem?
1. reading big csv files will cause memory error  


What is the expected output? What do you see instead?
-

What version of the product are you using? On what operating system?
the newest version (on a linux vps)

Please provide any additional information below.
"Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to 
allocate 81 bytes) in /home/.../.../parsecsv.lib.php on line 485"

After $data has been created by the parser, is there any reason that
$file_data should not be unset()?

My only concern is that parser a CSV file consumes anywhere from two to
three times its filesize in memory. With relatively small files parsed
sequentially, memory isn't much of an issue; however, as soon as sizable
CSVs are introduced and/or dealt with concurrently, the script could easily
run into memory limits.

hello i want to insert multi language data use csv file but i have get data 
like '??????' even i have used 

$csv->encoding('UTF-8', 'UTF-8');

Please fix my issue. thanks

What steps will reproduce the problem?
1. Use an array with two rows with same column value
2. Use orderby method
3. print_r results

What is the expected output? What do you see instead?
It should order it correctly

What version of the product are you using? On what operating system?
0.4.3 beta

Please provide any additional information below.

It can be fixed using array_multisort; using additional temporal array and then 
using array_multisort.
I have attached the working file.

Hi, i have csv file with cyrillic names. It outputs them like ??????, while 
with latin i got no problem. i use 4.3 beta. Thanks.

Hi, nice script. My website thanks you.

A quick convenience would be to handle ksort()'s required distinction
between numeric and textual data. Took me by surprise when I first came
across my ill-sorted numbers.

Thanks again!
Steve

What steps will reproduce the problem?
1. Go for simple data and keep $csv->header = false
2. Upload a single lined data(I checked with numbers only.)
3. print_r or see the output in $csv->data

What is the expected output? What do you see instead?
It should bring the real values where I got something like this:
Array
(
    [0] => Array
        (
            [0] => 235886.
            [1] => 
            [2] => ,85565.
            [3] => 
            [4] => ,
            [5] => .
            [6] => 
            [7] => ,-157461.
            [8] => 
            [9] => ,97
            [10] => 78.
            [11] => 
            [12] => ,179
            [13] => 34.
            [14] => 
            [15] => ,2
            [16] => 7
            [17] => 48.
            [18] => 
            [19] => ,386
            [20] => 82.
            [21] => 
            [22] => ,
            [23] => .
            [24] => 
            [25] => 
        )

)


What version of the product are you using? On what operating system?
parsecsv-0.3.2.zip 

Please provide any additional information below.
I am attaching the testing csv file.

I've adjusted the write-modes in save($file, $data, $append, $fields) to be 
'ab' and 'wb' instead of 'at' and 'wt'. The reason behind this change is that 
PHP, with the mode of t, will replace \n with \r\n on Windows. We are already 
appending \r\n to the end of our lines, in unparse(), however. Unfortunately, 
this means that our line endings go from \r\n to \r\r\n on Windows, resulting 
in an additional line. Changing $linefeed to \n does not solve the problem. The 
php docs for fopen() also suggest using binary mode for portability across 
platforms.

I have tested this fix with PHP 5.4.9 on OS X (10.7.5), Fedora (12), and on 
Windows 8, and the outputted CSV files are valid, with the correct number of 
lines.

Hey, Just wanted to let you know that your script is great but i have one 
question. Can I hide certain columns or is there anyway that I can 
disclude them from grabbing from the CSV file? 

Thanks,
Alan

When testing the parsecsv() beta release with a CSV consisting of 500k+ 
records, the script fails to produce a fully initialized CSV.

Hello,

parseCsv needs to much time to parse big files (20 MB with 36000 lines) - over 
10 minutes on Windows XP, PHP 5.2, Intel Core 2 2.40

Do you have experience with such large files?

How can I do sorting for certain column for CSV file that don't have
header? Is is possible now?

In parseCSV v0.3.2

When parsing a csv, I was finding the first key value of each row was prepended 
with a \n or \r, not sure which.

I think the problem lies on line 380:
if ( !$was_enclosed ) $current = trim($current);

I fixed the issue I was experiencing by changing line 382 to:
$row[$key] = trim($current);

Works good parsing a whole file. But is there any way to parse just a string (a 
single line/record of csv text)?  Thank you.

What steps will reproduce the problem?
1. Initialize parsing with incorrect data source

What is the expected output? What do you see instead?

I should get exception explaining what is wrong. Instead I see nothing.

This is an enhancement request to allow the developer to track the lines
that fail the conditional check instead of just ignoring them. Suggest
adding a $rejected array variable that will store all of the lines in the
file that did not meet the conditional check. This will allow the developer
to report the failures to the end user so they will know what needs to be
corrected in the file.

What steps will reproduce the problem?
1. Create new parser object
2. Set conditional on parser
3. Parse file that contains a line which fails the conditional

What is the current response?
$data array contains all lines from file that passed conditional checks.
Lines that failed the conditional check are not stored in any variable for
reference.

What is the desired response?
$rejected array is populated with the line(s) that failed the conditional
check. Entry should contain the entire record contents, as well as which
conditional check failed.

Not sure if this list is even watched by anyone...but here goes anyway

What steps will reproduce the problem?
1. Create a CSV file with 15 rows of data 
2. Set $csv->limit = 10;
3. Run the csv file through the parser

What is the expected output? What do you see instead?
I expect to see only 10 rows of data imported, but instead I get 15

What version of the product are you using? On what operating system?
Latest version in Windows

Hi.

Is there a way, to create a csv file with your class.. adding categories and 
subcategories, for each post?

If yes, you know where I could learn how to do that?

Thanks!

What steps will reproduce the problem?
1. Upload the attached CSV
2. and see the result of datas.
3. one digit missing.

What is the expected output? What do you see instead?

my CSV has single column values like 2445,2446,.... in my coding i added the 
code for heading false. after upload i echo the values,

//my coding
include_once(YII_BASE_PATH . "/lib/parsecsv.lib.php");
        $file = $orgfile;
$csv = new parseCSV();
        $csv->heading = false;
        $csv->auto($file);
        $result =  $csv->data;
        print_r($result);
        die();

the values print like,

Array
(
    [0] => Array
        (
            [0] => 
            [1] => 445
        )

    [1] => Array
        (
            [0] => 
            [1] => 446
        )

    [2] => Array
        (
            [0] => 
            [1] => 447
        )

    [3] => Array
        (
            [0] => 
            [1] => 448
        )

    [4] => Array
        (
            [0] => 
            [1] => 449
        )

    [5] => Array
        (
            [0] => 
            [1] => 450
        )

    [6] => Array
        (
            [0] => 
            [1] => 451
        )

    [7] => Array
        (
            [0] => 
            [1] => 45
            [2] => 
        )

    [8] => Array
        (
            [0] => 
            [1] => 453
        )

note: in the above printed values i t removes the first digit and [7] value it 
removes the the number "2". because the first number value is "2" i think.




What version of the product are you using? On what operating system?

parseCSV v0.3.2, windows

Please provide any additional information below.