stevehansen / csv Goto Github PK

Hi there,
I'm using you library reading data out of a csv file (as expected), with HeaderMode.HeaderPresent active.
The iteration through the items starts with the second line, where the data starts...
How can the headers being iterated?

Does not see any Header property having the list of all column names.
Would be a good thing to have directly (read) access to the headers.

Thank you for any hint.
// SiL

PS: Got it working, when switched the HeaderMode to HeaderAbsent ;)

This csv row:

"Normal,\"quoted with nested \"double\" quotes, and comma at the end,\",normal 3,normal 4,normal 5"

will count 2 columns instead of 5.

Note, there is a nested double quotes, which with that alone it passes the test correctly (5 columns); but if you add a comma at the very end of the column (2nd column in this case) it will evaluate to only 2 columns instead of 5.

My code:

string importDataString = "Test;\"A\nB\nC\nD\nE\nF\nG\nH\";testing with very long string;123123";
var options = new CsvOptions
{
    Separator = ";",
    HeaderMode = HeaderMode.HeaderAbsent,
    AllowNewLineInEnclosedFieldValues = true,
    AllowBackSlashToEscapeQuote = false,
};

CsvReader.ReadFromText(importDataString, options)

The result of my reader is a list with one record.

Record:

• ColumnCount = 4 (Ok)
• Headers = ["Column1", "Column2"] (Not ok)
• Values = ["Test","A\nB\nC\nD\nE\nF\nG\nH","testing with very long string","123123"] (Ok)

Why is Headers an array of 2 records and not 4 records? Since ColumnCount is actually 4 and Values also contains 4 records.

From CsvWriter.cs
var cell = data[i]; // add "?? string.empty"

Am I correct in that this library currently only reads csv files? I don't see any tests or implementation for writing, but I wanted to make sure I wasn't missing something....

Thanks!

Hi, Steve! Thank you a lot for the great tool!
Could you please explain how to change (if possible) the new line character (NLC) in CsvWriter.WriteToText? Yes, we can do similar thing through the CsvOptions in CsvReader.ReadFromText and it is very useful. I read my tables with \n\r as a general NLC, and \n as a NLC inside multiline cells. I would want to write the same way, but CsvWriter.WriteToText does it waywardly: both cases with \n\r. Though it is not a wrong behaviour and data remains consistent, it does mess up the appearance of my tables a bit.

Hi,

When I read a csv file with the HeaderMode.HeaderAbsent option, if the first row has two columns that are blank (I assume even with the same content) I get the error "an item with the same key has already been added".

I assume this is because it is using the values of the first row as the field/column name. It should just be manually using "Column1, Column2.." etc to avoid this.

Thanks for this little library, I'm working with some CSV files that will not have any header rows. So i'm modifying your project to optionally include the first row as data.

Does it sound like an idea you would care for in csv?

The CSV is simple but also one of the most difficult for developers to handle.
I love this simplicity and it helps me a lot.

Thanks,

I can't see why I would get an error like this. My file has no key of any kind, so why would the assembly assume that it does. Is it assuming that the first column is some kind of unique key? It shouldn't assume anything.

For the input data(with CR-LF return)

key,value
test1,test2
test3,"test4

test5"
test6,test7

which is expected to be parsed to

is actually parsed to

Breaks when reading Blank Fields

example:

,,First

I have a huge csv file.
I would like to read it line by line so it would not take a bunch of memory.
DoAre you thinking about adding that functionnality ?

Parsing

head1;head2;head3
text1;;text3

Throws an exception instead of the expected result that head2 contains an empty value.

In the case of a file like this:

nonsense1
nonsense2
header0,header1,header2
row0,row1,row2

Can the RowsToSkip not be applied before the header and separator is determined?

I tried reading a csv file with this package, but the separator is not detected, and even when I try passing in options specifying the separator, I keep getting only one column detected.

Note: This happened after opening the file in excel - works fine if the file wasn't opened in excel.

I tried to use this package but I can't find any documentation to direct me.

It seems most of our performance issues comes from the Regex based parser (it in itself is complex, it is invoked for each line, but we also need to clean up afterward if it was wrong for the splitting quoted values)

Going over the reader/memory character by character will allow us to do trimming at once, will handle quoting correctly, can escape correctly, and will probably handle multiline better as well.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency Microsoft.NET.Test.Sdk to 17.11.0

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• Update actions/checkout action to v4
• Update actions/stale action to v9
• Update dependency Microsoft.SourceLink.GitHub to v8
• Update github/codeql-action action to v3
• Update mstest monorepo to v3 (major) (MSTest.TestAdapter, MSTest.TestFramework)

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Can be used to map multiple unique headers to a single column

I test line.HasColumn(x) to see if a column exists, and then query line[x].
line.Headers is 55 elements wide, and line.Values is 40 elements wide. So, I get:

System.InvalidOperationException: Invalid row, missing line_item_1 header, expected 55 columns, got 40 columns.

I'd expect it to return "".

When a field contains the combination " at the end, parsing will fail.
The following test will provoke it in the second section:

    [TestMethod]
    public void BackslashBeforeQuote()
    {
        var withSpace = CsvReader.ReadFromText("\"A\";\"B\";\"C\"\n\"A \"\"\";\"B \\\"\" \";\"C\"", new CsvOptions { AllowNewLineInEnclosedFieldValues = true, AllowBackSlashToEscapeQuote = false, AllowSingleQuoteToEncloseFieldValues = false }).ToArray();
        Assert.AreEqual(1, withSpace.Length);
        Assert.AreEqual(3, withSpace[0].Headers.Length);
        Assert.AreEqual("A \"", withSpace[0][0]);
        Assert.AreEqual("B \\\" ", withSpace[0][1]);
        Assert.AreEqual("C", withSpace[0][2]);
        var withoutSpace = CsvReader.ReadFromText("\"A\";\"B\";\"C\"\n\"A \"\"\";\"B \\\"\"\";\"C\"", new CsvOptions { AllowNewLineInEnclosedFieldValues = true, AllowBackSlashToEscapeQuote = false, AllowSingleQuoteToEncloseFieldValues = false }).ToArray();
        Assert.AreEqual(1, withoutSpace.Length);
        Assert.AreEqual(3, withoutSpace[0].Headers.Length);
        Assert.AreEqual("A \"", withoutSpace[0][0]);
        Assert.AreEqual("B \\\"", withoutSpace[0][1]);
        Assert.AreEqual("C", withoutSpace[0][2]);
    }

Hi I encountered this problem in V1.0.51; I noticed there is a new commit regarding multiline issue, I am not sure whether those are relevant.

The code I used:

 static void Main(string[] args)
        {
            // V1.0.51
            string content =
@"A1,A2,A3,A4
1, 1,""Hello
World"",""""
2, 2,""Hello
World"",""""";
            // Case 1
            Console.WriteLine("Case 1:");
            foreach (var line in CsvReader.ReadFromText(content, 
                new CsvOptions() { AllowNewLineInEnclosedFieldValues = false }))
            {
                Console.WriteLine($"{line.ColumnCount}");
            }
            Console.ReadKey();

            // Case 2
            Console.WriteLine("Case 2:");
            foreach (var line in CsvReader.ReadFromText(content,
                new CsvOptions() { AllowNewLineInEnclosedFieldValues = true }))
            {
                Console.WriteLine($"{line.ColumnCount}");
                Console.WriteLine($"{line.Values}");
            }
            Console.ReadKey();
        }

The csv:

A1,A2,A3,A4
1, 1,"Hello
World",""
2, 2,""Hello
World",""

This is what I expect (as in Excel import csv):

This is what I am getting from program output:

Case 1:
3
2
3
2
Case 2:
7
System.String[]

Please check!
Thanks.

Visual Studio Solution: CsvMultilineIssue.zip

I have a file that uses tabs as separators, and the columns are unquoted, but quotes may exist in the data. The parser puts all remaining data into the first column containing a double quote. In c#, you can reproduce with this:

var header = "h1\th2\th3\r\n1\t\"2\" is 2\t3"; var res = CsvReader.ReadFromText(header);

This results in 3 headers but only 2 data elements in the first row. There isn't (or at least I can't find) a way to disable quote parsing.

As can be seen below, when the read text contains a quote-enclosed field containing the quote escape sequence "", followed by the separator character ,, the APIs for reading column values parse the data incorrectly.

Also, as can be seen below, even when the AllowSingleQuoteToEncloseFieldValues flag is set, the incorrect parsing behaviour remains unchanged. I tested this flag in order to see if, for some reason, quote-enclosed fields were only detected with this mode activated, even though I know it is specifically to be used when fields are enclosed with the single quote character '.

To summarize, it is valid for CSV data to contain fields enclosed by the " character, while also containing quote escape sequences such as "" followed by separator characters such as the comma ,, but this library does not seem to be able to parse this formatting correctly.

Is there going to be support for other types of encdoing?
Like letters as øæå?

De-facto there are at least three versions of CSV format with different separators: comma (canonical separator), semicolon and tab. CsvReader supports them through CsvOptions but CsvWriter doesn't.
As CsvOptions was intended for import purpose only I suggest not to use it but add a char parameter to Write() method like this:

public static void Write(TextWriter writer, string[] headers, IEnumerable<string[]> lines, char separator=',')

Would you consider signing assembly (making it strongly-named)?
When trying to use library in strongly-named assembly, I'm getting following error:
Unhandled Exception: System.IO.FileLoadException: Could not load file or assembly 'Csv, Version=1.0.58.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. A strongly-named assembly is required. (Exception from HRESULT: 0x80131044)
More info: https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/strong-naming

Issue:

A, ",," is parsed as four columns.

However, A,",," (no space) is correctly parsed as two columns.

Expected:

The line to be parsed as two columns.

Example Code:

        public static void ColumnBug()
        {
            string data =
"""
A, ",,"
A,",,"
""";

            using var stream = new MemoryStream(Encoding.UTF8.GetBytes(data));
            IEnumerable<ICsvLine> csvLines = Csv.CsvReader.ReadFromStream(stream, new CsvOptions()
            {
                HeaderMode = HeaderMode.HeaderAbsent,
                TrimData = true,
            });

            foreach (ICsvLine csvLine in csvLines)
            {
                Console.WriteLine($"Column Count {csvLine.ColumnCount}");
                Console.WriteLine($"Parsed:\t{string.Join("|", csvLine.Values)}");
                Console.WriteLine($"Raw:\t{csvLine.Raw}");
                Console.WriteLine();
            }
        }

    }

}

Output:

Column Count 4
Parsed: A|"||"
Raw:    A, ",,"

Column Count 2
Parsed: A|,,
Raw:    A,",,"

Assuming the CSV has a header but no data, there ought to be some way of telling that that has happened. As it stands, the best I can do is

        var hasData = false;
        var csvText = File.ReadAllText("c:\\tmp\\norecords.csv");
        foreach (var line in CsvReader.ReadFromText(csvText))
        {
            hasData = true;
        }
        if (hasData)
        {

        }

Hi I know this is well-intended feature and I support it most of the time, but lately I am parsing CSV files that have real values starting with # in the first letter and because there is no way to toggle off skipping comments, such CSV files cannot be parsed.

I understand that the authoring program should have exported the values with " to "escape" them properly, but it's not done by the authoring program in this case and it would not be easy to try to escape those values in-place.

A hypothetical data would look like below:

HashTag,Author
#New,Charles
#Feature,Tom

Ideally there would be an option to Csv.CsvOptions:

Csv.CsvReader.ReadFromText(content, new Csv.CsvOptions()
{
    IgnoreComments = false
})

Appreciated!

When there is a header and the header has duplicate header names it throws. InvalidOperationException("Duplicate headers detected in HeaderPresent mode. If you don't have a header you can set the HeaderMode to HeaderAbsent.");

Since the csv is provided to me can't be changed prior to processing the file, I added an option FixDuplicateColumnNames and some code to fix duplicate name issues.

If there is another way other than coding please let me know?

I would be happy to check in the changes. Please let me know if you wish to have the changes checked in.

Thanks

Using this as the input line: one,"two-a,two-b,""two-c"",two-d",three
The actual Lines property is [
"one",
""two-a, two-b, ""two-c"",
"two-d"",
"three" ]
Expected Lines property is [
"one",
"two-a,two-b,"two-c",two-d",
"three" ]

Test code that shows the issues is:

    [TestMethod]
    public void TestInternalSeperatorAfterEscapedQuote()
    {
        CsvOptions options = new CsvOptions();
        options.HeaderMode = HeaderMode.HeaderAbsent;
        foreach (ICsvLine line in CsvReader.ReadFromText("one,\"two - a, two - b, \"\"two - c\"\", two - d\",three", options))
            Assert.AreEqual(3, line.Values.Length);
    }