Comments (4)
Interesting, thanks for raising this! I currently see these possibilities why this could be happening:
- An escaped, unfinished line somewhere between row 500 and 700 (in which case it should not just stop but raise an error sometime) - essentially a CSV field that begins with a stray quote.
- A non-utf8 encoded file or a file with broken encoding (but also that should raise an error somewhen)
- A bug in the parser / decoder
Is the file publicly available? If yes I could take some time looking at this.
from csv.
I looked into a little more and the first issue I can into was that I didn't open the file using the right mode. If I do the following I get a little further:
file = File.open!("output.csv", [:write, :utf8])
This however only gets me through 600 rows. On row 601 it hangs but don't see what's different about this row. I've uploaded the data to my S3 bucket if you want to see for yourself. The data is a subset of what you can get from http://www.gbif.org/
from csv.
I had a look at this, there is this field in the file on line 686:
ecatalogue.LocCollectionEventLocal: "Australia, Western Australia, 10.9 Km S of The Turnoff To Athleen Valley" Along The Wiluna - Agnew Rd (27° 32' S, 120° 33' E) 23/09/1981 - 23/09/1981, A Greer, R Sadlier Et Al(Collector), Field Collected - Terrestrial"
There are other fields with stray quotes, however this is seen as an unfinished escape sequence because it is an odd number of quotes which are starting in the middle of the field - Fields with quotes should start with quotes, and quotes within the field should be double quoted to escape them, resulting in an even number of quotes in properly escaped and terminated fields. Otherwise the library will continue to collect lines as escaped, leading to the behavior you're seeing. Eventually you will see an error, but since the file is quite big that can take a while.
In the RFC 4180 this is marked as SHOULD so there may be deviations in files for good reasons. I do not see the reason why this field could not be properly encoded.
There are these things that the lib can improve to address these issues:
- Allow stray quotes in fields not starting with a quote (fields that are not escaped) [opened in #44]
- Add a maximum number of lines to be added to an escaped field with an option to allow configuration. This will make debugging easier for these files. [opened in #45]
- Add context of syntax errors to error message [opened in #46]
I will probably get to work on these some time next week. Thanks again for raising this!
from csv.
@beatrichartz Thanks for looking into this in such detail! I really appreciate it.
from csv.
Related Issues (20)
- Encode Formulas to prevent CSV injection HOT 1
- `headers: true` option doesn't work anymore in version 3.* HOT 5
- Allow parsing of other file encodings (`utf16`, `utf32`, `latin1`) HOT 1
- v3.0.1 module documentation missing on hexdocs.pm HOT 3
- final escaped field value lost when trailing newline is absent HOT 4
- Bug: duplicate error lines after upgrading to v3.0 HOT 2
- Dialyzer fails when using `escape_max_lines` option HOT 1
- Bug: argument error for escape_character HOT 2
- Dialyzer error for decode and decode! HOT 3
- Dealing with keep list intact HOT 2
- Ability to exclude column by name or data type from `escape_formulas`
- Is it possible to get values as nil if the column doesn't have a value when `headers: true`?
- How to export utf8 file with bom?
- CSV.decode! add additional \r\n to double quoted value containing \n HOT 2
- Need to manually strip newlines HOT 1
- Changelog is behind hex.pm versions HOT 1
- Line number is wrong in error message when headers: true HOT 3
- Missing key exception when raising StrayQuoteError HOT 2
- Unable to generate " HOT 2
- Unable to properly enclose cells with double quotes HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from csv.