mrodrig / json-2-csv-cli

NPM package providing CLI functionality for the json-2-csv NPM package

Hi, Welcome to my Profile!

I'm Mike Rodrigues (he/him/his)

• Lead Software Engineer and Architect
• Fullstack Developer - TypeScript/JavaScript (Node, Vue 2/3), Python 3, Swift, Java, Kotlin, Go
• Focused on Cloud-first Solutions - primarily working with Google Cloud Platform/Firebase, but also with Amazon Web Services and Microsoft Azure
• Network Security Enthusiast
• CI/CD Apprentice (Gitlab CI and GitHub Actions)
• Sole Developer, Architect, and Security Advisor for the now-retired Signature Limo Company.
  • Redesigned to use a PaaS serverless architecture
  • Previously designed using a full stack IaaS Linux VM that I hardened

About me ...

json-2-csv-cli's Issues

csv2json on command line, with no output file, prints corrupt JSON and limits to 100 items

Hi! 👋

Firstly, thanks for your work on this project! 🙂

Okay those first two lines are boilerplate from patch-package but seriously, thanks for the project. It's quite useful.

I discovered two things interesting though, for my use case:

1. I am streaming to a file from stdout, I am not specifying a file, and in this case csv2json was printing the raw javascript object via the node console printer, which is not the same as printing JSON. So it needs to be JSON.stringified I think

2. console.log has an implementation-specific limit on the number of lines of raw data it will print, with 100 as a suggestion. Node v16 at least sticks to that 100 limit, so if you have 101 CSV lines you will get 100 objects in your array then '... and 1 more item', instead of all your items. To fix this you can either use utils.inspect (I tested it, it works) or - the same JSON.stringify from 1 above fixes it.

I think if a tool is called csv2json it should print valid JSON in all cases, so I chose solution in 1 for a fix. It's possibly a breaking change though? Depending on how you feel about that, you might like option 2. On the breaking change philosophy front I maintain a lot of packages and at least for me, I'd just bump that major and in the changelog say "csv2json with no output file now prints valid JSON as output. If you relied on the old behavior, adapt to the new valid JSON" or something, and get on with my life. But we're adults and adults can disagree and that's fine :-)

I'll post 2 PRs so you can see both solutions and can decide for yourself - merge one, close the other

Cheers!

Here is the diff that solved my problem printing valid JSON:

diff --git a/node_modules/@mrodrig/json-2-csv-cli/bin/utils/utils.js b/node_modules/@mrodrig/json-2-csv-cli/bin/utils/utils.js
index abe837b..3fc9527 100644
--- a/node_modules/@mrodrig/json-2-csv-cli/bin/utils/utils.js
+++ b/node_modules/@mrodrig/json-2-csv-cli/bin/utils/utils.js
@@ -78,8 +78,8 @@ function processOutput(params) {
         // Pretty print the output when converting from CSV to JSON
         return writeToFile(params.output, JSON.stringify(params.outputData, null, 4));
     }
-    // Otherwise, no output was specified so just send it to stdout via the console
-    console.log(params.outputData); // eslint-disable-line no-console
+    // Otherwise, no output was specified so send JSON to stdout via the console
+    console.log(JSON.stringify(params.outputData, null, 2)); // eslint-disable-line no-console
 }
 
 function constructKeysList(key, keys) {

This issue body was partially generated by patch-package.

Missing new line breaks in output

version

@mrodrig/[email protected]

json chunk

[
  {
      "link": {
          "@href": "http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/1"
      },
      "latitude": "31.0069",
      "longitude": "-88.0103",
      "name": "Barry"
  }, {
      "link": {
          "@href": "http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/2"
      },
      "latitude": "34.0136",
      "longitude": "-85.9703",
      "name": "Gadsden"
  }, {
      "link": {
          "@href": "http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/3"
      },
      "latitude": "33.6446",
      "longitude": "-87.2003",
      "name": "Gorgas"
  }, {
      "link": {
          "@href": "http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/4"
      },
      "latitude": "32.6017",
      "longitude": "-87.7811",
      "name": "Greene County"
  }, {
     .....

---

json2csv current output

note the string 'undefined' after the headers

 $ json2csv Desktop/facilities.json
link.@href,latitude,longitude,nameundefinedhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/1,31.0069,-88.0103,Barry,http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/2,34.0136,-85.9703,Gadsden,http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/3,33.6446,-87.2003,Gorgas,http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/4,32.6017,-87.7811,Greene County,http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/5,33.2442,-86.4567,E C Gaston,http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/6,34.7439,-87.8486,Colbert,http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/7,34.8825,-85.7547,Widows Creek,http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/8,31.4833,-87.9125,Charles R Lowman,http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/9,31.4001,-86.4767,McWilliams,http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/10,33.6319,-87.0597,James H Miller Jr,http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/11,31.2542,-88.0319,McIntosh (7063),http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/12,31.2561,-88.0103,Washington County Cogen (Olin)

subsequently the spreadsheet import summary reflects the same problem

---

Same issue with --eol flags

current [with --eol flag]

note the string literal interpretation of the new line character anywhere we'd expect new line

$ json2csv Desktop/facilities.json --eol "\n"
link.@href,latitude,longitude,name\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/1,31.0069,-88.0103,Barry\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/2,34.0136,-85.9703,Gadsden\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/3,33.6446,-87.2003,Gorgas\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/4,32.6017,-87.7811,Greene County\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/5,33.2442,-86.4567,E C Gaston\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/6,34.7439,-87.8486,Colbert\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/7,34.8825,-85.7547,Widows Creek\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/8,31.4833,-87.9125,Charles R Lowman\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/9,31.4001,-86.4767,McWilliams\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/10,33.6319,-87.0597,James H Miller Jr\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/11,31.2542,-88.0319,McIntosh (7063)\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/12,31.2561,-88.0103,Washington County Cogen (Olin)

 $ json2csv Desktop/facilities.json --eol "\\n"
link.@href,latitude,longitude,name\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/1,31.0069,-88.0103,Barry\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/2,34.0136,-85.9703,Gadsden\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/3,33.6446,-87.2003,Gorgas\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/4,32.6017,-87.7811,Greene County\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/5,33.2442,-86.4567,E C Gaston\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/6,34.7439,-87.8486,Colbert\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/7,34.8825,-85.7547,Widows Creek\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/8,31.4833,-87.9125,Charles R Lowman\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/9,31.4001,-86.4767,McWilliams\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/10,33.6319,-87.0597,James H Miller Jr\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/11,31.2542,-88.0319,McIntosh (7063)\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/12,31.2561,-88.0103,Washington County Cogen (Olin)

 $ json2csv Desktop/facilities.json --eol \n
link.@href,latitude,longitude,namenhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/1,31.0069,-88.0103,Barrynhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/2,34.0136,-85.9703,Gadsdennhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/3,33.6446,-87.2003,Gorgasnhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/4,32.6017,-87.7811,Greene Countynhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/5,33.2442,-86.4567,E C Gastonnhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/6,34.7439,-87.8486,Colbertnhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/7,34.8825,-85.7547,Widows Creeknhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/8,31.4833,-87.9125,Charles R Lowmannhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/9,31.4001,-86.4767,McWilliamsnhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/10,33.6319,-87.0597,James H Miller Jrnhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/11,31.2542,-88.0319,McIntosh (7063)nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/12,31.2561,-88.0103,Washington County Cogen (Olin)

 $ json2csv Desktop/facilities.json --eol \\n
link.@href,latitude,longitude,name\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/1,31.0069,-88.0103,Barry\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/2,34.0136,-85.9703,Gadsden\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/3,33.6446,-87.2003,Gorgas\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/4,32.6017,-87.7811,Greene County\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/5,33.2442,-86.4567,E C Gaston\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/6,34.7439,-87.8486,Colbert\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/7,34.8825,-85.7547,Widows Creek\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/8,31.4833,-87.9125,Charles R Lowman\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/9,31.4001,-86.4767,McWilliams\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/10,33.6319,-87.0597,James H Miller Jr\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/11,31.2542,-88.0319,McIntosh (7063)\nhttp://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/12,31.2561,-88.0103,Washington County Cogen (Olin)

---

expected output

 $ json2csv Desktop/facilities.json
link.@href,latitude,longitude,name
http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/1,31.0069,-88.0103,Barry
http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/2,34.0136,-85.9703,Gadsden
http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/3,33.6446,-87.2003,Gorgas
http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/4,32.6017,-87.7811,Greene County
http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/5,33.2442,-86.4567,E C Gaston
http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/6,34.7439,-87.8486,Colbert
http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/7,34.8825,-85.7547,Widows Creek
http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/8,31.4833,-87.9125,Charles R Lowman
http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/9,31.4001,-86.4767,McWilliams
http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/10,33.6319,-87.0597,James H Miller Jr
http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/11,31.2542,-88.0319,McIntosh (7063)
http://10.1.6.16:8080/emit/resources/nations/1/states/1/facilities/12,31.2561,-88.0103,Washington County Cogen (Olin)

and equally expected is the spreadsheet import summary based on good output

---

my temporary patch

json2csv.js

220c220
<         }).join(options.delimiter.eol);
---
>         }).join(options.delimiter.eol ? options.delimiter.eol : constants.defaultOptions.delimiter.eol);
351c351
<                 (options.prependHeader ? header + options.delimiter.eol : '') +
---
>                 (options.prependHeader ? header + (options.delimiter.eol ? options.delimiter.eol : constants.defaultOptions.delimiter.eol) : '') +

NestedDotKeys with false and 0 issue [report from json-2-csv]

mrodrig/json-2-csv#230

Pull in latest fix from json-2-csv@3.20.0

converter require path missing one more folder step out/up

More of an fyi than an issue (for me anyways).
From the context of having installed both json-2-csv and @mrodrig/json-2-csv-cli with the global flag npm install -g

Within: utils.js
this doesn't work: converter = require('../../../json-2-csv');
this works: converter = require('../../../../json-2-csv');

CLI apps don't appear to be parsing parameters correctly

I noticed that json2csv doesn't appear to be respecting any of the parameters passed. After a bit of digging, I think the problem is that opts() isn't being called on the commander object, so all of the fields are being passed in as undefined.

From a "clean" environment:

root@4a9ea7587e5e:/usr/src/app# npm i @mrodrig/json-2-csv-cli

added 5 packages, and audited 6 packages in 2s

found 0 vulnerabilities
npm notice
npm notice New minor version of npm available! 8.12.1 -> 8.16.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v8.16.0
npm notice Run npm install -g [email protected] to update!
npm notice
root@4a9ea7587e5e:/usr/src/app# echo "{}" > a
root@4a9ea7587e5e:/usr/src/app# ./node_modules/.bin/json2csv a
"\n"
root@4a9ea7587e5e:/usr/src/app# ./node_modules/.bin/json2csv -o foo a
"\n"
root@4a9ea7587e5e:/usr/src/app# cat foo
cat: foo: No such file or directory

I've never used commander, so I might be totally off base here, but assuming that looks right to you, I'd be happy to send over a pull request with a suggested fix.