Some numbers convert but other numbers and texts getting lost. Thank you!! about json

[konklone]

Hi! I appreciate you filing this and sharing the JSON in question. This JSON is, in general, not an ideal candidate for CSV conversion because most of the data is in one big object. CSVs are designed to present lists of data structures, so generally it works well when there's a clear list of data inside the JSON. A lot of times, the array won't be the top level of the object, there will be some document-level metadata or something at the top level of the object, but isn't what people are looking for in their CSV.

So, this converter will "hunt for the array" inside an object to find the list of things that the user likely wants CSV-ized. In this case, it finds the small array of numbers that you see it convert into a small CSV. That's obviously not ideal, but it's also not obvious what this object should be converted into instead. Were you looking for a large one-row CSV with every field compressed into it?

JSON-to-CSV is an imperfect fit from a data perspective, since JSON is rarely tabular and CSV is only tabular. This may be one of those that isn't well suited to it, but if you can describe the ideal end state, I can consider whether it's reasonable to have general purpose logic there for it.

[alabamian2]

Thank you so much for your super quick response and happy holidays. Truly appreciated. My use case is with Google PageSpeed Insights API and that json file represents one report for one URL. And I have thousands of URLs. The API returns json and I need to load that data into Google BigQuery (then Data Studio), but BigQuery only allows Newline Delimited JSON so that file doesn't work. Even to load JSON into Google Dataprep (and Google stack in general) to do transformation/etc, you have to transform to ndjson to load into Dataprep where you are really supposed to do all the cleaning. Many folks out there seem to be having the same issue, but there isn't a good tool and I came across yours. My options (I think) are to convert this json to ndjson or to flatten to csv in order to load into BigQuery/Google. This JSON tends to contain many elements (https://developers.google.com/speed/docs/insights/v5/reference/pagespeedapi/runpagespeed#response).

Thank you again so much for responding with very detailed guidance, especially in the end of the year period. Please stay healthy.

[konklone]

So each JSON response is one object, and you have many - so really what you want is for each of these objects to be one row, which means you want to make one big JSON object that has all of these objects as members.

But there is a way to do this easily - my converter does detect "JSON lines" (newline-separated JSON objects), and so if you paste in multiple objects with a newline between them, I suspect this will do what you want. I took the object from your first link above and just pasted it again after itself, and got this:

This makes the converter detect the "primary list" as the newline-separated objects, rather than digging into an object for the first array.

So, what you could do here is first paste them into a text editor (newline-separated) and then paste the collected object into my converter. But these are a bit big, and if you have thousands the site will probably choke on doing one massive paste of all of them. You could do them a few (at least 2) at a time and then paste the results as well (you can use the "show the raw data" link to get he raw CSV) into a new file, appending each one. It's a bit tedious but there's no real way around that tedium whichever way you take, since fundamentally you're accessing independent URLs and needing to move that JSON around.

It sounds like you have a task that is better suited to dedicated scripting, but if you want to use this web-based tool to do it, pasting in multiple objects would get you what I think you want.

[alabamian2]

Yes, this totally DOES work. This is wonderful!! I QA'ed it and looks to be complete. Thank you so much again for your help and hope this thread gets indexed/picked up by others since you gave us so much details. Thank you and have a nice and safe holiday.

[konklone]

Glad to hear it! =)