Comments (1)
Closing this as I figured out why this doesn't work and what does instead. The first point to note is that get arrays, you need to use the repeated
field of a Field
.
Nevertheless, the problem with above data in JSON lines format is that Croissant (via Pandas) will try to read each array index in the line as a separate column in a dataframe, which is definitely not what we want. This is to say that if you want to to use JSON lines you will need to do something like this (presumably, haven't tried it yet):
{"values": [3, 1, 4]}
{"values": [1, 5, 9, 2]}
{"values": [6, 5]}
From there, I think you can just copy off of the introduction recipe (https://github.com/mlcommons/croissant/blob/main/python/mlcroissant/recipes/introduction.ipynb). Alternatively, you can just store the array of arrays one big JSON array and use extract=mlc.Extract(json_path="$[*]")
under your Source
.
from croissant.
Related Issues (20)
- Joins as described by Croissant Format Specification are not supported by mlcroissant python library.
- Add containedIn fileProperty to the Extract class
- [NeurIPS] How do i create a Croissant File using this library HOT 2
- [NeurIPS] Fileld has no source HOT 8
- Feature Request: support for CoNLL format HOT 2
- [NeurIPS] How do i create a Croissant File using this library for my dataset? HOT 1
- [NeurIPS] Compatibility with HuggingFace Image dataset HOT 1
- [NeurIPS] Data type extension (Video and multichannel time series ) HOT 3
- [NeurIPS] URL to dataset metedata
- Example for audio dataset HOT 2
- [NeurIPS] How to "ingest" multiple datasets made of .xz files as data/samples and space-separated .txt files as ground truth
- Please improve documentation
- Intro example not working HOT 3
- [NeurIPS] Support for (Sequential) Dataset Stored in TDMS Files HOT 3
- [Neurips]: any support for large h5 files? tried various encodings but no luck. HOT 1
- JSON LD parsing fails when multiple strings given to FileSet.includes HOT 1
- [NeurIPS] Join fails with reference if read from fileObject HOT 2
- Feature request: loading a record set as a pandas dataframe HOT 3
- Expose the underlying pd.DataFrames instead of an iterator
- Feature request: Add a mapping from FieldId to RecordSetId to Metadata.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from croissant.