Comments (4)
Writing a data frame is not a part of DataFrames.jl functionality. I would assume that serializing it using the Serialization
module should use least memory to write data (I assume this is what you ask for).
from dataframes.jl.
Which data format is the most efficient in Julia? The .arrow format consumes a lot of memory. For example, in R, the .fst format is considered the best for efficient memory usage and high performance. Similarly, in Julia, which format is optimal for writing DataFrames with high performance and memory efficiency?
from dataframes.jl.
Which data format is the most efficient in Julia?
There is no single format that is best in all aspects, so "the best" depends on many factors.
If you want to see a comparison of various formats performance you can have a look here: https://github.com/bkamins/Julia-DataFrames-Tutorial/blob/master/04_loadsave.ipynb. Note that these benchmarks report:
- write time
- load time
- size on disk
- memory used to write
- memory used to read
(and even here you see that your question ends up a 5-criteria problem)
Also these are reported when run on a laptop using 1 thread. Benchmarks might be different when wanting best performance on a mulit-core server scenario.
Your question is essentially open ended and unrelated with DataFrames.jl (it is a general Julia question). Such questions are welcome, but it is best to discuss them in an open-ended forum, as you might get the best advice there (as me or other DataFrames.jl maintainers might not be aware of all the options). I recommend you to post it on https://discourse.julialang.org/.
from dataframes.jl.
Thank you very much for this great information.
from dataframes.jl.
Related Issues (20)
- error when unique! a empty dataframe HOT 1
- ggplot2 has added an `after_stat`
- feature request: allow `skipmissing` column types HOT 3
- `combine` on grouped df return empty df when args is empty HOT 2
- DF description/missing color is unreadable on dark themes HOT 44
- Add rename!(::DataFrame, ::Pair{Regex, SubstitutionString}) method
- GroupBy then combine changes column order HOT 2
- Inconsistent Mean Calculation in Grouped DataFrame Compared to Overall DataFrame HOT 2
- Segmentation Fault when reading compressed file HOT 1
- Revisit spreading for `AsTable` output` HOT 6
- Better error message when forming a DataFrame from a vector of dictionaries with missing data. HOT 2
- `describe` is slow HOT 3
- CartesianIndex error in Julia 1.11 HOT 4
- `DataFrame(x=Int[], y=Int)` HOT 3
- Add comparison function for dataframes which can handle both isapprox and isequal column types HOT 2
- unique fails with column-type FixedDecimal HOT 5
- mapcols! should modify the parent of a SubDataFrame HOT 11
- Feature request: Pairs in stack HOT 2
- Grouped DataFrame with array elements fails to combine HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dataframes.jl.