apache / arrow

Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing

Apache Arrow

Powering In-Memory Analytics

Apache Arrow is a development platform for in-memory analytics. It contains a set of technologies that enable big data systems to process and move data fast.

Major components of the project include:

• The Arrow Columnar In-Memory Format: a standard and efficient in-memory representation of various datatypes, plain or nested
• The Arrow IPC Format: an efficient serialization of the Arrow format and associated metadata, for communication between processes and heterogeneous environments
• The Arrow Flight RPC protocol: based on the Arrow IPC format, a building block for remote services exchanging Arrow data with application-defined semantics (for example a storage server or a database)
• C++ libraries
• C bindings using GLib
• C# .NET libraries
• Gandiva: an LLVM-based Arrow expression compiler, part of the C++ codebase
• Go libraries
• Java libraries
• JavaScript libraries
• Python libraries
• R libraries
• Ruby libraries
• Rust libraries

Arrow is an Apache Software Foundation project. Learn more at arrow.apache.org.

What's in the Arrow libraries?

The reference Arrow libraries contain many distinct software components:

• Columnar vector and table-like containers (similar to data frames) supporting flat or nested types
• Fast, language agnostic metadata messaging layer (using Google's Flatbuffers library)
• Reference-counted off-heap buffer memory management, for zero-copy memory sharing and handling memory-mapped files
• IO interfaces to local and remote filesystems
• Self-describing binary wire formats (streaming and batch/file-like) for remote procedure calls (RPC) and interprocess communication (IPC)
• Integration tests for verifying binary compatibility between the implementations (e.g. sending data from Java to C++)
• Conversions to and from other in-memory data structures
• Readers and writers for various widely-used file formats (such as Parquet, CSV)

Implementation status

The official Arrow libraries in this repository are in different stages of implementing the Arrow format and related features. See our current feature matrix on git main.

How to Contribute

Please read our latest project contribution guide.

Getting involved

Even if you do not plan to contribute to Apache Arrow itself or Arrow integrations in other projects, we'd be happy to have you involved:

• Join the mailing list: send an email to [email protected]. Share your ideas and use cases for the project.
• Follow our activity on GitHub issues
• Learn the format
• Contribute code to one of the reference implementations

arrow's Issues

C++: establish a basic function evaluation model

We don't exactly have a working model for executing functions on Arrow objects that require memory allocations (either for intermediate storage of results).

For example:

F(input1, input2) -> output

Since the output type may depend on the input types, F in general will need to be able to request memory to be allocated. This will have to be figured out (some sort of ArrowContext with a memory pool attached) in due course. For example:

F(ctx, input1, input2) -> output

Where ctx is a Context object provided access to memory resources and other system state (user-configurable).

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-34. Please see the migration documentation for further details.}

Set up Travis CI

I will ask INFRA to enable Travis CI for the repo, and then will propose a patch that runs the C++ test suite to start (unless some kind soul beats me to it with a Java patch). We can use a build matrix with one build per language SDK (so gcc and clang for arrow-cpp) to start.

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-8. Please see the migration documentation for further details.}

C++: Add google/benchmark to the 3rd-party build toolchain

https://github.com/google/benchmark, to assist with building a microbenchmark suite.

Reporter: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-28. Please see the migration documentation for further details.}

C++: Algorithms for using nested types in a hash table context

Computing hash values (and performing equality comparisons) for top-level slots in nested-type data (for example, computing DISTINCT on a List<List<Int32>>, related: ARROW-32) can be fairly complex. Additionally, value slots at any level of the type tree can be null.

We should explore various algorithms for their performance and memory use in practical settings. For example, one can compute a contiguous "record" / byte array resulting from a depth-first traversal of a single value slot for the purposes of computing a hash value or comparing with another slot. If anyone has other ideas from past experiences I would be keen to learn more.

Reporter: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-38. Please see the migration documentation for further details.}

C++: Enable library builds for 3rd-party users without having to build thirdparty googletest

While the thirdparty dependencies for libarrow may not always remain so minimal, for now it would be useful to have a library build option where only necessary thirdparty dependencies are built (currently there are none).

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-50. Please see the migration documentation for further details.}

Get Github activity mirrored to JIRA

Reporter: Jacques Nadeau / @jacques-n
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-12. Please see the migration documentation for further details.}

C++: Add in-memory schema metadata container

The schema will consist of an array of fields / table columns (name + data type). The scope is limited to what will suffice for converting other schema metadata representations (especially Parquet) to Arrow schemas.

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-21. Please see the migration documentation for further details.}

Fix mismatch of javadoc names and method parameters

Reporter: Liwei Lin(Inactive) / @lw-lin

PRs and other links:

• PR#3

_{Note: This issue was originally created as ARROW-10. Please see the migration documentation for further details.}

Error when run maven install

when I run maven to install, I got following problem:

Failed to execute goal
org.apache.drill.tools:drill-fmpp-maven-plugin:1.4.0:generate
(generate-fmpp) on project vector: Execution generate-fmpp of goal
org.apache.drill.tools:drill-fmpp-maven-plugin:1.4.0:generate failed:
Plugin org.apache.drill.tools:drill-fmpp-maven-plugin:1.4.0 or one of its
dependencies could not be resolved: Failure to find
org.freemarker:freemarker:jar:2.3.24-SNAPSHOT in
http://repository.apache.org/snapshots was cached in the local repository

btw, I just clone repo and run mvn clean install.

dev mailing link
http://mail-archives.apache.org/mod_mbox/arrow-dev/201602.mbox/%3CCAABsKVCSEULDTL2hoANL8-wrWMDO8%3Dgv0RFmSQMXt3MdiqUcPw%40mail.gmail.com%3E

Environment: Ubuntu Maven 3.2
Reporter: AllenFang
Assignee: Liwei Lin(Inactive) / @lw-lin

_{Note: This issue was originally created as ARROW-5. Please see the migration documentation for further details.}

C++: Externalize memory allocations and add a MemoryPool abstract interface to builder classes

Currently memory allocations in the C++ implementation are all internal / self-contained, but in practice the array builders will need to integrate with some other memory allocator / tracker.

I'll define a basic abstract memory pool interface and add an example default implementation (with a resizable arrow::Buffer subclass that requests memory from the pool) with all memory managed internally.

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-19. Please see the migration documentation for further details.}

C++: Add developer instructions for building parquet-cpp integration

To avoid dependency management problems I added build instructions in cpp/doc/Parquet.md for now. See patch #12

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-26. Please see the migration documentation for further details.}

[Python] Add unnest/flatten function for List types

Reporter: Wes McKinney / @wesm
Assignee: Krisztian Szucs / @kszucs

PRs and other links:

• GitHub Pull Request #2757

_{Note: This issue was originally created as ARROW-45. Please see the migration documentation for further details.}

C++: Reinterpret Struct arrays as tables

This is mostly a question of layering container types, but will be provided as an API convenience.

Reporter: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-40. Please see the migration documentation for further details.}

Import Initial Codebase

From Drill vector module

Reporter: Jacques Nadeau / @jacques-n
Assignee: Steven Phillips / @StevenMPhillips

_{Note: This issue was originally created as ARROW-1. Please see the migration documentation for further details.}

Add PR merge tool similar to that used in Parquet

See https://github.com/apache/parquet-mr/tree/master/dev

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-13. Please see the migration documentation for further details.}

C++: Represent boolean array data in bit-packed form

This is not explicit in the Arrow specification (we should make this concrete), but has the additional benefit of making code for null bitmaps and boolean arrays interchangeable.

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-37. Please see the migration documentation for further details.}

C++: Add logical "Table" container

Depends on ARROW-21 and ARROW-23. Logical tables consist of a schema and logical columns, each of which may contain zero or more Arrow arrays

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-24. Please see the migration documentation for further details.}

[C++] Implement delimited file scanner / CSV reader

Like Parquet and binary file formats, text files will be an important data medium for converting to and from in-memory Arrow data.

pandas has some (Apache-compatible) business logic we can learn from here (as one of the gold-standard CSV readers in production use)

https://github.com/pydata/pandas/blob/master/pandas/src/parser/tokenizer.h
https://github.com/pydata/pandas/blob/master/pandas/parser.pyx

While very fast, this this should be largely written from scratch to target the Arrow memory layout, but we can reuse certain aspects like the tokenizer DFA (which originally came from the Python interpreter csv module implementation)

https://github.com/pydata/pandas/blob/master/pandas/src/parser/tokenizer.c#L713

Reporter: Wes McKinney / @wesm
Assignee: Antoine Pitrou / @pitrou

PRs and other links:

• GitHub Pull Request #2576

_{Note: This issue was originally created as ARROW-25. Please see the migration documentation for further details.}

Fix a naming typo for memory.AllocationManager.AllocationOutcome

Rename FORCED_SUCESS to FORCED_SUCCESS.

Reporter: Liwei Lin(Inactive) / @lw-lin
Assignee: Liwei Lin(Inactive) / @lw-lin

PRs and other links:

• PR#4

_{Note: This issue was originally created as ARROW-15. Please see the migration documentation for further details.}

Python: Add rudimentary console __repr__ for array types

This will improve development productivity in testing and prototyping

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-43. Please see the migration documentation for further details.}

Add a short call-to-action / how-to-get-involved to the main README.md

• Engage on the mailing list
• Read the format documentation
• Contribute code and design ideas to the reference implementations

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-35. Please see the migration documentation for further details.}

C++: Upload gcov coverage data to coveralls.io

I recently did this for Parquet (apache/parquet-cpp@9cab887); it will be nice to have the coverage report available to assist with unit testing.

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

PRs and other links:

• GitHub Pull Request #1241

_{Note: This issue was originally created as ARROW-27. Please see the migration documentation for further details.}

C++: Add null count member to Array containers, remove nullable member

Nullability/non-nullability may become a property of the metadata (if at all, TBD based on implementation-wide discussions); in the meantime the array containers and associated algorithms will be simplified.

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-20. Please see the migration documentation for further details.}

Add Python library build toolchain

I will be working on a patch to make the initial Arrow C++ libarrow.so library callable from Cython (http://cython.org/) extensions. For the uninitiated, Cython is the modern "gold standard" for interoperability with C++ libraries. I have used it recently to create Kudu's Python client (https://github.com/apache/incubator-kudu/tree/master/python/kudu).

A significant amount of Python "glue code" will be needed to interoperate with pandas, NumPy, and other standard Python libraries (in addition to Python's built-in scalar data types), but these will be the subject of many follow up JIRAs.

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-7. Please see the migration documentation for further details.}

Python: basic PyList <-> Arrow marshaling code

To start, this will encompass only lists containing flat data (i.e. no list or dict members). For example:

[1, 2, None, 3, 4]
['foo', 'bar', None, 'baz']
[True, False, False, True, None]

Type inference on more complicated Python lists (as occurring from json.loads, for example) is something we can explore later (if it is deemed useful).

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-31. Please see the migration documentation for further details.}

Python: Implement basic object model for scalar values (i.e. results of arrow_arr[i])

This will enable Arrow arrays to behave more like other Python collections (lists, tuples)

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-44. Please see the migration documentation for further details.}

Remove fixVersions from patch tool (until we have them)

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-36. Please see the migration documentation for further details.}

C++: Implement zero-copy array slicing

Sliced array results will retain a reference to the parent's data buffer

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

Related issues:

• Arrow release 0.2 (blocks)

_{Note: This issue was originally created as ARROW-33. Please see the migration documentation for further details.}

Post Initial Arrow Format Spec

Reporter: Jacques Nadeau / @jacques-n
Assignee: Wes McKinney / @wesm

Original Issue Attachments:

• 0001-arrow-format-draft.patch.gz

_{Note: This issue was originally created as ARROW-3. Please see the migration documentation for further details.}

Rename some unchanged "Drill" to "Arrow"

There are still some unchanged "Drill" in codes, comments, tests. Let's rename them to "Arrow".

Reporter: Liwei Lin(Inactive) / @lw-lin
Assignee: Liwei Lin(Inactive) / @lw-lin

PRs and other links:

• PR#2
• PR#18

_{Note: This issue was originally created as ARROW-9. Please see the migration documentation for further details.}

Mirror JIRA activity to [email protected]

I presume this is an ask of INFRA but tracking the task here.

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-11. Please see the migration documentation for further details.}

C++: Add logical "Column" container for chunked data

In some in-memory applications, logical columns will contain a list of primitive (contiguous) Arrow arrays. In particular, this strategy would be applied to represent very large in-memory tables (e.g. exceeding INT32_MAX rows)

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-23. Please see the migration documentation for further details.}

Post Simple Website

Reporter: Jacques Nadeau / @jacques-n
Assignee: Jason Altekruse

_{Note: This issue was originally created as ARROW-2. Please see the migration documentation for further details.}

Port DRILL-4410 to Arrow

This fixes a bug in ListVector which causes OversizeAllocation

Reporter: Steven Phillips / @StevenMPhillips
Assignee: Steven Phillips / @StevenMPhillips

_{Note: This issue was originally created as ARROW-46. Please see the migration documentation for further details.}

C++: Logical chunked arrays / columns: conforming to fixed chunk sizes

Implementing algorithms on large arrays assembled in physical chunks is problematic if:

• The chunks are not all the same size (except possibly the last chunk, which can be less). Otherwise, retrieving a particular element is in general a O(log num_chunks) operation

• The chunk size is not a power of 2. Computing integer modulus with a non-multiple of 2 requires more clock cycles (in other words, i % p is much more expensive to compute than i & (p - 1), but the latter only works if p is a power of 2)

Most of the Arrow data adapters will either feature contiguous data (1 chunk, so chunking is not an issue) or a regular chunk size, so this isn't as much of an immediate concern, but we should consider making it a contract of any data structures dealing in multiple arrays.

In general, it would be preferable to reorganize memory into either a regular chunksize (like 64K values per chunk) or a contiguous memory region. I would prefer for the moment to not to invest significant energy in writing algorithms for data with irregular chunk sizes.

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-39. Please see the migration documentation for further details.}

Initial Arrow CPP Implementation

Reporter: Jacques Nadeau / @jacques-n
Assignee: Wes McKinney / @wesm

Original Issue Attachments:

• 0001-arrow-initial-cpp.patch.gz

_{Note: This issue was originally created as ARROW-4. Please see the migration documentation for further details.}

C++: Add schema adapter routines for converting flat Parquet schemas to in-memory Arrow schemas

Depends on ARROW-21. There's many cases to implement here (e.g. 3 kinds of Parquet list encoding that all map onto the single Arrow list type) so I may create follow-up JIRAs for some of the less-commonly-occurring Parquet schemas.

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-22. Please see the migration documentation for further details.}

Set some vector fields to default access level for Drill compatibility

Drill has methods for serializing and deserializing value vectors, and this functionality was not included in the code checked into the Arrow repo. It was kept in drill. For these Drill methods to work, some private vector fields need to be made package level so that the Drill helper methods can access them.

Reporter: Steven Phillips / @StevenMPhillips
Assignee: Steven Phillips / @StevenMPhillips

_{Note: This issue was originally created as ARROW-17. Please see the migration documentation for further details.}

Hope to add development document

Awesome project, great job :)
Anyway, is possible to add some useful documents for development

Thanks

Reporter: AllenFang

_{Note: This issue was originally created as ARROW-6. Please see the migration documentation for further details.}

C++: add hash table classes for fixed-byte-width and variable-length primitive arrays

Some of the most important in-memory analytical routines are:

• unique
• contains / is-in
• match (see base::match in R or pandas.match)
• dictionary-encode (aka "factorize" as I call it)
• frequency-table (unique + observed frequencies)

At their lowest level these all involve either iterative hash table construction or construct-then-sweep (for the routines involving multiple arrays, e.g. contains/match).

Hashing more complex Arrow structures (e.g. structs or lists-of-structs) will require some more thought, but performing these operations on fixed-byte-width types and lists thereof (e.g. strings as List) is fairly straightforward and can be used to craft more complex hash-table based routines.

Reporter: Wes McKinney / @wesm
Assignee: Antoine Pitrou / @pitrou

_{Note: This issue was originally created as ARROW-32. Please see the migration documentation for further details.}

C++: Add re2 as optional 3rd-party toolchain dependency

We will want to eventually add RE2 wrapper functions which yield Arrow data structures resulting from applying regular expressions to Arrow data representing strings (for example, to implement operations like split_all(String) -> List)

Reporter: Wes McKinney / @wesm
Assignee: Max Risuhin / @maxhora

_{Note: This issue was originally created as ARROW-29. Please see the migration documentation for further details.}

Building cpp issues on XCode 7.2.1

Compiler options aren't quite right.

Environment: OS X 10.11.13. Last commit e6905ef
Reporter: Joseph Winston
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-16. Please see the migration documentation for further details.}

Add JIRA components

This will make it easier to look at JIRAs in component-level views. In the style of Parquet, we could do something like this to start:

• Format
• Java
• C++
• Python

Reporter: Wes McKinney / @wesm
Assignee: Jacques Nadeau / @jacques-n

_{Note: This issue was originally created as ARROW-14. Please see the migration documentation for further details.}

Python: Add to Travis CI build

A bit of wrangling to do here, but it will be good to have green builds in pull requests

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-42. Please see the migration documentation for further details.}

Fix bug with decimal precision and scale

The precision and scale parameters are reversed in the MapWriters template.

Reporter: Steven Phillips / @StevenMPhillips
Assignee: Steven Phillips / @StevenMPhillips

_{Note: This issue was originally created as ARROW-18. Please see the migration documentation for further details.}

C++: Convert RecordBatch to StructArray, and back

With arrow::TableBatchReader, we can turn a Table into a sequence of one or more RecordBatches. It would be useful to be able to easily convert between RecordBatch and a StructArray (which can be semantically equivalent in some contexts)

Reporter: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-41. Please see the migration documentation for further details.}

Python: Add Schema object wrapper

See arrow::Schema in C++

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-48. Please see the migration documentation for further details.}

Python: Add Column and Table wrapper interface

Per arrow::Column and arrow::Table. This will be the target data structure for most pandas.DataFrame-like data

Reporter: Wes McKinney / @wesm
Assignee: Uwe Korn / @xhochy

_{Note: This issue was originally created as ARROW-49. Please see the migration documentation for further details.}

[C++] Consider adding a scalar type object model

Just did this on the Python side. In later analytics routines, passing in scalar values (example: Array + Scalar) requires some kind of container. Some systems, like the R language, solve this problem with length-1 arrays, but we should do some analysis of use cases and figure out what will work best for Arrow.

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

Related issues:

• [C++] Add kernels for comparison operations to scalars (blocks)

PRs and other links:

• GitHub Pull Request #3604

_{Note: This issue was originally created as ARROW-47. Please see the migration documentation for further details.}

Python: pandas/NumPy to/from Arrow conversion routines

These will use the Python/NumPy C API in tandem with the Arrow C++ API.

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-30. Please see the migration documentation for further details.}