(Somewhat) silly question: what is a 'dataframe'? about explorer

[josevalim]

Good point. As things evolve, should we make sure to write the docs in a way we don't assume everyone has a ML/Pandas background.

[kenny-evitt]

This post (on a blog about a new Unix shell, of all things) actually seems to be trying to answer this very question:

• What Is a Data Frame? (In Python, R, and SQL)

From that post:

What Is a Data Frame? (In Python, R, and SQL)

A data frame is a table-like data structure available in languages like R and Python. Statisticians, scientists, and programmers use them in data analysis code.

...

1. In the last post, I analyzed Oil's code with data frames.

   When writing the praise in that section, I looked for a friendly intro to the idea, but the best hit was from an inactive Clojure project. Surprisingly, Wikipedia doesn't have a page on data frames! This post aims to fill that gap.

...

Typical Operations

Before I show source code, here's some more color on how data frames are used. You may:

• Create a data frame from the results of a SQL query, or from a CSV file. The columns have types like string, number, and date.
• Filter a data frame down to the rows and columns of interest.
• Clean its values with arithmetic and string operations.
• Summarize groups of rows.
• Compute new columns based on existing columns.
• Join a data frame with others, for further analysis.
• Plot one column vs. another (in many different ways)
• Mathematically model one column as a function of another — e.g. with linear regression.

Logically, data frames look like SQL tables, but:

• They're manipulated directly in the memory of your program. They don't live on a remote server.
• They more suited to complex, "offline" analysis with statistics, visualization, and mathematical modeling.

The first item in that last list doesn't seem to be entirely true – or at least not according to Databricks about their Spark DataFrames:

... a Spark DataFrame can span thousands of computers

So SQL tables are kinda 'dataframes' but typically a 'dataframe' 'lives' closer to the 'processing', e.g. 'in-memory' versus accessible over the network or (for Elixir) outside the BEAM VM (?).