Cannot create dataframe containing null values about nodejs-polars

[daolanfler]

for those who want a temporary solution, you can check if such column in the df.columns array. then use with_columns to add a column full of null values

const data3 = [
  { col1: "b", col2: "d", str: "1 -> 1 -> 2", time: 1691565256703, col3: null },
  { col1: "b", col2: "c", str: "2 -> 2 -> 1", col3: null },
  { col1: "c", col2: "a", str: "3 -> 3 -> 3", col3: null },
];

let df3 = pl.DataFrame(data3 );
if (!df3.columns.includes('col3')) {
    df3 = df3.withColumn(pl.lit(null).alias("col3"));
}

┌──────┬──────┬─────────────┬───────────┬──────┐
│ col1 ┆ col2 ┆ str         ┆ time      ┆ col3 │
│ ---  ┆ ---  ┆ ---         ┆ ---       ┆ ---  │
│ str  ┆ str  ┆ str         ┆ f64       ┆ null │
╞══════╪══════╪═════════════╪═══════════╪══════╡
│ b    ┆ d    ┆ 1 -> 1 -> 2 ┆ 1.6916e12 ┆ null │
│ b    ┆ c    ┆ 2 -> 2 -> 1 ┆ null      ┆ null │
│ c    ┆ a    ┆ 3 -> 3 -> 3 ┆ null      ┆ null │
└──────┴──────┴─────────────┴───────────┴──────┘

[johanroelofsen]

While looking into this one, I noticed that providing arrays into the object:

const lf = pl.DataFrame({ a: [1], b: [2], c: [null] });
console.log(lf);

returns

shape: (1, 3)
┌─────┬─────┬──────┐
│ a   ┆ b   ┆ c    │
│ --- ┆ --- ┆ ---  │
│ f64 ┆ f64 ┆ f64  │
╞═════╪═════╪══════╡
│ 1.0 ┆ 2.0 ┆ null │
└─────┴─────┴──────┘

Because of this notice of the documentation:

Interface DataFrame

A DataFrame is a two-dimensional data structure that represents data as a table with rows and columns.
Param

Object, Array, or Series Two-dimensional data in various forms. object must contain Arrays. Array may contain Series or other Arrays.

I am a little bit hesistant whether I should further check this?

[forgetso]

It's probably not a high priority ticket as you've demonstrated a workaround. I am coming from pandas where the other syntax does work.

>>> pd.DataFrame([{'a':1, 'b':2, 'c':None}])
   a  b     c
0  1  2  None

[Bidek56]

I have a created a PR for this issue.

[Bidek56]

But I do see a problem after my PR.
pl.DataFrame([{a:1, b:2, c:null}]) will omit column c, unless one of the rows has a non-null value.

[universalmind303]

But I do see a problem after my PR.
pl.DataFrame([{a:1, b:2, c:null}]) will omit column c, unless one of the rows has a non-null value.

I believe this is a known issue in the polars json parser. which the JS parser is mostly based off of.

pola-rs/polars#7858