chiphuyen / just-pandas-things

An ongoing list of pandas quirks

just-pandas-things

This repo contains a few peculiar things I've learned about pandas that have made my life easier and my code faster. This post isn't a friendly tutorial for beginners, but a friendly introduction to pandas weirdness.

What's in this repo?

1. pandas is column-major, which is why row-based operations are slow
2. SettingWithCopyWarning, or why we can't have nice things
3. Indexing and slicing
4. Accessors
5. Data exploration
6. Common pitfalls

I'll continue updating this repo as I have more time. As I'm still learning pandas quirks, feedback is much appreciated.

Thanks Luke Metz, Vikram Tiwari, and Karson Elmgren for reviewing!

just-pandas-things's Issues

Typo?

In just-pandas-things.ipynb

Note: You can also just use df.loc[0, "Review"] to calculate the memory address to retrieve the item. Its performance is comparable to gret row then get review.

gret row then get review should be get review then get row?

Hidden Chaining responsible for SettingWithCopyWarning

Hidden chaining is another frequent issue responsible for SettingWithCopyWarning. It may occur the following way:

import pandas as pd
import numpy as np
d = {'col1': [1, 2, 3], 'col2': [3, 4, np.nan]}
df = pd.DataFrame(data=d)

# This will cause the warning
df_no_na = df.dropna()
df_no_na.loc[:, 'new_column'] = df_no_na['col1'] + df_no_na['col2']

# This won't cause the warning
df_no_na = df.dropna().copy()
df_no_na.loc[: ,'new_column'] = df_no_na['col1'] + df_no_na['col2']

The point is that it also occurs when using df.loc[:, 'new_column'] so using .loc[] does not always prevent this warning from happening. The reason for the warning is obviously the same as in the example in your notebook, however i think this issue of chaining a get and an assignment is less obvious, hence 'hidden' :-).