dlab-berkeley / python-data-wrangling-legacy Goto Github PK
View Code? Open in Web Editor NEWD-Lab's 3 hour introduction to data wrangling in Python. Learn how to import and manipulate dataframes using pandas in Python.
License: Other
D-Lab's 3 hour introduction to data wrangling in Python. Learn how to import and manipulate dataframes using pandas in Python.
License: Other
Some of the participants asked if there could be more time spent on groupby()
, as they need it for their research. I feel it's important to spend at least 20' on this section and challenge. Not sure if I have an immediate idea for how to do this -- another challenge after #10, giving more examples of groupby()
functionalities or scenarios, introducing it earlier (e.g. before merging).
Name should be changed from "Introduction to Pandas" to "Python Data Wrangling and Manipulation With Pandas"
A minor error, but in introduction-to-pandas.ipynb
file, the header:
Should actually read:
as the challenge requires to get the year component of the Dataframe column and not the month.
unemployment_rate
is a bit of a weird name (as it’s not actual unemployment rates but a DF with null proportions for unemployment rates
Another thing that could be useful is using filtering functions along with groupby
The challenge 12 solution seems too complex. We can just do .dropna
on our newly created ps
column?
challenge 7 says: Another way to get the month
month should be "year" since that's what is being extracted
Part 7 has a line of code:
!head -5 ../data/unemployment_missing.csv
Head is a Unix/Linux specific function and folks who were using Windows locally cannot run this line.
One thing that seems to be missing is how to join two tables (or more) using pandas
https://youtu.be/lXPogGKR-AU
Using .round()
in Manipulating Columns needs to be explained. If we don’t use it, we will not get to a full integer and then calling int()
will truncate. Compare ((unemployment['year_month'] - unemployment['year']) * 100).astype(int)
and ((unemployment['year_month'] - unemployment['year']) * 100).round(0).astype(int)
- the first starts at 0, second starts at 1
Part 7 says Exporting A DataFrame to csv
it looks at the file using the "!" but that is only viable in mac, not windows
!head -5 data/unemployment_missing.csv
to do: add windows version of terminal "!"
Click the "Launch" button under "Jupyter Notebooks" and navigate through your file system to the Python-Data-Visualization folder you downloaded above.
Should read Python-Data-Wrangling instead of Data Visualization
Add examples and descriptions for how to plot with pandas (e.g., pandas.DataFrame.plot
).
Understand what it is that we’re looking at in the data is another useful skill - that i think most social science / stem grad students have, but for people who are new to data analysis, we can include more of that type of practice
Remove some values from the main CSV file
.fillna()
method
parameter.dropna()
@pattyf, @deniederhut, @davclark, @jackspaceBerkeley, @jonstiles, @sbenthall
The notebook for the intro to pandas workshop is almost ready to go, I think. I'd appreciate some feedback when you have time.
One specific thing I need help with is determining when we're dealing with methods, attributes, or properties.
Thanks!
unemployment_rate
is a bit of a weird name (as it’s not actual unemployment rates but a DF with null proportions for unemployment ratesunemployment_rate_missing = unemployment[unemployment['unemployment_rate'].isnull()]
-> no need for double subsetround()
in Manipulating Columns needs to be explained. If we don’t use it, we will not get to a full integer and then calling int() will truncate. Compare ((unemployment['year_month'] - unemployment['year']) * 100).astype(int)
and ((unemployment['year_month'] - unemployment['year']) * 100).round(0).astype(int)
- first starts at 0, second starts at 1.dropna
on our newly created ps
column?Minor issue, but the 10 minutes to pandas link in the resources section of the Readme is broken.
Instead of the current link:
https://pandas.pydata.org/pandas-docs/stable/getting_started/10min.html
it should be:
https://pandas.pydata.org/pandas-docs/stable/user_guide/10min.html
We might want to add a little section at the start of the notebook to explain the difference between DataFrame and Series objects.
unemployment_rate_missing = unemployment[unemployment['unemployment_rate'].isnull()]
-> no need for double subset
Possible things to consider for intermediate/advanced pandas class:
We can both also look up other pandas materials and think of other things we might like to add.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.