Tyler Caraza-Harter
Many datasets you'll encounter are tabular; in other words, the data can be organized with tables and columns. We've seen how to organize this data with lists of lists, but this is cumbersome. Now we'll learn Pandas, a Python module built specifically for tabular data. If you become comfortable with Pandas, you'll likely start preferring it over Excel for analyzing tables.
We need to install Pandas:
pip install pandasThen import it:
import pandas as pd
The as pd expression may be new for you. This just gives the pandas module a new name in our code, so we can type things like pd.some_function() to call a function named some_function rather than type out pandas.some_function. You could also have just used import pandas or even given it another name with an import like import pandas as silly_bear, but we recommend you import as "pd", because most pandas users do so by convention.
We'll also be using two new data types (Series and DataFrame) from Pandas very often, so let's import these directly so we don't even need to prefix their use with pd..
from pandas import Series, DataFrame
Pandas tables are built as collections of Pandas Series. A Series is a sophisticated data structure that combines many of the features of both Python lists and dicts.
You can think of a Series as a dictionary where the values are ordered and, in addition to having a key, are labeled with integer positions (0, 1, 2, etc).
The terms we'll use when talking about Series are not very consistent with the terms we used for lists and dicts. I would happily invent new terminology and teach it to you, but I think it's better to teach you the vocubalary you'll see used outside of CS 301.
A pandas integer position refers to a 0, 1, 2, etc. label, and is equivalent to a list's index.
A pandas index refers to the programmer-chosen label (which could be a string, int, etc) on a value, and is equivalent to a dict's key.
After all the fuss we've made about disntinguishing a list's index from a dict's key, it's fair for you to be unhappy about what "index" means in the context of pandas.
It is easy to convert a dict to a Series:
d = {"one": 7, "two": 8, "three": 9}
d
# dict to Series
s = Series(d)
s
In this case, the values are 7, 8, and 9, and the corresponding indexes are "one", "two", and "three".
dtype stands for data type. In this case, it means that the series contains integers, each of which require 64 bits of memory (this detail is not important for us). Although you could create a Series containing different types of data (as with lists), we'll avoid doing so because working with Series of one type will often be more convenient. You may mix values of different types in a Series (just like we regularly do with lists), but this is discouraged.
A Series can be expected to keep the same order (unless we intentionally change it), unlike a dict.
Let's lookup a value using some indexes, using .loc[???]:
print(s.loc["one"])
print(s.loc["three"])
print(s.loc["two"])
Instead of .loc we can use .iloc to do lookup by integer position:
print(s.iloc[0])
print(s.iloc[2])
If you don't use .loc or .iloc, pandas tries to figure out which one you mean:
print(s["one"])
print(s[0])
Of course, this can create ambiguities (should s[0] produce "A" or "C"?).
s = Series({2: "A", 1: "B", 0: "C"})
s
s[0]
We can always convert a Series back to a dict. The pandas indexes become dict keys.
# Series to dict
dict(s)
We can also convert back and forth between lists as Series:
num_list = [100, 200, 300]
print(type(num_list))
num_series = Series(num_list) # create Series from list
print(type(num_series))
# displaying a list:
num_list
# displaying a Series:
num_series
Notice that both the list and the Series contain the same values. However, there are some differences:
When we create a Series from a list, there is no difference between the integer position and index. s.iloc[X] is the same as s.loc[X].
num_series.iloc[0], num_series.loc[0]
Going from a Series back to a list is just as easy as going from a list to a Series:
list(num_series)
Except for negative indexing, indexing and slicing for a Series is much like it is for a list.
letter_list = ["A", "B", "C", "D"]
letter_series = Series(letter_list)
letter_series
letter_list[0]
letter_series[0]
letter_list[3]
letter_series[3]
letter_list[-1]
# but be careful! Series don't support negative indexes to the extent that lists do
try:
print(letter_series[-1]) # BAD
except Exception as e:
print(type(e))
letter_series.iloc[-1] # OK
Series slicing works much like list slicing:
print("list slice:")
print(letter_list[:2])
print("\nseries slice:")
print(letter_series[:2])
print("list slice:")
print(letter_list[2:])
print("\nseries slice:")
print(letter_series[2:])
Be careful! Notice the indices for the slice. It is not creating a new Series indexed from zero, as you would expect with a list.
# although we CANNOT always do negative indexing with a Series
# we CAN use negative numbers in a Series slice
print("list slice:")
print(letter_list[:-1])
print("\nseries slice:")
print(letter_series[:-1])
You should think of Series(["A", "B", "C"]) as being similar to this:
s = Series({0: "A", 1: "B", 2: "C"})
s
We can also slice a Series constructed from a dictionary (remember that you may not slice a regular Python dict):
s[1:]
With Series, it is easy to apply the same operation to every value in the Series with a single line of code (instead of with a loop).
For example, suppose we wanted to add 1 to every item in a list. We would need to write something like this:
orig_nums = [100, 200, 300]
new_nums = []
for x in orig_nums:
new_nums.append(x+1)
new_nums
With a Series, we can do the same like this:
nums = Series([100, 200, 300])
nums + 1
This probably feels more intuitive for those of you familar with vector math.
It also means multiplication means something very different for lists than for Series.
[1,2,3] * 3
Series([1,2,3]) * 3
Whereas a "+" means concatenate for lists, it means element-wise addition for Series:
[10, 20] + [3, 4]
Series([10, 20]) + Series([3, 4])
One implication of this is that you might not get what you expect if you add Series of different sizes:
Series([10,20,30]) + Series([1,2])
The 10 gets added with the 1, and the 20 gets added with the 2, but there's nothing in the second series to add with 30. 30 plus nothing doesn't make sense, so Pandas gives "NaN". This stands for "Not a Number".
Consider the following:
nums = Series([1,9,8,2])
nums
nums > 5
This example shows that you can do element-wise comparisons as well. The result is a Series of booleans. If the value in the original Series is greater than 5, we see True at the same position in the output Series. Otherwise, the value at the same position in the output Series is False.
We can also chain these operations together:
nums = Series([7,5,8,2,3])
nums
mod_2 = nums % 2
mod_2
odd = mod_2 == 1
odd
As you can see, we first obtained an integer Series (mod_2) by computing the value of every number modulo 2 (mod_2) will of course contain only 1's and 0's).
We then create a Boolean series (odd) by comparing the mod_2 series to 1.
If a number in the nums Series is odd, then the value at the same position in the odd series will be True.
Notice what happens when we create a series from a list:
Series([100,200,300])
We see the following:
One interesting difference between lists and Series is that with Series, the index does not always need to correspond so closely with the position; that's just a default that can be overridden. We've already seen one way to do this (creating a Series from a dict). We can also build a Series with two aligned lists (one for the values and one for the index).
nums1 = Series([100, 200, 300], index=[2,1,0])
nums1
Now we see indexes are assigned based on the argument we passed for index (not the position):
When we do element-wise operations between two Series, Pandas lines up the data based on index, not position. As a concrete example, consider three Series:
X = Series([100, 200, 300])
Y = Series([10, 20, 30])
Z = Series([10, 20, 30], index=[2,1,0])
X
Y
Z
Note: Y and Z are nearly the same (numbers 10, 20, and 30, in that order), except for the index. Let's see the difference between X+Y and Y+Z:
X+Y
X+Z
For X+Y, Pandas adds the number at index 0 in X (100) with the value at index 0 in Y (10), such that the value in the output at index 0 is 110.
For X+Z, Pandas adds the number at index 0 in X (100) with the value at index 0 in Y (30), such that the value in the output at index 0 is 130. It doesn't matter that the first number in Z is 10, because Pandas does element-wise operations based on index, not position.
We've seen this syntax before:
obj[X]For a dictionary, X is a key, and for a list, X is an index. With a Series, X could be either of these things, or, interestingly, obj and X could both be a Series. In this last scenario, X must specifically be a Series of booleans. This type of lookup is often called "boolean indexing", or sometimes "fancy indexing."
letters = Series(["A", "B", "C", "D"])
letters
bool_series = Series([True, True, False, False])
bool_series
# we can used the bool_series almost like an index
# to pull values out of letters:
letters[bool_series]
# We could also create the Boolean Series on the fly:
letters[Series([True, True, False, False])]
# Let's grab the last two letterrs:
letters[Series([False, False, True, True])]
# Let's grab the first and last (can't do this with a slice):
letters[Series([True, False, False, True])]
As with element wise operations, fancy indexing aligns both Series:
s = Series({"w": 6, "x": 7, "y": 8, "z": 9})
b = Series({"w": True, "x": False, "y": False, "z": True})
s[b]
As we just saw, we can use a Boolean series (let's call it B) to select values from another Series (let's call it S).
A common pattern is to create B by performing operation on S, then using B to select from S. Let's try doing this to pull all the numbers greater than 5 from a Series.
# we want to pull out 9 and 8
S = Series([1,9,2,3,8])
S
B = S > 5
B
# this will pull out values from S at index 1 and 4,
# because the values in B at index 1 and 4 are True
S[B]
Let's try to pull out all the upper case strings from a series:
words = Series(["APPLE", "boy", "CAT", "dog"])
words
# we can use .str.upper() to get upper case version of words
upper_words = words.str.upper()
upper_words
# B will be True where the original word equals the upper-case version
B = words == upper_words
B
# pull out the just words that were orginally uppercase
words[B]
We have done this example in several steps to illustrate what is happening, but it could have been simplified. Recall that B is words == upper_words. Thus we could have done this without ever storing a Boolean series in B:
words[words == upper_words]
Let's simplify one step further (instead of using upper_words, let's paste the expression we used to compute it earlier):
words[words == words.str.upper()]
Let's try to pull out all the odd numbers from this Series:
nums = Series([11,12,19,18,15,17])
nums
nums % 2 well produce a Series of 1's (for odd numbers) and 0's (for even numbers). Thus nums % 2 == 1 produces a Boolean Series of True's (for odd numbers) and False's (for even numbers). Let's use that Boolean Series to pull out the odd numbers:
nums[nums % 2 == 1]
One might be able to perform operations like this in Pandas:
Series([True, False]) or Series([False, False])
Unfortunately, that doesn't work, because Python doesn't let modules like Pandas override the behavior of and and or. Instead, you must use & and | for these respectively.
Let's try to get the numbers between 10 and 20:
s = Series([5, 55, 11, 12, 999])
s
s >= 10
s <= 20
(s >= 10) & (s <= 20)
s[(s >= 10) & (s <= 20)]
Cool, we got all the numbers between 10 and 20! Notice we needed extra parentheses, though. & and | are high precedence, so we need those to make the logical operators occur last.
Pandas will often be used to deal with tabular data (much as in Excel).
In many tables, all the data in the same column is similar, so Pandas represents each column in a table as a Series object. A table is represented as a DataFrame, which is just a collection of named Series (one for each column).
We can use a dictionary of aligned Series objects to create a dictionary. For example:
name_column = Series(["Alice", "Bob", "Cindy", "Dan"])
score_column = Series([100, 150, 160, 120])
table = DataFrame({'name': name_column, 'score': score_column})
table
Or, if we want, we can create a DataFrame table from a dictionary of lists, and Pandas will implicitly create the Series for each column for us:
data = {"name": ["Alice", "Bob", "Cindy", "Dan"],
"score": [100, 150, 160, 120]}
df = DataFrame(data)
df
There are a few things we might want to do:
# we'll use the DataFrame of scores defined
# in the previous section
df
# let's grab the name cell using DataFrame["COL NAME"]
df["name"]
# or we could extract the score column:
df["score"]
# if we want to generate some simple stats over a column,
# we can use .describe()
df["score"].describe()
# lookup is done for columns by default (df[x] looks up column named x)
# we can also lookup a row, but we need to use df.loc[y]. ("loc" stands for location)
# for example, let's get Bob's row:
df.loc[1]
# if we want a particular cell, we can use df.loc[row,col].
# for example, this is Bob's score:
df.loc[1, "score"]
# we can also use this to modify cells:
df.loc[1, "score"] += 5
df
Most of the time, we'll let Pandas directly load a CSV file to a DataFrame (instead of creating a dictionary of lists ourselves). We can easily do this with pd.read_csv(path) (recall that we imported pandas as import pandas as pd):
# movies is a DataFrame
movies = pd.read_csv('IMDB-Movie-Data.csv')
# how many are there?
print("Number of movies:", len(movies))
# it's large, but we can preview the first few with DataFrame.head()
movies.head()
# we can pull out Runtime minutes if we like
runtime = movies["Runtime"]
# it's still long (same length as movies), but let's preview the first 10 runtime minutes
runtime.head(10)
# what is the mean runtime, in hours?
runtime.mean() / 60
# what if we want stats about movies from 2016?
# use .head() on results to make it shorter
(movies["Year"] == 2016).head()
Observe:
Let's pull out the movies from 2016 using this Boolean Series:
movies_2016 = movies[movies["Year"] == 2016]
print("there are " + str(len(movies_2016)) + " movies in 2016")
movies_2016.head(10)
# let's get some general stats about movies from 2016
movies_2016.describe()
We see (among other things) that the average Runtime is 107.34 minutes.
Data comes in many different forms, but tabular data is especially common. The Pandas module helps us work with tabular data and integrates with ipython, making it fast and easy to compute simple statistics over columns within our dataset. In this lesson, we learned to do the following: