Introduction

For this tutorial we'll be looking at a sample of dischage information containing de-identified patient information, hospital information, services rendered by the hospitals and billing information.

We want to:

• Load the data using pandas
• Clean the data so we can work with it
• Produce Histograms of the Total Charges
• See how we can "zoom in" on the data to identify areas of interest

Objectives

• Learn how to use the "apply" function to make changes to the data in a column
• Learn how to call matplotlib functions from panda objects
• Learn how to use pandas binary vectors to select subsets of data
• learn some basic uses of the groupby function to manipulate your data

To begin we import the libraries we'll be needing today, including the pylab submodule (named 'plt' for short) and pandas (renamed 'pd' to make typing easier). Additionally we include the %matplotlib inline statement to ensure our plots show up on the webpage.

Now we load the discharge information into our dataset by using panda's read_csv() function. This will load all the information along with the header and provide us with a variable df that we can use to manipulate the data and generate our plots.

Right now we're primarily interested in Total Charges, but before we can start asking interesting questions about total charges and plotting its distribution we need to resolve a data cleanliness issue. Lets see what is in the 'Total Charges' column

Total Charges contains dollar values for the services rendered to the patient while at the hospital. Unfortunately the data currently includes the dollar sign ($). This leads to some difficult problems when trying to add, subtract or do any other kind of operation on these values.

As an example lets take the the first value of Total Charges (df['Total Charges'].iloc[0]) and add it to the second value of Total Charges (df['Total Charges'].iloc[1]). (iloc is a method, supported by panda objects, that can be used to index an axis by an integer.)

This is certainly not what we want! But what is going on here?

Instead of adding the numbers 62328.96 and 30063.34 Python has gone ahead and appended the strings $62328.96 and $30063.34. If we want to do any kind of plotting or take any kind of summary statistics we're going to need to convert the strings contained in Total Charges to actual numbers.

From a high level this means 1. removing the dollar sign ($) from our string 2. converting the type of the variable from a string to a number 3. Doing this for every variable in the 'Total Charges' column.

To achieve this practically we're going to write a function that takes a string like $62328.96 and converts it to a number 62328.96. Then we're going to apply that function to each of the cells in the 'Total Charges' column. That is the purpose of functions, to take an input and return a transformed output. In this case our input is total charge as a string and our output will be total charge as a number.

Lets take a look at some of the building blocks we'll need.

Replace is a function that we can use to return a new string which replaces certain characters with other characters. It takes two arguments, the first is the string to look for, the second is the string to replace with. In this case we replace with the empty string. Lets see an example of this:

This is a good start, we've deleted the dollar sign ($) by "replacing" it with the empty string. Unfortunately this is not enough! our "number" is still considered a string, if we try and add two replaced strings we're still in the same situation:

Here we've removed the dollar signs ($) from each string and tried to add them together. Python still sees these as two strings, and this can be an easy point to miss. We need to tell python that these two 'strings' are actually numbers. For numbers that have floating point decimals we do this with the float() function.

Putting them both together

Making it General

We get the number that we expect. This is all well and good, but we'd like to be able to convert any string with a dollar sign to a floating point number. To do this we need a function:

Now we can pass any value to our function translate_charge and it will replace the dollar sign ($) and return the number

We can even get the addition we were looking for before

Now we just need to call this function on every value in 'Total Charges.' Luckily pandas has a very simple function that allows us to apply this function to each value. Not surprisingly it is called apply

Careful though, because apply will not change df['Total Charges'] it will just return a new column. If you want to keep that column you have to set Total Charges equal to it.

Now that we have cleaned up Total Charges and have proper numbers for values, we can start looking at some of the summary statistics for our sample. The quickest way to do this is to use the describe() function. This gives us the count, mean, the standard deviation and the min and max.

We can use the hist function to plot a histogram of our data to get a better sense of our distribution. Note that panda data series support the hist method (among other plotting functions). The method draws a histogram using matplotlib and using itself as the input data. hist takes an argument bins to determine the number of bins we want to display. Finally we can use the argument figsize to increase the size of the figure.

Zooming In

This looks like a distribution with a pretty long tail. This makes it hard to see exactly what is going on around the mean of the distribution. What we'd really like to be able to do is to "zoom in" and see the distribution without some of these outliers.

To do this we're going to use binary vectors to select subsets of the data we want.

Binary vectors are just columns of data that contain either True or False. We can generate these vectors by comparing columns to values. For example:

This simple operation has compared each cell in Total Charges to the value 50,000. If the value is less than 50,000 it gives a True, otherwise it gives a False. Once we have this I can use this selector to grab only the values that are True (meet our condition)

In this example we are going to plot the data using a pie chart. First we want to take a look at the organization of our data. We use pprint (which stands for "pretty print") to print the columns of the dataframe.

Before creating the pie charts we want to learn how to organize the data using the groupby function. groupby returns a DataFrameGroupby object which allows us to retrieve information about the groups (http://pandas.pydata.org/pandas-docs/stable/api.html#groupby). As an example here we create groups by county. We then use the size function to get a series containing the group names (it should be the county names) and the size of each group. We print out the type of the objects and the objects themselves to verify.

We can also group by 2 categories. Here we group by county and type of admission. Then we use ix (an indexing function) to retrieve the data for Albany county.

Before we create our plot lets import the numpy library and the interact widget and also set the font type for our plot.

Now we define a function that draws our plot. We first group by county and type of admission. Then we get the data for the county we are interested in. Finaly we use matplotlib to create a pie chart.

At this point we've defined a function to draw a plot. But have not actually created a plot. We are going to use the interact function to create an interactive plot. interact takes a function (in this case the by_county function we defined above, and a list of values as inputs. It creates a widget (gui component) using the values of the list. Each time we select a new value from the widget it will call our function thereby redrawing the plot using a different county.

In the below example we first take a look at the unique Age Group column. Then we mmodify our function such that we plot a second pie chart containing only the 0 to 17 data. An advanced excersise would be to add a second widget to the plot, such that one could select the age group of the second plot.

We can start to see how powerful pandas and matplotlib can be. In this example we use groupby to organize the data by Admit Day of Week then plot a bar graph (using only 1 line of code!).

And we can group by 2 columns to create a detailed bar plot of both Admit Day of Week and Type of Admission.