Intermediate Exploratory Data Analysis in R

Introduction

In this tutorial, we will be applying some of the tidyverse functions to an initial exploratory data analysis of state-level data from the Eviction Lab. Specifically, we’ll be learning how to:

• Use the %in% operator.

• Perform partial string matching.

• Compute your own variables, including per capita rates and percent changes.

• Use the subscript notation to pull out specific information.

• Perform lagged calculations like year-over-year changes.

• Calculate the linear correlation between two variables.

• Recode values.

The Dataset

The Eviction Lab has collected data from 80 million records around the country and you can read about the methodology here. As always, it is helpful to review the data dictionary to understand the meaning of the different variables (what each variable name corresponds to). If you want more detail about any of the variables, I encourage you to review the full Methodology Report.

Pre-Analysis Steps

Load the data

The first step, of course, is to read in the data. Like before, we’ll use the readr::read_csv() function:

library(tidyverse)

## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.1 ──

## ✓ ggplot2 3.3.5     ✓ purrr   0.3.4
## ✓ tibble  3.1.3     ✓ dplyr   1.0.7
## ✓ tidyr   1.1.3     ✓ stringr 1.4.0
## ✓ readr   2.0.1     ✓ forcats 0.5.1

## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## x dplyr::filter()     masks stats::filter()
## x dplyr::group_rows() masks kableExtra::group_rows()
## x dplyr::lag()        masks stats::lag()

us_states <- read_csv("https://dds.rodrigozamith.com/files/evictions_us_states.csv")

## Rows: 867 Columns: 27

## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr  (3): GEOID, name, parent-location
## dbl (24): year, population, poverty-rate, renter-occupied-households, pct-re...

##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

Confirm it imported correctly

Normally, we would make sure the data imported correctly. Because we’ve previously evaluated the data from this data source, we can just take a quick look at the first few observations to confirm that the data were imported correctly.

head(us_states)

Just like before, it appears we imported the data correctly. Each observation represents a different state (name) at a different point in time (year).

Filtering Strings

Using the %in% operator

While most of the data crunching we do as data journalists involves numbers, we sometimes have to work with strings, too.

So, I’d like to begin this more advanced data analysis by asking my data: Which New England state had the highest eviction rate in 2016?

Our dataset does not have a variable covering the region represented by the state. Instead, we’ll have to create our own filter that only includes those states. We can turn to the U.S. Census Bureau-designated regions and divisions for help in determining which states are part of New England.

Rather than creating a filter with a bunch of different OR conditions, I can just use the %in% operator and pair it with a vector of values that I would like to include. Put another way, I can use the filter() function and supply the conditional as variable %in% c("vector", "with", "multiple", "values").

With this information in mind, here’s what I’d need to do to answer that question:

1. Call the data from my data frame.

2. Apply a filter to cover the desired year and the desired states.

3. Sort the data in descending order based on the variable of interest.

4. Optional: Select only the variables of interest to me, so I can more quickly see the answer.

Now, let’s translate that logic into code:

us_states %>%
  filter(name %in% c("Connecticut", "Maine", "Massachusetts", "New Hampshire", "Rhode Island", "Vermont") & year == 2016) %>%
  arrange(desc(`eviction-rate`)) %>%
  select(year, name, `eviction-rate`)

As we can see, Rhode Island was the New England state with the highest eviction rate in 2016.

Using str_starts() and str_ends()

Similarly, sometimes we want to create a filter for our data so we only include observations that begin with or end with certain text.

In this case, I’d like to ask my data: Which of the Virginias had the highest average eviction rate between 2013 and 2016?

One approach would be to create a filter with just those two state names. However, I can also use the stringr::str_ends() function to perform a search that looks for partial matches by evaluating the end of the string.

The arguments I need to provide str_ends() function are (1) the variable I’m evaluating (the string argument, which comes first) and (2) the pattern or value that must be present at the end for it to yield a TRUE response (the pattern argument, which comes second). As a reminder, the TRUE responses are the ones included in the results when we apply a filter with the filter() function.

So, let’s try to create a filter (filter()) that only includes observations where the name variable ends (str_ends()) with the string "Virginia":

us_states %>%
  filter(str_starts(name, "Virginia"))

Unlike prior instances of dplyr::filter() where we specified a variable, an evaluation operator, and the condition (e.g., name == "Virginia"), the str_ends() function expects a different syntax. This is common in R, especially when you are using functions from different packages. Thus, be sure to read each function’s documentation to understand what it’s looking for.

We can similarly look at the start of a string by using the str_starts() function, which also comes with R. For example, let’s include only the states (name) that begin with the string "North":

us_states %>%
  filter(str_starts(name, "North"))

So, let’s return to our original question: Which of the Virginias had the highest average eviction rate between 2013 and 2016?

Here’s what I’d need to do to answer that question:

1. Call the data from my data frame.

2. Apply a filter to cover the observations ending with my desired string and matching the years within my desired range.

3. Group the data based on the grouping variable of interest.

4. Calculate the summary statistic for the mean of the desired variable.

We can translate that into code like so:

us_states %>%
  filter(str_ends(name, "Virginia") & year >= 2013 & year <= 2016) %>%
  group_by(name) %>%
  summarize(mean_eviction_rate=mean(`eviction-rate`))

As we can see here, the state of Virginia had the higher eviction rate during that period of time.

Advanced string matching

What if we want to just filter observations based on some text that is present in the middle of a string? Here, we can use stringr::str_detect().

str_detect() allows you to match text based on patterns, or regular expressions. These patterns can get very complicated (but are quite powerful), and you can read more about them here.

For the sake of simplicity, we won’t cover patterns here. Instead, we’ll just use str_detect() to look for entries that include a specified string anywhere within a given variable. For example, we can filter in any state with “y” in its name and count the number of unique states like so:

us_states %>%
  filter(str_detect(name, "y")) %>%
  count(name)

Calculating Your Own Rates

The third question I’d like to ask my data is: What were the eviction rates in Massachusetts between 2010 and 2016, expressed on a per 100,000 people basis?

We already have data about eviction rates in the eviction-rate variable. However, those data are on a per 100 renter homes basis. In order to address that question, I would need to calculate my own rate. Thankfully, I have the variables necessary to do that (evictions and population).

To compute a new variable, we can use the dplyr::mutate() function. mutate() operates just like summarize() in the sense that you give it a variable name and provide the operation necessary to calculate it. However, the key difference is that the variables you create are simply appended to the existing data frame—they do not reduce the data frame into a set of summary statistics.

A rate like the one we are interested in can be easily computed through the following calculation: X / Y (where X refers to the number of evictions and Y to the population). To get the per 100,000 people basis, we would multiply that result by 100,000.

So, here’s what I’d need to do to answer that third question:

1. Call the data from my data frame.

2. Apply a filter to cover the desired state and the timeframe.

3. Create a new variable (we’ll call it eviction_rate_pop) based on our rate calculation.

4. Optional: Select only the variables of interest to me, so I can more quickly see the answer.

And here is how I can translate that into code:

us_states %>%
  filter(name == "Massachusetts" & between(year, 2010, 2016)) %>%
  mutate(eviction_rate_pop = (evictions/population)*100000) %>%
  select(year, name, population, evictions, eviction_rate_pop)

Looking at this output, I can see that there were 270 evictions for every 100,000 people who lived in Massachusetts in 2013. (And, for other years between 2010 and 2016, too!)

Things to be mindful of

One thing to be mindful of is that the population statistics in our dataset change pretty infrequently. For example, we have the same population value for all years between 2011-2016. That’s probably because the Eviction Lab team are relying on the most comprehensive counts from the decennial U.S. Census.

However, the U.S. Census also offers less precise, but more frequently updated, annual data from its annual American Community Survey. If we wanted more recent population counts, we would thus need to merge the Eviction Lab data with data from the American Community Survey. This is possible, but there may be a good reason for keeping the less-recent counts, and we should therefore see if the methodological details explain the reason for that decision.

A second thing to be mindful of is that deciding the basis for a rate (e.g., 1,000 vs. 100,000) is more of an art than a science. It requires an understanding of the context and the base rates that your audiences are most familiar with. For example, we tend to see rates for states and cities expressed on a per 100,000 people basis, and it is thus an already familiar anchor for most folks. As such, that’s often a helpful basis to use. But that may not be the case in all contexts.

Alternatively, the Eviction Lab likely provides its rates on a per-100 renters basis because it is easy to also present that as a percentage (e.g., 2.1 per 100 can be translated to 2.1%). In fact, they present that rate as a percentage on the visuals on their website.

The key is to use a basis (‘per’ number) that you believe would be most informative to your audience—and not the one that makes a statistic seem most or least alarming.

Calculating a Percent Change

A fourth question I would like to ask my data is: How much higher (as a percentage) was Massachusetts’ number of evictions in 2016, relative to Florida’s?

As a reminder, we can calculate a percent change as follows: (Y2-Y1)/abs(Y1)*100. (The abs() function allows us to get the absolute value of a number.)

I could apply that formula as part of a two-step operation where I manually look up Massachusetts’ number (Y2) of evictions as well as Florida’s (Y1), and then (1) manually subtract Massachusetts’ number from Florida’s, (2) divide the result by Florida’s number, and (3) multiply that by 100.

However, we can also do this in a single operation by calculating a summary statistic using that equation. In order to get the data necessary for that operation, we can use the subscripting approach that is native to R.

Subscripting

R allows us to pull out specific information from its outputs, which can be handy when you want to perform calculations using filtered or mutated data. We have already performed some calculations with different dplyr functions, but this is a good opportunity to demonstrate an alternative that is sometimes necessary for functions that don’t play nicely with tidyverse or to do some things we want within tidyverse.

First, let’s try getting information from a vector. As a reminder, we can call information from a variable in a data frame with the object$variable notation, such as when we run:

head(us_states$evictions)

## [1]  4583 10106  9603  7379  6170  8300

Let’s say we wanted to get just the first value from that vector (4583). Here’s where a subscript would be useful. Subscripts allow you to call specific elements from an object by specifying it in brackets. As such, us_states$evictions[1] would produce only the first element from that vector.

us_states$evictions[1]

## [1] 4583

If your object is a data frame, which has observations (rows) and variables (columns), we subscript using matrix notation [(observation), (variable)] (or [(row), (column)]; note the comma in both. For example, if I wanted to get the value in the first row and third column of our us_states data frame, I’d write:

us_states[1,3]

I could also provide just one of those values (either observation or variable), but keep the comma, in order to get the information from my desired observation or variable. For example, here’s the entire first row in our dataset.

us_states[1,]

Notably, we can also use subscripting to filter information based on a condition (like we do with the filter() function). For example, if I just wanted rows that had the name variable be equivalent to “Wyoming”, I could write:

us_states[us_states$name == "Wyoming",]

Returning to our question

Let’s return to our fourth question: How much higher (as a percentage) was Massachusetts’ number of evictions in 2016, relative to Florida’s?

Here’s how I can approach the question:

1. Call the data from my data frame.

2. Apply a filter to cover the desired timeframe.

3. Optional: Select the relevant columns to make it easier to read the output.

4. Summarize the data by creating a new variable called pct_chg that uses subscripting to (1) manually subtract Massachusetts’ number of evictions from Florida’s, (2) divide the result by Florida’s number, and (3) multiply that by 100.

Here’s what that logic looks like as code:

us_states %>%
  filter(year==2016) %>%
  select(year, name, evictions) %>%
  summarize(pct_chg=((evictions[name=="Massachusetts"]-evictions[name=="Florida"])/abs(evictions[name=="Florida"]))*100)

As we can see, the number of evictions in Massachusetts was 78.1% lower than the number in Florida in 2016.

Of course, we need to keep in mind that Florida’s population is much larger than Massachusetts’. Thus, it would usually make the most sense to calculate the difference in their rates for a between-states comparison. If we have rates that are already expressed as percentages, it also makes the most sense to calculate the difference in percentage points (i.e., noting that 30% is a 20 percentage point increase from 10%) rather than trying to calculate a percent difference between percents (i.e., noting that 30% is a 200% increase from 10%).

Lagged Calculations

The fifth question I would like to ask my data is: What was the three-year percent change in eviction rate for each year between 2013 and 2016 in Massachusetts and Florida?

To answer this question, the first thing I would need to do is filter out all the states but Florida and Massachusetts. Then, I’d want to group the data by state so I can calculate the percent changes relative to the previous years for that state (and not other states). Then, I would want to order the data sequentially (to make sure the previous year is always the preceding row). Finally, I’d need to tell R to perform our percent change calculation using information from the current row as Y2 and the information from three rows back as Y1.

To perform that last step, we’ll use the dplyr::lag() function.

lag() allows us to look up a value from n elements (or rows in the case of a data frame) back in object x with the syntax, lag(x, n). For example, let’s consider the following code:

lag(c(1, 5, 7, 10, 13), 3)

## [1] NA NA NA  1  5

What we are getting is NA for the first three elements (because there were no values that far back, relative to those elements). The fourth value in our lagged vector was the first in our original vector, and the fifth lagged value was the second original value.

Answering the question

Let’s go back to our original question: What was the three-year percent change in eviction rate for each year between 2013 and 2016 in Massachusetts and Florida?

Here’s what I’d need to do to answer that question:

1. Call the data from my data frame.

2. Apply a filter to cover just the desired states.

3. Optional: Select only the variables of interest to me, so I can more quickly see the answer and increase computational efficiency.

4. Group the data based on my grouping variable of interest.

5. Sort the data by year (and add the .by_group=TRUE argument to the sorting function so that we’re only sorting in-group and thus keeping our state observations in sequential order).

6. Compute a new variable (we’ll call it evictions_pct_chg) using the percent change formula we’re already familiar with (((Y2-Y1)/Y1)*100), using the present value as Y2 and the value three years earlier as Y1.

7. Optional: Apply a second filter that only includes values within the desired timeframe. (We do this at the end since we need data from the earlier years to compute our new evictions_pct_chg variable.)

Here’s what that logic looks like as code:

us_states %>%
  filter(name %in% c("Massachusetts", "Florida")) %>%
  select(year, name, `eviction-rate`) %>%
  group_by(name) %>%
  arrange(year, .by_group=TRUE) %>%
  mutate(evictions_pct_chg=((`eviction-rate`-lag(`eviction-rate`, 3))/lag(`eviction-rate`, 3))*100) %>%
  filter(between(year, 2013, 2016))

As we can see, Florida had the higher three-year percent change in 2015.

Calculating Correlations

Data journalists are often interested in looking at relationships between variables.

Scatterplots are generally best for spotting potential relationships, but it is sometimes helpful to also quickly calculate a linear correlation using Pearson’s correlation coefficient to gauge the strength of the linear relationship. (Important note: Relationships can be curvilinear in nature. How to evaluate that is better covered in a formal statistics course.)

To calculate a simple, linear correlation, we just use the base::cor() function, which is loaded automatically in R. With cor(), we just specify the two variables of interest as separate arguments (e.g., cor(dataset$variable1, dataset$variable2)).

Piping into cor()

We could also calculate multiple correlations at the same time by incorporating cor() into our summarize() function to calculate the linear correlations between (1) population size and the rent-burden and (2) population size and the eviction rate:

us_states %>%
  summarize(cor_pop_burden=cor(population, `rent-burden`), cor_pop_evicrate=cor(population, `eviction-rate`, use="pairwise.complete.obs"))

What the above output tells us is that, when looking at every observation in our data frame, there is a positive (the r coefficient has a positive value) but weak (the r coefficient is less than +/-0.3) relationship between population size and the rent burden, and between the population size and the eviction rate. Thus, an increase in population size results in a linear increase of some proportion in the rent burden (or vice versa) and eviction rate (or vice versa) some of the time.

The nice thing about using our multi-step workflow (instead of just running cor() in isolation) is that it allows us to apply filters that can help narrow our inquiry (correlation in recent years vs. many years ago) or to calculate the correlation for computed (mutated) variables.

Causality and the nature of the relationship

As a reminder, there is a big difference between correlation and causation.

Even if we had found a strong correlation between population size and rent burden, the causal variable might be very different. (Put differently, it’s not the size of the population that causes a place to have a high rent burden but some other variable that is common in cities that have larger populations.)

And, again, it is crucial to stress here that we are only measuring a linear correlation (using Pearson’s r) right now. A strong relationship might exist but simply operate in a non-linear way (e.g., increase Y by a low magnitude at the low end of X but then increase Y at increasingly high magnitudes as you move up the values of X). Thus, be careful with how you interpret such statistics.

Recoding Data

It is not uncommon for a dataset to include typos (e.g., Massachusetts being expressed as Masachussets) or to have variations in capitalization (e.g., Massachusetts and massachusetts). To ensure that those values are grouped correctly, we would want to give it a consistent label.

One easy way to do that is to use the dplyr::recode() function to recode some of our values. With the recode() function, we first specify the variable we want to inspect and the values we’d like to recode in the following format: OriginalValue = ReplacedValue. Within a piped operation, we will typically pair it with the mutate() function (thereby making recode() the operation used to compute the new variable).

This dataset is in good shape and we do not need to recode any of the information. Thus, for demonstrative purposes, I would like to recode Massachusetts to The Best Commonwealth and Kentucky as The Worst Commonwealth. (Can you guess the other Commonwealths in the U.S.

Here’s how I could do that:

us_states %>%
  filter(name %in% c("Kentucky", "Massachusetts"), year == 2016) %>%
  mutate(new_name = recode(name, "Massachusetts" = "The Best Commonwealth", "Kentucky" = "The Worst Commonwealth")) %>%
  select(year, name, new_name)

There are two important things to note here:

• First, I could easily just replace the values in the original variable (name) by simply using mutate to compute a variable called name (rather than new_name). I often like to create new variables when I make changes, but you do not have to.

• Second, by default, recode() will keep the original value for non-matching values (e.g., Alabama stays as Alabama). We can also tell recode() to replace non-matching values with a specific string (or operation) by appending the .default="REPLACEMENT STRING" argument. We can do the same for missing values (NA) by appending the .missing="REPLACEMENT STRING" argument.

Changing the case

In instances where we want to standardize the case (e.g., make everything lower- or upper-case), we can also very simply use the tolower() and toupper() functions within our mutate() operation.

The tolower() and toupper() functions only require a single argument: the vector to apply the case change to.

For example, we can easily make all of our state names lower-case like this:

us_states %>%
  head() %>%
  mutate(name=tolower(name))