Exploratory Data Analysis in R

Introduction

In this tutorial, we will be applying some of the tidyverse functions to an initial, exploratory data analysis of city-level data about evictions in Massachusetts from the Eviction Lab. Specifically, we’ll be learning how to:

• Deal with NA values.

• Deal with special characters in variable names.

• Perform a ‘sanity check.’

• Apply filters based on different conditions, including partial string matching.

• Sort data based on a variable.

• Select specific variables to make it easier to read output.

• Calculate summary statistics, such as a mean for a variable.

• Group data in order to facilitate comparisons.

The Dataset

The Eviction Lab data are derived from 80 million records painstakingly collected by the Lab’s research team. You can read about the methodology here, but the key things to keep in mind for this tutorial is that the dataset we’ll be using covers the city level in the state of Massachusetts between the years of 2000 and 2016. Desmond’s team also makes data available at more micro levels (U.S. Census tracts and block groups, as well as more macro levels (county, statewide).

Because these data aren’t collected in a consistent manner (across states or within them), the Lab is forced to estimate some of the variables. You may find, for example, that a city had 0.45 evictions one year. How’s it possible to only evict 0.45 of a home? Well, that’s just an estimate and it would be reasonable for our purposes to round that tally to the closest whole number. (The Eviction Lab’s website does this for the sake of simplicity but the datasets they provide are more precise.)

As always, it is helpful to review the data dictionary to understand the meaning of the different variables (what each variable name corresponds to). If you want more detail about any of the variables, I encourage you to review the full Methodology Report.

Pre-Analysis Steps

Load the data

The first step, as usual, is to read in the data. We’ll use the readr::read_csv() function (which can be loaded through tidyverse) to read a CSV file with the data and assign it to an object called ma_cities using the assignment operator (<-).

library(tidyverse)

## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.1 ──

## ✓ ggplot2 3.3.5     ✓ purrr   0.3.4
## ✓ tibble  3.1.3     ✓ dplyr   1.0.7
## ✓ tidyr   1.1.3     ✓ stringr 1.4.0
## ✓ readr   2.0.1     ✓ forcats 0.5.1

## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## x dplyr::filter()     masks stats::filter()
## x dplyr::group_rows() masks kableExtra::group_rows()
## x dplyr::lag()        masks stats::lag()

ma_cities <- read_csv("https://dds.rodrigozamith.com/files/evictions_ma_cities.csv")

## Rows: 4245 Columns: 27

## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr  (2): name, parent-location
## dbl (25): GEOID, year, population, poverty-rate, renter-occupied-households,...

##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

Confirm it imported correctly

The next step is to confirm the data were imported the way we wanted it to. Here, I would recommend that you follow three steps.

Step 1: Check the first few observations

Start by checking the first few observations (rows) of the object to check if there’s anything odd using the head() function:

head(ma_cities)

At first glance, the data seem to have been imported cleanly. Each observation represents a different city (name) at a different point in time (year).

Step 2: Check the structure

Next, we’ll use the glimpse() function to check all the column names and make sure they’re associated with the right data type:

glimpse(ma_cities)

## Rows: 4,245
## Columns: 27
## $ GEOID                        <dbl> 2500135, 2500135, 2500135, 2500135, 25001...
## $ year                         <dbl> 2000, 2001, 2002, 2003, 2004, 2005, 2006,...
## $ name                         <chr> "Abington", "Abington", "Abington", "Abin...
## $ `parent-location`            <chr> "Massachusetts", "Massachusetts", "Massac...
## $ population                   <dbl> 14605, 14605, 14605, 14605, 14605, 16399,...
## $ `poverty-rate`               <dbl> 3.60, 3.60, 3.60, 3.60, 3.60, 3.62, 3.62,...
## $ `renter-occupied-households` <dbl> NA, NA, NA, NA, NA, NA, NA, NA, 1752.00, ...
## $ `pct-renter-occupied`        <dbl> 28.22, 28.22, 28.22, 28.22, 28.22, 26.64,...
## $ `median-gross-rent`          <dbl> 676, 676, 676, 676, 676, 1060, 1060, 1060...
## $ `median-household-income`    <dbl> 57100, 57100, 57100, 57100, 57100, 73956,...
## $ `median-property-value`      <dbl> 178900, 178900, 178900, 178900, 178900, 3...
## $ `rent-burden`                <dbl> 23.8, 23.8, 23.8, 23.8, 23.8, 27.9, 27.9,...
## $ `pct-white`                  <dbl> 96.97, 96.97, 96.97, 96.97, 96.97, 94.85,...
## $ `pct-af-am`                  <dbl> 0.73, 0.73, 0.73, 0.73, 0.73, 2.26, 2.26,...
## $ `pct-hispanic`               <dbl> 0.71, 0.71, 0.71, 0.71, 0.71, 0.96, 0.96,...
## $ `pct-am-ind`                 <dbl> 0.12, 0.12, 0.12, 0.12, 0.12, 0.00, 0.00,...
## $ `pct-asian`                  <dbl> 0.46, 0.46, 0.46, 0.46, 0.46, 1.50, 1.50,...
## $ `pct-nh-pi`                  <dbl> 0.01, 0.01, 0.01, 0.01, 0.01, 0.00, 0.00,...
## $ `pct-multiple`               <dbl> 0.76, 0.76, 0.76, 0.76, 0.76, 0.27, 0.27,...
## $ `pct-other`                  <dbl> 0.25, 0.25, 0.25, 0.25, 0.25, 0.16, 0.16,...
## $ `eviction-filings`           <dbl> NA, NA, NA, NA, NA, NA, NA, NA, 49.00, 65...
## $ evictions                    <dbl> NA, NA, NA, NA, NA, NA, NA, NA, 39.00, 53...
## $ `eviction-rate`              <dbl> NA, NA, NA, NA, NA, NA, NA, NA, 2.23, 2.9...
## $ `eviction-filing-rate`       <dbl> NA, NA, NA, NA, NA, NA, NA, NA, 2.80, 3.6...
## $ `low-flag`                   <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1,...
## $ imputed                      <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,...
## $ subbed                       <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,...

All of our variables appear to have the data types we’d expect for them (e.g., dbl for variables we would think would be numerical and chr for those we expect to be text).

Step 3: Check the range of values

We’ll also want to check to see if there are any extreme values in our dataset that may have been the result of information being entered incorrectly. To do that, we’ll use the summary() function:

summary(ma_cities)

##      GEOID              year          name           parent-location
##  Min.   :2500135   Min.   :2000   Length:4245        Length:4245
##  1st Qu.:2523840   1st Qu.:2004   Class :character   Class :character
##  Median :2541960   Median :2008   Mode  :character   Mode  :character
##  Mean   :2542110   Mean   :2008
##  3rd Qu.:2561205   3rd Qu.:2012
##  Max.   :2582595   Max.   :2016
##
##    population      poverty-rate    renter-occupied-households
##  Min.   :     0   Min.   : 0.000   Min.   :     1.74
##  1st Qu.:  2245   1st Qu.: 2.480   1st Qu.:   199.61
##  Median :  4754   Median : 5.000   Median :   623.02
##  Mean   : 19266   Mean   : 6.476   Mean   :  3757.21
##  3rd Qu.: 23464   3rd Qu.: 8.895   3rd Qu.:  2990.25
##  Max.   :650281   Max.   :78.600   Max.   :178128.00
##  NA's   :175      NA's   :175      NA's   :1583
##  pct-renter-occupied median-gross-rent median-household-income
##  Min.   :  0.00      Min.   :   0.0    Min.   :     0
##  1st Qu.: 18.80      1st Qu.: 707.0    1st Qu.: 47360
##  Median : 30.41      Median : 906.0    Median : 60067
##  Mean   : 31.60      Mean   : 952.1    Mean   : 64479
##  3rd Qu.: 42.99      3rd Qu.:1141.0    3rd Qu.: 76328
##  Max.   :100.00      Max.   :2908.0    Max.   :250001
##  NA's   :175         NA's   :346       NA's   :206
##  median-property-value  rent-burden     pct-white        pct-af-am
##  Min.   :      0       Min.   : 0.0   Min.   :  0.00   Min.   : 0.000
##  1st Qu.: 199025       1st Qu.:24.9   1st Qu.: 84.84   1st Qu.: 0.310
##  Median : 285400       Median :28.2   Median : 92.28   Median : 1.090
##  Mean   : 319087       Mean   :29.1   Mean   : 87.73   Mean   : 2.402
##  3rd Qu.: 383200       3rd Qu.:31.9   3rd Qu.: 95.76   3rd Qu.: 2.690
##  Max.   :1000001       Max.   :50.0   Max.   :100.00   Max.   :41.210
##  NA's   :213           NA's   :311    NA's   :175      NA's   :175
##   pct-hispanic      pct-am-ind       pct-asian        pct-nh-pi
##  Min.   : 0.000   Min.   :0.0000   Min.   : 0.000   Min.   :0.00000
##  1st Qu.: 0.980   1st Qu.:0.0000   1st Qu.: 0.420   1st Qu.:0.00000
##  Median : 2.100   Median :0.0600   Median : 1.260   Median :0.00000
##  Mean   : 4.711   Mean   :0.1531   Mean   : 2.727   Mean   :0.01555
##  3rd Qu.: 4.940   3rd Qu.:0.1800   3rd Qu.: 3.260   3rd Qu.:0.01000
##  Max.   :76.350   Max.   :2.3600   Max.   :26.630   Max.   :0.71000
##  NA's   :175      NA's   :175      NA's   :175      NA's   :175
##   pct-multiple      pct-other       eviction-filings    evictions
##  Min.   : 0.000   Min.   : 0.0000   Min.   :   0.00   Min.   :   0.000
##  1st Qu.: 0.640   1st Qu.: 0.0000   1st Qu.:   3.13   1st Qu.:   2.255
##  Median : 1.210   Median : 0.1400   Median :  11.09   Median :   8.000
##  Mean   : 1.506   Mean   : 0.5807   Mean   : 116.84   Mean   :  72.593
##  3rd Qu.: 2.020   3rd Qu.: 0.5000   3rd Qu.:  61.00   3rd Qu.:  40.370
##  Max.   :11.890   Max.   :30.3200   Max.   :6447.00   Max.   :2616.000
##  NA's   :175      NA's   :175       NA's   :1583      NA's   :1583
##  eviction-rate    eviction-filing-rate    low-flag         imputed      subbed
##  Min.   : 0.000   Min.   : 0.000       Min.   :0.0000   Min.   :0   Min.   :0
##  1st Qu.: 0.700   1st Qu.: 1.090       1st Qu.:0.0000   1st Qu.:0   1st Qu.:0
##  Median : 1.425   Median : 2.090       Median :0.0000   Median :0   Median :0
##  Mean   : 1.693   Mean   : 2.479       Mean   :0.3925   Mean   :0   Mean   :0
##  3rd Qu.: 2.370   3rd Qu.: 3.467       3rd Qu.:1.0000   3rd Qu.:0   3rd Qu.:0
##  Max.   :10.730   Max.   :16.650       Max.   :1.0000   Max.   :0   Max.   :0
##  NA's   :1583     NA's   :1583

Glancing at that output, we see a few things.

First, there don’t appear to be any extreme values for any of our variables. Percentages don’t exceed 100 or go below 0 and the year goes from 2000 to 2016, as expected. The maximum values for renter-occupied-households and evictions are quite a bit higher than the values at the third quartile, but that’s probably because most cities in Massachusetts are quite small — and then you have the likes of Boston. We’ll want to look at these a bit more closely. (I’ll show you how to do that shortly.)

Second, some of our variables have missing data (NA values). Given that the number of NA values is pretty consistent (1,526), I’m guessing that data simply are not available for some cities at given years. However, it could also mean that I have some extra observations in ma_cities that are junk (e.g., blank lines at the bottom of the file). Again, we should check this out. (Again, more on this shortly.)

The special case of NA

Let’s start by examining some of those NA values. This is a special class of values because NA is used to denote missing data in R. NA values don’t have a numerical meaning, so we can’t perform operations on vectors that include them.

For example, R will simply output NA if we try to take the median of the following vector:

median(c(1, 3, 5, NA, 9))

## [1] NA

We can get around this in any of two ways. First, we can simply rewrite our vector without the NA value (i.e., c(1, 3, 5, 9)). Alternatively, we can see if the function has an argument for excluding NA values. In this case, we can add the argument na.rm=TRUE to tell the median() function to remove NA values before performing the operation:

median(c(1, 3, 5, NA, 9), na.rm=TRUE)

## [1] 4

Filtering out NA values

Similarly, because they lack a numerical value, we cannot filter out NA values based on a mathematical criterion.

If we want to filter out NA values, we will want to pair the base::is.na() function (which assesses whether each value in a given vector is NA or not) with the dplyr::filter() function (which allows us to exclude observations based on some criterion). This is how we would get a listing of our rows that have NA values in the evictions variable:

ma_cities %>%
  filter(is.na(evictions))