Tidyverse Functions (Part 1)

Introduction

An incredibly useful metapackage for data journalists is tidyverse. This metapackage loads a handful of packages originally authored by Hadley Wickham that help users load data, clean them, reshape them, and aggregate them in different ways.

In this tutorial, I will outline some of the key functions from the dplyr package (which is part of tidyverse).

First, here is a summary of the functions covered in this tutorial and a quick description of when you’ll want to use each:

• Use pipes (%>%) whenever you want to pass information to another function (i.e., when you’re adding a step to your analysis)

• Use select() whenever you want to only include information about certain variables (columns)

• Use filter() whenever you want to only include observations (rows) that match specified criteria

• Use count() whenever you want to count how often a unique value comes up (e.g., how many times Rodrigo Zamith appears within a given variable)

• Use arrange() whenever you want to sort the data (e.g., from high to low)

• Use summarize() whenever you want to reduce many values to a single one (e.g., get the median value)

• Use group_by() whenever you want to apply an operation separately for each value within a variable (e.g., every different department)

• Use ggplot() whenever you want to create data visualizations

Our dataset for this tutorial

I will be illustrating each function by applying them to a dataset of gross salaries for Town of Amherst employees in 2020. These data were extracted from the PDF file using Tabula and exported to this CSV file.

I can load this dataset by using the read_csv() function from the readr package (part of tidyverse), supplying the URL for the dataset as my lone argument.

options(scipen=999)
library(tidyverse)
salaries <- read_csv("https://dds.rodrigozamith.com/files/amherst_gross_wages_2020.csv")

## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.1 ──

## ✓ ggplot2 3.3.5     ✓ purrr   0.3.4
## ✓ tibble  3.1.3     ✓ dplyr   1.0.7
## ✓ tidyr   1.1.3     ✓ stringr 1.4.0
## ✓ readr   2.0.1     ✓ forcats 0.5.1

## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## x dplyr::filter() masks stats::filter()
## x dplyr::lag()    masks stats::lag()

## Rows: 802 Columns: 4

## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (3): last_name, first_name, department
## dbl (1): gross_wages

##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

Let’s review the first few rows in the dataset to see what we’re working with.

head(salaries)

As we can see, there are four variables in this dataset (and at least six observations).

Piping

The tidyr package (part of tidyverse) gives us access to a special kind of operator (%>%) that permits us to send the output (or result) of one line of code forward to the next line of code. In effect, what this does is pre-fill the first argument of the function appearing in the line of code following the pipe.

Consider the following code:

head(salaries, n=10)

In that example, we are supplying the head() function with two arguments. The first argument is unnamed; unless specified otherwise, head() assumes that the first argument you are providing it is for an attribute it calls x (which is expected to be the name of an object in our environment). Put another way, the above code is functionally equivalent to head(x=salaries, n=10). The x argument is required; without it, head() will give you an error:

head()

## Error in checkHT(n, dx <- dim(x)): argument "x" is missing, with no default

Then, by adding a comma, we conveyed that we’d be specifying a second argument. That second argument is n, which tells the head() function to show me the first X rows instead of six (the default). By specifying the number 10 (n=10), I am telling it to show me the first 10 rows.

Returning to the piping mechanism, we can use it as follows. First, we’ll load the salaries object. Then, we’ll use the piping operator to pipe the output of that line of code (all of the information about the salaries object) to another function. In the next line, we write out the head() function, but we only supply the second argument (n=10). That’s because the piping operator is pre-filling the first argument (x) that head() expects. The result is equivalent to what we saw above.

salaries %>%
  head(n=10)

While this approach may not seem particularly useful on such a simple example, it becomes extremely useful when you start chaining multiple functions together. Put another way, instead of nesting a bunch of functions inside one another in a way that is difficult to read (e.g., arrange(summarize(group_by(filter( ...​ )))))), you can place each function (step in the analysis) on its own line and just pipe the output from one function to the next.

This produces cleaner code that is much easier to follow. (Consequently, it makes it easier to avoid mistakes and to think logically about problem-solving.)

select()

The select() function is used to only include certain variables (columns) in the output. (Put another way, you are dropping the non-selected variables.) This is particularly useful when we are looking at the dataset ourselves. For example, if you have a lot of variables but only care about a few of them, you’ll want to use select(). Additionally, this function can be quite useful in advanced applications, as we’ll see later in the book.

With the select() function, you simply need to list the variables you are interested in (separating them with commas).

For example, let’s use that function to only examine the first_name, last_name, and department variables in our dataset:

salaries %>%
  select(first_name, last_name, department)

We now see a data frame object that has just those three variables (it omits the gross_wages variable).

This isn’t a big deal with a small dataset like this one. However, if you have a dataset with 50+ variables and want to take a look at just a few variables from the left and right ends of the dataset, this will prove quite helpful.

filter()

While select() is designed to include only certain variables (columns), filter() is designed to include only certain observations (rows) based on the criteria we specify. Use filter() when a question asks you to only include certain kinds of observations (e.g., when a variable is equivalent to, greater than, etc., a particular value).

For example, if we only want to include people from the Police Department, we’d use a filter that specifies an equivalency between our desired variable (department) and value (Police Department):

salaries %>%
  filter(department=="Police Department")

The equivalency operators most likely to be useful to you at this stage are:

• == (is equivalent to)

• != (is not equivalent to)

• > (is greater than)

• < (is less than)

• >= (is greater than or equal to)

• ⇐ (is less than or equal to)

This function also allows us to specify multiple criteria (e.g., include people from the Police Department with the title Police Officer) using Boolean operators.

The Boolean operators that are most likely to be useful to you at this stage are:

• & (AND), which is useful if we want a filter for observations where the individual’s gross_wages exceeds 100000 AND they work for in a department called Police Department

• | (OR), which is useful if we want a filter for observations where the individual works for either the department called Police Department OR the department called Fire Department.

• ! (NOT), which is useful if we want a filter for observations where the individual works for any department except the Public Works.

It is important to note that we can also group sets of conditions by using parentheses (). For example, we can specify the following filter to give me all individuals making 100000 or more except those working for either the Police Department OR Fire Department:

salaries %>%
  filter(gross_wages >= 100000 & !(department == "Police Department" | department == "Fire Department"))

Similarly, here is a filter that outputs the individuals who work for either the Public Works department OR the Town Clerk's office AND make more than 75000:

salaries %>%
  filter((department == "Public Works" | department == "Town Clerk's") & gross_wages > 75000)

As you can see, 16 individuals matched my filter criteria.

count()

We use the count() function whenever we want to count how often a unique value comes up inside a variable. This is useful whenever we want to see how many of something we have in our dataset.

For example, if we want to see how many people there are in each department (i.e., how often each unique value appears within the department variable, since each observation in our dataset refers to a different person), we can just supply the variable we want to count by as the lone argument.

salaries %>%
  count(department)

As we can see, the “Animal Control” department had the fewest employees in our dataset.

Do note that you can actually supply multiple variables to count by, simply by providing them as additional arguments. For example, if we want all of the unique last_name values within all of the unique department values, we would simply write:

salaries %>%
  count(department, last_name)

(In this case, very few departments have more than one employee sharing a last name. However, if you look closely at the full output on your computer, you will see that most do!)

arrange()

We use the arrange() function whenever we want to sort our data in some fashion. We pair it with a particular variable (what to sort by) by supplying the variable as an argument. We can also specify if we want it in ascending order (low-to-high, which is the default) or descending order (high-to-low, by using the desc() function within it).

To illustrate this, let’s add a third step to our earlier code counting the unique department values by supplying an additional pipe (%>%). In that third step, we will sort the resulting data frame (count of department) in descending order based on the n variable:

salaries %>%
  count(department) %>%
  arrange(desc(n))

As we can see, the “Town Clerk’s” department had the most employees in our dataset.

summarize()

We use summarize() whenever we want to take multiple values from a variable and return a single value summarizing some aspect of that variable. This includes measures of central tendency and dispersion for that variable, such as the minimum, maximum, mean, median, variance, or standard deviation.

We supply the summarize() function with an argument structured with (1) the name of the variable we want to create/compute (e.g., mean_wages), (2) followed by the equal sign (=) to denote what we want that variable to be equal to, which is followed by (3) the operation we want to perform to calculate the summary statistic (e.g., mean(gross_wages)).

This is what that would look like in practice:

salaries %>%
  summarize(mean_wages=mean(gross_wages))

Notice how we now get a dataset with a single observation and a single variable. That is because we have calculated a single summary statistic for all of our observations. Put another way, the summarize() function aggregates our data by producing a new data frame with the new variables we want to calculate (e.g., aggregate by). As the dataset has been aggregated, we cannot go back to the original data within this operation.

If we want to calculate multiple summary statistics, we would just supply them as additional arguments to the summarize() function. For example, here is how we would calculate the mean, median, and maximum salaries in our dataset:

salaries %>%
  summarize(mean_wages=mean(gross_wages), median_wages=median(gross_wages), max_wages=max(gross_wages))

group_by()

We use the group_by() function whenever we want to apply an operation(s) separately for each unique value within a variable. This is particularly useful when we want to treat groups independently.

With the group_by() function, we simply supply the grouping variable (the variable containing the unique values we want to apply operations independently to) as an argument.

For example, if we wanted to calculate the mean, median, and max values for the gross_wages variable for each department in our variable, we would make the department our grouping variable. However, we would need to specify our groupings before we calculate our summary statistics. This is how that would work out:

salaries %>%
  group_by(department) %>%
  summarize(mean_wages=mean(gross_wages), median_wages=median(gross_wages), max_wages=max(gross_wages))

As with the count() variable, we can also nest our groupings (i.e., perform the operation separately on each last_name within each department) by supplying them as additional arguments to the group_by() function.

Additionally, it is worth noting here that the count() function is effectively just a wrapper (shortcut) for calculating a summary statistic. We could replicate our count() results from above by performing an operation with the n() function, like this:

salaries %>%
  group_by(department) %>%
  summarize(n=n())

ggplot()

We use ggplot() and its child functions whenever we want to create visualizations of our datasets. This function comes from the powerful ggplot2 package (which is loaded by tidyverse).

ggplots use what we call a grammar of graphics, which is basically a general scheme for data visualization that breaks up the visual into semantic components such as scales and layers.

You can think of this layering process as being conceptually similar to the piping (%>%) we’ve done throughout this tutorial — it is just putting one thing on top of the other. However, unlike the pipes we’ve used, ggplot’s semantic uses the plus sign (+) at the end of a line of code to signal the addition of a layer in the subsequent line.

Let’s illustrate this by creating a bar chart with the mean wage for the 5 departments with the highest mean salaries.

We’ll start by creating the data frame we’ll use to plot:

salaries %>%
  group_by(department) %>%
  summarize(median_wages=median(gross_wages)) %>%
  arrange(desc(median_wages)) %>% %>%
  head(5)

The result that you see above is the information that we will be providing to our ggplot. Put another way, it is a manipulated (in this case, aggregated) slice of our original data.

The ggplot semantic is quite powerful, so we will start with a very basic example. With any ggplot, the first layer will include some information about how to map the data onto a coordinate plane (e.g., X and Y axes).

In our example, let’s go ahead and add our base layer by using the ggplot() function and setting the mapping attribute by using the aes() (aesthetic mappings) function. Within that aes() function, we will map the department variable onto the X axis and the mean_wages variable onto the Y axis. Here’s what that looks like now.

salaries %>%
  group_by(department) %>%
  summarize(median_wages=median(gross_wages)) %>%
  arrange(desc(median_wages)) %>% %>%
  head(5) %>%
  ggplot(aes(x=department, y=median_wages))

Notice that we have our five departments in the X axis and the correct range (lowest and highest values) of the salaries in the Y axis. However, for us to see the values for each department, we need to add a second layer containing some geometric shape that represent the data.

There are a lot of different geometric shapes you can add to your plot as additional layers (e.g., to create line charts, scatterplots, etc.). You can also re-label your axes and change their ranges by adding those functions as layers. You can also simply flip the bars to be horizontal by adding a coord_flip() layer.

For a simple vertical bar graph, we can use the geom_col() function. Let’s use that function by adding a second layer at the end of our code.

salaries %>%
  group_by(department) %>%
  summarize(median_wages=median(gross_wages)) %>%
  arrange(desc(median_wages)) %>% %>%
  head(5) %>%
  ggplot(aes(x=department, y=median_wages)) +
    geom_col()

We could go on to style the visualization to look much nicer (including organizing the bars from high-to-low!), but simple visualizations like this will usually do the trick during the exploratory phase of data analysis.

Approaching a Question

When you start to answer the question you’ve posed the data, it is helpful to break up the question logically into multiple steps before you even write a single line of code.

For example, consider the question: Among the Police, Fire, and Public Works departments, which one had the highest median wage?

This question asks us to:

• Include only certain kinds of observations (filter(), matching certain department criteria)

• Apply an operation separately to each group (group_by(), using the department variable)

• Calculate a summary statistic (summarize(), to get the median() value of the different gross_wages vectors)

• Sort the information (arrange(), in descending (desc()) order by the median wage variable)

Now that we know the logic of the question, we can translate it into the code below.

salaries %>%
  filter(department == "Police Department" | department == "Fire Department" | department == "Public Works") %>%
  group_by(department) %>%
  summarize(median_wages=median(gross_wages)) %>%
  arrange(desc(median_wages))