R for data science by Garrett Grolemund and Hadley Wickham


Lists

In this chapter, you’ll learn how to handle lists, the data structure R uses for complex, hierarchical objects. You’re already familiar with vectors, R’s data structure for 1d objects. Lists extend these ideas to model objects that are like trees. You can create a hierarchical structure with a list because unlike vectors, a list can contain other lists.

If you’ve worked with list-like objects before, you’re probably familiar with the for loop. I’ll talk a little bit about for loops here, but the focus will be functions from the purrr package. purrr makes it easier to work with lists by eliminating common for loop boilerplate so you can focus on the specifics. The apply family of functions in base R (apply(), lapply(), tapply(), etc) solve a similar problem, but purrr is more consistent and easier to learn.

The goal of using purrr functions instead of for loops is to allow you break common list manipulation challenges into independent pieces:

  1. How can you solve the problem for a single element of the list? Once you’ve solved that problem, purrr takes care of generalising your solution to every element in the list.

  2. If you’re solving a complex problem, how can you break it down into bite sized pieces that allow you to advance one small step towards a solution? With purrr, you get lots of small pieces that you can compose together with the pipe.

This structure makes it easier to solve new problems. It also makes it easier to understand your solutions to old problems when you re-read your old code.

In later chapters you’ll learn how to apply these ideas when modelling. You can often use multiple simple models to help understand a complex dataset, or you might have multiple models because you’re bootstrapping or cross-validating. The techniques you’ll learn in this chapter will be invaluable.

List basics

You create a list with list():

x <- list(1, 2, 3)
str(x)
#> List of 3
#>  $ : num 1
#>  $ : num 2
#>  $ : num 3

x_named <- list(a = 1, b = 2, c = 3)
str(x_named)
#> List of 3
#>  $ a: num 1
#>  $ b: num 2
#>  $ c: num 3

Unlike atomic vectors, lists() can contain a mix of objects:

y <- list("a", 1L, 1.5, TRUE)
str(y)
#> List of 4
#>  $ : chr "a"
#>  $ : int 1
#>  $ : num 1.5
#>  $ : logi TRUE

Lists can even contain other lists!

z <- list(list(1, 2), list(3, 4))
str(z)
#> List of 2
#>  $ :List of 2
#>   ..$ : num 1
#>   ..$ : num 2
#>  $ :List of 2
#>   ..$ : num 3
#>   ..$ : num 4

str() is very helpful when looking at lists because it focusses on the structure, not the contents.

Visualising lists

To explain more complicated list manipulation functions, it’s helpful to have a visual representation of lists. For example, take these three lists:

x1 <- list(c(1, 2), c(3, 4))
x2 <- list(list(1, 2), list(3, 4))
x3 <- list(1, list(2, list(3)))

I draw them as follows:

  • Lists are rounded rectangles that contain their children.

  • I draw each child a little darker than its parent to make it easier to see the hierarchy.

  • The orientation of the children (i.e. rows or columns) isn’t important, so I’ll pick a row or column orientation to either save space or illustrate an important property in the example.

Subsetting

There are three ways to subset a list, which I’ll illustrate with a:

a <- list(a = 1:3, b = "a string", c = pi, d = list(-1, -5))
  • [ extracts a sub-list. The result will always be a list.

    str(a[1:2])
    #> List of 2
    #>  $ a: int [1:3] 1 2 3
    #>  $ b: chr "a string"
    str(a[4])
    #> List of 1
    #>  $ d:List of 2
    #>   ..$ : num -1
    #>   ..$ : num -5

    Like subsetting vectors, you can use an integer vector to select by position, or a character vector to select by name.

  • [[ extracts a single component from a list. It removes a level of hierarchy from the list.

    str(y[[1]])
    #>  chr "a"
    str(y[[4]])
    #>  logi TRUE
  • $ is a shorthand for extracting named elements of a list. It works similarly to [[ except that you don’t need to use quotes.

    a$a
    #> [1] 1 2 3
    a[["b"]]
    #> [1] "a string"

Or visually:

Lists of condiments

It’s easy to get confused between [ and [[, but it’s important to understand the difference. A few months ago I stayed at a hotel with a pretty interesting pepper shaker that I hope will help you remember these differences:

If this pepper shaker is your list x, then, x[1] is a pepper shaker containing a single pepper packet:

x[2] would look the same, but would contain the second packet. x[1:2] would be a pepper shaker containing two pepper packets.

x[[1]] is:

If you wanted to get the content of the pepper package, you’d need x[[1]][[1]]:

Exercises

  1. Draw the following lists as nested sets.

  2. Generate the lists corresponding to these nested set diagrams.

  3. What happens if you subset a data frame as if you’re subsetting a list? What are the key differences between a list and a data frame?

For loops vs functionals

Imagine you have a data frame and you want to compute the mean of each column. You might write code like this:

df <- data.frame(
  a = rnorm(10),
  b = rnorm(10),
  c = rnorm(10),
  d = rnorm(10)
)

results <- numeric(length(df))
for (i in seq_along(df)) {
  results[i] <- mean(df[[i]])
}
results
#> [1] -0.441 -0.179 -0.124  0.152

(Here we’re taking advantage of the fact that a data frame is a list of the individual columns, so length() and seq_along() are useful.)

You realise that you’re going to want to compute the means of every column pretty frequently, so you extract it out into a function:

col_mean <- function(df) {
  results <- numeric(length(df))
  for (i in seq_along(df)) {
    results[i] <- mean(df[[i]])
  }
  results
}

But then you think it’d also be helpful to be able to compute the median or the standard deviation:

col_median <- function(df) {
  results <- numeric(length(df))
  for (i in seq_along(df)) {
    results[i] <- median(df[[i]])
  }
  results
}
col_sd <- function(df) {
  results <- numeric(length(df))
  for (i in seq_along(df)) {
    results[i] <- sd(df[[i]])
  }
  results
}

I’ve now copied-and-pasted this function three times, so it’s time to think about how to generalise it. Most of the code is for-loop boilerplate and it’s hard to see the one piece (mean(), median(), sd()) that differs.

What would you do if you saw a set of functions like this:

f1 <- function(x) abs(x - mean(x)) ^ 1
f2 <- function(x) abs(x - mean(x)) ^ 2
f3 <- function(x) abs(x - mean(x)) ^ 3

Hopefully, you’d notice that there’s a lot of duplication, and extract it out into an additional argument:

f <- function(x, i) abs(x - mean(x)) ^ i

You’ve reduce the chance of bugs (because you now have 1/3 less code), and made it easy to generalise to new situations. We can do exactly the same thing with col_mean(), col_median() and col_sd(), by adding an argument that contains the function to apply to each column:

col_summary <- function(df, fun) {
  out <- vector("numeric", length(df))
  for (i in seq_along(df)) {
    out[i] <- fun(df[[i]])
  }
  out
}
col_summary(df, median)
#> [1] -0.2458 -0.2873 -0.0567  0.1443
col_summary(df, min)
#> [1] -2.44 -1.86 -1.51 -1.91

The idea of using a function as an argument to another function is extremely powerful. It might take you a while to wrap your head around it, but it’s worth the investment. In the rest of the chapter, you’ll learn about and use the purrr package which provides a set of functions that eliminate the need for for-loops for many common scenarios.

Exercises

  1. Read the documentation for apply(). In the 2d case, what two for loops does it generalise?

  2. Adapt col_summary() so that it only applies to numeric columns You might want to start with an is_numeric() function that returns a logical vector that has a TRUE corresponding to each numeric column.

The map functions

This pattern of looping over a list and doing something to each element is so common that the purrr package provides a family of functions to do it for you. Each function always returns the same type of output so there are six variations based on what sort of result you want:

  • map() returns a list.
  • map_lgl() returns a logical vector.
  • map_int() returns a integer vector.
  • map_dbl() returns a double vector.
  • map_chr() returns a character vector.
  • map_df() returns a data frame.
  • walk() returns nothing. Walk is a little different to the others because it’s called exclusively for its side effects, so it’s described in more detail later in walk.

Each function takes a list as input, applies a function to each piece, and then returns a new vector that’s the same length as the input. The type of the vector is determined by the specific map function. Usually you want to use the most specific available, using map() only as a fallback when there is no specialised equivalent available.

We can use these functions to perform the same computations as the previous for loops:

map_int(x, length)
#> [1] 1 1 1
map_dbl(x, mean)
#> [1] 1 2 3
map_dbl(x, median)
#> [1] 1 2 3

Compared to using a for loop, focus is on the operation being performed (i.e. length(), mean(), or median()), not the book-keeping required to loop over every element and store the results.

There are a few differences between map_*() and compute_summary():

  • All purrr functions are implemented in C. This means you can’t easily understand their code, but it makes them a little faster.

  • The second argument, .f, the function to apply, can be a formula, a character vector, or an integer vector. You’ll learn about those handy shortcuts in the next section.

  • Any arguments after .f will be passed on to it each time it’s called:

    map_dbl(x, mean, trim = 0.5)
    #> [1] 1 2 3
  • The map functions also preserve names:

    z <- list(x = 1:3, y = 4:5)
    map_int(z, length)
    #> x y 
    #> 3 2

Shortcuts

There are a few shortcuts that you can use with .f in order to save a little typing. Imagine you want to fit a linear model to each group in a dataset. The following toy example splits the up the mtcars dataset in to three pieces (one for each value of cylinder) and fits the same linear model to each piece:

models <- mtcars %>% 
  split(.$cyl) %>% 
  map(function(df) lm(mpg ~ wt, data = df))

The syntax for creating an anonymous function in R is quite verbose so purrr provides a convenient shortcut: a one-sided formula.

models <- mtcars %>% 
  split(.$cyl) %>% 
  map(~lm(mpg ~ wt, data = .))

Here I’ve used . as a pronoun: it refers to the current list element (in the same way that i referred to the current index in the for loop). You can also use .x and .y to refer to up to two arguments. If you want to create a function with more than two arguments, do it the regular way!

When you’re looking at many models, you might want to extract a summary statistic like the R2. To do that we need to first run summary() and then extract the component called r.squared. We could do that using the shorthand for anonymous functions:

models %>% 
  map(summary) %>% 
  map_dbl(~.$r.squared)
#>     4     6     8 
#> 0.509 0.465 0.423

But extracting named components is a common operation, so purrr provides an even shorter shortcut: you can use a string.

models %>% 
  map(summary) %>% 
  map_dbl("r.squared")
#>     4     6     8 
#> 0.509 0.465 0.423

You can also use a numeric vector to select elements by position:

x <- list(list(1, 2, 3), list(4, 5, 6), list(7, 8, 9))
x %>% map_dbl(2)
#> [1] 2 5 8

Base R

If you’re familiar with the apply family of functions in base R, you might have noticed some similarities with the purrr functions:

  • lapply() is basically identical to map(). There’s no advantage to using map() over lapply() except that it’s consistent with all the other functions in purrr.

  • The base sapply() is a wrapper around lapply() that automatically tries to simplify the results. This is useful for interactive work but is problematic in a function because you never know what sort of output you’ll get:

    x1 <- list(
      c(0.27, 0.37, 0.57, 0.91, 0.20),
      c(0.90, 0.94, 0.66, 0.63, 0.06), 
      c(0.21, 0.18, 0.69, 0.38, 0.77)
    )
    x2 <- list(
      c(0.50, 0.72, 0.99, 0.38, 0.78), 
      c(0.93, 0.21, 0.65, 0.13, 0.27), 
      c(0.39, 0.01, 0.38, 0.87, 0.34)
    )
    
    threshold <- function(x, cutoff = 0.8) x[x > cutoff]
    str(sapply(x1, threshold))
    #> List of 3
    #>  $ : num 0.91
    #>  $ : num [1:2] 0.9 0.94
    #>  $ : num(0)
    str(sapply(x2, threshold))
    #>  num [1:3] 0.99 0.93 0.87
  • vapply() is a safe alternative to sapply() because you supply an additional argument that defines the type. The only problem with vapply() is that it’s a lot of typing: vapply(df, is.numeric, logical(1)) is equivalent to map_lgl(df, is.numeric).

    One of advantage of vapply() over the map functions is that it can also produce matrices - the map functions only ever produce vectors.

  • map_df(x, f) is effectively the same as do.call("rbind", lapply(x, f)) but under the hood is much more efficient.

Exercises

  1. How can you determine which columns in a data frame are factors? (Hint: data frames are lists.)

  2. What happens when you use the map functions on vectors that aren’t lists? What does map(1:5, runif) do? Why?

  3. What does map(-2:2, rnorm, n = 5) do. Why?

  4. Rewrite map(x, function(df) lm(mpg ~ wt, data = df)) to eliminate the anonymous function.

Handling hierarchy

The map functions apply a function to every element in a list. They are the most commonly used part of purrr, but not the only part. Since lists are often used to represent complex hierarchies, purrr also provides tools to work with hierarchy:

  • You can extract deeply nested elements in a single call by supplying a character vector to the map functions.

  • You can remove a level of the hierarchy with the flatten functions.

  • You can flip levels of the hierarchy with the transpose function.

Extracting deeply nested elements

Some times you get data structures that are very deeply nested. A common source of such data is JSON from a web API. I’ve previously downloaded a list of GitHub issues related to this book and saved it as issues.json. Now I’m going to load it into a list with jsonlite. By default fromJSON() tries to be helpful and simplifies the structure a little for you. Here I’m going to show you how to do it with purrr, so I set simplifyVector = FALSE:

# From https://api.github.com/repos/hadley/r4ds/issues
issues <- jsonlite::fromJSON("issues.json", simplifyVector = FALSE)

There are eight issues, and each issue is a nested list:

length(issues)
#> [1] 8
str(issues[[1]])
#> List of 20
#>  $ url         : chr "https://api.github.com/repos/hadley/r4ds/issues/11"
#>  $ labels_url  : chr "https://api.github.com/repos/hadley/r4ds/issues/11/labels{/name}"
#>  $ comments_url: chr "https://api.github.com/repos/hadley/r4ds/issues/11/comments"
#>  $ events_url  : chr "https://api.github.com/repos/hadley/r4ds/issues/11/events"
#>  $ html_url    : chr "https://github.com/hadley/r4ds/pull/11"
#>  $ id          : int 117521642
#>  $ number      : int 11
#>  $ title       : chr "Typo correction in file expressing-yourself.Rmd"
#>  $ user        :List of 17
#>   ..$ login              : chr "shoili"
#>   ..$ id                 : int 8914139
#>   ..$ avatar_url         : chr "https://avatars.githubusercontent.com/u/8914139?v=3"
#>   ..$ gravatar_id        : chr ""
#>   ..$ url                : chr "https://api.github.com/users/shoili"
#>   ..$ html_url           : chr "https://github.com/shoili"
#>   ..$ followers_url      : chr "https://api.github.com/users/shoili/followers"
#>   ..$ following_url      : chr "https://api.github.com/users/shoili/following{/other_user}"
#>   ..$ gists_url          : chr "https://api.github.com/users/shoili/gists{/gist_id}"
#>   ..$ starred_url        : chr "https://api.github.com/users/shoili/starred{/owner}{/repo}"
#>   ..$ subscriptions_url  : chr "https://api.github.com/users/shoili/subscriptions"
#>   ..$ organizations_url  : chr "https://api.github.com/users/shoili/orgs"
#>   ..$ repos_url          : chr "https://api.github.com/users/shoili/repos"
#>   ..$ events_url         : chr "https://api.github.com/users/shoili/events{/privacy}"
#>   ..$ received_events_url: chr "https://api.github.com/users/shoili/received_events"
#>   ..$ type               : chr "User"
#>   ..$ site_admin         : logi FALSE
#>  $ labels      : list()
#>  $ state       : chr "open"
#>  $ locked      : logi FALSE
#>  $ assignee    : NULL
#>  $ milestone   : NULL
#>  $ comments    : int 0
#>  $ created_at  : chr "2015-11-18T06:26:09Z"
#>  $ updated_at  : chr "2015-11-18T06:26:09Z"
#>  $ closed_at   : NULL
#>  $ pull_request:List of 4
#>   ..$ url      : chr "https://api.github.com/repos/hadley/r4ds/pulls/11"
#>   ..$ html_url : chr "https://github.com/hadley/r4ds/pull/11"
#>   ..$ diff_url : chr "https://github.com/hadley/r4ds/pull/11.diff"
#>   ..$ patch_url: chr "https://github.com/hadley/r4ds/pull/11.patch"
#>  $ body        : chr "The discussion of the code in lines 236-243 was a little confusing with x and y so I proposed changing it to a and b. Not sure "| __truncated__

To work with this sort of data, you typically want to turn it into a data frame by extracting the related vectors that you’re most interested in:

issues %>% map_int("id")
#> [1] 117521642 110795521 109680972 107925580 107506216  99430051  99430007
#> [8]  99429843
issues %>% map_lgl("locked")
#> [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
issues %>% map_chr("state")
#> [1] "open" "open" "open" "open" "open" "open" "open" "open"

You can use the same technique to extract more deeply nested structure. For example, imagine you want to extract the name and id of the user. You could do that in two steps:

users <- issues %>% map("user")
users %>% map_chr("login")
#> [1] "shoili"     "benmarwick" "datalove"   "hadley"     "hadley"    
#> [6] "hadley"     "hadley"     "hadley"
users %>% map_int("id")
#> [1] 8914139 1262179  222907    4196    4196    4196    4196    4196

But by supplying a character vector to map_*, you can do it in one:

issues %>% map_chr(c("user", "login"))
#> [1] "shoili"     "benmarwick" "datalove"   "hadley"     "hadley"    
#> [6] "hadley"     "hadley"     "hadley"
issues %>% map_int(c("user", "id"))
#> [1] 8914139 1262179  222907    4196    4196    4196    4196    4196

Removing a level of hierarchy

As well as indexing deeply into hierarchy, it’s sometimes useful to flatten it. That’s the job of the flatten family of functions: flatten(), flatten_lgl(), flatten_int(), flatten_dbl(), and flatten_chr(). In the code below we take a list of lists of double vectors, then flatten it to a list of double vectors, then to a double vector.

x <- list(list(a = 1, b = 2), list(c = 3, d = 4))
str(x)
#> List of 2
#>  $ :List of 2
#>   ..$ a: num 1
#>   ..$ b: num 2
#>  $ :List of 2
#>   ..$ c: num 3
#>   ..$ d: num 4

y <- flatten(x) 
str(y)
#> List of 4
#>  $ a: num 1
#>  $ b: num 2
#>  $ c: num 3
#>  $ d: num 4
flatten_dbl(y)
#> [1] 1 2 3 4

Graphically, that sequence of operations looks like:

Whenever I get confused about a sequence of flattening operations, I’ll often draw a diagram like this to help me understand what’s going on.

Base R has unlist(), but I recommend avoiding it for the same reason I recommend avoiding sapply(): it always succeeds. Even if your data structure accidentally changes, unlist() will continue to work silently the wrong type of output. This tends to create problems that are frustrating to debug.

Switching levels in the hierarchy

Other times the hierarchy feels “inside out”. You can use transpose() to flip the first and second levels of a list:

x <- list(
  x = list(a = 1, b = 3, c = 5),
  y = list(a = 2, b = 4, c = 6)
)
x %>% str()
#> List of 2
#>  $ x:List of 3
#>   ..$ a: num 1
#>   ..$ b: num 3
#>   ..$ c: num 5
#>  $ y:List of 3
#>   ..$ a: num 2
#>   ..$ b: num 4
#>   ..$ c: num 6
x %>% transpose() %>% str()
#> List of 3
#>  $ a:List of 2
#>   ..$ x: num 1
#>   ..$ y: num 2
#>  $ b:List of 2
#>   ..$ x: num 3
#>   ..$ y: num 4
#>  $ c:List of 2
#>   ..$ x: num 5
#>   ..$ y: num 6

Graphically, this looks like:

You’ll see an example of this in the next section, as transpose() is particularly useful in conjunction with adverbs like safely() and quietly().

It’s called transpose by analogy to matrices. When you subset a transposed matrix, you switch indices: x[i, j] is the same as t(x)[j, i]. It’s the same idea when transposing a list, but the subsetting looks a little different: x[[i]][[j]] is equivalent to transpose(x)[[j]][[i]]. Similarly, a transpose is its own inverse so transpose(transpose(x)) is equal to x.

Transpose is also useful when working with JSON APIs. Many JSON APIs represent data frames in a row-based format, rather than R’s column-based format. transpose() makes it easy to switch between the two:

df <- dplyr::data_frame(x = 1:3, y = c("a", "b", "c"))
df %>% transpose() %>% str()
#> List of 3
#>  $ :List of 2
#>   ..$ x: int 1
#>   ..$ y: chr "a"
#>  $ :List of 2
#>   ..$ x: int 2
#>   ..$ y: chr "b"
#>  $ :List of 2
#>   ..$ x: int 3
#>   ..$ y: chr "c"

Turning lists into data frames

  • Have a deeply nested list with missing pieces
  • Need a tidy data frame so you can visualise, transform, model etc.
  • What do you do?
  • By hand with purrr, talk about fromJSON and tidyJSON

Exercises

Dealing with failure

When you do many operations on a list, sometimes one will fail. When this happens, you’ll get an error message, and no output. This is annoying: why does one failure prevent you from accessing all the other successes? How do you ensure that one bad apple doesn’t ruin the whole barrel?

In this section you’ll learn how to deal this situation with a new function: safely(). safely() is an adverb: it takes a function (a verb) and returns a modified version. In this case, the modified function will never throw an error. Instead, it always returns a list with two elements:

  1. result is the original result. If there was an error, this will be NULL.

  2. error is an error object. If the operation was successful this will be NULL.

(You might be familiar with the try() function in base R. It’s similar, but because it sometimes returns the original result and it sometimes returns an error object it’s more difficult to work with.)

Let’s illustrate this with a simple example: log():

safe_log <- safely(log)
str(safe_log(10))
#> List of 2
#>  $ result: num 2.3
#>  $ error : NULL
str(safe_log("a"))
#> List of 2
#>  $ result: NULL
#>  $ error :List of 2
#>   ..$ message: chr "non-numeric argument to mathematical function"
#>   ..$ call   : language .f(...)
#>   ..- attr(*, "class")= chr [1:3] "simpleError" "error" "condition"

When the function succeeds the result element contains the result and the error element is NULL. When the function fails, the result element is NULL and the error element contains an error object.

safely() is designed to work with map:

x <- list(1, 10, "a")
y <- x %>% map(safely(log))
str(y)
#> List of 3
#>  $ :List of 2
#>   ..$ result: num 0
#>   ..$ error : NULL
#>  $ :List of 2
#>   ..$ result: num 2.3
#>   ..$ error : NULL
#>  $ :List of 2
#>   ..$ result: NULL
#>   ..$ error :List of 2
#>   .. ..$ message: chr "non-numeric argument to mathematical function"
#>   .. ..$ call   : language .f(...)
#>   .. ..- attr(*, "class")= chr [1:3] "simpleError" "error" "condition"

This would be easier to work with if we had two lists: one of all the errors and one of all the results. That’s easy to get with transpose().

y <- y %>% transpose()
str(y)
#> List of 2
#>  $ result:List of 3
#>   ..$ : num 0
#>   ..$ : num 2.3
#>   ..$ : NULL
#>  $ error :List of 3
#>   ..$ : NULL
#>   ..$ : NULL
#>   ..$ :List of 2
#>   .. ..$ message: chr "non-numeric argument to mathematical function"
#>   .. ..$ call   : language .f(...)
#>   .. ..- attr(*, "class")= chr [1:3] "simpleError" "error" "condition"

It’s up to you how to deal with the errors, but typically you’ll either look at the values of x where y is an error or work with the values of y that are ok:

is_ok <- y$error %>% map_lgl(is_null)
x[!is_ok]
#> [[1]]
#> [1] "a"
y$result[is_ok] %>% flatten_dbl()
#> [1] 0.0 2.3

Purrr provides two other useful adverbs:

  • Like safely(), possibly() always succeeds. It’s simpler than safely(), because you give it a default value to return when there is an error.

    x <- list(1, 10, "a")
    x %>% map_dbl(possibly(log, NA_real_))
    #> [1] 0.0 2.3  NA
  • quietly() performs a similar role to safely(), but instead of capturing errors, it captures printed output, messages, and warnings:

    x <- list(1, -1)
    x %>% map(quietly(log)) %>% str()
    #> List of 2
    #>  $ :List of 4
    #>   ..$ result  : num 0
    #>   ..$ output  : chr ""
    #>   ..$ warnings: chr(0) 
    #>   ..$ messages: chr(0) 
    #>  $ :List of 4
    #>   ..$ result  : num NaN
    #>   ..$ output  : chr ""
    #>   ..$ warnings: chr "NaNs produced"
    #>   ..$ messages: chr(0)

Exercises

  1. Challenge: read all the csv files in this directory. Which ones failed and why?

    files <- dir("data", pattern = "\\.csv$")
    files %>%
      set_names(., basename(.)) %>%
      map_df(safely(readr::read_csv), .id = "filename") %>%

Parallel maps

So far we’ve mapped along a single list. But often you have multiple related lists that you need iterate along in parallel. That’s the job of the map2() and pmap() functions. For example, imagine you want to simulate some random normals with different means. You know how to do that with map():

mu <- list(5, 10, -3)
mu %>% map(rnorm, n = 10)
#> [[1]]
#>  [1] 5.25 4.45 6.41 4.20 3.43 3.96 6.02 4.30 5.97 4.92
#> 
#> [[2]]
#>  [1] 10.89  9.22 10.44 10.41 10.98 11.15 11.22 10.00 10.76 10.34
#> 
#> [[3]]
#>  [1] -2.83 -1.60 -3.68 -2.26 -3.86 -2.58 -1.55 -2.81 -3.69 -1.66

What if you also want to vary the standard deviation? You need to iterate along a vector of means and a vector of standard deviations in parallel. That’s a job for map2() which works with two parallel sets of inputs:

sigma <- list(1, 5, 10)
map2(mu, sigma, rnorm, n = 10)
#> [[1]]
#>  [1] 7.74 4.06 3.22 4.28 5.91 4.23 4.22 4.57 4.33 6.39
#> 
#> [[2]]
#>  [1] 14.56 11.03 22.92  6.05 12.94  6.44 17.92 13.38  8.84 13.19
#> 
#> [[3]]
#>  [1] -16.71 -17.26 -15.46  -9.83 -12.80  -7.63   9.15 -15.78   4.48  30.92

map2() generates this series of function calls:

The arguments that vary for each call come before the function name, and arguments that are the same for every function call come afterwards.

Like map(), map2() is just a wrapper around a for loop:

map2 <- function(x, y, f, ...) {
  out <- vector("list", length(x))
  for (i in seq_along(x)) {
    out[[i]] <- f(x[[i]], y[[i]], ...)
  }
  out
}

You could also imagine map3(), map4(), map5(), map6() etc, but that would get tedious quickly. Instead, purrr provides pmap() which takes a list of arguments. You might use that if you wanted to vary the mean, standard deviation, and number of samples:

n <- list(1, 3, 5)
args1 <- list(n, mu, sigma)
args1 %>% pmap(rnorm) %>% str()
#> List of 3
#>  $ : num 6.62
#>  $ : num [1:3] 0.746 15.277 5.973
#>  $ : num [1:5] 12.96 4.76 8.52 18.33 4.04

That looks like:

However, instead of relying on position matching, it’s better to name the arguments. This is more verbose, but it makes the code clearer.

args2 <- list(mean = mu, sd = sigma, n = n)
args2 %>% pmap(rnorm) %>% str()
#> List of 3
#>  $ : num 5.72
#>  $ : num [1:3] 4.55 12.01 12.02
#>  $ : num [1:5] 17.4301 8.3891 -10.7706 -5.8046 -0.0261

That generates longer, but safer, calls:

Since the arguments are all the same length, it makes sense to store them in a data frame:

params <- dplyr::data_frame(mean = mu, sd = sigma, n = n)
params$result <- params %>% pmap(rnorm)
params
#> Source: local data frame [3 x 4]
#> 
#>       mean       sd        n   result
#>      (chr)    (chr)    (chr)    (chr)
#> 1 <dbl[1]> <dbl[1]> <dbl[1]> <dbl[1]>
#> 2 <dbl[1]> <dbl[1]> <dbl[1]> <dbl[3]>
#> 3 <dbl[1]> <dbl[1]> <dbl[1]> <dbl[5]>

As soon as your code gets complicated, I think a data frame is a good approach because it ensures that each column has a name and is the same length as all the other columns. We’ll come back to this idea when we explore the intersection of dplyr, purrr, and model fitting.

Invoking different functions

There’s one more step up in complexity - as well as varying the arguments to the function you might also vary the function itself:

f <- c("runif", "rnorm", "rpois")
param <- list(
  list(min = -1, max = 1), 
  list(sd = 5), 
  list(lambda = 10)
)

To handle this case, you can use invoke_map():

invoke_map(f, param, n = 5) %>% str()
#> List of 3
#>  $ : num [1:5] 0.919 -0.403 -0.9 0.152 -0.564
#>  $ : num [1:5] -5.731 4.231 0.409 -6.526 -4.725
#>  $ : int [1:5] 11 7 11 5 9

The first argument is a list of functions or character vector of function names. The second argument is a list of lists giving the arguments that vary for each function. The subsequent arguments are passed on to every function.

You can use dplyr::frame_data() to make creating these matching pairs a little easier:

# Needs dev version of dplyr
sim <- dplyr::frame_data(
  ~f,      ~params,
  "runif", list(min = -1, max = -1),
  "rnorm", list(sd = 5),
  "rpois", list(lambda = 10)
)
sim %>% dplyr::mutate(
  samples = invoke_map(f, params, n = 10)
)

Walk

Walk is an alternative to map that you use when you want to call a function for its side effects, rather than for its return value. You typically do this because you want to render output to the screen or save files to disk - the important thing is the action, not the return value. Here’s a very simple example:

x <- list(1, "a", 3)

x %>% 
  walk(print)
#> [1] 1
#> [1] "a"
#> [1] 3

walk() is generally not that useful compared to walk2() or pwalk(). For example, if you had a list of plots and a vector of file names, you could use pwalk() to save each file to the corresponding location on disk:

library(ggplot2)
plots <- mtcars %>% 
  split(.$cyl) %>% 
  map(~ggplot(., aes(mpg, wt)) + geom_point())
paths <- paste0(names(plots), ".pdf")

pwalk(list(paths, plots), ggsave, path = tempdir())
#> Saving 7 x 5 in image
#> Saving 7 x 5 in image
#> Saving 7 x 5 in image

walk(), walk2() and pwalk() all invisibly return the .x, the first argument. This makes them suitable for use in the middle of pipelines.

Predicates

Imagine we want to summarise each numeric column of a data frame. We could do it in two steps:

  1. Find all numeric columns.
  2. Summarise each column.

In code, that would look like:

col_sum <- function(df, f) {
  is_num <- df %>% map_lgl(is_numeric)
  df[is_num] %>% map_dbl(f)
}

is_numeric() is a predicate: a function that returns either TRUE or FALSE. There are a number of of purrr functions designed to work specifically with predicates:

  • keep() and discard() keeps/discards list elements where the predicate is true.

  • head_while() and tail_while() keep the first/last elements of a list until you get the first element where the predicate is true.

  • some() and every() determine if the predicate is true for any or all of the elements.

  • detect() and detect_index()

We could use keep() to simplify the summary function to:

col_sum <- function(df, f) {
  df %>%
    keep(is.numeric) %>%
    map_dbl(f)
}

I like this formulation because you can easily read the sequence of steps.

Built-in predicates

Purrr comes with a number of predicate functions built-in:

lgl int dbl chr list null
is_logical() x
is_integer() x
is_double() x
is_numeric() x x
is_character() x
is_atomic() x x x x
is_list() x
is_vector() x x x x x
is_null() x

Compared to the base R functions, they only inspect the type of the object, not its attributes. This means they tend to be less surprising:

is.atomic(NULL)
#> [1] TRUE
is_atomic(NULL)
#> [1] FALSE

is.vector(factor("a"))
#> [1] FALSE
is_vector(factor("a"))
#> [1] TRUE

Each predicate also comes with “scalar” and “bare” versions. The scalar version checks that the length is 1 and the bare version checks that the object is a bare vector with no S3 class.

y <- factor(c("a", "b", "c"))
is_integer(y)
#> [1] TRUE
is_scalar_integer(y)
#> [1] FALSE
is_bare_integer(y)
#> [1] FALSE

Exercises

  1. A possible base R equivalent of col_sum() is:

    col_sum3 <- function(df, f) {
      is_num <- sapply(df, is.numeric)
      df_num <- df[, is_num]
    
      sapply(df_num, f)
    }

    But it has a number of bugs as illustrated with the following inputs:

    df <- data.frame(z = c("a", "b", "c"), x = 1:3, y = 3:1)
    # OK
    col_sum3(df, mean)
    # Has problems: don't always return numeric vector
    col_sum3(df[1:2], mean)
    col_sum3(df[1], mean)
    col_sum3(df[0], mean)

    What causes the bugs?

  2. Carefully read the documentation of is.vector(). What does it actually test for?