5 Variation

If you measure any quantity twice—and precisely enough, you will get two different results. This is true even for quantities that should be constant, like the speed of light (below).

This phenomenon, called variation, is the beginning of data science. To understand anything you must decipher patterns of variation. But variation does more than just obscure, it is an incredibly useful tool. Patterns of variation provide evidence of causal relationships.

The best way to study variation is to collect data, particularly rectangular data: data that is made up of variables, observations, and values.

  • A variable is a quantity, quality, or property that you can measure.

  • A value is the state of a variable when you measure it. The value of a variable may change from measurement to measurement.

  • An observation is a set of measurements you make under similar conditions (usually all at the same time or on the same object). Observations contain values that you measure on different variables.

Rectangular data provides a clear record of variation, but that doesn’t mean it is easy to understand. The human mind isn’t built to process tables of data. This section will show you the best ways to comprehend your own data, which is the most important challenge of data science.

Table: (#tab:unnamed-chunk-3)The speed of light is the universal constant, but variation obscures its value, here demonstrated by Albert Michelson in 1879. Michelson measured the speed of light 100 times and observed 30 different values (in km/sec).


3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05 3e+05


Tip: Throughout this section, we will rely on a distinction between two types of variables:

  • A variable is continuous if you can arrange its values in order and an infinite number of values can exist between any two values of the variable. For example, numbers and date-times are continuous variables. ggplot2 will treat your variable as continuous if it is a numeric, integer, or a recognizable date-time class (but not a factor, see ?factor).

  • A variable is discrete if it is not continuous. Discrete variables can only contain a finite (or countably infinite) set of unique values. For example, character strings and boolean values are discrete variables. ggplot2 will treat your variable as discrete if it is not a numeric, integer, or recognizable date-time class.


5.0.1 Visualizing Distributions

The first group of geoms visualizes the distribution of the values in a variable.

Recall that a variable is a quantity, quality, or property whose value can change between measurements. This unique property—that the values of a variable can vary—gives the word “variable” its name. It also motivates all of data science. Scientists attempt to understand what determines the value of a variable. They then use that information to predict or control the value of the variable under a variety of circumstances.

One of the most useful tools in this quest are the values themselves, the values that you have already observed for a variable. These values reveal which states of the variable are common, which are rare, and which are seemingly impossible. The pattern of values that emerges as you collect large amounts of data is known as the variable’s distribution.

The distribution of a variable reveals information about the probabilities associated with the variable. As you collect more data, the proportion of observations that occur at a value (or in an interval) will match the probability that the variable will take that value (or take a value in that interval) in a future measurement.

In theory, it is easy to visualize the distribution of a variable: simply display how many observations occur at each value of the variable. In practice, how you do this will depend on the type of variable that you wish to visualize.

5.0.1.0.1 Discrete distributions

Use geom_bar() to visualize the distribution of a discrete variable. geom_bar() counts the number of observations that are associated with each value of the variable, and it displays the results as a series of bars. The height of each bar reveals the count of observations that are associated with the x value of the bar.

ggplot(data = diamonds) +
  geom_bar(mapping = aes(x = cut))


Tip - Since each of the geoms in this subsection visualizes the values of a single variable, you do not need to provide a \(y\) aesthetic.


Useful aesthetics for geom_bar() are:

  • x (required)
  • alpha
  • color
  • fill
  • linetype
  • size
  • weight

Useful position adjustments for geom_bar() are

  • “stack” (default)
  • “dodge”
  • “fill”

Useful stats for geom_bar() are

  • “bin” (default)
  • “identity” (to map bar heights to a y variable)

The width argument of geom_bar() controls the width of each bar. The bars will touch when you set width = 1.

ggplot(data = diamonds) +
  geom_bar(mapping = aes(x = cut), width = 1)


Tip: You can compute the counts of a discrete variable quickly with R’s table() function. These are the numbers that geom_bar() visualizes.

table(diamonds$cut)
#> 
#>      Fair      Good Very Good   Premium     Ideal 
#>      1610      4906     12082     13791     21551

5.0.1.0.2 Continuous distributions

The strategy of counting the number of observations at each value breaks down for continuous data. If your data is truly continuous, then no two observations will have the same value—so long as you measure the data precisely enough (e.g. without rounding to the _n_th decimal place).

To get around this, data scientists divide the range of a continuous variable into equally spaced intervals, a process called binning.

They then count how many observations fall into each bin.

And display the count as a bar, or some other object.

This method is temperamental because the appearance of the distribution can change dramatically if the bin size changes. As no bin size is “correct,” you should explore several bin sizes when examining data.

Several geoms exist to help you visualize continuous distributions. They almost all use the “bin” stat to implement the above strategy. For each of these geoms, you can set the following arguments for “bin” to use:

  • binwidth - the width to use for the bins in the same units as the x variable
  • origin - origin of the first bin interval
  • right - if TRUE bins will be right closed (e.g. points that fall on the border of two bins will be counted with the bin to the left)
  • breaks - a vector of actual bin breaks to use. If you set the breaks argument, it will override the binwidth and origin arguments.

Use geom_histogram() to make a traditional histogram. The height of each bar reveals how many observations fall within the width of the bar.

ggplot(data = diamonds) +
  geom_histogram(aes(x = carat))
#> `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

By default, geom_histogram() will divide the range of the variable into 30 equal length bins. The quickest way to change this behavior is to set the binwidth argument.

ggplot(data = diamonds) +
  geom_histogram(aes(x = carat), binwidth = 1)

Notice how different binwidths reveal different information. The plot above shows that the availability of diamonds decreases quickly as carat size increases. The plot below shows that there are more diamonds than you would expect at whole carat sizes (and common fractions of carat sizes). Moreover, for each popular size, there are more diamonds slightly larger than the size than diamonds slightly smaller than the size.

ggplot(data = diamonds) +
  geom_histogram(aes(x = carat), binwidth = 0.01)

Useful aesthetics for geom_histogram() are:

  • x (required)
  • alpha
  • color
  • fill
  • linetype
  • size
  • weight

Useful position adjustments for geom_histogram() are

  • “stack” (default)
  • “fill”

geom_freqpoly() uses a line to display the same information as geom_histogram(). You can think of geom_freqpoly() as drawing a line that connects the tops of the bars that would appear in a histogram.

ggplot(data = diamonds) +
  geom_freqpoly(aes(x = carat))

ggplot(data = diamonds) +
  geom_histogram(aes(x = carat))

It is easier to compare levels of a third variable with geom_freqpoly() than with geom_histogram(). geom_freqpoly() displays the shape of the distribution faithfully for each subgroup because you can plot multiple lines in the same graph without adjusting their position. Notice that geom_histogram() must stack each new subgroup on top of the others, which obscures the shape of the distributions.

ggplot(data = diamonds) +
  geom_freqpoly(aes(x = carat, color = cut))

ggplot(data = diamonds) +
  geom_histogram(aes(x = carat, fill = cut))

Useful aesthetics for geom_freqpoly() are:

  • x (required)
  • y
  • alpha
  • color
  • linetype
  • size

Although the name of geom_freqpoly() suggests that it draws a polygon, it actually draws a line. You can draw the same information as a true polygon (and thus fill in the area below the line) if you combine geom_area() with stat = "bin". You will learn more about geom_area() in Visualizing functions between two variables.

ggplot(data = diamonds) +
  geom_area(aes(x = carat, fill = cut), stat = "bin", position = "stack")
#> `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

geom_density() plots a one dimensional kernel density estimate of a variable’s distribution. The result is a smooth version of the information contained in a histogram or a freqpoly.

ggplot(data = diamonds) +
  geom_density(aes(x = carat))

geom_density() displays \(density\)—not \(count\)—on the y axis, which makes it easier to compare the shape of the distributions of multiple subgroups; the area under each curve will be normalized to one, no matter how many total observations occur in the subgroup.

geom_density() does not use the binwidth argument. You can control the smoothness of the density with adjust, and you can select the kernel to use to estimate the density with kernel. Set kernel to one of “gaussian” (default), “epanechikov”, “rectangular”, “triangular”, “biweight”, “cosine”, “optcosine”.

ggplot(data = diamonds) +
  geom_density(aes(x = carat, color = cut), kernel = "gaussian", adjust = 4)

Useful aesthetics for geom_density() are:

  • x (required)
  • y
  • alpha
  • color
  • fill
  • linetype
  • size

Useful position adjustments for geom_density() are

  • “identity” (default)
  • “stack” (when using the fill aesthetic)
  • “fill” (when using the fill aesthetic)

geom_dotplot() provides a final way to visualize distributions. This unique geom displays a point for each observation, but it stacks points that appear in the same bin on top of each other. The result is similar to a histogram, the height of each stack reveals the number of points in the stack.

ggplot(data = mpg) +
  geom_dotplot(aes(x = displ), binwidth = 0.2)

Useful aesthetics for geom_dotplot() are:

  • x (required)
  • y
  • alpha
  • color
  • fill

Useful arguments that apply to geom_dotplot()

  • binaxis - the axis to bin along (“x” or “y”)
  • binwidth - the interval width to use when binning
  • dotsize - diameter of dots relative to binwidth
  • stackdir - which direction to stack the dots (“up” (default), “down”, “center”, “centerwhole”)
  • stackgroups - Has the equivalent of position = "stack" when set to true.
  • stackratio - how close to stack the dots. Values less than 1 cause dots to overlap, which shortens stacks.

In practice, I find that geom_dotplot() works best with small data sets and takes a lot of tweaking of the binwidth, dotsize, and stackratio arguments to fit the dots within the graph (the stack heights depend entirely on the organization of the dots, which renders the y axis ambiguous). That said, dotplots can be useful as a learning aid. They provide an intuitive representation of a histogram.

5.0.2 Compare Distributions

5.0.3 Visualize Covariation

5.0.3.1 Visualize functions between two variables

Distributions provide useful information about variables, but the information is general. By itself, a distribution cannot tell you how the value of a variable in one set of circumstances will differ from the value of the same variable in a different set of circumstances.

Covariation can provide more specific information. Covariation is a relationship between the values of two or more variables.

To see how covariation works, consider two variables: the \(volume\) of an object and its \(temperature\). If the \(volume\) of the object usually increases when the \(temperature\) of the object increases, then you could use the value of \(temperature\) to help predict the value of \(volume\).

You’ve probably heard that “correlation (covariation) does not prove causation.” This is true, two variables can covary without one causing the other. However, covariation is often the first clue that two variables have a causal relationship.

Visualization is one of the best ways to spot covariation. How you look for covariation will depend on the structural relationship between two variables. The simplest structure occurs when two continuous variables have a functional relationship, where each value of one variable corresponds to a single value of the second variable.

In this scenario, covariation will appear as a pattern in the relationship. If two variables do not covary, their functional relationship will look like a random walk.

The variables date and unemploy in the economics data set have a functional relationship. The economics data set comes with ggplot2 and contains various economic indicators for the United States between 1967 and 2007. The unemploy variable measures the number of unemployed individuals in the United States in thousands.

A scatterplot of the data reveals the functional relationship between date and unemploy.

ggplot(data = economics) +
  geom_point(aes(x = date, y = unemploy))

geom_line() makes the relationship clear. geom_line() creates a line chart, one of the most used—and most efficient—devices for visualizing a function.

ggplot(data = economics) +
  geom_line(aes(x = date, y = unemploy))

Useful aesthetics for geom_line() are:

  • x (required)
  • y (required)
  • alpha
  • color
  • linetype
  • size

Use geom_step() to turn a line chart into a step function. Here, the result will be easier to see with a subset of data.

ggplot(data = economics[1:150, ]) +
  geom_step(aes(x = date, y = unemploy))

Control the step direction by giving geom_step() a direction argument. direction = "hv" will make stairs that move horizontally then vertically to connect points. direction = "vh" will do the opposite.

Useful aesthetics for geom_step() are:

  • x (required)
  • y (required)
  • alpha
  • color
  • linetype
  • size

geom_area() creates a line chart with a filled area under the line.

ggplot(data = economics) +
  geom_area(aes(x = date, y = unemploy))

Useful aesthetics for geom_area() are:

  • x (required)
  • y (required)
  • alpha
  • color
  • fill
  • linetype
  • size
5.0.3.1.1 Visualize correlations between two variables

Many variables do not have a functional relationship. As a result, a single value of one variable can correspond to multiple values of another variable.

Height and weight are two variables that are often related, but do not have a functional relationship. You could examine a classroom of students and notice that three different students, with three different weights all have the same height, 5’4“. In this case, there is not a one to one relationship between height and weight.

The easiest way to plot the relationship between two variables is with a scatterplot, i.e. geom_point(). If the variables covary, a pattern will appear in the points. If they do not, the points will look like a random cloud of points.

ggplot(data = mpg) +
  geom_point(mapping = aes(x = displ, y = hwy))

Useful aesthetics for geom_point() are:

  • x (required)
  • y (required)
  • alpha
  • color
  • fill (for some shapes)
  • shape
  • size

Useful position adjustments for geom_point() are:

  • “identity” (default)
  • “jitter”

In fact, the jitter adjustment is so useful that ggplot2 provides the geom_jitter(), which is identical to geom_point() but comes with position = "jitter" by default.

ggplot(data = mpg) +
  geom_jitter(mapping = aes(x = displ, y = hwy))

geom_jitter() can be a useful way to visualize the distribution between two discrete variables. Can you tell why geom_point() would be less useful here?

ggplot(data = mpg) +
  geom_jitter(mapping = aes(x = cyl, y = fl, color = fl))

Use geom_rug() to visualize the distribution of each variable in the scatterplot. geom_rug() adds a tickmark along each axis for each value observed in the data. geom_rug() works best as a second layer in the plot (see Section 3 for more info on layers).

ggplot(data = mpg) +
  geom_point(mapping = aes(x = displ, y = hwy)) +
  geom_rug(mapping = aes(x = displ, y = hwy), position = "jitter")

Use the sides argument to control which axes to place a “rug” on.

  • sides = "bl" - (default) Places a rug on each axis
  • sides = "b" - Places a rug on the bottom axis
  • sides = "l" - Places a rug on the left axis

Useful aesthetics for geom_rug() are:

  • x (required)
  • y (required)
  • alpha
  • color
  • linetype
  • size

Useful position adjustments for geom_rug() are:

  • “identity” (default)
  • “jitter”

Use geom_text() to display a label, instead of a point, for each observation in a scatterplot. geom_text() lets you add information to the scatterplot, but is less effective when you have many data points.

ggplot(data = mpg[sample(1:234, 10), ]) +
  geom_text(mapping = aes(x = displ, y = hwy, label = class))

Useful aesthetics for geom_text() are:

  • x (required)
  • y (required)
  • alpha
  • angle
  • color
  • family
  • fontface
  • hjust
  • label (geom_text() displays the values of this variable)
  • lineheight
  • linetype
  • size
  • vjust

Control the appearance of the labels with the following arguments. You can also use each of these arguments as an aesthetic. To do so, set them inside the aes() call in geom_text()’s mapping argument.

  • angle - angle of text
  • family - font family of text
  • fontface - bold, italic, etc.
  • hjust - horizontal adjustment
  • vjust- vertical adjustment

Scatterplots do not work well with large data sets because individual points will begin to occlude each other. As a result, you cannot tell where the mass of the data lies. Does a black region contain a single layer of points? Or hundreds of points stacked on top of each other?

You can see this type of plotting in the diamonds data set. The data set only contains 53,940 points, but the points overplot each other in a way that we cannot fix with jittering.

ggplot(data = diamonds) +
  geom_point(mapping = aes(x = carat, y = price))

For large data, it is more useful to plot summary information that describes the raw data than it is to plot the raw data itself. Several geoms can help you do this.

The simplest way to summarise covariance between two variables is with a model line. The model line displays the trend of the relationship between the variables.

Use geom_smooth() to display a model line between any two variables. As with geom_rug(), geom_smooth() works well as a second layer for a plot (See Section 3 for details).

ggplot(data = diamonds) +
  geom_point(mapping = aes(x = carat, y = price)) +
  geom_smooth(mapping = aes(x = carat, y = price))

geom_smooth() will add a loess line to your data if the data contains less than 1000 points, otherwise it will fit a general additive model to your data with a cubic regression spline, and plot the resulting model line. In either case, geom_smooth() will display a message in the console to tell you what it is doing. This is not a warning message; you do not need to worry when you see it.

geom_smooth() will also plot a standard error band around the model line. You can remove the standard error band by setting the se argument of geom_smooth() to FALSE.

Use the method argument of geom_smooth() to add a specific type of model line to your data. method takes the name of an R modeling function. geom_smooth() will use the function to calculate the model line. For example, the code below uses R’s lm() function to fit a linear model line to the data.

ggplot(data = diamonds) +
  geom_point(mapping = aes(x = carat, y = price)) +
  geom_smooth(mapping = aes(x = carat, y = price), method = lm)

By default, geom_smooth() will use the formula y ~ x to model your data. You can modify this formula by setting the formula argument to a different formula. If you do this, be sure to refer to the variable on your \(x\) axis as x and the variable on your \(y\) axis as y, e.g.

ggplot(data = diamonds) +
  geom_point(mapping = aes(x = carat, y = price)) +
  geom_smooth(mapping = aes(x = carat, y = price), 
    method = lm, formula = y ~ poly(x, 4))

Useful aesthetics for geom_smooth() are:

  • x (required)
  • y (required)
  • alpha
  • color
  • fill
  • linetype
  • size
  • weight

Useful arguments for geom_smooth() are:

  • formula - the formula to use in the smoothing function
  • fullrange - Should the fit span the full range of the plot, or just the data?
  • level - Confidence level to use for standard error ribbon
  • method - Smoothing function to use, a model function in R
  • n - The number of points to evaluate smoother at (defaults to 80)
  • se - If TRUE (the default), geom_smooth() will include a standard error ribbon

Be careful, geom_smooth() will overlay a trend line on every data set, even if the underlying data is uncorrelated. You can avoid being fooled by also inspecting the raw data or calculating the correlation between your variables, e.g. cor(diamonds$carat, diamonds$price).

geom_quantile() fits a different type of model to your data. Use it to display the results of a quantile regression (see ?rq for details). Like geom_smooth(), geom_quantile() takes a formula argument that describes the relationship between \(x\) and \(y\).

ggplot(data = diamonds) +
  geom_point(mapping = aes(x = carat, y = price)) +
  geom_quantile(mapping = aes(x = carat, y = price),
                quantiles = c(0.05, 0.5, 0.95), 
                formula = y ~ poly(x, 2))
#> Warning: Computation failed in `stat_quantile()`:
#> Package `quantreg` required for `stat_quantile`.
#> Please install and try again.

Useful aesthetics for geom_quantile() are:

  • x (required)
  • y (required)
  • alpha
  • color
  • linetype
  • size
  • weight

Useful arguments for geom_quantile() are:

  • formula - the formula to use in the smoothing function
  • quantiles - Conditional quantiles of \(y\) to display. Each quantile is displayed with a line.

geom_smooth() and geom_quantile() summarise the relationship between two variables as a function, but you can also summarise the relationship as a bivariate distribution.

geom_bin2d() divides the coordinate plane into a two dimensional grid and then displays the number of observations that fall into each bin in the grid. This technique let’s you see where the mass of the data lies; bins with a light fill color contain more data than bins with a dark fill color. Bins with no fill contain no data at all.

ggplot(data = diamonds) +
  geom_bin2d(mapping = aes(x = carat, y = price), binwidth = c(0.1, 500))

Useful aesthetics for geom_bin2d() are:

  • x (required)
  • y (required)
  • alpha
  • color
  • fill
  • size
  • weight

Useful arguments for geom_bin2d() are:

  • bins - A vector like c(30, 40) that gives the number of bins to use in the horizontal and vertical directions.
  • binwidth - A vector like c(0.1, 500) that gives the binwidths to use in the horizontal and vertical directions. Overrides bins when set.
  • drop - If TRUE (default) geom_bin2d() removes the fill from all bins that contain zero observations.

geom_hex() works similarly to geom_bin2d(), but it divides the coordinate plane into hexagon shaped bins. This can reduce visual artifacts that are introduced by the aligning edges of rectangular bins.

ggplot(data = diamonds) +
  geom_hex(mapping = aes(x = carat, y = price), binwidth = c(0.1, 500))
#> Loading required package: methods

geom_hex() requires the hexbin package, which you can install with install.packages("hexbin").

geom_density2d() uses density contours to display similar information. It is the two dimensional equivalent of geom_density(). Interpret a two dimensional density plot the same way you would interpret a contour map. Each line connects points of equal density, which makes changes of slope easy to see.

As with other summary geoms, geom_density2d() makes a useful second layer.

ggplot(data = diamonds) +
  geom_point(mapping = aes(x = carat, y = price)) +
  geom_density2d(mapping = aes(x = carat, y = price))

Useful aesthetics for geom_density2d() are:

  • x (required)
  • y (required)
  • alpha
  • color
  • linetype
  • size

Useful arguments for geom_density2d() are:

  • h - A vector like c(0.2, 500) that gives the bandwiths to use to estimate the density in the horizontal and vertical directions.
  • n - number of gridpoints to use when estimating the density (defaults to 100).
5.0.3.1.2 Visualize correlations between three variables

There are two ways to add three (or more) variables to a two dimensional plot. You can map additional variables to aesthetics within the plot, or you can use a geom that is designed to visualize three variables.

ggplot2 provides three geoms that are designed to display three variables: geom_raster(), geom_tile() and geom_contour(). These geoms generalize geom_bin2d() and geom_density() to display a third variable instead of a count, or a density.

geom_raster() and geom_tile()