Expressions

Learning objectives:

• Understand the idea of the abstract syntax tree (AST).
• Discuss the data structures that underlie the AST:
  • Constants
  • Symbols
  • Calls
• Explore the idea behind parsing.
• Explore some details of R’s grammar.
• Discuss the use or recursive functions to compute on the language.
• Work with three other more specialized data structures:
  • Pairlists
  • Missing arguments
  • Expression vectors

library(rlang)
library(lobstr)

Introduction

To compute on the language, we first need to understand its structure.

• This requires a few things:
  • New vocabulary.
  • New tools to inspect and modify expressions.
  • Approach the use of the language with new ways of thinking.
• One of the first new ways of thinking is the distinction between an operation and its result.

y <- x * 10

#> Error: object 'x' not found

• We can capture the intent of the code without executing it using the rlang package.

z <- rlang::expr(y <- x * 10)

z

#> y <- x * 10

• We can then evaluate the expression using the base::eval function.

x <- 4

base::eval(expr(y <- x * 10))

y

#> [1] 40

Evaluating multiple expressions

• The function expression() allows for multiple expressions, and in some ways it acts similarly to the way files are source()d in. That is, we eval()uate all of the expressions at once.

• expression() returns a vector and can be passed to eval().

z <- expression(x <- 4, x * 10)

eval(z)

#> [1] 40

is.atomic(z)

#> [1] FALSE

is.vector(z)

#> [1] TRUE

• exprs() does not evaluate everything at once. To evaluate each expression, the individual expressions must be evaluated in a loop.

for (i in exprs(x <- 4, x * 10)) {
print(i)
print(eval(i))
}

#> x <- 4
#> [1] 4
#> x * 10
#> [1] 40

Abstract Syntax Tree (AST)

• Expressions are objects that capture the structure of code without evaluating it.
• Expressions are also called abstract syntax trees (ASTs) because the structure of code is hierarchical and can be naturally represented as a tree.
• Understanding this tree structure is crucial for inspecting and modifying expressions.
  • Branches = Calls
  • Leaves = Symbols and constants

f(x, "y", 1)

With lobstr::ast():

lobstr::ast(f(x, "y", 1))

#> █─f 
#> ├─x 
#> ├─"y" 
#> └─1

• Some functions might also contain more calls like the example below:

f(g(1, 2), h(3, 4, i())):

lobstr::ast(f(g(1, 2), h(3, 4, i())))

#> █─f 
#> ├─█─g 
#> │ ├─1 
#> │ └─2 
#> └─█─h 
#>   ├─3 
#>   ├─4 
#>   └─█─i

• Read the hand-drawn diagrams from left-to-right (ignoring vertical position)
• Read the lobstr-drawn diagrams from top-to-bottom (ignoring horizontal position).
• The depth within the tree is determined by the nesting of function calls.
• Depth also determines evaluation order, as evaluation generally proceeds from deepest-to-shallowest, but this is not guaranteed because of lazy evaluation.

Infix calls

Every call in R can be written in tree form because any call can be written in prefix form.

An infix operator is a function where the function name is placed between its arguments. Prefix form is when then function name comes before the arguments, which are enclosed in parentheses. [Note that the name infix comes from the words prefix and suffix.]

y <- x * 10
`<-`(y, `*`(x, 10))

• A characteristic of the language is that infix functions can always be written as prefix functions; therefore, all function calls can be represented using an AST.

lobstr::ast(y <- x * 10)

#> █─`<-` 
#> ├─y 
#> └─█─`*` 
#>   ├─x 
#>   └─10

lobstr::ast(`<-`(y, `*`(x, 10)))

#> █─`<-` 
#> ├─y 
#> └─█─`*` 
#>   ├─x 
#>   └─10

• There is no difference between the ASTs for the infix version vs the prefix version, and if you generate an expression with prefix calls, R will still print it in infix form:

rlang::expr(`<-`(y, `*`(x, 10)))

#> y <- x * 10

Expression

• Collectively, the data structures present in the AST are called expressions.
• These include:
  1. Constants
  2. Symbols
  3. Calls
  4. Pairlists

Constants

• Scalar constants are the simplest component of the AST.
• A constant is either NULL or a length-1 atomic vector (or scalar)
  • e.g., TRUE, 1L, 2.5, "x", or "hello".
• We can test for a constant with rlang::is_syntactic_literal().
• Constants are self-quoting in the sense that the expression used to represent a constant is the same constant:

identical(expr(TRUE), TRUE)

#> [1] TRUE

identical(expr(1), 1)

#> [1] TRUE

identical(expr(2L), 2L)

#> [1] TRUE

identical(expr("x"), "x")

#> [1] TRUE

identical(expr("hello"), "hello")

#> [1] TRUE

Symbols

• A symbol represents the name of an object.
  • x
  • mtcars
  • mean
• In base R, the terms symbol and name are used interchangeably (i.e., is.name() is identical to is.symbol()), but this book used symbol consistently because “name” has many other meanings.
• You can create a symbol in two ways:
  1. by capturing code that references an object with expr().
  2. turning a string into a symbol with rlang::sym().

expr(x)

#> x

sym("x")

#> x

• A symbol can be turned back into a string with as.character() or rlang::as_string().
• as_string() has the advantage of clearly signalling that you’ll get a character vector of length 1.

as_string(expr(x))

#> [1] "x"

• We can recognize a symbol because it is printed without quotes

expr(x)

#> x

• str() tells you that it is a symbol, and is.symbol() is TRUE:

str(expr(x))

#>  symbol x

is.symbol(expr(x))

#> [1] TRUE

• The symbol type is not vectorised, i.e., a symbol is always length 1.
• If you want multiple symbols, you’ll need to put them in a list, using rlang::syms().

Note that as_string() will not work on expressions which are not symbols.

as_string(expr(x+y))

#> Error in `as_string()`:
#> ! Can't convert a call to a string.

Calls

• A call object represents a captured function call.
• Call objects are a special type of list.
  • The first component specifies the function to call (usually a symbol, i.e., the name fo the function).
  • The remaining elements are the arguments for that call.
• Call objects create branches in the AST, because calls can be nested inside other calls.
• You can identify a call object when printed because it looks just like a function call.
• Confusingly typeof() and str() print language for call objects (where we might expect it to return that it is a “call” object), but is.call() returns TRUE:

lobstr::ast(read.table("important.csv", row.names = FALSE))

#> █─read.table 
#> ├─"important.csv" 
#> └─row.names = FALSE

x <- expr(read.table("important.csv", row.names = FALSE))

typeof(x)

#> [1] "language"

is.call(x)

#> [1] TRUE

Subsetting

• Calls generally behave like lists.
• Since they are list-like, you can use standard subsetting tools.
• The first element of the call object is the function to call, which is usually a symbol:

x[[1]]

#> read.table

is.symbol(x[[1]])

#> [1] TRUE

• The remainder of the elements are the arguments:

is.symbol(x[-1])

#> [1] FALSE

as.list(x[-1])

#> [[1]]
#> [1] "important.csv"
#> 
#> $row.names
#> [1] FALSE

• We can extract individual arguments with [[ or, if named, $:

x[[2]]

#> [1] "important.csv"

x$row.names

#> [1] FALSE

• We can determine the number of arguments in a call object by subtracting 1 from its length:

length(x) - 1

#> [1] 2

• Extracting specific arguments from calls is challenging because of R’s flexible rules for argument matching:
  • It could potentially be in any location, with the full name, with an abbreviated name, or with no name.
• To work around this problem, you can use rlang::call_standardise() which standardizes all arguments to use the full name:

rlang::call_standardise(x)

#> Warning: `call_standardise()` is deprecated as of rlang 0.4.11
#> This warning is displayed once every 8 hours.

#> read.table(file = "important.csv", row.names = FALSE)

• But If the function uses … it’s not possible to standardise all arguments.
• Calls can be modified in the same way as lists:

x$header <- TRUE
x

#> read.table("important.csv", row.names = FALSE, header = TRUE)

Function position

• The first element of the call object is the function position. This contains the function that will be called when the object is evaluated, and is usually a symbol.

lobstr::ast(foo())

#> █─foo

• While R allows you to surround the name of the function with quotes, the parser converts it to a symbol:

lobstr::ast("foo"())

#> █─foo

• However, sometimes the function doesn’t exist in the current environment and you need to do some computation to retrieve it:
  • For example, if the function is in another package, is a method of an R6 object, or is created by a function factory. In this case, the function position will be occupied by another call:

lobstr::ast(pkg::foo(1))

#> █─█─`::` 
#> │ ├─pkg 
#> │ └─foo 
#> └─1

lobstr::ast(obj$foo(1))

#> █─█─`$` 
#> │ ├─obj 
#> │ └─foo 
#> └─1

lobstr::ast(foo(1)(2))

#> █─█─foo 
#> │ └─1 
#> └─2

Constructing

• You can construct a call object from its components using rlang::call2().
• The first argument is the name of the function to call (either as a string, a symbol, or another call).
• The remaining arguments will be passed along to the call:

call2("mean", x = expr(x), na.rm = TRUE)

#> mean(x = x, na.rm = TRUE)

call2(expr(base::mean), x = expr(x), na.rm = TRUE)

#> base::mean(x = x, na.rm = TRUE)

• Infix calls created in this way still print as usual.

call2("<-", expr(x), 10)

#> x <- 10

Parsing and grammar

• Parsing - The process by which a computer language takes a string and constructs an expression. Parsing is governed by a set of rules known as a grammar.
• We are going to use lobstr::ast() to explore some of the details of R’s grammar, and then show how you can transform back and forth between expressions and strings.
• Operator precedence - Conventions used by the programming language to resolve ambiguity.
• Infix functions introduce two sources of ambiguity.
• The first source of ambiguity arises from infix functions: what does 1 + 2 * 3 yield? Do you get 9 (i.e., (1 + 2) * 3), or 7 (i.e., 1 + (2 * 3))? In other words, which of the two possible parse trees below does R use?

• Programming languages use conventions called operator precedence to resolve this ambiguity. We can use ast() to see what R does:

lobstr::ast(1 + 2 * 3)

#> █─`+` 
#> ├─1 
#> └─█─`*` 
#>   ├─2 
#>   └─3

• PEMDAS (or BEDMAS or BODMAS, depending on where in the world you grew up) is pretty clear on what to do. Other operator precedence isn’t as clear.
• There’s one particularly surprising case in R:
  • ! has a much lower precedence (i.e., it binds less tightly) than you might expect.
  • This allows you to write useful operations like:

lobstr::ast(!x %in% y)

#> █─`!` 
#> └─█─`%in%` 
#>   ├─x 
#>   └─y

• R has over 30 infix operators divided into 18 precedence groups.
• While the details are described in ?Syntax, very few people have memorized the complete ordering.
• If there’s any confusion, use parentheses!

# override PEMDAS
lobstr::ast((1 + 2) * 3)

#> █─`*` 
#> ├─█─`(` 
#> │ └─█─`+` 
#> │   ├─1 
#> │   └─2 
#> └─3

Associativity

• The second source of ambiguity is introduced by repeated usage of the same infix function.

1 + 2 + 3

#> [1] 6

# What does R do first?
(1 + 2) + 3

#> [1] 6

# or
1 + (2 + 3)

#> [1] 6

• In this case it doesn’t matter. Other places it might, like in ggplot2.

• In R, most operators are left-associative, i.e., the operations on the left are evaluated first:

lobstr::ast(1 + 2 + 3)

#> █─`+` 
#> ├─█─`+` 
#> │ ├─1 
#> │ └─2 
#> └─3

• There are two exceptions to the left-associative rule:
  1. exponentiation
  2. assignment

lobstr::ast(2 ^ 2 ^ 3)

#> █─`^` 
#> ├─2 
#> └─█─`^` 
#>   ├─2 
#>   └─3

lobstr::ast(x <- y <- z)

#> █─`<-` 
#> ├─x 
#> └─█─`<-` 
#>   ├─y 
#>   └─z

Parsing and deparsing

• Parsing - turning characters you’ve typed into an AST (i.e., from strings to expressions).
• R usually takes care of parsing code for us.
• But occasionally you have code stored as a string, and you want to parse it yourself.
• You can do so using rlang::parse_expr():

x1 <- "y <- x + 10"
x1

#> [1] "y <- x + 10"

is.call(x1)

#> [1] FALSE

x2 <- rlang::parse_expr(x1)
x2

#> y <- x + 10

is.call(x2)

#> [1] TRUE

• parse_expr() always returns a single expression.
• If you have multiple expression separated by ; or ,, you’ll need to use rlang::parse_exprs() which is the plural version of rlang::parse_expr(). It returns a list of expressions:

x3 <- "a <- 1; a + 1"

rlang::parse_exprs(x3)

#> [[1]]
#> a <- 1
#> 
#> [[2]]
#> a + 1

• If you find yourself parsing strings into expressions often, quasiquotation may be a safer approach.
  • More about quasiquaotation in Chapter 19.
• The inverse of parsing is deparsing.
• Deparsing - given an expression, you want the string that would generate it.
• Deparsing happens automatically when you print an expression.
• You can get the string with rlang::expr_text():
• Parsing and deparsing are not symmetric.
  • Parsing creates the AST which means that we lose backticks around ordinary names, comments, and whitespace.

cat(expr_text(expr({
  # This is a comment
  x <-             `x` + 1
})))

#> {
#>     x <- x + 1
#> }

Using the AST to solve more complicated problems

• Here we focus on what we learned to perform recursion on the AST.
• Two parts of a recursive function:
  • Recursive case: handles the nodes in the tree. Typically, you’ll do something to each child of a node, usually calling the recursive function again, and then combine the results back together again. For expressions, you’ll need to handle calls and pairlists (function arguments).
  • Base case: handles the leaves of the tree. The base cases ensure that the function eventually terminates, by solving the simplest cases directly. For expressions, you need to handle symbols and constants in the base case.

Two helper functions

• First, we need an epxr_type() function to return the type of expression element as a string.

expr_type <- function(x) {
  if (rlang::is_syntactic_literal(x)) {
    "constant"
  } else if (is.symbol(x)) {
    "symbol"
  } else if (is.call(x)) {
    "call"
  } else if (is.pairlist(x)) {
    "pairlist"
  } else {
    typeof(x)
  }
}

expr_type(expr("a"))

#> [1] "constant"

expr_type(expr(x))

#> [1] "symbol"

expr_type(expr(f(1, 2)))

#> [1] "call"

• Second, we need a wrapper function to handle exceptions.

switch_expr <- function(x, ...) {
  switch(expr_type(x),
    ...,
    stop("Don't know how to handle type ", typeof(x), call. = FALSE)
  )
}

• Lastly, we can write a basic template that walks the AST using the switch() statement.

recurse_call <- function(x) {
  switch_expr(x,
    # Base cases
    symbol = ,
    constant = ,

    # Recursive cases
    call = ,
    pairlist =
  )
}

Specific use cases for recurse_call()

Example 1: Finding F and T

• Using F and T in our code rather than FALSE and TRUE is bad practice.
• Say we want to walk the AST to find times when we use F and T.
• Start off by finding the type of T vs TRUE.

expr_type(expr(TRUE))

#> [1] "constant"

expr_type(expr(T))

#> [1] "symbol"

• With this knowledge, we can now write the base cases of our recursive function.
• The logic is as follows:
  • A constant is never a logical abbreviation and a symbol is an abbreviation if it is “F” or “T”:

logical_abbr_rec <- function(x) {
  switch_expr(x,
    constant = FALSE,
    symbol = as_string(x) %in% c("F", "T")
  )
}

logical_abbr_rec(expr(TRUE))

#> [1] FALSE

logical_abbr_rec(expr(T))

#> [1] TRUE

• It’s best practice to write another wrapper, assuming every input you receive will be an expression.

logical_abbr <- function(x) {
  logical_abbr_rec(enexpr(x))
}

logical_abbr(T)

#> [1] TRUE

logical_abbr(FALSE)

#> [1] FALSE

Next step: code for the recursive cases

• Here we want to do the same thing for calls and for pairlists.
• Here’s the logic: recursively apply the function to each subcomponent, and return TRUE if any subcomponent contains a logical abbreviation.
• This is simplified by using the purrr::some() function, which iterates over a list and returns TRUE if the predicate function is true for any element.

logical_abbr_rec <- function(x) {
  switch_expr(x,
  # Base cases
  constant = FALSE,
  symbol = as_string(x) %in% c("F", "T"),
  # Recursive cases
  call = ,
  # Are we sure this is the correct function to use?
  # Why not logical_abbr_rec?
  pairlist = purrr::some(x, logical_abbr_rec)
  )
}

logical_abbr(mean(x, na.rm = T))

#> [1] TRUE

logical_abbr(function(x, na.rm = T) FALSE)

#> [1] TRUE

Example 2: Finding all variables created by assignment

• Listing all the variables is a little more complicated.
• Figure out what assignment looks like based on the AST.

ast(x <- 10)

#> █─`<-` 
#> ├─x 
#> └─10

• Now we need to decide what data structure we’re going to use for the results.
  • Easiest thing will be to return a character vector.
  • We would need to use a list if we wanted to return symbols.

Dealing with the base cases

find_assign_rec <- function(x) {
  switch_expr(x,
    constant = ,
    symbol = character()
  )
}
find_assign <- function(x) find_assign_rec(enexpr(x))

find_assign("x")

#> character(0)

find_assign(x)

#> character(0)

Dealing with the recursive cases

• Here is the function to flatten pairlists.

flat_map_chr <- function(.x, .f, ...) {
  purrr::flatten_chr(purrr::map(.x, .f, ...))
}

flat_map_chr(letters[1:3], ~ rep(., sample(3, 1)))

#> [1] "a" "a" "b" "b" "c" "c"

• Here is the code needed to identify calls.

find_assign_rec <- function(x) {
  switch_expr(x,
    # Base cases
    constant = ,
    symbol = character(),

    # Recursive cases
    pairlist = flat_map_chr(as.list(x), find_assign_rec),
    call = {
      if (is_call(x, "<-")) {
        as_string(x[[2]])
      } else {
        flat_map_chr(as.list(x), find_assign_rec)
      }
    }
  )
}

find_assign(a <- 1)

#> [1] "a"

find_assign({
  a <- 1
  {
    b <- 2
  }
})

#> [1] "a" "b"

Make the function more robust

• Throw cases at it that we think might break the function.
• Write a function to handle these cases.

find_assign_call <- function(x) {
  if (is_call(x, "<-") && is_symbol(x[[2]])) {
    lhs <- as_string(x[[2]])
    children <- as.list(x)[-1]
  } else {
    lhs <- character()
    children <- as.list(x)
  }

  c(lhs, flat_map_chr(children, find_assign_rec))
}

find_assign_rec <- function(x) {
  switch_expr(x,
    # Base cases
    constant = ,
    symbol = character(),

    # Recursive cases
    pairlist = flat_map_chr(x, find_assign_rec),
    call = find_assign_call(x)
  )
}

find_assign(a <- b <- c <- 1)

#> [1] "a" "b" "c"

find_assign(system.time(x <- print(y <- 5)))

#> [1] "x" "y"

• This approach certainly is more complicated, but it’s important to start simple and move up.

Specialised data structures

• Pairlists
• Missing arguments
• Expression vectors

Pairlists

• Pairlists are a remnant of R’s past and have been replaced by lists almost everywhere.
• The only place you are likely to see pairlists in R is when working with calls to the function, as the formal arguments to a function are stored in a pairlist:

f <- expr(function(x, y = 10) x + y)

args <- f[[2]]
args

#> $x
#> 
#> 
#> $y
#> [1] 10

typeof(args)

#> [1] "pairlist"

• Fortunately, whenever you encounter a pairlist, you can treat it just like a regular list:

pl <- pairlist(x = 1, y = 2)

length(pl)

#> [1] 2

pl$x

#> [1] 1

Missing arguments

• Empty symbols
• To create an empty symbol, you need to use missing_arg() or expr().

missing_arg()

typeof(missing_arg())

#> [1] "symbol"

• Empty symbols don’t print anything.
  • To check, we need to use rlang::is_missing()

is_missing(missing_arg())

#> [1] TRUE

• These are usually present in function formals:

f <- expr(function(x, y = 10) x + y)

args <- f[[2]]


is_missing(args[[1]])

#> [1] TRUE

Expression vectors

• An expression vector is just a list of expressions.
  • The only difference is that calling eval() on an expression evaluates each individual expression.
  • Instead, it might be more advantageous to use a list of expressions.
• Expression vectors are only produced by two base functions: expression() and parse():

exp1 <- parse(text = c(" 
x <- 4
x
"))
exp1

#> expression(x <- 4, x)

exp2 <- expression(x <- 4, x)
exp2

#> expression(x <- 4, x)

typeof(exp1)

#> [1] "expression"

typeof(exp2)

#> [1] "expression"

• Like calls and pairlists, expression vectors behave like lists:

length(exp1)

#> [1] 2

exp1[[1]]

#> x <- 4