---
title: "Week 3 Tutorial"
subtitle: "Cleaning Data — Answer Key, Part 1"
format:
  html:
    toc: true
    toc-location: left
    toc-depth: 3
    highlight-style: github
    code-copy: true
    embed-resources: true
    df-print: paged
    include-in-header:
      text: |
        <style>
          .cell-output pre { max-height: 250px; overflow-y: auto; }
        </style>
editor: visual
---

First things first: Make a new project in RStudio, *and be sure to place it somewhere where you can easily find it* (e.g., Documents, Desktop). Then download this file (the `.qmd` file) from the syllabus website and place it in the folder associated with that project. Then open this file in RStudio and follow along.

I encourage you to work with/talk to others throughout this tutorial, especially during times where I have you code on your own and use AI. When you code, think about the SPARCK framework:

**S: Set up** (What is your end goal?)

**P: Pseudocode** (What are the steps?)

**A: AI** (What are your AI prompts?)

**R: Run** (Run it!)

**C: Check** (Did your code work?)

**K: Knowledge** (What does your code mean?)

## Agenda

- Tidyverse syntax
- Data Moves
- Quarto

For today's tutorial, we're going to be working with [this dataset](https://data.ny.gov/Education/Headcount-Enrollment-by-Student-Level-and-Student-/4fyc-bf8i/about_data) on enrollment in the State University of New York (SUNY) system. This data comes from New York State's open data portal (<https://data.ny.gov/>). It contains information on enrollment at different types of SUNY colleges every fall since 2011.

Place the data in the folder you created for this R Project. Load the data into R. (Note: The date at the end of the file name is the date the file was downloaded, so your file name may be slightly different from mine.)

```{r}
library(tidyverse)
suny <- read_csv('Headcount_Enrollment_by_Student_Level_and_Student_Load_by_Institutions_of_the_State_University_of_New_York__Beginning_Fall_2011_20260928.csv')
```

You can call your data whatever you'd like, but I will call mine `suny`.

One other thing to note: When using Gemini, you can **upload your data file** so that Gemini knows exactly what your data looks like. If you don't upload it, you'll have to explain a bit what your data looks like so it knows exactly what you're trying to do and how to help you. So it's not a requirement, but it's useful when trying to get help on how to work with your data specifically.

# Tidyverse syntax

We've already learned a bit about the "syntax" of R - functions, objects, arguments, etc. One important aspect of the `tidyverse` package is that it comes with its own syntax that differs slightly from base R. This syntax is a little bit more "logical" and easier to read (i.e. "tidier"), which is part of why it's so popular.

In this class, you'll primarily be using `tidyverse` functions to clean and analyze data.

## The pipe operator: `|>`

Before we start moving data around, we need one of the most important pieces of `tidyverse` syntax: the **pipe operator**, `|>`. The pipe lets you chain several operations together, one after another, so that the output of one step becomes the input of the next. (This section is adapted from Ismay et al., 2025.)

Imagine you have an object `x` and three functions `f()`, `g()`, and `h()`. You want to:

1.  Take `x`, *then*
2.  Use `x` as the input to `f()`, *then*
3.  Use the output of `f(x)` as the input to `g()`, *then*
4.  Use the output of `g(f(x))` as the input to `h()`.

Without the pipe, you have to **nest** the functions inside each other:

```{r}
#| eval: false
h(g(f(x)))
```

This works, but it's hard to read: you have to start from the *inside* and work your way out, which is the opposite of the order we'd describe the steps in.

With the pipe, you can write the same thing like this:

```{r}
#| eval: false
x |>
  f() |>
  g() |>
  h()
```

Now the code reads top-to-bottom, left-to-right, in the same order as the steps. A good way to read `|>` out loud is **"and then."** "Take `x`, *and then* do `f()`, *and then* do `g()`, *and then* do `h()`."

A few things to notice:

- The pipe takes whatever is on its left and passes it in as the **first argument** of the function on its right. `x |> f()` is the same as `f(x)`.
- Put each step on its own line, with the `|>` at the *end* of the line. R knows the command isn't finished yet because the line ends in a pipe.
- The keyboard shortcut for the pipe is **Cmd + Shift + M** (Mac) or **Ctrl + Shift + M** (Windows).
- You will often see an older version of the pipe, `%>%`, especially in code written by AI or found online. For our purposes it does the same thing.

In our case, `x` will almost always be a **data frame**, and `f()`, `g()`, and `h()` will be `tidyverse` functions that each do one thing to that data frame. For example, use the `clean_names()` function from the `janitor` package on `suny`. (When prompting Gemini, it may be useful to explicitly say "using the pipe" or "using tidyverse".)

```{r}
library(janitor)
suny <- suny |>
  clean_names()
```

Notice that we saved the result. If you don't assign the result with `<-`, R will print it but *won't save it*.

## The `$` (or lack thereof)

Last class we learned about the `$` sign, which we use to index columns in R. But in the `tidyverse`, we don't really need it. Because we're "piping in" our dataframe, the `tidyverse` functions know what dataframe our columns come from, and so we only need to call the column name.

For example, let's use the `pull()` function to display the college names. (Don't forget that you can start typing the column name and use tab to auto-complete! That will save you a lot of time.)

```{r}
suny |> pull(college_or_institution_name)
```

This is the same as typing `suny$college_or_institution_name`, but many people find it tidier and easier to read when using the pipe.

# Data Moves

In this class, we're using the Data Moves framework (Erickson et al., 2019) to understand data wrangling. A "data move" is "an action that alters a dataset's contents, structure, or values." When we download data, it often isn't in the structure that we need to conduct analyses. Data moves allow us to get it into that structure, and to derive basic insights that will provide context for our analyses.

## Filtering

One of the first things we often want to do with our data is filter it, because our data may contain way more data points than we're actually interested in. To filter our data, we use the `filter()` function.

There are certain rows of `suny` that are "summary" rows (totals for each type of college, and for SUNY as a whole). Those rows have no value for the college name column. Get rid of those rows and save the output as a new data frame. (Use Gemini and the SPARCK framework to help you complete this task and the following.)

```{r}
suny_clean <- suny |>
  filter(!is.na(college_or_institution_name))
```

Why is it important to get rid of these rows before we do anything else?

**Answer:** The summary rows are totals of the other rows. If we kept them, any sums or averages we calculate would count the same students more than once.

Create a new data frame that only contains data from after 2020 (i.e., 2021 onward).

```{r}
suny_post2020 <- suny_clean |>
  filter(year > 2020)
```

Create a new data frame that only shows comprehensive colleges with more than 4,000 full time undergraduate students.

```{r}
comp_big <- suny_clean |>
  filter(college_or_institution_type == "Comprehensive Colleges",
         undergraduate_full_time > 4000)
```

## Summarizing

Data frames contain many rows that represent individual data points, but often, we want to reduce our data down into a summary statistic. We can do this with the `summarize()` function. Summaries are useful not just to get a sense of our data, but because summaries can often be used for further analysis.

Get the minimum and maximum values for full time undergraduates and full time graduate students across the entire dataset. Make sure you're using your data frame where you excluded the summary rows with blank values for college name. (Hint: Some colleges have no graduate students, so those values are missing, or `NA`. In order to get a real number returned, we need to tell R to ignore `NA` values.)

```{r}
suny_clean |>
  summarize(min_ug_ft = min(undergraduate_full_time, na.rm = TRUE),
            max_ug_ft = max(undergraduate_full_time, na.rm = TRUE),
            min_grad_ft = min(graduate_full_time, na.rm = TRUE),
            max_grad_ft = max(graduate_full_time, na.rm = TRUE))
```

Get the mean number of full time undergraduate students for the year 2025 only.

```{r}
suny_clean |>
  filter(year == 2025) |>
  summarize(mean_ug_ft = mean(undergraduate_full_time, na.rm = TRUE))
```

## Grouping

Sometimes we want to calculate summaries for specific subgroups all at the same time. To do this, we can use the `group_by()` function in combination with `summarize()`.

Calculate the average number of full time graduate students for each year.

```{r}
suny_clean |>
  group_by(year) |>
  summarize(mean_grad_ft = mean(graduate_full_time, na.rm = TRUE))
```

Calculate the median number of part time undergraduates for each type of college, in 2017.

```{r}
suny_clean |>
  filter(year == 2017) |>
  group_by(college_or_institution_type) |>
  summarize(median_ug_pt = median(undergraduate_part_time, na.rm = TRUE))
```

## Calculating

The `mutate()` function should be your best friend. It allows you to create new data frame columns based on calculations done on existing columns. These calculations might combine several columns, or describe some attribute of a single column.

Create a new column that represents the total number of full time students (undergraduate + graduate) for each row. (Hint: What happens when you add a number to `NA`?)

```{r}
suny_clean <- suny_clean |>
  mutate(full_time_total = replace_na(undergraduate_full_time, 0) +
           replace_na(graduate_full_time, 0))
```

Let's say we consider a school to be "big" if it has over 4,000 full time undergraduate students. Create a new variable that indicates, for each row, whether or not a school is "big."

```{r}
suny_clean <- suny_clean |>
  mutate(big = if_else(undergraduate_full_time > 4000, "big", "small"))
```

## Merging/joining

*The answers for the rest of this tutorial (Merging/joining, Pivoting, and Practice) will be posted after we finish them in lab.*

Sometimes, we want to merge two datasets together based on a common characteristic. Often, the data that we want to use to do analyses is spread across many different sources.

Download the dataset [on this page](https://data.ny.gov/Education/State-University-of-New-York-SUNY-Campus-Locations/3cij-nwhw/about_data), which has information on the location of each SUNY campus. Place it in your project folder and load it into R. Clean the column names.

```{r}
# Write your code here

```

What is the name of the column from each data frame that contains information that will allow us to join them?

**Answer:**

Now go ahead and merge them together using one of the `_join()` functions (e.g., `left_join()`).

```{r}
# Write your code here

```

## Pivoting

While not one of the core Data Moves, knowing how to "pivot" your data is very important because certain visualization and analysis tools expect our data to be in a particular structure. There are two functions in the `tidyverse` that help us pivot: `pivot_longer()` and `pivot_wider()`. These functions allow us to switch our data from "long" format to "wide" format and vice versa.

First, calculate the total number of full time undergraduates at each type of institution, in each year. Then use `pivot_wider()` so that each type of institution has its own column.

```{r}
# Write your code here

```

Make sure you compare the two data frames so you can understand what has changed!

Use `pivot_longer()` on `suny_clean` so that there is a "student type" column, and all of the enrollment numbers for the different student types are combined into one column.

```{r}
# Write your code here

```

## Practice

Let's imagine you have a research question: How has enrollment for each kind of student changed at doctoral degree granting institutions before vs. after Covid? How would you investigate this? Let's use **SPARCK** to figure it out. (For this question, count 2011–2019 as "pre-Covid" and 2020–2025 as "post-Covid.")

**S:**

**P:**

**A:**

**R:**

```{r}
# Write your code here

```

**C:**

**K:**

# Quarto

As a reminder, this file, which has a `.qmd` extension, is called a Quarto file. Quarto files are special files that we can use in RStudio because they allow us to combine plain text - like this - and code chunks, which are in the gray boxes. Quarto documents also have a header at the top that specifies the document's settings.

## Quarto and homework

For your homework, you're going to need to submit an `.html` file, which is a **rendered** version of your Quarto file. I have provided you with a Quarto file for your homework. You will need to fill it in, render it to html, and submit that html file.

**You must submit a fully rendered html file in order to receive full credit on the homework.** To get your file to render, you simply click the Render button above once you've completed the homework. If you are having trouble getting your file to render, you may ask me for help, or ask AI to help you troubleshoot your errors. Some of the most common reasons rendering fails:

- An object was created in the console but not in a code chunk.

- A package was loaded in the console but never in the document.

- The code uses an object before the code chunk that creates it.

- Install packages in the console only. `install.packages()` in a Quarto document can cause it to fail to render.

Think of your homework as a cohesive document that guides us through your data analysis. A cohesive document will be easy to read and will contain a clear narrative.

## The header

You'll want to make basic edits to the Quarto file's header so that it includes your name. You don't need to worry about other aspects of the header for now.

## The plain text

In your homework, you will need to explain what you are doing in plain text. Quarto files use a special type of syntax called Markdown to control what text looks like in the file. In RStudio, most of the tools you need to format text can be accessed in the tool bar at the top of the Editor pane.

But you don't need to click these tools to format with Markdown. Some examples of Markdown formatting:

- The \# sign in front of text (with a space between the sign and the text) turns the text into a heading. More hashtag signs create smaller headings.

- You can bold text by wrapping it in two asterisks (`**`) and italicize it by wrapping it in one asterisk (`*`).

## Code chunks

This is where all of your code goes. At baseline, your code chunk runs the code inside of it and displays the output. But you can tinker with some of the settings of the code chunk by putting a chunk option inside of it.

- `#| eval: false` means the code in that chunk will not be run when rendering

- `#| echo: false` displays the code output but not the code itself

- `#| error: true` will allow the document to render even if that code chunk contains an error

- `#| results: hide` prevents the output from being shown, but still runs the code

- `#| message: false` prevents messages from showing in the rendered file

# References

Erickson, T., Wilkerson, M., Finzer, W., & Reichsman, F. (2019). Data moves. *Technology Innovations in Statistics Education, 12*(1). <https://doi.org/10.5070/T5121038001>

Ismay, C., Kim, A. Y., & Valdivia, A. (2025). The pipe operator: `|>`. In *Statistical inference via data science: A ModernDive into R and the tidyverse* (2nd ed., Section 3.1). <https://moderndive.com/v2/03-wrangling.html>