---
title: "Week 2 Tutorial"
subtitle: "Finding Data"
format:
  html:
    toc: true
    toc-location: left
    toc-depth: 3
    highlight-style: github
    code-copy: true
    embed-resources: true
    df-print: paged
    include-in-header:
      text: |
        <style>
          .cell-output pre { max-height: 250px; overflow-y: auto; }
        </style>
editor: visual
---

First things first: Make a new project in RStudio, *and be sure to place it somewhere where you can easily find it* (e.g., Documents, Desktop). Then download this file (the `.qmd` file) from the syllabus website and place it in the folder associated with that project. Then open this file in RStudio and follow along.

I encourage you to work with/talk to others throughout this tutorial, especially during times where I have you code on your own and use AI. When you code, think about the SPARCK framework:

**S: Set up** (What is your end goal?)

**P: Pseudocode** (What are the steps?)

**A: AI** (What are your AI prompts?)

**R: Run** (Run it!)

**C: Check** (Did your code work?)

**K: Knowledge** (What does your code mean?)

There will also be several instances where I will have you go through the full SPARCK framework explicitly.

## Agenda

- Types of data in R
- Simple data operations
- Importing data
- Exploring data
- Data sources

# Types of data in R

There are different "types" of data that R knows how to handle. Each is encoded in a slightly different way, and can have different things done to it.

## Numeric data

Numeric data encompasses any (real) number. Positive, negative, integer, decimal, all fine. You can do your usual math on numeric data.

The basic arithmetic operators are specified as:

- `+` addition
- `-` subtraction
- `*` multiplication
- `/` division
- `^` exponent
- `()` parentheses

Create two numeric objects (variables) and multiply them together.

```{r}
first.var <- 280
second.var <- 41.5

first.var * second.var
```

## Character data

This is letter/word-based data. A piece of character data is called a **string.**

R understands that this data is made up of text, and treats it in a special way. In order to indicate to R that your data is a string, you need to enclose it in quotation marks. You can do either single or double quotes, as long as it's the same on either side.

Create a character object of your favorite fruit and display it. Then make all of the letters uppercase and display that. Then display the number of characters in your object.

```{r}
favefruit <- "passionfruit"
favefruit
toupper(favefruit)
nchar(favefruit)
```

What functions did you use? What did they do?

**Answer:** I used `toupper()` which made all of the letters uppercase, and I used `nchar()` which told me how many letters were in my fruit string.

# Simple data operations

## Relational operators

Most of the time, when you want to do things in R, you want to do them in *some conditions but not others.*

**Relational operators** are the first key to making this happen. These are essentially inequality operators like the ones you would encounter in algebra.

The following is a **relational statement.** This is a command involving a relational operator that returns `TRUE` or `FALSE` based on whether the statement is true or false.

```{r}
2 > 1
```

Here are the relational operators:

- `>` (greater than)
- `<` (less than)
- `>=` (greater than or equal to)
- `<=` (less than or equal to)
- `==` (is equal to--NOTE that it is *two* equals signs, *one* equals sign does something different)
- `!=` (is not equal to)
- `%in%` (is contained by; this is useful when you need to see whether the element on the left matches any of a vector of elements on the right)

Create two statements using `==` and two using `!=`. For each one, one output should be `TRUE` and one should be `FALSE`.

```{r}
5 == 5
5 != 5
7 == 10
7 != 10
```

The `%in%` statement is a little less intuitive than the others. Create a character vector of your four favorite fruits and write an `%in%` statement that outputs `FALSE`. Let's go through the full **SPARCK** framework here.

**S:** Output `FALSE`

**P:** Create a character vector and save it to an object. Use `%in%` with something that isn't in that vector

**A:** I need you to create a vector in R with these items: passionfruit, mango, apple, plum. Then save it to an object. Then use %in% to see if "peach" is present in the vector.

**R:**

```{r}
fruits <- c('passionfruit','mango','apple','plum')
fruits
'peach' %in% fruits
```

**C:** Peach outputs to `FALSE` so I know it's not in the vector. And when I type `fruits` it shows my vector correctly.

**K:** First I used `c()` to make a vector of fruits. Then I used `%in%` to test if `"peach"` was in the vector.

## Logical statements

Relational statements will get you a long way. Sometimes, though, you need to know whether groups of conditions are true or false. You can combine the `TRUE`s and `FALSE`s of relational statements using Boolean logical operators.

These are the core Boolean operators you'll need in most cases:

- `!` (NOT operator; this returns the opposite of whatever follows it)
- `&` (AND operator; this returns `TRUE` if the statements on both sides are true)
- `|` (Pipe, or shift-backslash--OR operator; this returns `TRUE` if *at least one* of the statements on either side is true)

For example:

```{r}
1 == 1 & 1 == 2
```

Create two statements using `&` and two using `|`. For each one, one output should be `TRUE` and one should be `FALSE`.

```{r}
1 == 1 & 2 == 2
1 == 1 & 4 == 5
5 == 5 | 3 != 3
7 != 7 | 4 == 6
```

# Importing data

In the last tutorial, we used a dataset that is built into R. More often, we're going to be reading in **external** datasets. Datasets can be retrieved from many different places:

- Most governments (cities, states, counties, etc.) have a central repository of data related to government activities

- Many non-governmental organizations collect and distribute data related to their work

- Scientific research studies increasingly make the data associated with their study publicly accessible.

## The working directory

Before we can actually get started looking at external data in R, we need to understand how R interacts with files on your computer. Any data that gets loaded in or saved out is saved on your computer proper, and you can access all of these files through R.

Every R session has a **working directory**, or a "home base" folder. Essentially, this is the folder that R is "in". R is not actually installed in this folder, mind you! The working directory is the first folder where R looks for raw data files to load in.

Type the function `getwd()` in the console to figure out what our current working directory is.

```{r, eval=FALSE}
getwd()
```

If you run this command on your own computer, you'll get a different folder. And that's okay! Our folder structures on our computers are all different. What you're looking at is a **file path**.

## File paths

Every single file and folder on your computer has an address. You can navigate these paths in your computer's built-in file browser, but in order for R to be able to access these files, you have to tell R the address.

You can think of your computer's main drive as a building, with only one front door, and a series of rooms (folders), which may contain stuff (files) or doors leading to more rooms (sub-folders). You can walk through any series of connected doors to get to the room you're looking for. A file path is a set of directions of which rooms to walk through to get to a particular room (folder) or object (file) in a room.

There are two ways of specifying file paths: **absolute** and **relative**.

### Absolute file paths

An **absolute file path** is a file's full address on your computer. An absolute path tells you how to get to a file starting from the **root folder,** which is essentially the one invisible giant folder holding literally everything saved on your computer. In your computer-building, the root folder is the front door.

If you give the directions to a room in your computer-building by *starting at the front entrance,* your computer will always be able to follow those directions correctly by "teleporting" to the front entrance and then following the directions to get to the room of interest.

The way you specify that a file path starts from the root folder (the front entrance to your computer) differs depending on the operating system.

### Relative file paths

Meanwhile, **relative file paths** don't involve any sort of teleportation. Relative file paths are directions that assume you're starting in the current working directory (remember from earlier?).

Relative file paths are shorter to type (because you don't need to specify all the directions to get from the root folder to the current working directory), but they do depend on what the current working directory is. If your working directory is set to a folder other than what you expect, your computer will not be able to follow the relative path you provide because it will be looking for file/folder names that may not exist in its starting location.

To specify a relative path, you only need to start your path with a file or folder name that exists in your current working directory.

## RStudio Projects and file paths

One of the benefits of RStudio Projects is that you can set your working directory ahead of time and put all the files you need (R script, Quarto file, data) in there. This means that you can put external data files in this same folder, and to load them into your R session, you don't need to worry about "where" on your computer they are located; calling the relative file path will do the trick.

## Reading in data

Okay, we're finally ready to actually read in some data. For today's tutorial, we're going to use a data file from NYC Open Data on the homeless shelter population. Go to [NYC Open Data](https://www.nyc.gov/opendata), search "DHS Daily Report," go to the dataset, and click Export. Then put the downloaded file in the same folder that you put this `.qmd` file.

This is a dataset that reports the number of people in NYC homeless shelters every day from 2021 until the present. The data is broken down by different categories - adults, families, children, etc. You can learn more about the dataset on the [dataset page](https://data.cityofnewyork.us/Social-Services/DHS-Daily-Report/k46n-sa2m/about_data).

Use the `read_csv()` function to save this data to an object in R. Make sure you're using `read_csv()`, which is from the `tidyverse`. Don't use `read.csv()`.

The downloaded file's name includes the date you downloaded it, so yours will probably be different from what's in this answer key.

```{r}
library(tidyverse)
dhs <- read_csv('DHS_Daily_Report_20260922.csv')
```

How do you know that you did this properly?

**Answer:** I can see `dhs` in my environment

You can give the object any name you like. I'm going to call it `dhs`, for "Department of Homeless Services," so that's how I'll be referring to it moving forward.

You'll now see an object called `dhs` (or whatever you called yours) in your environment (top right window). This is a dataframe, and it contains all of your data. You can click on it to open it up as a tab in the editor window.

# Exploring data

## Dataframes

This is the data type where it happens! Imagine your usual rectangular spreadsheet, where each column of the spreadsheet contains a meaningful category of information (e.g. subject ID, age, gender, race, task performance, etc.), and each row contains one observation of information (one subject, or one observation). A dataframe is basically that.

A **dataframe** is a special rectangular data structure in R. Each column contains all the observations of a particular meaningful grouping. Each row contains one complete observation from all the columns. Each observation can be a subject, a trial, a group, whatever--any one meaningful something.

Notice that dataframes can be read in from CSV files. These files can be opened in other programs like Microsoft Excel, where you can see that they look like any garden-variety spreadsheet.

## Basic data explorations

Before diving into the dataset, look at it in the Editor window and make sure you understand what data it contains.

### Inspecting

There are a number of different functions you might want to use right off the bat to get to know your data. These include:

- `names()`: Display all of the column names.

- `head()`: Display the first few rows of your data.

- `summary()`: Display some basic summary stats of the spread of your numeric data.

Play around with these functions below.

```{r}
names(dhs)
head(dhs)
summary(dhs)
```

### Indexing

Sometimes you might want to access one specific column of a data frame, rather than the whole thing.

We use a special syntax to do this: the dollar sign `$`.

`$` tells R when a variable name, in this case a column name, is a sub-variable of a larger variable, in this case a data frame.

So let's say you wanted to display just the total number of children in shelters. To do so, you would type the following:

```{r}

dhs$`Total Children in Shelter`

```

In this way, we can tell R that `` `Total Children in Shelter` `` is not floating around on its own, but is a column in the data frame `dhs`. If we were to type `` `Total Children in Shelter` `` on its own, we'd get an error:

```{r}
#| error: true
`Total Children in Shelter`
```

You may also notice when typing in RStudio, that if you type `dhs$`, you'll suddenly see a little tab-complete selector pop up, listing out the names of all the columns in `dhs`.

You can either type more to narrow down to the column you want, and then hit Tab to auto-complete the column name, or use the arrow keys to scroll up and down through the column names to select the one you want (and hit Tab to auto-complete in this case as well).

### Renaming columns

One quirk of this dataset: the column names have spaces in them, which is usually something we want to avoid. In R, we can indicate that the space is part of the column name by wrapping it in the `` ` `` symbol.

But that's kind of annoying. You can rename individual columns using the `rename()` function. There's also a package called `janitor` with a nifty function called `clean_names()`, which automatically removes spaces and special characters from column names. It's really useful when downloading external data, since external data can be so messy.

Use `clean_names()` to clean up the column names in `dhs`. (Note: You don't *need* to upload your data to Gemini to get it to help you, but you're totally allowed to! And as things become more complicated, it will likely be very helpful for Gemini to know exactly what your data looks like. It cuts down on explanations you need to provide as well.)

**SPARCK**

**S:** Make it so my column names don't have spaces

**P:** Install `janitor`. Load it. Use the `clean_names()` function on `dhs`.

**A:** Use clean_names from the janitor package to clean up my column names. My dataframe is called dhs.

**R:**

```{r}
#| eval: false
install.packages('janitor')
```

```{r}
library(janitor)
dhs <- clean_names(dhs)
```

**C:** When I open up the dhs dataframe the column names look different!

**K:** The `clean_names()` function cleaned all of my column names

### Dataframe column calculations

Next week we're going to get into data cleaning/wrangling, and we'll learn all sorts of functions from `tidyverse` that help us clean up our data. For today, we'll keep it simple and focus on basic mathematical calculations you can do on your data frame columns, just like you can do on any vector.

Calculate the daily mean number of total individuals in shelters across this entire dataset.

```{r}
mean(dhs$total_individuals_in_shelter)
```

Now calculate the minimum and maximum daily number of single women in shelters.

```{r}
max(dhs$single_adult_women_in_shelter)
min(dhs$single_adult_women_in_shelter)
```

Divide the families with children variable by 1,000.

```{r}
dhs$families_with_children_in_shelter/1000
```

Just like our small vector last week, R "vectorizes" mathematical calculations, meaning it applies them to every value in your vector.

### Date-time data

One last thing!

Note that you can only do mathematical calculations on numeric data. You can check if a column is numeric with the `class()` function. Check the class of the date column.

```{r}
class(dhs$date_of_census)
```

You'll see that it's a character type. R actually has a special data-type for dates and times. This can prove very useful if you ever want to sort your data by time, extract the year or month from a full date, etc.

Use the `mdy()` function (from the `tidyverse`) on the date column to display it in date-time format.

```{r}
mdy(dhs$date_of_census)
```

One final challenge that will set the scene for next week: Extract each row's year from the date column. (Remember that you can only extract date information from data that is in date-time format.) This will require multiple lines of code and likely some functions you haven't used before.

**S:** Extract years from data

**P:** Convert column to date-time format. Extract year.

**A:** Help me extract the year from the date of census column (upload data)

**R:**

```{r}
dhs$date_of_census_dt <- mdy(dhs$date_of_census)
dhs$year <- year(dhs$date_of_census_dt)
```

**C:** The year column is in my data and matches what's in the date column.

**K:** The mdy function converted my character data to date-time data. The year function extracted the year.

# Data sources

There are many other sources of data besides NYC Open Data. Let's move over to some PowerPoint slides to check them out.