--- title: "Lab Assignment 1 EXAMPLE" format: html: embed-resources: true execute: warning: false message: false editor: visual author: Example --- ## Instructions Welcome to your first lab assignment! Are you excited? I sure am. Please read the following instructions before you begin. For your lab assignments, you may use AI to help you code. You will be required to explicitly fill out the **SPARCK** framework for each question. You may not use AI to write the non-coding parts of the framework, although you may consult it for assistance. Generally, your lab assignments will ask you to access a dataset, and then to investigate some questions in regards to the data. These questions will be relatively broad, meaning that you will have some flexibility in *how* exactly you answer them. This is the point, and the benefit of using AI to help you code: We don't need to be worried about walking step by step through an analysis, but can instead think about it from a broader perspective; the specific code you use to tackle the analysis is up to you. The goal is for you to get practice thinking about how to answer a question with data. As you write code, think about what we've learned in class so far, and especially what we learned in the Data Wrangling/Cleaning tutorial. You don't need to exclusively use functions we used in class, but they should serve as a reminder of the types of things you might wish to do. You should guide me through the answer the questions. This means you shouldn't just copy and paste code from Gemini and be done with it; your **SPARCK** responses should guide me through your thought process and analysis. You might also wish to spread your code across several different chunks so you can display output along the way if you think that makes it easier to understand. Remember that you must render your Quarto document into an html file. This html is what you should hand in on Canvas. If you encounter errors, use Gemini to help you troubleshoot! **Make sure you put this Quarto file in your Week 3 folder, and then open the file from that folder so that R's working directory is your Week 3 folder.** **S: Set up** (What is your end goal?) **P: Pseudocode** (What are the steps?) **A: AI** (What are your AI prompts?) **R: Run** (Run it!) **C: Check** (Did your code work?) **K: Knowledge** (What does your code mean?) ## The data For today's homework, we are going to be using a dataset from the Prison Policy Initiative about the costs of making phone calls from inside carceral institutions. You can learn more about this project [here](https://www.prisonpolicy.org/phones/state_of_phone_justice_2022.html). Specifically, we will be using a dataset that tracks phone call rates in over 3,000 jails and other detention facilities across the U.S. in 2021. (Jails are different from prisons. Jails are run by counties and cities and mostly hold people who are awaiting trial and haven't been convicted of a crime, along with people serving short sentences. Prisons are run by states or the federal government and hold people who have been convicted and sentenced, usually to more than a year.) Most facilities in this dataset are jails, but it also includes juvenile detention centers, immigrant detention centers, tribal jails, and a few other types. For this homework, I have provided you with the dataset. You may download it from either our syllabus website or Canvas. You should place it in the same folder as this document. ## Question 1 Pick one state that is meaningful to you, and give me a sense of the phone rate landscape in that state. Some questions you might wish to answer: - What is the average cost of making a phone call? - How much variety is there between facilities? - Is there a large difference in in-state vs out of state rates? Remember, *you* are choosing what point you want to get across, and how exactly you want to do it. **S:** I chose New York. I want to know how much a jail phone call costs there and how much that varies between facilities. **P:** Load the data and turn the rate columns into numbers. Filter to New York. Calculate the average, median, lowest, and highest rates. Compare in-state and out-of-state rates. Look at the most and least expensive facilities. **A:** *Load jail_phone_rates_2021.csv with tidyverse and clean the column names. Turn the rate columns into numbers. Then filter to New York and show me the average, median, min, and max rates, and the most and least expensive facilities.* **R:** First, I load the packages I need. ```{r} library(tidyverse) library(janitor) ``` Next, I read in the data, clean up the column names, and look at the first few rows. ```{r} phones_raw <- read_csv("jail_phone_rates_2021.csv") phones_raw <- clean_names(phones_raw) head(phones_raw) ``` The rate columns are stored as text with a "\$" (like "\$0.21"), so R can't do math on them yet. Some rows also say things like "N/A \*" (rate unknown), "N/A (facility pays the cost)" (the call is free), or "\$4.35 for first minute; \$0.40 for each subsequent minute." This chunk turns the rate columns into numbers. Unknown rates become `NA`, free calls become \$0, and first-minute pricing becomes `NA` because it doesn't have one per-minute rate. ```{r} phones <- phones_raw |> mutate( in_state = parse_number(in_state_per_minute_rate, na = c("N/A *", "N/A (facility pays the cost)")), out_state = parse_number(out_of_state_per_minute_rate, na = c("N/A *", "N/A (facility pays the cost)")), adp = parse_number(average_daily_population_adp, na = "N/A **"), in_state = if_else(str_detect(in_state_per_minute_rate, "facility pays"), 0, in_state), out_state = if_else(str_detect(out_of_state_per_minute_rate, "facility pays"), 0, out_state), in_state = if_else(str_detect(in_state_per_minute_rate, "first minute"), NA, in_state) ) ``` Now I keep only the New York facilities and calculate summary statistics for in-state and out-of-state rates. ```{r} ny <- phones |> filter(state == "New York") ny |> summarize( facilities = n(), mean_in_state = mean(in_state, na.rm = TRUE), median_in_state = median(in_state, na.rm = TRUE), min_in_state = min(in_state, na.rm = TRUE), max_in_state = max(in_state, na.rm = TRUE), mean_out_state = mean(out_state, na.rm = TRUE), median_out_state = median(out_state, na.rm = TRUE) ) ``` There are 64 facilities in New York. The median in-state rate is \$0.20 per minute and the average is \$0.175, so a typical 15-minute call costs about \$3. Out-of-state rates are almost identical (they're the same at 62 of the 64 facilities). Next, I sort the facilities from most to least expensive, and then from least to most expensive. ```{r} ny |> select(facility, carrier_provider, in_state_per_minute_rate, out_of_state_per_minute_rate) |> arrange(desc(parse_number(in_state_per_minute_rate))) ny |> select(facility, carrier_provider, in_state, out_state) |> arrange(in_state) ``` The most expensive facility is Nassau County, which charges \$4.35 for the first minute and \$0.40 after that. (You may insert as many code chunks as you'd like. You can do so by going to Insert \> Executable Cell \> R in the toolbar above.) **C:** There are 64 facilities in New York, which matches the number of rows in my sorted tables. The highest per-minute rate after cleaning is \$0.21, and I can see in the sorted table that Nassau County's first-minute price didn't sneak in as a \$4.35 per-minute rate. **K:** `parse_number()` pulls the number out of text like "\$0.21" so R can do math on it. `filter()` keeps only the rows I want (New York). `summarize()` collapses all the rows into one row of statistics, and `na.rm = TRUE` tells R to skip missing values instead of returning `NA`. `arrange()` sorts the rows, and `desc()` sorts from highest to lowest. The mean and median tell me different things: the median is the middle facility, while the mean can be pulled up or down by a few extreme values. ## Question 2 Compare different types of facilities across the whole country. Some questions you might want to answer: - How many of each are included in this data? - How do different facilities differ in terms of call rates? **S:** I want to know how many of each type of facility are in the data, and how their call rates compare. **P:** Group by facility type. Count the facilities and how many have rate data. Calculate the median, mean, min, and max in-state rate and the median out-of-state rate. Sort from most to fewest facilities. **A:** *Using my data, group by facility type. For each type, show the number of facilities and the median and average call rates. Sort from most to fewest facilities.* **R:** This chunk groups the facilities by type, and calculates the number of facilities and summary statistics for each type. ```{r} phones |> group_by(facility_type) |> summarize( facilities = n(), median_in_state = median(in_state, na.rm = TRUE), mean_in_state = mean(in_state, na.rm = TRUE), min_in_state = min(in_state, na.rm = TRUE), max_in_state = max(in_state, na.rm = TRUE), ) |> arrange(desc(facilities)) ``` Jails make up 91% of the data (2,886 facilities). Military has 7 facilities, and Miscellaneous and Territorial have 3 each. Median in-state rates differ by type: \$0.21 for jails and juvenile facilities, \$0.14 for private prisons, \$0.12 for community corrections, and \$0.07 for immigrant detention. (You may insert as many code chunks as you'd like. You can do so by going to Insert \> Executable Cell \> R in the toolbar above.) **C:** The facility counts add up to 3,172, the total number of rows, and there are 10 rows, one per facility type. **K:** `group_by()` splits the data into groups (one per facility type), so `summarize()` calculates the statistics separately for each group instead of for the whole dataset. `n()` counts the rows in each group. ## Question 3 Compare different carriers/providers. Some questions you might want to answer: - Do certain states rely more on certain carriers over others? - Which carriers charge the highest rates? **S:** I want to know which carriers charge the most, and whether some states rely mostly on one carrier. **P:** Group the partnership names under their main company. Summarize rates by carrier, keeping carriers with at least 20 facilities. For each state, find the carrier with the most facilities and its share of that state's facilities. **A:** *Combine carrier names that include "NCIC" or "ViaPath" into one group each. Then show the number of facilities and the median and average rate for each carrier with at least 20 facilities. Also show the most common carrier in each state, and which states have the highest median rate.* **R:** Some carriers show up under several names (e.g., "NCIC" and "NCIC/Crown Correctional"). This chunk combines those into one group, then calculates the number of facilities, number of states, and rates for each carrier with at least 20 facilities. ```{r} phones <- phones |> mutate(carrier_group = case_when( str_detect(carrier_provider, "NCIC") ~ "NCIC (incl. partnerships)", str_detect(carrier_provider, "ViaPath") ~ "ViaPath (incl. partnerships)", .default = carrier_provider )) phones |> group_by(carrier_group) |> summarize( facilities = n(), median_in_state = median(in_state, na.rm = TRUE), mean_in_state = mean(in_state, na.rm = TRUE) ) |> filter(facilities >= 20) |> arrange(desc(mean_in_state)) ``` The biggest national carriers aren't the most expensive. Securus (677 facilities in 45 states), NCIC, and ViaPath have median in-state rates of \$0.17–\$0.21. Smaller regional carriers charge the most: Custom Teleconnect (median \$0.50) and City Tele Coin (median \$0.35, 243 facilities in only 8 states). Next, for each state, I find the carrier that serves the most facilities and what share of the state's facilities it serves. ```{r} phones |> count(state, carrier_group) |> group_by(state) |> mutate( state_total = sum(n), share = n / state_total ) |> slice_max(share, n = 1, with_ties = FALSE) |> ungroup() |> filter(state_total >= 20) |> arrange(desc(share)) |> print(n = Inf) ``` Some states rely heavily on one carrier. ViaPath serves 88% of New Jersey's facilities and 70% of New York's. City Tele Coin serves 57% of Arkansas's facilities. (You may insert as many code chunks as you'd like. You can do so by going to Insert \> Executable Cell \> R in the toolbar above.) **C:** After grouping, NCIC and ViaPath each appear once. New York's top carrier is ViaPath (45 of 64 facilities), which matches what I saw in Question 1. Encartele has 0 facilities with rate data, and it has 143 facilities, so it accounts for all 143 "N/A \*" values. **K:** `case_when()` creates a new column based on rules: if the carrier name contains "NCIC," label it NCIC, and so on. `str_detect()` checks whether a piece of text contains a word. `n_distinct()` counts how many different states each carrier serves. `filter(facilities >= 20)` drops tiny carriers whose averages could be based on just a few facilities. `count()` counts rows for each state and carrier combination, and `slice_max()` keeps the row with the largest share in each state. Combining the partnerships was my own judgment call; someone else could reasonably keep them separate. ## Common issues 1. Use `read_csv()`, NOT `read.csv()`. When prompting AI, it can be helpful to include in your prompt something like "Using tidyverse functions..." 2. You had to use `parse_number()` to convert the price columns into numbers that you can do math on. 3. A benefit of large-scale secondary data is that it can withstand small inaccuracies. But still, it's important to look through and make sure everything is being calculated correctly. Are the correct numbers being extracted? Are the correct things being labeled `NA` vs 0? 4. Make sure your "check" is filled in and well-explained. "Knowledge" is about explaining what your code does. You should still leave notes about your findings throughout - and I've tried to make that more explicit in future labs. 5. Your document should be organized, linear, non-repetitive, and easy to follow. This includes checking that the rendered output looks good and isn't a bajillion lines long! Use `head()` to avoid that.