library(tidyverse)
library(janitor)Lab Assignment 1 EXAMPLE
Instructions
Welcome to your first lab assignment! Are you excited? I sure am. Please read the following instructions before you begin.
For your lab assignments, you may use AI to help you code. You will be required to explicitly fill out the SPARCK framework for each question. You may not use AI to write the non-coding parts of the framework, although you may consult it for assistance.
Generally, your lab assignments will ask you to access a dataset, and then to investigate some questions in regards to the data. These questions will be relatively broad, meaning that you will have some flexibility in how exactly you answer them. This is the point, and the benefit of using AI to help you code: We don’t need to be worried about walking step by step through an analysis, but can instead think about it from a broader perspective; the specific code you use to tackle the analysis is up to you. The goal is for you to get practice thinking about how to answer a question with data.
As you write code, think about what we’ve learned in class so far, and especially what we learned in the Data Wrangling/Cleaning tutorial. You don’t need to exclusively use functions we used in class, but they should serve as a reminder of the types of things you might wish to do.
You should guide me through the answer the questions. This means you shouldn’t just copy and paste code from Gemini and be done with it; your SPARCK responses should guide me through your thought process and analysis. You might also wish to spread your code across several different chunks so you can display output along the way if you think that makes it easier to understand.
Remember that you must render your Quarto document into an html file. This html is what you should hand in on Canvas. If you encounter errors, use Gemini to help you troubleshoot!
Make sure you put this Quarto file in your Week 3 folder, and then open the file from that folder so that R’s working directory is your Week 3 folder.
S: Set up (What is your end goal?)
P: Pseudocode (What are the steps?)
A: AI (What are your AI prompts?)
R: Run (Run it!)
C: Check (Did your code work?)
K: Knowledge (What does your code mean?)
The data
For today’s homework, we are going to be using a dataset from the Prison Policy Initiative about the costs of making phone calls from inside carceral institutions. You can learn more about this project here.
Specifically, we will be using a dataset that tracks phone call rates in over 3,000 jails and other detention facilities across the U.S. in 2021. (Jails are different from prisons. Jails are run by counties and cities and mostly hold people who are awaiting trial and haven’t been convicted of a crime, along with people serving short sentences. Prisons are run by states or the federal government and hold people who have been convicted and sentenced, usually to more than a year.) Most facilities in this dataset are jails, but it also includes juvenile detention centers, immigrant detention centers, tribal jails, and a few other types.
For this homework, I have provided you with the dataset. You may download it from either our syllabus website or Canvas. You should place it in the same folder as this document.
Question 1
Pick one state that is meaningful to you, and give me a sense of the phone rate landscape in that state. Some questions you might wish to answer:
What is the average cost of making a phone call?
How much variety is there between facilities?
Is there a large difference in in-state vs out of state rates?
Remember, you are choosing what point you want to get across, and how exactly you want to do it.
S: I chose New York. I want to know how much a jail phone call costs there and how much that varies between facilities.
P: Load the data and turn the rate columns into numbers. Filter to New York. Calculate the average, median, lowest, and highest rates. Compare in-state and out-of-state rates. Look at the most and least expensive facilities.
A: Load jail_phone_rates_2021.csv with tidyverse and clean the column names. Turn the rate columns into numbers. Then filter to New York and show me the average, median, min, and max rates, and the most and least expensive facilities.
R:
First, I load the packages I need.
Next, I read in the data, clean up the column names, and look at the first few rows.
phones_raw <- read_csv("jail_phone_rates_2021.csv")
phones_raw <- clean_names(phones_raw)
head(phones_raw)# A tibble: 6 x 7
state facility facility_type average_daily_popula~1 carrier_provider
<chr> <chr> <chr> <chr> <chr>
1 Alabama Adamsville Poli~ Jail 6 NCIC
2 Alabama Albertville Cit~ Jail 27 NCIC
3 Alabama Arab City Polic~ Jail 9 NCIC
4 Alabama Attalla Alabama~ Jail 5 CPC
5 Alabama Autauga County ~ Jail 127 ATN
6 Alabama Baldwin County ~ Jail 656 ViaPath (GTL)
# i abbreviated name: 1: average_daily_population_adp
# i 2 more variables: out_of_state_per_minute_rate <chr>,
# in_state_per_minute_rate <chr>The rate columns are stored as text with a “$” (like “$0.21”), so R can’t do math on them yet. Some rows also say things like “N/A *” (rate unknown), “N/A (facility pays the cost)” (the call is free), or “$4.35 for first minute; $0.40 for each subsequent minute.”
This chunk turns the rate columns into numbers. Unknown rates become NA, free calls become $0, and first-minute pricing becomes NA because it doesn’t have one per-minute rate.
phones <- phones_raw |>
mutate(
in_state = parse_number(in_state_per_minute_rate,
na = c("N/A *", "N/A (facility pays the cost)")),
out_state = parse_number(out_of_state_per_minute_rate,
na = c("N/A *", "N/A (facility pays the cost)")),
adp = parse_number(average_daily_population_adp, na = "N/A **"),
in_state = if_else(str_detect(in_state_per_minute_rate, "facility pays"), 0, in_state),
out_state = if_else(str_detect(out_of_state_per_minute_rate, "facility pays"), 0, out_state),
in_state = if_else(str_detect(in_state_per_minute_rate, "first minute"), NA, in_state)
)Now I keep only the New York facilities and calculate summary statistics for in-state and out-of-state rates.
ny <- phones |> filter(state == "New York")
ny |>
summarize(
facilities = n(),
mean_in_state = mean(in_state, na.rm = TRUE),
median_in_state = median(in_state, na.rm = TRUE),
min_in_state = min(in_state, na.rm = TRUE),
max_in_state = max(in_state, na.rm = TRUE),
mean_out_state = mean(out_state, na.rm = TRUE),
median_out_state = median(out_state, na.rm = TRUE)
)# A tibble: 1 x 7
facilities mean_in_state median_in_state min_in_state max_in_state
<int> <dbl> <dbl> <dbl> <dbl>
1 64 0.175 0.2 0.03 0.21
# i 2 more variables: mean_out_state <dbl>, median_out_state <dbl>There are 64 facilities in New York. The median in-state rate is $0.20 per minute and the average is $0.175, so a typical 15-minute call costs about $3. Out-of-state rates are almost identical (they’re the same at 62 of the 64 facilities).
Next, I sort the facilities from most to least expensive, and then from least to most expensive.
ny |>
select(facility, carrier_provider, in_state_per_minute_rate, out_of_state_per_minute_rate) |>
arrange(desc(parse_number(in_state_per_minute_rate)))# A tibble: 64 x 4
facility carrier_provider in_state_per_minute_~1 out_of_state_per_min~2
<chr> <chr> <chr> <chr>
1 "Nassau Count~ ViaPath (GTL) "$4.35 for first minu~ $0.21
2 "Albany Count~ Securus "$0.21" $0.21
3 "Depew Villag~ NCIC/SD Communi~ "$0.21" $0.21
4 "Erie County ~ ICSolutions "$0.21" $0.21
5 "Genesee Coun~ Securus "$0.21" $0.21
6 "Greece Polic~ NCIC "$0.21" $0.21
7 "Madison Coun~ ViaPath (GTL) "$0.21" $0.21
8 "Monroe Count~ Securus "$0.21" $0.10
9 "Ontario Coun~ Securus "$0.21" $0.21
10 "Orleans Coun~ ICSolutions "$0.21" $0.21
# i 54 more rows
# i abbreviated names: 1: in_state_per_minute_rate,
# 2: out_of_state_per_minute_rateny |>
select(facility, carrier_provider, in_state, out_state) |>
arrange(in_state)# A tibble: 64 x 4
facility carrier_provider in_state out_state
<chr> <chr> <dbl> <dbl>
1 New York CIanna M Kross Center, New York~ Securus 0.03 0.03
2 Buffalo Federal Detention Facility (DHS ~ Talton Communic~ 0.07 0.07
3 Central New York Psychiatric Center (Pat~ NCIC 0.12 0.12
4 Central New York Psychiatric Center (Res~ NCIC 0.12 0.12
5 St. Lawrence Psychiatric Center NCIC 0.12 0.12
6 Onondaga County Justice Center ICSolutions 0.13 0.13
7 Onondaga Department Of Correction ICSolutions 0.13 0.13
8 Ulster County Jail Securus 0.14 0.14
9 Allegany County Jail ViaPath (GTL) 0.15 0.15
10 Cattaraugus County Jail ViaPath (GTL) 0.15 0.15
# i 54 more rowsThe most expensive facility is Nassau County, which charges $4.35 for the first minute and $0.40 after that.
(You may insert as many code chunks as you’d like. You can do so by going to Insert > Executable Cell > R in the toolbar above.)
C: There are 64 facilities in New York, which matches the number of rows in my sorted tables. The highest per-minute rate after cleaning is $0.21, and I can see in the sorted table that Nassau County’s first-minute price didn’t sneak in as a $4.35 per-minute rate.
K: parse_number() pulls the number out of text like “$0.21” so R can do math on it. filter() keeps only the rows I want (New York). summarize() collapses all the rows into one row of statistics, and na.rm = TRUE tells R to skip missing values instead of returning NA. arrange() sorts the rows, and desc() sorts from highest to lowest. The mean and median tell me different things: the median is the middle facility, while the mean can be pulled up or down by a few extreme values.
Question 2
Compare different types of facilities across the whole country. Some questions you might want to answer:
How many of each are included in this data?
How do different facilities differ in terms of call rates?
S: I want to know how many of each type of facility are in the data, and how their call rates compare.
P: Group by facility type. Count the facilities and how many have rate data. Calculate the median, mean, min, and max in-state rate and the median out-of-state rate. Sort from most to fewest facilities.
A: Using my data, group by facility type. For each type, show the number of facilities and the median and average call rates. Sort from most to fewest facilities.
R:
This chunk groups the facilities by type, and calculates the number of facilities and summary statistics for each type.
phones |>
group_by(facility_type) |>
summarize(
facilities = n(),
median_in_state = median(in_state, na.rm = TRUE),
mean_in_state = mean(in_state, na.rm = TRUE),
min_in_state = min(in_state, na.rm = TRUE),
max_in_state = max(in_state, na.rm = TRUE),
) |>
arrange(desc(facilities))# A tibble: 10 x 6
facility_type facilities median_in_state mean_in_state min_in_state
<chr> <int> <dbl> <dbl> <dbl>
1 Jail 2886 0.21 0.210 0
2 Juvenile 78 0.21 0.189 0
3 Private Prison 54 0.14 0.139 0
4 Community Corrections 44 0.12 0.146 0.07
5 Immigrant Detention 42 0.07 0.0807 0.07
6 Indian Country 36 0.21 0.196 0.13
7 Treatment 19 0.12 0.153 0.05
8 Military 7 0.14 0.183 0.07
9 Miscellaneous 3 0.2 0.167 0.09
10 Territorial 3 0.1 0.123 0.1
# i 1 more variable: max_in_state <dbl>Jails make up 91% of the data (2,886 facilities). Military has 7 facilities, and Miscellaneous and Territorial have 3 each.
Median in-state rates differ by type: $0.21 for jails and juvenile facilities, $0.14 for private prisons, $0.12 for community corrections, and $0.07 for immigrant detention.
(You may insert as many code chunks as you’d like. You can do so by going to Insert > Executable Cell > R in the toolbar above.)
C: The facility counts add up to 3,172, the total number of rows, and there are 10 rows, one per facility type.
K: group_by() splits the data into groups (one per facility type), so summarize() calculates the statistics separately for each group instead of for the whole dataset. n() counts the rows in each group.
Question 3
Compare different carriers/providers. Some questions you might want to answer:
Do certain states rely more on certain carriers over others?
Which carriers charge the highest rates?
S: I want to know which carriers charge the most, and whether some states rely mostly on one carrier.
P: Group the partnership names under their main company. Summarize rates by carrier, keeping carriers with at least 20 facilities. For each state, find the carrier with the most facilities and its share of that state’s facilities.
A: Combine carrier names that include “NCIC” or “ViaPath” into one group each. Then show the number of facilities and the median and average rate for each carrier with at least 20 facilities. Also show the most common carrier in each state, and which states have the highest median rate.
R:
Some carriers show up under several names (e.g., “NCIC” and “NCIC/Crown Correctional”). This chunk combines those into one group, then calculates the number of facilities, number of states, and rates for each carrier with at least 20 facilities.
phones <- phones |>
mutate(carrier_group = case_when(
str_detect(carrier_provider, "NCIC") ~ "NCIC (incl. partnerships)",
str_detect(carrier_provider, "ViaPath") ~ "ViaPath (incl. partnerships)",
.default = carrier_provider
))
phones |>
group_by(carrier_group) |>
summarize(
facilities = n(),
median_in_state = median(in_state, na.rm = TRUE),
mean_in_state = mean(in_state, na.rm = TRUE)
) |>
filter(facilities >= 20) |>
arrange(desc(mean_in_state))# A tibble: 16 x 4
carrier_group facilities median_in_state mean_in_state
<chr> <int> <dbl> <dbl>
1 Custom Teleconnect 22 0.5 0.344
2 City Tele Coin 243 0.35 0.331
3 Correct Solutions 74 0.25 0.281
4 Consolidated Telecom 31 0.3 0.280
5 Reliance Telephone 157 0.25 0.251
6 ATN 25 0.19 0.227
7 Turnkey Corrections 39 0.21 0.208
8 Prodigy 41 0.21 0.208
9 CPC 290 0.2 0.201
10 Pay Tel 143 0.21 0.200
11 Securus 677 0.21 0.194
12 NCIC (incl. partnerships) 555 0.21 0.181
13 Smart Communications 20 0.195 0.175
14 ICSolutions 246 0.205 0.167
15 ViaPath (incl. partnerships) 440 0.17 0.164
16 Encartele 143 NA NaN The biggest national carriers aren’t the most expensive. Securus (677 facilities in 45 states), NCIC, and ViaPath have median in-state rates of $0.17–$0.21. Smaller regional carriers charge the most: Custom Teleconnect (median $0.50) and City Tele Coin (median $0.35, 243 facilities in only 8 states).
Next, for each state, I find the carrier that serves the most facilities and what share of the state’s facilities it serves.
phones |>
count(state, carrier_group) |>
group_by(state) |>
mutate(
state_total = sum(n),
share = n / state_total
) |>
slice_max(share, n = 1, with_ties = FALSE) |>
ungroup() |>
filter(state_total >= 20) |>
arrange(desc(share)) |>
print(n = Inf)# A tibble: 41 x 5
state carrier_group n state_total share
<chr> <chr> <int> <int> <dbl>
1 New Jersey ViaPath (incl. partnerships) 23 26 0.885
2 Minnesota Reliance Telephone 59 79 0.747
3 New York ViaPath (incl. partnerships) 45 64 0.703
4 South Dakota Reliance Telephone 20 29 0.690
5 Kentucky CPC 48 77 0.623
6 Utah Securus 14 24 0.583
7 Pennsylvania ViaPath (incl. partnerships) 40 69 0.580
8 Arkansas City Tele Coin 43 75 0.573
9 Massachusetts NCIC (incl. partnerships) 16 29 0.552
10 New Mexico Securus 16 29 0.552
11 North Dakota Reliance Telephone 12 22 0.545
12 Nebraska Encartele 31 58 0.534
13 Idaho ViaPath (incl. partnerships) 18 34 0.529
14 Indiana CPC 51 98 0.520
15 North Carolina Pay Tel 48 95 0.505
16 Iowa Reliance Telephone 56 115 0.487
17 South Carolina Securus 23 49 0.469
18 Nevada Securus 11 24 0.458
19 Alabama NCIC (incl. partnerships) 49 108 0.454
20 Mississippi City Tele Coin 44 98 0.449
21 Oregon ViaPath (incl. partnerships) 13 30 0.433
22 Montana Securus 17 40 0.425
23 Michigan Securus 38 91 0.418
24 California NCIC (incl. partnerships) 55 132 0.417
25 Texas NCIC (incl. partnerships) 137 330 0.415
26 Missouri Encartele 44 112 0.393
27 Colorado Securus 18 46 0.391
28 Oklahoma City Tele Coin 36 92 0.391
29 Virginia ViaPath (incl. partnerships) 28 74 0.378
30 Florida ViaPath (incl. partnerships) 29 78 0.372
31 Maryland ICSolutions 9 25 0.36
32 Kansas CPC 34 100 0.34
33 Ohio Securus 42 125 0.336
34 Illinois CPC 31 93 0.333
35 Wyoming Securus 7 22 0.318
36 Wisconsin Securus 21 68 0.309
37 Louisiana NCIC (incl. partnerships) 31 104 0.298
38 Washington Securus 17 62 0.274
39 Georgia Pay Tel 45 165 0.273
40 Arizona ViaPath (incl. partnerships) 11 42 0.262
41 Tennessee CPC 25 101 0.248Some states rely heavily on one carrier. ViaPath serves 88% of New Jersey’s facilities and 70% of New York’s. City Tele Coin serves 57% of Arkansas’s facilities.
(You may insert as many code chunks as you’d like. You can do so by going to Insert > Executable Cell > R in the toolbar above.)
C: After grouping, NCIC and ViaPath each appear once. New York’s top carrier is ViaPath (45 of 64 facilities), which matches what I saw in Question 1. Encartele has 0 facilities with rate data, and it has 143 facilities, so it accounts for all 143 “N/A *” values.
K: case_when() creates a new column based on rules: if the carrier name contains “NCIC,” label it NCIC, and so on. str_detect() checks whether a piece of text contains a word. n_distinct() counts how many different states each carrier serves. filter(facilities >= 20) drops tiny carriers whose averages could be based on just a few facilities. count() counts rows for each state and carrier combination, and slice_max() keeps the row with the largest share in each state. Combining the partnerships was my own judgment call; someone else could reasonably keep them separate.
Common issues
Use
read_csv(), NOTread.csv(). When prompting AI, it can be helpful to include in your prompt something like “Using tidyverse functions…”You had to use
parse_number()to convert the price columns into numbers that you can do math on.A benefit of large-scale secondary data is that it can withstand small inaccuracies. But still, it’s important to look through and make sure everything is being calculated correctly. Are the correct numbers being extracted? Are the correct things being labeled
NAvs 0?Make sure your “check” is filled in and well-explained. “Knowledge” is about explaining what your code does. You should still leave notes about your findings throughout - and I’ve tried to make that more explicit in future labs.
Your document should be organized, linear, non-repetitive, and easy to follow. This includes checking that the rendered output looks good and isn’t a bajillion lines long! Use
head()to avoid that.