install.packages(c('tidycensus', 'ngramr', 'congress', 'httr2'))
install.packages(c('gssr', 'gssrdoc'), repos = c('https://kjhealy.r-universe.dev', 'https://cloud.r-project.org'))Week 7 Tutorial
Web Scraping
First things first: Make a new project in RStudio, and be sure to place it somewhere where you can easily find it (e.g., Documents, Desktop). Then download this file (the .qmd file) from the syllabus website and place it in the folder associated with that project. Then open this file in RStudio and follow along.
I encourage you to work with/talk to others throughout this tutorial, especially during times where I have you code on your own and use AI. When you code, think about the SPARCK framework:
S: Set up (What is your end goal?)
P: Pseudocode (What are the steps?)
A: AI (What are your AI prompts?)
R: Run (Run it!)
C: Check (Did your code work?)
K: Knowledge (What does your code mean?)
Agenda
Scraping with
rvestPulling data with
gssr,ngramr, andtidycensusUsing APIs with
congressandhttr2Practice
So far, almost every dataset we’ve used has been a file that we downloaded from a website and then read into R. Today, you’re going to learn how to pull data from the web directly into R. We’ll go through the full range of ways to do this:
Scraping: You can have R read the content of a webpage. This is called web scraping. It’s the most flexible way to get data from the web, because it works on almost any website, but it’s also the most confusing, because it requires a lot of knowledge about how webpages are built. We’ll look at 2 brief examples of how to web-scrape in R, but won’t go into too much detail.
Packages that get data for you: For a lot of popular data sources, someone has already written an R package that gets the data for you. This is by far the easiest way to get data from the web.
APIs: Many websites that hold data have an API (Application Programming Interface), which is a way to access a website without clicking around. Some APIs require an API key to use them. And when there’s no package for an API, you can use the API yourself.
One thing to keep in mind: most of these packages are made by individual researchers, not by the websites themselves. Some are well-maintained, while others stop working when the website they connect to changes.
If you haven’t already, install the packages we’ll need today in the console. (gssr and gssrdoc need to be installed a slightly different way because the data is too big to sit in R’s package repository)
Load tidyverse. We’ll load each of the other packages when we get to it.
# Write your code hereScraping with rvest
The most flexible way to get data from the web is to scrape it from the webpage itself, using the rvest package. Every webpage is really just a text file written in HTML (HyperText Markup Language). Your web browser reads that file and turns it into the page you see. rvest reads the same file, and lets you pull out the data you want.
To scrape a website, you need to know a fair amount about how HTML is organized. An HTML page is made of elements. Each element starts with a tag in angle brackets, like <p>, and ends with a closing tag, like </p>. Elements can sit inside other elements, and they can have attributes, like a class or an id, which web designers use to label elements. Here’s a tiny example:
<ul id="reports">
<li>
<span class="date">October 2026</span>
<a href="/reports/report1.html">Title of the report</a>
</li>
</ul>A few tags you’ll see a lot: <p> (paragraph), <table> (table), <a> (link), <ul> and <li> (a list, and an item in the list), <span> and <div> (boxes that hold other elements), and <h1>, <h2>, etc. (headings).
Web scraping can get confusing quickly, so we won’t go beyond a couple of examples in this class. You don’t need to run the code in this section. It’s just here to show you what web scraping looks like, in case you need it for your project.
Scraping a table
The easiest thing to scrape is a table. This Wikipedia page has a table with the incarceration rate in every US state and territory. Here’s how we could get it into R:
# Example (you don't need to run this)
library(rvest)
library(janitor)
wiki_page <- read_html('https://en.wikipedia.org/wiki/List_of_U.S._states_by_incarceration_and_correctional_supervision_rate')
incarceration <- html_table(wiki_page)[[1]] |>
clean_names() |>
mutate(incarceration_rate = parse_number(incarceration_rate),
location = str_trim(str_remove(location, '\\*')))
incarcerationread_html() reads in the HTML of the page. html_table() finds every table on the page and turns each one into a dataframe. It gives back a list of tables, so [[1]] takes the first one. Then we clean it up: clean_names() fixes the column names, parse_number() turns the incarceration rates (which have commas) into numbers, and str_remove() removes the * (a footnote on the Wikipedia page) from the end of the place names.
Scraping other content
For non-tables, you tell rvest exactly which elements you want, using a CSS selector, which is a short code that points to certain elements on the page. For example:
'p'means “every paragraph.”'.date'means “every element with the classdate” (.means class).'#reports'means “the element with the idreports” (#means id).'#reports li'means “every list item inside the element with the idreports.”
The Prison Policy Initiative (PPI) is a research and advocacy organization that publishes reports about mass incarceration. Their list of reports is at https://www.prisonpolicy.org/reports.html. Here’s how we could make a dataframe with one row for each report, and columns for the date, title, link, and authors:
# Example (you don't need to run this)
ppi_page <- read_html('https://www.prisonpolicy.org/reports.html')
reports <- ppi_page |>
html_elements('#reports li')
ppi_reports <- tibble(
date = reports |> html_element('.date') |> html_text2(),
title = reports |> html_element('a:last-of-type') |> html_text2(),
link = reports |> html_element('a:last-of-type') |> html_attr('href'),
authors = reports |> html_element('.attrib') |> html_text2()
)
ppi_reportshtml_elements() (with an s) finds every element that matches the selector: here, every report, and each report becomes a row. Then html_element() (no s) finds one element inside each report (the date, the title, etc.), and each one becomes a column. html_text2() pulls the text out of each element, and html_attr() pulls out an attribute. (The web address a link goes to is stored in its href attribute.)
Using AI with html web-scraping
If you ever need to scrape a website and don’t know html, AI can help a lot. Here’s the best way to do it:
Open the page in Google Chrome. Right-click on the thing you want to scrape (e.g., the title of the first report), and click Inspect. A panel will open with the page’s HTML, and the element you clicked on will be highlighted.
Hover over the lines just above the highlighted one. As you do, the matching part of the page lights up. Find a line that lights up the whole area you want.
Right-click that line, and click Copy > Copy element.
Paste it into your AI chatbot, along with what you want.
Webpages can have thousands of lines of HTML, so copy only the part you need, not the whole page.
rvest can scrape almost any website that isn’t built with JavaScript. If you want to learn more about rvest, check out the web scraping chapter of R for Data Science (Wickham et al., 2023), which you can find here.
Luckily, for a lot of the data you might want, you don’t need to use rvest. Many people have written packages that get data from specific websites for you. Let’s learn about a few of them.
Pulling data with gssr, ngramr, and tidycensus
ngramr (Data from Google ngram)
The ngramr package gets data from Google Books Ngram Viewer, which tracks how often words and phrases appear in millions of books from 1500 to 2022. This can be useful if you want to know how ideas, terms, or concerns have changed over time. Go to https://books.google.com/ngrams/ to play around with it and see how it works.
Load ngramr. Then use ngram() to get the frequency of the phrase “mass incarceration” from 1960 onward, and save it as ngram_mi. (Hint: use the year_start argument.)
The Frequency column is the proportion of all phrases of the same length (here, all two-word phrases) that year that were “mass incarceration.” These numbers are really small, so also add a column called per_million that multiplies Frequency by 1,000,000. This is the number of times the phrase appeared per million two-word phrases.
# Write your code hereYou can also search for several phrases at once by giving ngram() a vector. Get the frequencies of “war on drugs” and “mass incarceration” from 1960 onward, add the per_million column, and make a line graph with a different colored line for each phrase.
# Write your code hereNow make the same kind of graph for the term “African American” since 1900.
# Write your code heretidycensus (Census data)
The tidycensus package pulls data from the US Census Bureau straight into R. Many of your projects might involve population statistics broken down by demographic group or by place, and this is the easiest way to get them.
Population estimates
The simplest tidycensus function is get_estimates(), which gets the Census Bureau’s yearly population estimates. It has a few important arguments:
geography: The level of the data you want (e.g.,'state','county'). Type?get_estimatesin the console to see all of the options.product: The kind of data you want.'population'gives you the total population.vintage: The year of the estimates.
Load tidycensus. Then use get_estimates() to get the population of every county in New York State in 2024. Save it as ny_pop. If you look at it, you’ll see each county has two rows. POPESTIMATE is the population, and NPOPCHG is how much the population changed in the last year. Keep only the POPESTIMATE rows, and sort the counties from largest to smallest. (Hint: get_estimates() has a state argument.)
# Write your code herePopulation by group
If you set product = 'characteristics', you can break the population down by age, sex, race, or Hispanic/Non-Hispanic, using the breakdown argument. Add breakdown_labels = TRUE to get readable labels instead of codes.
Get the population of each New York county broken down by race, and save it as ny_race. Take a look at the RACE column. Each county has a row for “All races,” which is the total population. Remove the “All races” rows, and then add a column called percent with the percentage of each county’s population in each group. (Hint: group_by() the county first.)
# Write your code hereMake a horizontal stacked bar graph comparing the racial makeup of the five counties in New York City: Bronx County, Kings County (Brooklyn), New York County (Manhattan), Queens County, and Richmond County (Staten Island). (Hint: str_detect() with | can find all five names at once.)
# Write your code hereComponents of population change
A place’s population changes for four reasons: people are born, people die, people move in from (or out to) other parts of the US (domestic migration), and people move in from (or out to) other countries (international migration). If you set product = 'components', get_estimates() gives you each of these numbers.
Get the components of population change for every New York county in 2024, and save it as ny_components. Take a look at the variable column. Then make a horizontal bar graph comparing BIRTHS, DEATHS, DOMESTICMIG, and INTERNATIONALMIG in the five counties in New York City, with the bars side by side. (Hint: %in% lets you filter() for several values at once, and position = 'dodge' puts bars side by side.)
# Write your code hereget_estimates() only has a few variables. For much more detail, there’s another tidycensus function, get_acs(), which gets data from the American Community Survey (ACS). Every year, the ACS surveys a sample of about 3.5 million households about things like income, education, housing, and health insurance, and it has tens of thousands of variables. We won’t use get_acs() today, because it uses the Census Bureau’s API, which is slow and requires an API key. But keep it in mind if you need more detailed Census data for your project. If you want to learn more about tidycensus, the free book Analyzing US Census Data (Walker, 2023) is a great place to start. You can find it here.
Using APIs with congress and httr2
An API (Application Programming Interface) is a way to ask a website for data in an automated way. When you visit a website in your browser, the website sends back a page designed for people to read. When R asks an API for data, the website sends back just the data, in a format that’s easy for a computer to read. Packages like tidycensus and congress are mostly just a set of functions that ask an API for data, and then turn what comes back into a dataframe.
API keys
Many APIs require an API key. An API key is a long string of letters and numbers that identifies you to a website’s API, kind of like a username and password rolled into one. Every time R asks the API for data, it sends your key along with the request, so the website knows who’s asking.
API keys are usually free, and you usually get one by signing up on the website with your email. Many US government agencies share one system for API keys, called api.data.gov. Go to https://api.data.gov/signup/ and sign up with your name and email. You will receive your key by email (check your spam). The same key works for many government APIs, like the FBI’s Crime Data Explorer, which has crime statistics from police departments across the US.
Since an API key identifies you, treat it like a password: don’t put it in a document you’re going to share. If someone else uses your key, it counts against your limits, and anything they do with it looks like it came from you.
congress (Using an API with a package)
Congress.gov is the official website for US federal legislation. Its API has information about every member of Congress, every bill, every law, and more, and the congress package gets this data into R. The Congress.gov API uses api.data.gov keys, so your new key will work with it.
Packages that use an API key usually have a function where you give it your key. In the congress package, it’s set_congress_key(). Load congress, and then run the code below in the console, with your own key pasted in. install = TRUE saves your key in a special file on your computer (called .Renviron), so you only have to do this once, and so your key never has to appear in your Quarto document.
library(congress)
set_congress_key('PASTE YOUR KEY HERE', install = TRUE)Now you can use any function in the package. They all start with cong_, and each one matches a part of the Congress.gov API (members, bills, laws, votes, etc.). You can see all of the functions and what they do on the congress package website.
Members of Congress
Use cong_member() to get every current member of Congress from New York, and save it as ny_members. (Hint: look at the help page for cong_member() to find the arguments you need. You’ll want to set limit = 250, which is the most results the API will give at once.)
Then count how many members are in each party. Look closely at the data first, though: some members show up more than once (e.g., if they’ve served more than one term), so you’ll want to use distinct() to keep only one row for each member. (Hint: every member of Congress has a unique ID in the bioguide_id column.)
# Write your code hereLaws passed by a Congress
Each Congress lasts two years (e.g., the 118th Congress was 2023-2024). cong_law() gets the laws passed by a Congress. The 118th Congress passed 274 laws, but the API only gives us 250 at a time. APIs often split their results into pages, like the pages of search results on Google. To get the rest, you can use the offset argument, which tells the API to skip a certain number of results. For example, offset = 250 skips the first 250 laws, and gives you the rest.
Use cong_law() twice, once without offset and once with offset = 250, and use bind_rows() to stack the two results into one dataframe called laws_118. Then check that you have all 274 laws, by counting how many laws started in the House (HR) and how many started in the Senate (S). (The type column says what kind of bill each law started as. You’ll also see a few HJRES, which are joint resolutions, a different kind of bill that can also become a law.)
# Write your code hereA single bill
You can also get detailed information about a single bill. To do that, you need to know three things about it: which Congress it was in, its type (e.g., 'hr' for a House bill, 's' for a Senate bill), and its number. You can find all three by searching for the bill on Congress.gov.
The First Step Act was a major criminal justice reform law, which changed federal sentencing laws and prison programs. It passed as Senate bill 756 (S. 756) in the 115th Congress (2017-2018). Use cong_bill() to get information about the bill, and save it as first_step. Then look at the title, introduced_date, and cosponsors_count columns.
# Write your code hereA cosponsor is a member of Congress who signs on to support a bill. The item argument lets you get more detail about a bill, and setting item = 'cosponsors' gets the list of cosponsors. Get the cosponsors of the First Step Act, save them as first_step_cosponsors, and count how many are in each party.
# Write your code hereOther functions
There’s a lot more in the congress package. A few other functions you might find useful:
cong_bill()with otheritems, like'actions'(everything that happened to the bill on the way to becoming a law),'summaries'(summaries of what the bill does), and'text'(links to the full text of the bill).cong_member()withitem = 'sponsored-legislation', which gets every bill a member of Congress has sponsored.cong_house_vote(), which gets roll call votes in the House of Representatives.cong_committee()andcong_hearing(), which get information about congressional committees and their hearings.cong_crs_report(), which gets reports from the Congressional Research Service, which writes nonpartisan research reports for members of Congress.cong_nomination(), which gets presidential nominations (e.g., for judges) that the Senate votes on.
If you want to learn more about what’s in the Congress.gov API, its documentation is here, and there’s a more detailed guide to each part of it here.
httr2 (Using an API without a package)
What if there’s no package for the API you want to use? Then you can ask the API for data yourself, using the httr2 package. To do that, you need to learn how that particular API works. Let’s try it with Wikipedia.
The Wikipedia API
Wikipedia keeps track of how many times people view each of its articles, and it shares this data through the Wikimedia Analytics API. Asking this API (or almost any API) for data is really just visiting a specific web address (a URL). The trick is knowing what to put in the web address.
Every API has documentation that explains how to build the address for the data you want. The documentation for this API is here. It explains that the address for the number of views of an article looks like this, where each part in {curly brackets} is something you fill in:
https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/{project}/{access}/{agent}/{article}/{granularity}/{start}/{end}https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/: The start of the address, which is the same for every request.{project}: Which website (e.g.,en.wikipediafor English Wikipedia).{access}: Which devices (all-accessfor all of them, ordesktop,mobile-app, ormobile-web).{agent}: Which visitors (userfor people, as opposed to automated programs).{article}: Which article. This has to be the article’s title exactly as it appears in the article’s web address, including capital letters.{granularity}: How often (dailyormonthly).{start}and{end}: The start and end dates, written as year, month, day (e.g.,20150701). The data starts in July 2015.
So here’s the address for the number of times people viewed the Wikipedia article on Minneapolis each month, from July 2015 to the end of 2026:
https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia/all-access/user/Minneapolis/monthly/20150701/20261231Try pasting it into your web browser. Instead of a webpage, you get the raw data, written in a format called JSON (JavaScript Object Notation), which is a common way for APIs to send data. If you change any part of the address, you get different data.
Getting the data with httr2
The httr2 package lets R visit these addresses for you, and turn what comes back into data you can work with. It works in three steps:
request()creates a request for a web address.req_perform()sends the request to the API, and gets back a response.resp_body_json()reads the data out of the response.
You can learn more about httr2 on its website, here.
Load httr2, and use request() and req_perform() to send a request to the web address above. Save the response as wiki_response, and print it out. Wikipedia asks people to say who they are when they use their API, so add req_user_agent('PSYC1495 tutorial') before you perform the request.
# Write your code hereStatus: 200 OK means the request worked.
Now use resp_body_json() to get the data out of the response. Add simplifyVector = TRUE to turn the JSON into something that looks like a dataframe. The data you want is in a part of the response called items, which you can get with $items. Save it as wiki_views, and then make a line graph of the number of views each month. (Hint: The timestamp column is written like 2015070100. You can turn it into a date with ymd_h(). And scale_y_continuous(labels = scales::label_comma()) writes big numbers with commas, instead of like 2e+06.)
# Write your code herePractice
Use SPARCK to pull data from the web into R. Your analysis must:
Use
gssr,ngramr,tidycensus,congress, orhttr2, in a way that we didn’t in this tutorial (a different variable, place, year, phrase, bill, member, or article).Include a visualization.
For example, you could use gss_doc to find a GSS question you’re interested in and compare how different groups answered it, compare the cosponsors of a bill you’re interested in, compare the policy areas of two members of Congress, or compare the Wikipedia views of a few related articles.
S:
P:
A:
R:
# Write your code hereC:
K:
References
Carmody, S. (2025). ngramr: Retrieve and plot Google n-gram data (Version 1.10.0) [R package]. https://CRAN.R-project.org/package=ngramr
Healy, K. (2026). gssr: General Social Survey data for use in R (Version 0.9.0) [R package]. http://kjhealy.github.io/gssr
Kenny, C. (2025). congress: Access the Congress.gov API (Version 0.1.0) [R package]. https://CRAN.R-project.org/package=congress
Walker, K. (2023). Analyzing US Census data: Methods, maps, and models in R. CRC Press. https://walker-data.com/census-r/
Walker, K., & Herman, M. (2026). tidycensus: Load US Census boundary and attribute data as ‘tidyverse’ and ‘sf’-ready data frames [R package]. https://CRAN.R-project.org/package=tidycensus
Wickham, H. (2024). rvest: Easily harvest (scrape) web pages (Version 1.0.4) [R package]. https://CRAN.R-project.org/package=rvest
Wickham, H. (2025). httr2: Perform HTTP requests and process the responses (Version 1.2.2) [R package]. https://CRAN.R-project.org/package=httr2
Wickham, H., Çetinkaya-Rundel, M., & Grolemund, G. (2023). R for data science (2nd ed.). O’Reilly. https://r4ds.hadley.nz/