Week 6 Tutorial

Text Data

First things first: Make a new project in RStudio, and be sure to place it somewhere where you can easily find it (e.g., Documents, Desktop). Then download this file (the .qmd file) from the syllabus website and place it in the folder associated with that project. Then open this file in RStudio and follow along.

I encourage you to work with/talk to others throughout this tutorial, especially during times where I have you code on your own and use AI. When you code, think about the SPARCK framework:

S: Set up (What is your end goal?)

P: Pseudocode (What are the steps?)

A: AI (What are your AI prompts?)

R: Run (Run it!)

C: Check (Did your code work?)

K: Knowledge (What does your code mean?)

Agenda

• Working with strings

• Patterns (regular expressions)

• Tidy text

• Word frequency

• Pairs of words (bigrams)

• Sentiment analysis

• Practice

Today, you’re going to learn how to work with text data. Most of the data we’ve worked with so far is structured: every column has one kind of information (a number, a category, a date). Text is unstructured: a single paragraph can contain any number of words, in any order. To analyze text, we have to turn it into something we can count, like how long a text is, whether it contains a certain word, or which words show up the most.

If you want to learn more about text analysis in R, a great (free) place to start is Text Mining with R (Silge & Robinson, 2017), which you can find here.

Today’s data contains 17,257 news headlines from the HuffPost (formerly the Huffington Post), published between January 2018 and September 2022 (Misra, 2022). Each row is one article, with its headline, a short description of the article, the news category, and the date.

If you haven’t already, install the packages we’ll need today in the console. (Remember: install packages in the console, not in your Quarto document.)

install.packages(c('tidytext', 'textdata'))

Download News_Headlines.csv from the course website and put it in your project folder. Then load tidyverse and read in the data.

library(tidyverse)

Warning: package 'ggplot2' was built under R version 4.4.3

Warning: package 'tidyr' was built under R version 4.4.3

Warning: package 'purrr' was built under R version 4.4.3

Warning: package 'dplyr' was built under R version 4.4.3

-- Attaching core tidyverse packages ------------------------ tidyverse 2.0.0 --
v dplyr     1.2.0     v readr     2.1.5
v forcats   1.0.0     v stringr   1.5.1
v ggplot2   4.0.2     v tibble    3.2.1
v lubridate 1.9.4     v tidyr     1.3.2
v purrr     1.2.1     
-- Conflicts ------------------------------------------ tidyverse_conflicts() --
x dplyr::filter() masks stats::filter()
x dplyr::lag()    masks stats::lag()
i Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

news <- read_csv('News_Headlines.csv')

Rows: 17257 Columns: 6
-- Column specification --------------------------------------------------------
Delimiter: ","
chr  (5): link, headline, category, short_description, authors
date (1): date

i Use `spec()` to retrieve the full column specification for this data.
i Specify the column types or set `show_col_types = FALSE` to quiet this message.

Working with strings

In programming, a piece of text is called a string. The stringr package (which comes with tidyverse) has lots of functions for working with strings, and all of them start with str_. Most of them work the same way: the first argument is the text you want to work with, and the second argument is the pattern you’re looking for.

Replacing text and changing case

Use count() to look at the category column.

news |>
  count(category)

The categories are in ALL CAPS, and a few of them are the same thing with different names. Use AI to help you do three things in a single mutate(): change “ARTS & CULTURE” to “CULTURE & ARTS” and “PARENTS” to “PARENTING” (using str_replace()), and then change category to title case (using str_to_title()).

news <- news |>
  mutate(category = str_replace(category, 'ARTS & CULTURE', 'CULTURE & ARTS'),
         category = str_replace(category, 'PARENTS', 'PARENTING'),
         category = str_to_title(category))

Text length

There are two ways to measure the length of a text: the number of characters (str_length()) and the number of words (str_count() with boundary('word')). Use AI to help you create two new columns, headline_chars and headline_words, with the length of each headline in characters and in words. Then make a histogram of headline_words.

news <- news |>
  mutate(headline_chars = str_length(headline),
         headline_words = str_count(headline, boundary('word')))

ggplot(data = news, aes(x = headline_words)) +
  geom_histogram(binwidth = 1)

Detecting words

str_detect() tells us whether a text contains a particular word or pattern. It returns TRUE or FALSE for each row. Use str_detect() and sum() to count how many headlines contain “COVID,” and how many contain “Covid.” (Hint: sum() counts the number of TRUEs.)

sum(str_detect(news$headline, 'COVID'))

[1] 298

sum(str_detect(news$headline, 'Covid'))

[1] 3

R is case sensitive, so “COVID” and “Covid” are different strings. To catch every version of a word, it’s usually a good idea to change your text to lowercase first. You can also look for several words at once using |, which means “or.” Create a new column called headline_lower, which is a lowercase version of headline (using str_to_lower()). Then create a column called mentions_covid that tells us whether each headline contains “covid” or “coronavirus,” and count how many do.

news <- news |>
  mutate(headline_lower = str_to_lower(headline),
         mentions_covid = str_detect(headline_lower, 'covid|coronavirus'))

news |>
  count(mentions_covid)

Comparing across years

Make a bar graph showing the percentage of headlines that mention COVID in each year. (Hint: year() will pull the year out of a date. And if you take the mean() of a TRUE/FALSE column, R treats TRUE as 1 and FALSE as 0, so you get the proportion of TRUEs.)

news <- news |>
  mutate(year = year(date))

covid_by_year <- news |>
  group_by(year) |>
  summarize(pct_covid = mean(mentions_covid) * 100)

ggplot(data = covid_by_year, aes(x = year, y = pct_covid)) +
  geom_col() +
  labs(x = NULL, y = 'Percent of headlines that mention COVID')

Now do the same thing for Trump, using SPARCK.

S: Make a bar graph showing the percentage of headlines that mention Trump in each year, from 2018 to 2022.

P: Create a new column that tells us whether each headline mentions Trump. Group by year. Calculate the percentage of headlines in each year that mention Trump. Make a bar graph.

A: I have a data frame in R called news. It has a column called headline_lower with lowercase headlines, and a column called year. Using tidyverse, create a new column called mentions_trump that is TRUE if headline_lower contains “trump”. Then calculate the percentage of rows where mentions_trump is TRUE, separately for each year, and make a bar graph of those percentages with ggplot.

R:

news <- news |>
  mutate(mentions_trump = str_detect(headline_lower, 'trump'))

trump_by_year <- news |>
  group_by(year) |>
  summarize(pct_trump = mean(mentions_trump) * 100)

ggplot(data = trump_by_year, aes(x = year, y = pct_trump)) +
  geom_col() +
  labs(x = NULL, y = 'Percent of headlines that mention Trump')

C: I checked that the percentages are between 0 and 100, and that there’s one bar for each year from 2018 to 2022.

K: str_detect() checked whether each lowercase headline contained “trump.” year() pulled the year out of each date. Taking the mean() of mentions_trump within each year gave the proportion of headlines that mention Trump, and multiplying by 100 turned it into a percentage. About 18 to 23% of headlines mentioned Trump from 2018 to 2020, which dropped to about 10% in 2021 and 7% in 2022.

Other stringr functions

There are many more stringr functions. You don’t need to memorize them, but you should know that they exist, so you can ask AI for them:

Function	What it does
str_to_lower(), str_to_upper(), str_to_title()	Change the case of text
str_length()	Number of characters
str_detect()	Is a pattern in the text? (TRUE/FALSE)
str_starts(), str_ends()	Does the text start or end with a pattern?
str_count()	How many times is a pattern in the text?
str_replace(), str_replace_all()	Replace a pattern with something else
str_remove(), str_remove_all()	Remove a pattern
str_extract(), str_extract_all()	Pull out the text that matches a pattern
str_sub()	Grab characters by position (e.g., the first 10 characters)
str_squish()	Remove extra spaces

Patterns (regular expressions)

Patterns like 'covid|coronavirus' are called regular expressions (or regex for short). Regular expressions are a language for describing patterns in text (Wickham et al., 2023). You don’t need to memorize them, but you need to know what kinds of patterns are possible, so that you can ask AI for the right one and check whether it does what you want. Here are some of the most useful building blocks:

Pattern	What it matches	Example
|	Or	'covid|coronavirus' matches either word
[ ]	Any one of the characters in the brackets	'[0-9]' matches any digit
^	The start of the text	'^The' matches “The” only at the very beginning
$	The end of the text	'\\?$' matches a question mark only at the very end
\\b	A word boundary (the start or end of a word)	'\\bice\\b' matches “ice” but not “police”
\\w	Any letter or number	'\\w+' matches a whole word
*, +	Repeat the thing before it (* = zero or more times, + = one or more times)	'\\w*ice\\w*' matches “police,” “justice,” “office,” etc.
.	Any single character	'b.t' matches “bat,” “bit,” and “but”

Because characters like . and ? have special meanings in regex, you need to put two backslashes (\\) in front of them when you’re looking for the actual character.

Let’s try some patterns on a few example phrases. Run the code below to create them.

examples <- c('ICE agents detain immigrants',
              'police officers respond to protest',
              'Justice Department opens investigation',
              'new ice cream flavor',
              'Olympic ice dancing team')

Use str_detect() to see which of the examples contain each of these patterns: 'ice', '\\bice\\b', and '\\bICE\\b'.

str_detect(examples, 'ice')

[1] FALSE  TRUE  TRUE  TRUE  TRUE

str_detect(examples, '\\bice\\b')

[1] FALSE FALSE FALSE  TRUE  TRUE

str_detect(examples, '\\bICE\\b')

[1]  TRUE FALSE FALSE FALSE FALSE

Now let’s try the same patterns on the real headlines. Use str_detect() and sum() to count how many headlines match each pattern: 'ice' and '\\bice\\b' in headline_lower, and '\\bICE\\b' in the original headline column.

sum(str_detect(news$headline_lower, 'ice'))

[1] 777

sum(str_detect(news$headline_lower, '\\bice\\b'))

[1] 66

sum(str_detect(news$headline, '\\bICE\\b'))

[1] 37

str_detect() tells us whether a pattern is in a text. str_extract_all() pulls out all of the text that matches it. Use str_extract_all() to find every word in headline_lower that contains “ice,” and count how many times each one appears. (Hint: Use the pattern '\\w*ice\\w*'. str_extract_all() can find more than one match in each headline, so you’ll need to unnest() the result to give each match its own row.)

news |>
  mutate(ice_words = str_extract_all(headline_lower, '\\w*ice\\w*')) |>
  unnest(ice_words) |>
  count(ice_words, sort = TRUE)

Finally, use filter() and str_detect() to keep only the headlines that contain '\\bICE\\b', and look at their headline and category. Then count how many of them are in each category.

ice_headlines <- news |>
  filter(str_detect(headline, '\\bICE\\b')) |>
  select(headline, category)

ice_headlines

ice_headlines |>
  count(category, sort = TRUE)

Tidy text

So far, we’ve treated each piece of text as one long string. To analyze the words in a collection of texts, we need to break the texts into pieces. For that, we’re going to use a package called tidytext (Silge & Robinson, 2016). Load it in.

library(tidytext)

The big idea of tidytext is the tidy text format: one token per row. A token is a meaningful unit of text, usually a single word. Splitting text into tokens is called tokenizing. Once our data is in this format, every function we already know (filter(), count(), group_by(), ggplot(), etc.) works on text, too.

When we split each article into words, we need a way to keep track of which article each word came from. Use mutate() to add a column called id that gives each article its own ID number. (Hint: row_number() gives you the number of each row.)

news <- news |>
  mutate(id = row_number())

To tokenize, we use unnest_tokens(). The first argument is the name of the new column you want to create (we’ll call it word), and the second argument is the column with the text you want to split up. Tokenize the short_description column, and save the result as a new object called news_words. (Hint: First use select() to keep only the columns we’ll need: id, headline, category, year, and short_description.)

news_words <- news |>
  select(id, headline, category, year, short_description) |>
  unnest_tokens(word, short_description)

Each row of news_words is now one word from one article’s description, and the id column tells us which article it came from. unnest_tokens() also makes all of the words lowercase and removes punctuation.

Word frequency

Stop words

Use count() to find the most common words in all of the descriptions.

news_words |>
  count(word, sort = TRUE)

Boring! The most common words in almost any English text are words like “the,” “and,” and “of.” These are called stop words. tidytext comes with a list of stop words called stop_words, which you can load with data(stop_words). Load it in, and use anti_join() to remove all of the stop words from news_words. (Remember: anti_join() keeps the rows in the first data frame that don’t have a match in the second.) Then, in the same step, use filter() and str_detect() to also remove any “words” that contain a number. (Hint: Remember the [0-9] pattern? And remember that ! means “not.”)

data(stop_words)

news_words <- news_words |>
  anti_join(stop_words) |>
  filter(!str_detect(word, '[0-9]'))

Joining with `by = join_by(word)`

Now make a horizontal bar graph of the 20 most common words, with the bars in order.

news_words |>
  count(word, sort = TRUE) |>
  slice_max(n, n = 20) |>
  ggplot(aes(x = n, y = reorder(word, n))) +
  geom_col() +
  labs(x = 'Number of times used', y = NULL)

Comparing categories

Now let’s compare the most common words in different categories, using SPARCK.

S: Make a bar graph of the 10 most common words in the Crime, Politics, and Entertainment categories, with a separate panel for each category.

P: Keep only the words from the three categories. Count how many times each word appears in each category. Keep the top 10 words in each category. Make a bar graph with a separate panel for each category.

A: I have a data frame in R called news_words, with one row per word. It has a column called word, and a column called category. Using tidyverse and tidytext, keep only the rows where category is “Crime”, “Politics”, or “Entertainment”. Count how many times each word appears in each category, and keep the top 10 words in each category. Then make a horizontal bar graph with ggplot, with a separate panel for each category, and with the bars in order within each panel using reorder_within() and scale_y_reordered().

R:

news_words |>
  filter(category %in% c('Crime', 'Politics', 'Entertainment')) |>
  count(category, word) |>
  group_by(category) |>
  slice_max(n, n = 10, with_ties = FALSE) |>
  ungroup() |>
  ggplot(aes(x = n, y = reorder_within(word, n, category))) +
  geom_col() +
  scale_y_reordered() +
  facet_wrap(~ category, scales = 'free') +
  labs(x = 'Number of times used', y = NULL)

C: I checked that there are three panels with 10 bars each, and that the bars are in order within each panel. I also checked the counts for a few words against a count() table, to make sure they matched.

K: filter() kept only the words from the three categories. count() counted how many times each word appears in each category. group_by() and slice_max() kept the top 10 words in each category, instead of the top 10 overall. facet_wrap() made a separate panel for each category, and reorder_within() and scale_y_reordered() put the bars in order within each panel. (Without them, the bars would only be in order across all three panels combined.)

Comparing years

Make the same kind of graph, but compare the 10 most common words in 2018, 2020, and 2022. (Hint: You can reuse your code from above. Just change the column and the values you filter on.)

news_words |>
  filter(year %in% c(2018, 2020, 2022)) |>
  count(year, word) |>
  group_by(year) |>
  slice_max(n, n = 10, with_ties = FALSE) |>
  ungroup() |>
  ggplot(aes(x = n, y = reorder_within(word, n, year))) +
  geom_col() +
  scale_y_reordered() +
  facet_wrap(~ year, scales = 'free') +
  labs(x = 'Number of times used', y = NULL)

Pairs of words (bigrams)

So far, our tokens have been single words. Tokens can also be n-grams: sequences of n words in a row. Two-word sequences are called bigrams.

Use unnest_tokens() to split the headline column into bigrams, and save it as a new object called news_bigrams. Then count the most common bigrams. (Hint: unnest_tokens() has an argument called token, which you can set to 'ngrams', and an argument called n, which sets how many words are in each token.)

news_bigrams <- news |>
  select(id, category, year, headline) |>
  unnest_tokens(bigram, headline, token = 'ngrams', n = 2)

news_bigrams |>
  count(bigram, sort = TRUE)

Lots of the most common bigrams include stop words, like “of the” and “in the.” To remove them, we need to split each bigram into two columns, one for each word. Use AI to help you split bigram into two columns called word1 and word2, and remove any bigram where either word is a stop word. Then put the two words back together in a column called bigram. (Hint: separate_wider_delim() can split a column wherever there’s a space. To check whether a word is a stop word, you can use %in% stop_words$word. To put the words back together, use paste().)

news_bigrams <- news_bigrams |>
  separate_wider_delim(bigram, delim = ' ', names = c('word1', 'word2')) |>
  filter(!word1 %in% stop_words$word,
         !word2 %in% stop_words$word) |>
  mutate(bigram = paste(word1, word2))

Now make a bar graph of the 10 most common bigrams in 2018, 2020, and 2022, with a separate panel for each year. (Hint: This is just like the graph you made for words by year!)

news_bigrams |>
  filter(year %in% c(2018, 2020, 2022)) |>
  count(year, bigram) |>
  group_by(year) |>
  slice_max(n, n = 10, with_ties = FALSE) |>
  ungroup() |>
  ggplot(aes(x = n, y = reorder_within(bigram, n, year))) +
  geom_col() +
  scale_y_reordered() +
  facet_wrap(~ year, scales = 'free') +
  labs(x = 'Number of times used', y = NULL)

Sentiment analysis

Sentiment analysis measures how positive or negative the language in a text is. The simplest kind uses a dictionary (also called a lexicon): a list of words that someone has labeled as positive or negative. To score a text, you look up each of its words in the dictionary, and add up the positive and negative words.

tidytext comes with a few different sentiment dictionaries. We’ll use one called “bing” (Hu & Liu, 2004). Use get_sentiments() to load it, and count how many positive and negative words it has.

bing <- get_sentiments('bing')

bing |>
  count(sentiment)

To find the sentiment of each word in our descriptions, we join the dictionary to our words. Use inner_join() to join bing to news_words, and save it as a new object called news_sentiment. (Remember: inner_join() only keeps rows that have a match in both data frames, so any word that isn’t in the dictionary gets dropped.) Then calculate what percentage of the words in news_words got a sentiment score.

news_sentiment <- news_words |>
  inner_join(bing)

Joining with `by = join_by(word)`

nrow(news_sentiment) / nrow(news_words) * 100

[1] 12.4625

Make a bar graph of the 15 most common positive words and the 15 most common negative words, with a separate panel for positive and negative words. (Hint: This is very similar to the graphs you made for categories and years!)

news_sentiment |>
  count(sentiment, word) |>
  group_by(sentiment) |>
  slice_max(n, n = 15, with_ties = FALSE) |>
  ungroup() |>
  ggplot(aes(x = n, y = reorder_within(word, n, sentiment), fill = sentiment)) +
  geom_col(show.legend = FALSE) +
  scale_y_reordered() +
  facet_wrap(~ sentiment, scales = 'free') +
  labs(x = 'Number of times used', y = NULL)

The dictionary labels the word “trump” as positive, “vice” as negative, and “supreme” as positive. In news headlines, these are usually part of names and titles (Donald Trump, vice president, Supreme Court). Use filter() to create a new dictionary called bing_news, which is bing without these three words. Then redo the inner_join() with the new dictionary.

bing_news <- bing |>
  filter(!word %in% c('trump', 'vice', 'supreme'))

news_sentiment <- news_words |>
  inner_join(bing_news)

Joining with `by = join_by(word)`

Now use AI to help you calculate a sentiment score for each article: the number of positive words minus the number of negative words. Then make a horizontal bar graph of the average score in each category, with the bars in order. (Hint: After you count the positive and negative words for each article, pivot_wider() can put them in separate columns, so you can subtract them. Use values_fill = 0 so that articles with no negative words get a 0 instead of NA.)

article_sentiment <- news_sentiment |>
  count(id, headline, category, sentiment) |>
  pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) |>
  mutate(net_sentiment = positive - negative)

article_sentiment |>
  group_by(category) |>
  summarize(avg_sentiment = mean(net_sentiment)) |>
  ggplot(aes(x = avg_sentiment, y = reorder(category, avg_sentiment))) +
  geom_col() +
  labs(x = 'Average sentiment (positive words minus negative words)', y = NULL)

Practice

Now it’s your turn. Use SPARCK to answer your own question about the HuffPost articles. Your analysis must:

• Use at least one stringr function or one tidytext function.

• Include a visualization.

For example, you could track how often a word of your choice appeared in headlines over time, compare the most common bigrams in two different categories, compare how often headlines in different categories end with a question mark (str_ends()), or try a different sentiment dictionary, like “afinn” (Nielsen, 2011), which scores words from -5 to +5, or “nrc” (Mohammad & Turney, 2013), which labels words with emotions like fear, anger, and trust. (The first time you load “afinn” or “nrc,” R will ask you in the console whether you want to download it. Type 1 and press enter.) Show us something interesting!

S:

P:

A:

R:

C:

K:

References

Hu, M., & Liu, B. (2004). Mining and summarizing customer reviews. Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 168–177. https://doi.org/10.1145/1014052.1014073

Misra, R. (2022). News category dataset. arXiv. https://doi.org/10.48550/arXiv.2209.11429

Mohammad, S. M., & Turney, P. D. (2013). Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3), 436–465. https://doi.org/10.1111/j.1467-8640.2012.00460.x

Nielsen, F. Å. (2011). A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. Proceedings of the ESWC2011 Workshop on “Making Sense of Microposts,” 93–98. https://arxiv.org/abs/1103.2903

Silge, J., & Robinson, D. (2016). tidytext: Text mining and analysis using tidy data principles in R. Journal of Open Source Software, 1(3), 37. https://doi.org/10.21105/joss.00037

Silge, J., & Robinson, D. (2017). Text mining with R: A tidy approach. O’Reilly. https://www.tidytextmining.com/

Wickham, H., Çetinkaya-Rundel, M., & Grolemund, G. (2023). R for data science (2nd ed.). O’Reilly. https://r4ds.hadley.nz/