install.packages(c('tidytext', 'textdata'))Week 6 Tutorial
Text Data
First things first: Make a new project in RStudio, and be sure to place it somewhere where you can easily find it (e.g., Documents, Desktop). Then download this file (the .qmd file) from the syllabus website and place it in the folder associated with that project. Then open this file in RStudio and follow along.
I encourage you to work with/talk to others throughout this tutorial, especially during times where I have you code on your own and use AI. When you code, think about the SPARCK framework:
S: Set up (What is your end goal?)
P: Pseudocode (What are the steps?)
A: AI (What are your AI prompts?)
R: Run (Run it!)
C: Check (Did your code work?)
K: Knowledge (What does your code mean?)
Agenda
Working with strings
Patterns (regular expressions)
Tidy text
Word frequency
Pairs of words (bigrams)
Sentiment analysis
Practice
Today, you’re going to learn how to work with text data. Most of the data we’ve worked with so far is structured: every column has one kind of information (a number, a category, a date). Text is unstructured: a single paragraph can contain any number of words, in any order. To analyze text, we have to turn it into something we can count, like how long a text is, whether it contains a certain word, or which words show up the most.
If you want to learn more about text analysis in R, a great (free) place to start is Text Mining with R (Silge & Robinson, 2017), which you can find here.
Today’s data contains 17,257 news headlines from the HuffPost (formerly the Huffington Post), published between January 2018 and September 2022 (Misra, 2022). Each row is one article, with its headline, a short description of the article, the news category, and the date.
If you haven’t already, install the packages we’ll need today in the console. (Remember: install packages in the console, not in your Quarto document.)
Download News_Headlines.csv from the course website and put it in your project folder. Then load tidyverse and read in the data.
# Write your code hereWorking with strings
In programming, a piece of text is called a string. The stringr package (which comes with tidyverse) has lots of functions for working with strings, and all of them start with str_. Most of them work the same way: the first argument is the text you want to work with, and the second argument is the pattern you’re looking for.
Replacing text and changing case
Use count() to look at the category column.
# Write your code hereThe categories are in ALL CAPS, and a few of them are the same thing with different names. Use AI to help you do three things in a single mutate(): change “ARTS & CULTURE” to “CULTURE & ARTS” and “PARENTS” to “PARENTING” (using str_replace()), and then change category to title case (using str_to_title()).
# Write your code hereText length
There are two ways to measure the length of a text: the number of characters (str_length()) and the number of words (str_count() with boundary('word')). Use AI to help you create two new columns, headline_chars and headline_words, with the length of each headline in characters and in words. Then make a histogram of headline_words.
# Write your code hereDetecting words
str_detect() tells us whether a text contains a particular word or pattern. It returns TRUE or FALSE for each row. Use str_detect() and sum() to count how many headlines contain “COVID,” and how many contain “Covid.” (Hint: sum() counts the number of TRUEs.)
# Write your code hereR is case sensitive, so “COVID” and “Covid” are different strings. To catch every version of a word, it’s usually a good idea to change your text to lowercase first. You can also look for several words at once using |, which means “or.” Create a new column called headline_lower, which is a lowercase version of headline (using str_to_lower()). Then create a column called mentions_covid that tells us whether each headline contains “covid” or “coronavirus,” and count how many do.
# Write your code hereComparing across years
Make a bar graph showing the percentage of headlines that mention COVID in each year. (Hint: year() will pull the year out of a date. And if you take the mean() of a TRUE/FALSE column, R treats TRUE as 1 and FALSE as 0, so you get the proportion of TRUEs.)
# Write your code hereNow do the same thing for Trump, using SPARCK.
S:
P:
A:
R:
# Write your code hereC:
K:
Other stringr functions
There are many more stringr functions. You don’t need to memorize them, but you should know that they exist, so you can ask AI for them:
| Function | What it does |
|---|---|
str_to_lower(), str_to_upper(), str_to_title() |
Change the case of text |
str_length() |
Number of characters |
str_detect() |
Is a pattern in the text? (TRUE/FALSE) |
str_starts(), str_ends() |
Does the text start or end with a pattern? |
str_count() |
How many times is a pattern in the text? |
str_replace(), str_replace_all() |
Replace a pattern with something else |
str_remove(), str_remove_all() |
Remove a pattern |
str_extract(), str_extract_all() |
Pull out the text that matches a pattern |
str_sub() |
Grab characters by position (e.g., the first 10 characters) |
str_squish() |
Remove extra spaces |
Patterns (regular expressions)
Patterns like 'covid|coronavirus' are called regular expressions (or regex for short). Regular expressions are a language for describing patterns in text (Wickham et al., 2023). You don’t need to memorize them, but you need to know what kinds of patterns are possible, so that you can ask AI for the right one and check whether it does what you want. Here are some of the most useful building blocks:
| Pattern | What it matches | Example |
|---|---|---|
| |
Or | 'covid|coronavirus' matches either word |
[ ] |
Any one of the characters in the brackets | '[0-9]' matches any digit |
^ |
The start of the text | '^The' matches “The” only at the very beginning |
$ |
The end of the text | '\\?$' matches a question mark only at the very end |
\\b |
A word boundary (the start or end of a word) | '\\bice\\b' matches “ice” but not “police” |
\\w |
Any letter or number | '\\w+' matches a whole word |
*, + |
Repeat the thing before it (* = zero or more times, + = one or more times) |
'\\w*ice\\w*' matches “police,” “justice,” “office,” etc. |
. |
Any single character | 'b.t' matches “bat,” “bit,” and “but” |
Because characters like . and ? have special meanings in regex, you need to put two backslashes (\\) in front of them when you’re looking for the actual character.
Let’s try some patterns on a few example phrases. Run the code below to create them.
examples <- c('ICE agents detain immigrants',
'police officers respond to protest',
'Justice Department opens investigation',
'new ice cream flavor',
'Olympic ice dancing team')Use str_detect() to see which of the examples contain each of these patterns: 'ice', '\\bice\\b', and '\\bICE\\b'.
# Write your code hereNow let’s try the same patterns on the real headlines. Use str_detect() and sum() to count how many headlines match each pattern: 'ice' and '\\bice\\b' in headline_lower, and '\\bICE\\b' in the original headline column.
# Write your code herestr_detect() tells us whether a pattern is in a text. str_extract_all() pulls out all of the text that matches it. Use str_extract_all() to find every word in headline_lower that contains “ice,” and count how many times each one appears. (Hint: Use the pattern '\\w*ice\\w*'. str_extract_all() can find more than one match in each headline, so you’ll need to unnest() the result to give each match its own row.)
# Write your code hereFinally, use filter() and str_detect() to keep only the headlines that contain '\\bICE\\b', and look at their headline and category. Then count how many of them are in each category.
# Write your code hereTidy text
So far, we’ve treated each piece of text as one long string. To analyze the words in a collection of texts, we need to break the texts into pieces. For that, we’re going to use a package called tidytext (Silge & Robinson, 2016). Load it in.
# Write your code hereThe big idea of tidytext is the tidy text format: one token per row. A token is a meaningful unit of text, usually a single word. Splitting text into tokens is called tokenizing. Once our data is in this format, every function we already know (filter(), count(), group_by(), ggplot(), etc.) works on text, too.
When we split each article into words, we need a way to keep track of which article each word came from. Use mutate() to add a column called id that gives each article its own ID number. (Hint: row_number() gives you the number of each row.)
# Write your code hereTo tokenize, we use unnest_tokens(). The first argument is the name of the new column you want to create (we’ll call it word), and the second argument is the column with the text you want to split up. Tokenize the short_description column, and save the result as a new object called news_words. (Hint: First use select() to keep only the columns we’ll need: id, headline, category, year, and short_description.)
# Write your code hereEach row of news_words is now one word from one article’s description, and the id column tells us which article it came from. unnest_tokens() also makes all of the words lowercase and removes punctuation.
Word frequency
Stop words
Use count() to find the most common words in all of the descriptions.
# Write your code hereBoring! The most common words in almost any English text are words like “the,” “and,” and “of.” These are called stop words. tidytext comes with a list of stop words called stop_words, which you can load with data(stop_words). Load it in, and use anti_join() to remove all of the stop words from news_words. (Remember: anti_join() keeps the rows in the first data frame that don’t have a match in the second.) Then, in the same step, use filter() and str_detect() to also remove any “words” that contain a number. (Hint: Remember the [0-9] pattern? And remember that ! means “not.”)
# Write your code hereNow make a horizontal bar graph of the 20 most common words, with the bars in order.
# Write your code hereComparing categories
Now let’s compare the most common words in different categories, using SPARCK.
S:
P:
A:
R:
# Write your code hereC:
K:
Comparing years
Make the same kind of graph, but compare the 10 most common words in 2018, 2020, and 2022. (Hint: You can reuse your code from above. Just change the column and the values you filter on.)
# Write your code herePairs of words (bigrams)
So far, our tokens have been single words. Tokens can also be n-grams: sequences of n words in a row. Two-word sequences are called bigrams.
Use unnest_tokens() to split the headline column into bigrams, and save it as a new object called news_bigrams. Then count the most common bigrams. (Hint: unnest_tokens() has an argument called token, which you can set to 'ngrams', and an argument called n, which sets how many words are in each token.)
# Write your code hereLots of the most common bigrams include stop words, like “of the” and “in the.” To remove them, we need to split each bigram into two columns, one for each word. Use AI to help you split bigram into two columns called word1 and word2, and remove any bigram where either word is a stop word. Then put the two words back together in a column called bigram. (Hint: separate_wider_delim() can split a column wherever there’s a space. To check whether a word is a stop word, you can use %in% stop_words$word. To put the words back together, use paste().)
# Write your code hereNow make a bar graph of the 10 most common bigrams in 2018, 2020, and 2022, with a separate panel for each year. (Hint: This is just like the graph you made for words by year!)
# Write your code hereSentiment analysis
Sentiment analysis measures how positive or negative the language in a text is. The simplest kind uses a dictionary (also called a lexicon): a list of words that someone has labeled as positive or negative. To score a text, you look up each of its words in the dictionary, and add up the positive and negative words.
tidytext comes with a few different sentiment dictionaries. We’ll use one called “bing” (Hu & Liu, 2004). Use get_sentiments() to load it, and count how many positive and negative words it has.
# Write your code hereTo find the sentiment of each word in our descriptions, we join the dictionary to our words. Use inner_join() to join bing to news_words, and save it as a new object called news_sentiment. (Remember: inner_join() only keeps rows that have a match in both data frames, so any word that isn’t in the dictionary gets dropped.) Then calculate what percentage of the words in news_words got a sentiment score.
# Write your code hereMake a bar graph of the 15 most common positive words and the 15 most common negative words, with a separate panel for positive and negative words. (Hint: This is very similar to the graphs you made for categories and years!)
# Write your code hereThe dictionary labels the word “trump” as positive, “vice” as negative, and “supreme” as positive. In news headlines, these are usually part of names and titles (Donald Trump, vice president, Supreme Court). Use filter() to create a new dictionary called bing_news, which is bing without these three words. Then redo the inner_join() with the new dictionary.
# Write your code hereNow use AI to help you calculate a sentiment score for each article: the number of positive words minus the number of negative words. Then make a horizontal bar graph of the average score in each category, with the bars in order. (Hint: After you count the positive and negative words for each article, pivot_wider() can put them in separate columns, so you can subtract them. Use values_fill = 0 so that articles with no negative words get a 0 instead of NA.)
# Write your code herePractice
Now it’s your turn. Use SPARCK to answer your own question about the HuffPost articles. Your analysis must:
Use at least one
stringrfunction or onetidytextfunction.Include a visualization.
For example, you could track how often a word of your choice appeared in headlines over time, compare the most common bigrams in two different categories, compare how often headlines in different categories end with a question mark (str_ends()), or try a different sentiment dictionary, like “afinn” (Nielsen, 2011), which scores words from -5 to +5, or “nrc” (Mohammad & Turney, 2013), which labels words with emotions like fear, anger, and trust. (The first time you load “afinn” or “nrc,” R will ask you in the console whether you want to download it. Type 1 and press enter.) Show us something interesting!
S:
P:
A:
R:
# Write your code hereC:
K:
References
Hu, M., & Liu, B. (2004). Mining and summarizing customer reviews. Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 168–177. https://doi.org/10.1145/1014052.1014073
Misra, R. (2022). News category dataset. arXiv. https://doi.org/10.48550/arXiv.2209.11429
Mohammad, S. M., & Turney, P. D. (2013). Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3), 436–465. https://doi.org/10.1111/j.1467-8640.2012.00460.x
Nielsen, F. Å. (2011). A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. Proceedings of the ESWC2011 Workshop on “Making Sense of Microposts,” 93–98. https://arxiv.org/abs/1103.2903
Silge, J., & Robinson, D. (2016). tidytext: Text mining and analysis using tidy data principles in R. Journal of Open Source Software, 1(3), 37. https://doi.org/10.21105/joss.00037
Silge, J., & Robinson, D. (2017). Text mining with R: A tidy approach. O’Reilly. https://www.tidytextmining.com/
Wickham, H., Çetinkaya-Rundel, M., & Grolemund, G. (2023). R for data science (2nd ed.). O’Reilly. https://r4ds.hadley.nz/