--- title: "Week 6 Tutorial" subtitle: "Text Data" format: html: toc: true toc-location: left toc-depth: 3 highlight-style: github code-copy: true embed-resources: true df-print: paged include-in-header: text: | editor: visual --- First things first: Make a new project in RStudio, *and be sure to place it somewhere where you can easily find it* (e.g., Documents, Desktop). Then download this file (the `.qmd` file) from the syllabus website and place it in the folder associated with that project. Then open this file in RStudio and follow along. I encourage you to work with/talk to others throughout this tutorial, especially during times where I have you code on your own and use AI. When you code, think about the SPARCK framework: **S: Set up** (What is your end goal?) **P: Pseudocode** (What are the steps?) **A: AI** (What are your AI prompts?) **R: Run** (Run it!) **C: Check** (Did your code work?) **K: Knowledge** (What does your code mean?) ## Agenda - Working with strings - Patterns (regular expressions) - Tidy text - Word frequency - Pairs of words (bigrams) - Sentiment analysis - Practice Today, you're going to learn how to work with **text data**. Most of the data we've worked with so far is **structured**: every column has one kind of information (a number, a category, a date). Text is **unstructured**: a single paragraph can contain any number of words, in any order. To analyze text, we have to turn it into something we can count, like how long a text is, whether it contains a certain word, or which words show up the most. If you want to learn more about text analysis in R, a great (free) place to start is *Text Mining with R* (Silge & Robinson, 2017), which you can find [here](https://www.tidytextmining.com/). Today's data contains 17,257 news headlines from the *HuffPost* (formerly the *Huffington Post*), published between January 2018 and September 2022 (Misra, 2022). Each row is one article, with its headline, a short description of the article, the news category, and the date. If you haven't already, install the packages we'll need today in the console. (Remember: install packages in the console, not in your Quarto document.) ```{r} #| eval: false install.packages(c('tidytext', 'textdata')) ``` Download `News_Headlines.csv` from the course website and put it in your project folder. Then load `tidyverse` and read in the data. ```{r} library(tidyverse) news <- read_csv('News_Headlines.csv') ``` ## Working with strings In programming, a piece of text is called a **string**. The `stringr` package (which comes with `tidyverse`) has lots of functions for working with strings, and all of them start with `str_`. Most of them work the same way: the first argument is the text you want to work with, and the second argument is the pattern you're looking for. ### Replacing text and changing case Use `count()` to look at the `category` column. ```{r} news |> count(category) ``` The categories are in ALL CAPS, and a few of them are the same thing with different names. Use AI to help you do three things in a single `mutate()`: change "ARTS & CULTURE" to "CULTURE & ARTS" and "PARENTS" to "PARENTING" (using `str_replace()`), and then change `category` to title case (using `str_to_title()`). ```{r} news <- news |> mutate(category = str_replace(category, 'ARTS & CULTURE', 'CULTURE & ARTS'), category = str_replace(category, 'PARENTS', 'PARENTING'), category = str_to_title(category)) ``` ### Text length There are two ways to measure the length of a text: the number of **characters** (`str_length()`) and the number of **words** (`str_count()` with `boundary('word')`). Use AI to help you create two new columns, `headline_chars` and `headline_words`, with the length of each headline in characters and in words. Then make a histogram of `headline_words`. ```{r} news <- news |> mutate(headline_chars = str_length(headline), headline_words = str_count(headline, boundary('word'))) ggplot(data = news, aes(x = headline_words)) + geom_histogram(binwidth = 1) ``` ### Detecting words `str_detect()` tells us whether a text contains a particular word or pattern. It returns `TRUE` or `FALSE` for each row. Use `str_detect()` and `sum()` to count how many headlines contain "COVID," and how many contain "Covid." (Hint: `sum()` counts the number of `TRUE`s.) ```{r} sum(str_detect(news$headline, 'COVID')) sum(str_detect(news$headline, 'Covid')) ``` R is **case sensitive**, so "COVID" and "Covid" are different strings. To catch every version of a word, it's usually a good idea to change your text to lowercase first. You can also look for several words at once using `|`, which means "or." Create a new column called `headline_lower`, which is a lowercase version of `headline` (using `str_to_lower()`). Then create a column called `mentions_covid` that tells us whether each headline contains "covid" **or** "coronavirus," and count how many do. ```{r} news <- news |> mutate(headline_lower = str_to_lower(headline), mentions_covid = str_detect(headline_lower, 'covid|coronavirus')) news |> count(mentions_covid) ``` ### Comparing across years Make a bar graph showing the **percentage** of headlines that mention COVID in each year. (Hint: `year()` will pull the year out of a date. And if you take the `mean()` of a `TRUE`/`FALSE` column, R treats `TRUE` as 1 and `FALSE` as 0, so you get the proportion of `TRUE`s.) ```{r} news <- news |> mutate(year = year(date)) covid_by_year <- news |> group_by(year) |> summarize(pct_covid = mean(mentions_covid) * 100) ggplot(data = covid_by_year, aes(x = year, y = pct_covid)) + geom_col() + labs(x = NULL, y = 'Percent of headlines that mention COVID') ``` Now do the same thing for Trump, using SPARCK. **S:** Make a bar graph showing the percentage of headlines that mention Trump in each year, from 2018 to 2022. **P:** Create a new column that tells us whether each headline mentions Trump. Group by year. Calculate the percentage of headlines in each year that mention Trump. Make a bar graph. **A:** I have a data frame in R called `news`. It has a column called `headline_lower` with lowercase headlines, and a column called `year`. Using tidyverse, create a new column called `mentions_trump` that is TRUE if `headline_lower` contains "trump". Then calculate the percentage of rows where `mentions_trump` is TRUE, separately for each year, and make a bar graph of those percentages with ggplot. **R:** ```{r} news <- news |> mutate(mentions_trump = str_detect(headline_lower, 'trump')) trump_by_year <- news |> group_by(year) |> summarize(pct_trump = mean(mentions_trump) * 100) ggplot(data = trump_by_year, aes(x = year, y = pct_trump)) + geom_col() + labs(x = NULL, y = 'Percent of headlines that mention Trump') ``` **C:** I checked that the percentages are between 0 and 100, and that there's one bar for each year from 2018 to 2022. **K:** `str_detect()` checked whether each lowercase headline contained "trump." `year()` pulled the year out of each date. Taking the `mean()` of `mentions_trump` within each year gave the proportion of headlines that mention Trump, and multiplying by 100 turned it into a percentage. About 18 to 23% of headlines mentioned Trump from 2018 to 2020, which dropped to about 10% in 2021 and 7% in 2022. ### Other `stringr` functions There are many more `stringr` functions. You don't need to memorize them, but you should know that they exist, so you can ask AI for them: | Function | What it does | |------------------------------------|------------------------------------| | `str_to_lower()`, `str_to_upper()`, `str_to_title()` | Change the case of text | | `str_length()` | Number of characters | | `str_detect()` | Is a pattern in the text? (`TRUE`/`FALSE`) | | `str_starts()`, `str_ends()` | Does the text start or end with a pattern? | | `str_count()` | How many times is a pattern in the text? | | `str_replace()`, `str_replace_all()` | Replace a pattern with something else | | `str_remove()`, `str_remove_all()` | Remove a pattern | | `str_extract()`, `str_extract_all()` | Pull out the text that matches a pattern | | `str_sub()` | Grab characters by position (e.g., the first 10 characters) | | `str_squish()` | Remove extra spaces | ## Patterns (regular expressions) Patterns like `'covid|coronavirus'` are called **regular expressions** (or **regex** for short). Regular expressions are a language for describing patterns in text (Wickham et al., 2023). You don't need to memorize them, but you need to know what kinds of patterns are possible, so that you can ask AI for the right one and check whether it does what you want. Here are some of the most useful building blocks: | Pattern | What it matches | Example | |------------------------|------------------------|------------------------| | `|` | Or | `'covid|coronavirus'` matches either word | | `[ ]` | Any *one* of the characters in the brackets | `'[0-9]'` matches any digit | | `^` | The start of the text | `'^The'` matches "The" only at the very beginning | | `$` | The end of the text | `'\\?$'` matches a question mark only at the very end | | `\\b` | A word boundary (the start or end of a word) | `'\\bice\\b'` matches "ice" but not "police" | | `\\w` | Any letter or number | `'\\w+'` matches a whole word | | `*`, `+` | Repeat the thing before it (`*` = zero or more times, `+` = one or more times) | `'\\w*ice\\w*'` matches "police," "justice," "office," etc. | | `.` | Any single character | `'b.t'` matches "bat," "bit," and "but" | Because characters like `.` and `?` have special meanings in regex, you need to put two backslashes (`\\`) in front of them when you're looking for the actual character. Let's try some patterns on a few example phrases. Run the code below to create them. ```{r} examples <- c('ICE agents detain immigrants', 'police officers respond to protest', 'Justice Department opens investigation', 'new ice cream flavor', 'Olympic ice dancing team') ``` Use `str_detect()` to see which of the examples contain each of these patterns: `'ice'`, `'\\bice\\b'`, and `'\\bICE\\b'`. ```{r} str_detect(examples, 'ice') str_detect(examples, '\\bice\\b') str_detect(examples, '\\bICE\\b') ``` Now let's try the same patterns on the real headlines. Use `str_detect()` and `sum()` to count how many headlines match each pattern: `'ice'` and `'\\bice\\b'` in `headline_lower`, and `'\\bICE\\b'` in the original `headline` column. ```{r} sum(str_detect(news$headline_lower, 'ice')) sum(str_detect(news$headline_lower, '\\bice\\b')) sum(str_detect(news$headline, '\\bICE\\b')) ``` `str_detect()` tells us *whether* a pattern is in a text. `str_extract_all()` pulls out *all* of the text that matches it. Use `str_extract_all()` to find every word in `headline_lower` that contains "ice," and count how many times each one appears. (Hint: Use the pattern `'\\w*ice\\w*'`. `str_extract_all()` can find more than one match in each headline, so you'll need to `unnest()` the result to give each match its own row.) ```{r} news |> mutate(ice_words = str_extract_all(headline_lower, '\\w*ice\\w*')) |> unnest(ice_words) |> count(ice_words, sort = TRUE) ``` Finally, use `filter()` and `str_detect()` to keep only the headlines that contain `'\\bICE\\b'`, and look at their `headline` and `category`. Then count how many of them are in each category. ```{r} ice_headlines <- news |> filter(str_detect(headline, '\\bICE\\b')) |> select(headline, category) ice_headlines ice_headlines |> count(category, sort = TRUE) ``` ## Tidy text So far, we've treated each piece of text as one long string. To analyze the *words* in a collection of texts, we need to break the texts into pieces. For that, we're going to use a package called `tidytext` (Silge & Robinson, 2016). Load it in. ```{r} library(tidytext) ``` The **big idea** of `tidytext` is the **tidy text format**: one **token** per row. A token is a meaningful unit of text, usually a single word. Splitting text into tokens is called **tokenizing**. Once our data is in this format, every function we already know (`filter()`, `count()`, `group_by()`, `ggplot()`, etc.) works on text, too. When we split each article into words, we need a way to keep track of which article each word came from. Use `mutate()` to add a column called `id` that gives each article its own ID number. (Hint: `row_number()` gives you the number of each row.) ```{r} news <- news |> mutate(id = row_number()) ``` To tokenize, we use `unnest_tokens()`. The first argument is the name of the new column you want to create (we'll call it `word`), and the second argument is the column with the text you want to split up. Tokenize the `short_description` column, and save the result as a new object called `news_words`. (Hint: First use `select()` to keep only the columns we'll need: `id`, `headline`, `category`, `year`, and `short_description`.) ```{r} news_words <- news |> select(id, headline, category, year, short_description) |> unnest_tokens(word, short_description) ``` Each row of `news_words` is now one word from one article's description, and the `id` column tells us which article it came from. `unnest_tokens()` also makes all of the words lowercase and removes punctuation. ## Word frequency ### Stop words Use `count()` to find the most common words in all of the descriptions. ```{r} news_words |> count(word, sort = TRUE) ``` Boring! The most common words in almost any English text are words like "the," "and," and "of." These are called **stop words**. `tidytext` comes with a list of stop words called `stop_words`, which you can load with `data(stop_words)`. Load it in, and use `anti_join()` to remove all of the stop words from `news_words`. (Remember: `anti_join()` keeps the rows in the first data frame that **don't** have a match in the second.) Then, in the same step, use `filter()` and `str_detect()` to also remove any "words" that contain a number. (Hint: Remember the `[0-9]` pattern? And remember that `!` means "not.") ```{r} data(stop_words) news_words <- news_words |> anti_join(stop_words) |> filter(!str_detect(word, '[0-9]')) ``` Now make a horizontal bar graph of the 20 most common words, with the bars in order. ```{r} news_words |> count(word, sort = TRUE) |> slice_max(n, n = 20) |> ggplot(aes(x = n, y = reorder(word, n))) + geom_col() + labs(x = 'Number of times used', y = NULL) ``` ### Comparing categories Now let's compare the most common words in different categories, using SPARCK. **S:** Make a bar graph of the 10 most common words in the Crime, Politics, and Entertainment categories, with a separate panel for each category. **P:** Keep only the words from the three categories. Count how many times each word appears in each category. Keep the top 10 words in each category. Make a bar graph with a separate panel for each category. **A:** I have a data frame in R called `news_words`, with one row per word. It has a column called `word`, and a column called `category`. Using tidyverse and tidytext, keep only the rows where `category` is "Crime", "Politics", or "Entertainment". Count how many times each word appears in each category, and keep the top 10 words in each category. Then make a horizontal bar graph with ggplot, with a separate panel for each category, and with the bars in order within each panel using `reorder_within()` and `scale_y_reordered()`. **R:** ```{r} news_words |> filter(category %in% c('Crime', 'Politics', 'Entertainment')) |> count(category, word) |> group_by(category) |> slice_max(n, n = 10, with_ties = FALSE) |> ungroup() |> ggplot(aes(x = n, y = reorder_within(word, n, category))) + geom_col() + scale_y_reordered() + facet_wrap(~ category, scales = 'free') + labs(x = 'Number of times used', y = NULL) ``` **C:** I checked that there are three panels with 10 bars each, and that the bars are in order within each panel. I also checked the counts for a few words against a `count()` table, to make sure they matched. **K:** `filter()` kept only the words from the three categories. `count()` counted how many times each word appears in each category. `group_by()` and `slice_max()` kept the top 10 words *in each category*, instead of the top 10 overall. `facet_wrap()` made a separate panel for each category, and `reorder_within()` and `scale_y_reordered()` put the bars in order within each panel. (Without them, the bars would only be in order across all three panels combined.) ### Comparing years Make the same kind of graph, but compare the 10 most common words in 2018, 2020, and 2022. (Hint: You can reuse your code from above. Just change the column and the values you filter on.) ```{r} news_words |> filter(year %in% c(2018, 2020, 2022)) |> count(year, word) |> group_by(year) |> slice_max(n, n = 10, with_ties = FALSE) |> ungroup() |> ggplot(aes(x = n, y = reorder_within(word, n, year))) + geom_col() + scale_y_reordered() + facet_wrap(~ year, scales = 'free') + labs(x = 'Number of times used', y = NULL) ``` ## Pairs of words (bigrams) So far, our tokens have been single words. Tokens can also be **n-grams**: sequences of *n* words in a row. Two-word sequences are called **bigrams**. Use `unnest_tokens()` to split the `headline` column into bigrams, and save it as a new object called `news_bigrams`. Then count the most common bigrams. (Hint: `unnest_tokens()` has an argument called `token`, which you can set to `'ngrams'`, and an argument called `n`, which sets how many words are in each token.) ```{r} news_bigrams <- news |> select(id, category, year, headline) |> unnest_tokens(bigram, headline, token = 'ngrams', n = 2) news_bigrams |> count(bigram, sort = TRUE) ``` Lots of the most common bigrams include stop words, like "of the" and "in the." To remove them, we need to split each bigram into two columns, one for each word. Use AI to help you split `bigram` into two columns called `word1` and `word2`, and remove any bigram where *either* word is a stop word. Then put the two words back together in a column called `bigram`. (Hint: `separate_wider_delim()` can split a column wherever there's a space. To check whether a word is a stop word, you can use `%in% stop_words$word`. To put the words back together, use `paste()`.) ```{r} news_bigrams <- news_bigrams |> separate_wider_delim(bigram, delim = ' ', names = c('word1', 'word2')) |> filter(!word1 %in% stop_words$word, !word2 %in% stop_words$word) |> mutate(bigram = paste(word1, word2)) ``` Now make a bar graph of the 10 most common bigrams in 2018, 2020, and 2022, with a separate panel for each year. (Hint: This is just like the graph you made for words by year!) ```{r} news_bigrams |> filter(year %in% c(2018, 2020, 2022)) |> count(year, bigram) |> group_by(year) |> slice_max(n, n = 10, with_ties = FALSE) |> ungroup() |> ggplot(aes(x = n, y = reorder_within(bigram, n, year))) + geom_col() + scale_y_reordered() + facet_wrap(~ year, scales = 'free') + labs(x = 'Number of times used', y = NULL) ``` ## Sentiment analysis **Sentiment analysis** measures how positive or negative the language in a text is. The simplest kind uses a **dictionary** (also called a **lexicon**): a list of words that someone has labeled as positive or negative. To score a text, you look up each of its words in the dictionary, and add up the positive and negative words. `tidytext` comes with a few different sentiment dictionaries. We'll use one called "bing" (Hu & Liu, 2004). Use `get_sentiments()` to load it, and count how many positive and negative words it has. ```{r} bing <- get_sentiments('bing') bing |> count(sentiment) ``` To find the sentiment of each word in our descriptions, we **join** the dictionary to our words. Use `inner_join()` to join `bing` to `news_words`, and save it as a new object called `news_sentiment`. (Remember: `inner_join()` only keeps rows that have a match in *both* data frames, so any word that isn't in the dictionary gets dropped.) Then calculate what percentage of the words in `news_words` got a sentiment score. ```{r} news_sentiment <- news_words |> inner_join(bing) nrow(news_sentiment) / nrow(news_words) * 100 ``` Make a bar graph of the 15 most common positive words and the 15 most common negative words, with a separate panel for positive and negative words. (Hint: This is very similar to the graphs you made for categories and years!) ```{r} news_sentiment |> count(sentiment, word) |> group_by(sentiment) |> slice_max(n, n = 15, with_ties = FALSE) |> ungroup() |> ggplot(aes(x = n, y = reorder_within(word, n, sentiment), fill = sentiment)) + geom_col(show.legend = FALSE) + scale_y_reordered() + facet_wrap(~ sentiment, scales = 'free') + labs(x = 'Number of times used', y = NULL) ``` The dictionary labels the word "trump" as positive, "vice" as negative, and "supreme" as positive. In news headlines, these are usually part of names and titles (Donald Trump, vice president, Supreme Court). Use `filter()` to create a new dictionary called `bing_news`, which is `bing` without these three words. Then redo the `inner_join()` with the new dictionary. ```{r} bing_news <- bing |> filter(!word %in% c('trump', 'vice', 'supreme')) news_sentiment <- news_words |> inner_join(bing_news) ``` Now use AI to help you calculate a sentiment score for each article: the number of positive words minus the number of negative words. Then make a horizontal bar graph of the average score in each category, with the bars in order. (Hint: After you count the positive and negative words for each article, `pivot_wider()` can put them in separate columns, so you can subtract them. Use `values_fill = 0` so that articles with no negative words get a 0 instead of `NA`.) ```{r} article_sentiment <- news_sentiment |> count(id, headline, category, sentiment) |> pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) |> mutate(net_sentiment = positive - negative) article_sentiment |> group_by(category) |> summarize(avg_sentiment = mean(net_sentiment)) |> ggplot(aes(x = avg_sentiment, y = reorder(category, avg_sentiment))) + geom_col() + labs(x = 'Average sentiment (positive words minus negative words)', y = NULL) ``` ## Practice Now it's your turn. Use SPARCK to answer your own question about the HuffPost articles. Your analysis must: - Use at least one `stringr` function **or** one `tidytext` function. - Include a visualization. For example, you could track how often a word of your choice appeared in headlines over time, compare the most common bigrams in two different categories, compare how often headlines in different categories end with a question mark (`str_ends()`), or try a different sentiment dictionary, like "afinn" (Nielsen, 2011), which scores words from -5 to +5, or "nrc" (Mohammad & Turney, 2013), which labels words with emotions like fear, anger, and trust. (The first time you load "afinn" or "nrc," R will ask you in the console whether you want to download it. Type `1` and press enter.) **Show us something interesting!** **S:** **P:** **A:** **R:** ```{r} ``` **C:** **K:** ## References Hu, M., & Liu, B. (2004). Mining and summarizing customer reviews. *Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, 168–177. Misra, R. (2022). *News category dataset*. arXiv. Mohammad, S. M., & Turney, P. D. (2013). Crowdsourcing a word-emotion association lexicon. *Computational Intelligence, 29*(3), 436–465. Nielsen, F. Å. (2011). A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. *Proceedings of the ESWC2011 Workshop on "Making Sense of Microposts,"* 93–98. Silge, J., & Robinson, D. (2016). tidytext: Text mining and analysis using tidy data principles in R. *Journal of Open Source Software, 1*(3), 37. Silge, J., & Robinson, D. (2017). *Text mining with R: A tidy approach*. O'Reilly. Wickham, H., Çetinkaya-Rundel, M., & Grolemund, G. (2023). *R for data science* (2nd ed.). O'Reilly.