---
title: "Week 6 Tutorial"
subtitle: "Text Data"
format:
  html:
    toc: true
    toc-location: left
    toc-depth: 3
    highlight-style: github
    code-copy: true
    embed-resources: true
    df-print: paged
    include-in-header:
      text: |
        <style>
          .cell-output pre { max-height: 250px; overflow-y: auto; }
        </style>
editor: visual
---

First things first: Make a new project in RStudio, *and be sure to place it somewhere where you can easily find it* (e.g., Documents, Desktop). Then download this file (the `.qmd` file) from the syllabus website and place it in the folder associated with that project. Then open this file in RStudio and follow along.

I encourage you to work with/talk to others throughout this tutorial, especially during times where I have you code on your own and use AI. When you code, think about the SPARCK framework:

**S: Set up** (What is your end goal?)

**P: Pseudocode** (What are the steps?)

**A: AI** (What are your AI prompts?)

**R: Run** (Run it!)

**C: Check** (Did your code work?)

**K: Knowledge** (What does your code mean?)

## Agenda

- Working with strings

- Patterns (regular expressions)

- Tidy text

- Word frequency

- Pairs of words (bigrams)

- Sentiment analysis

- Practice

Today, you're going to learn how to work with **text data**. Most of the data we've worked with so far is **structured**: every column has one kind of information (a number, a category, a date). Text is **unstructured**: a single paragraph can contain any number of words, in any order. To analyze text, we have to turn it into something we can count, like how long a text is, whether it contains a certain word, or which words show up the most.

If you want to learn more about text analysis in R, a great (free) place to start is *Text Mining with R* (Silge & Robinson, 2017), which you can find [here](https://www.tidytextmining.com/).

Today's data contains 17,257 news headlines from the *HuffPost* (formerly the *Huffington Post*), published between January 2018 and September 2022 (Misra, 2022). Each row is one article, with its headline, a short description of the article, the news category, and the date.

If you haven't already, install the packages we'll need today in the console. (Remember: install packages in the console, not in your Quarto document.)

```{r}
#| eval: false
install.packages(c('tidytext', 'textdata'))
```

Download `News_Headlines.csv` from the course website and put it in your project folder. Then load `tidyverse` and read in the data.

```{r}
# Write your code here
```

## Working with strings

In programming, a piece of text is called a **string**. The `stringr` package (which comes with `tidyverse`) has lots of functions for working with strings, and all of them start with `str_`. Most of them work the same way: the first argument is the text you want to work with, and the second argument is the pattern you're looking for.

### Replacing text and changing case

Use `count()` to look at the `category` column.

```{r}
# Write your code here
```

The categories are in ALL CAPS, and a few of them are the same thing with different names. Use AI to help you do three things in a single `mutate()`: change "ARTS & CULTURE" to "CULTURE & ARTS" and "PARENTS" to "PARENTING" (using `str_replace()`), and then change `category` to title case (using `str_to_title()`).

```{r}
# Write your code here
```

### Text length

There are two ways to measure the length of a text: the number of **characters** (`str_length()`) and the number of **words** (`str_count()` with `boundary('word')`). Use AI to help you create two new columns, `headline_chars` and `headline_words`, with the length of each headline in characters and in words. Then make a histogram of `headline_words`.

```{r}
# Write your code here
```

### Detecting words

`str_detect()` tells us whether a text contains a particular word or pattern. It returns `TRUE` or `FALSE` for each row. Use `str_detect()` and `sum()` to count how many headlines contain "COVID," and how many contain "Covid." (Hint: `sum()` counts the number of `TRUE`s.)

```{r}
# Write your code here
```

R is **case sensitive**, so "COVID" and "Covid" are different strings. To catch every version of a word, it's usually a good idea to change your text to lowercase first. You can also look for several words at once using `|`, which means "or." Create a new column called `headline_lower`, which is a lowercase version of `headline` (using `str_to_lower()`). Then create a column called `mentions_covid` that tells us whether each headline contains "covid" **or** "coronavirus," and count how many do.

```{r}
# Write your code here
```

### Comparing across years

Make a bar graph showing the **percentage** of headlines that mention COVID in each year. (Hint: `year()` will pull the year out of a date. And if you take the `mean()` of a `TRUE`/`FALSE` column, R treats `TRUE` as 1 and `FALSE` as 0, so you get the proportion of `TRUE`s.)

```{r}
# Write your code here
```

Now do the same thing for Trump, using SPARCK.

**S:**

**P:**

**A:**

**R:**

```{r}
# Write your code here
```

**C:**

**K:**

### Other `stringr` functions

There are many more `stringr` functions. You don't need to memorize them, but you should know that they exist, so you can ask AI for them:

| Function | What it does |
|------------------------------------|------------------------------------|
| `str_to_lower()`, `str_to_upper()`, `str_to_title()` | Change the case of text |
| `str_length()` | Number of characters |
| `str_detect()` | Is a pattern in the text? (`TRUE`/`FALSE`) |
| `str_starts()`, `str_ends()` | Does the text start or end with a pattern? |
| `str_count()` | How many times is a pattern in the text? |
| `str_replace()`, `str_replace_all()` | Replace a pattern with something else |
| `str_remove()`, `str_remove_all()` | Remove a pattern |
| `str_extract()`, `str_extract_all()` | Pull out the text that matches a pattern |
| `str_sub()` | Grab characters by position (e.g., the first 10 characters) |
| `str_squish()` | Remove extra spaces |

## Patterns (regular expressions)

Patterns like `'covid|coronavirus'` are called **regular expressions** (or **regex** for short). Regular expressions are a language for describing patterns in text (Wickham et al., 2023). You don't need to memorize them, but you need to know what kinds of patterns are possible, so that you can ask AI for the right one and check whether it does what you want. Here are some of the most useful building blocks:

| Pattern | What it matches | Example |
|------------------------|------------------------|------------------------|
| `|` | Or | `'covid|coronavirus'` matches either word |
| `[ ]` | Any *one* of the characters in the brackets | `'[0-9]'` matches any digit |
| `^` | The start of the text | `'^The'` matches "The" only at the very beginning |
| `$` | The end of the text | `'\\?$'` matches a question mark only at the very end |
| `\\b` | A word boundary (the start or end of a word) | `'\\bice\\b'` matches "ice" but not "police" |
| `\\w` | Any letter or number | `'\\w+'` matches a whole word |
| `*`, `+` | Repeat the thing before it (`*` = zero or more times, `+` = one or more times) | `'\\w*ice\\w*'` matches "police," "justice," "office," etc. |
| `.` | Any single character | `'b.t'` matches "bat," "bit," and "but" |

Because characters like `.` and `?` have special meanings in regex, you need to put two backslashes (`\\`) in front of them when you're looking for the actual character.

Let's try some patterns on a few example phrases. Run the code below to create them.

```{r}
examples <- c('ICE agents detain immigrants',
              'police officers respond to protest',
              'Justice Department opens investigation',
              'new ice cream flavor',
              'Olympic ice dancing team')
```

Use `str_detect()` to see which of the examples contain each of these patterns: `'ice'`, `'\\bice\\b'`, and `'\\bICE\\b'`.

```{r}
# Write your code here
```

Now let's try the same patterns on the real headlines. Use `str_detect()` and `sum()` to count how many headlines match each pattern: `'ice'` and `'\\bice\\b'` in `headline_lower`, and `'\\bICE\\b'` in the original `headline` column.

```{r}
# Write your code here
```

`str_detect()` tells us *whether* a pattern is in a text. `str_extract_all()` pulls out *all* of the text that matches it. Use `str_extract_all()` to find every word in `headline_lower` that contains "ice," and count how many times each one appears. (Hint: Use the pattern `'\\w*ice\\w*'`. `str_extract_all()` can find more than one match in each headline, so you'll need to `unnest()` the result to give each match its own row.)

```{r}
# Write your code here
```

Finally, use `filter()` and `str_detect()` to keep only the headlines that contain `'\\bICE\\b'`, and look at their `headline` and `category`. Then count how many of them are in each category.

```{r}
# Write your code here
```

## Tidy text

So far, we've treated each piece of text as one long string. To analyze the *words* in a collection of texts, we need to break the texts into pieces. For that, we're going to use a package called `tidytext` (Silge & Robinson, 2016). Load it in.

```{r}
# Write your code here
```

The **big idea** of `tidytext` is the **tidy text format**: one **token** per row. A token is a meaningful unit of text, usually a single word. Splitting text into tokens is called **tokenizing**. Once our data is in this format, every function we already know (`filter()`, `count()`, `group_by()`, `ggplot()`, etc.) works on text, too.

When we split each article into words, we need a way to keep track of which article each word came from. Use `mutate()` to add a column called `id` that gives each article its own ID number. (Hint: `row_number()` gives you the number of each row.)

```{r}
# Write your code here
```

To tokenize, we use `unnest_tokens()`. The first argument is the name of the new column you want to create (we'll call it `word`), and the second argument is the column with the text you want to split up. Tokenize the `short_description` column, and save the result as a new object called `news_words`. (Hint: First use `select()` to keep only the columns we'll need: `id`, `headline`, `category`, `year`, and `short_description`.)

```{r}
# Write your code here
```

Each row of `news_words` is now one word from one article's description, and the `id` column tells us which article it came from. `unnest_tokens()` also makes all of the words lowercase and removes punctuation.

## Word frequency

### Stop words

Use `count()` to find the most common words in all of the descriptions.

```{r}
# Write your code here
```

Boring! The most common words in almost any English text are words like "the," "and," and "of." These are called **stop words**. `tidytext` comes with a list of stop words called `stop_words`, which you can load with `data(stop_words)`. Load it in, and use `anti_join()` to remove all of the stop words from `news_words`. (Remember: `anti_join()` keeps the rows in the first data frame that **don't** have a match in the second.) Then, in the same step, use `filter()` and `str_detect()` to also remove any "words" that contain a number. (Hint: Remember the `[0-9]` pattern? And remember that `!` means "not.")

```{r}
# Write your code here
```

Now make a horizontal bar graph of the 20 most common words, with the bars in order.

```{r}
# Write your code here
```

### Comparing categories

Now let's compare the most common words in different categories, using SPARCK.

**S:**

**P:**

**A:**

**R:**

```{r}
# Write your code here
```

**C:**

**K:**

### Comparing years

Make the same kind of graph, but compare the 10 most common words in 2018, 2020, and 2022. (Hint: You can reuse your code from above. Just change the column and the values you filter on.)

```{r}
# Write your code here
```

## Pairs of words (bigrams)

So far, our tokens have been single words. Tokens can also be **n-grams**: sequences of *n* words in a row. Two-word sequences are called **bigrams**.

Use `unnest_tokens()` to split the `headline` column into bigrams, and save it as a new object called `news_bigrams`. Then count the most common bigrams. (Hint: `unnest_tokens()` has an argument called `token`, which you can set to `'ngrams'`, and an argument called `n`, which sets how many words are in each token.)

```{r}
# Write your code here
```

Lots of the most common bigrams include stop words, like "of the" and "in the." To remove them, we need to split each bigram into two columns, one for each word. Use AI to help you split `bigram` into two columns called `word1` and `word2`, and remove any bigram where *either* word is a stop word. Then put the two words back together in a column called `bigram`. (Hint: `separate_wider_delim()` can split a column wherever there's a space. To check whether a word is a stop word, you can use `%in% stop_words$word`. To put the words back together, use `paste()`.)

```{r}
# Write your code here
```

Now make a bar graph of the 10 most common bigrams in 2018, 2020, and 2022, with a separate panel for each year. (Hint: This is just like the graph you made for words by year!)

```{r}
# Write your code here
```

## Sentiment analysis

**Sentiment analysis** measures how positive or negative the language in a text is. The simplest kind uses a **dictionary** (also called a **lexicon**): a list of words that someone has labeled as positive or negative. To score a text, you look up each of its words in the dictionary, and add up the positive and negative words.

`tidytext` comes with a few different sentiment dictionaries. We'll use one called "bing" (Hu & Liu, 2004). Use `get_sentiments()` to load it, and count how many positive and negative words it has.

```{r}
# Write your code here
```

To find the sentiment of each word in our descriptions, we **join** the dictionary to our words. Use `inner_join()` to join `bing` to `news_words`, and save it as a new object called `news_sentiment`. (Remember: `inner_join()` only keeps rows that have a match in *both* data frames, so any word that isn't in the dictionary gets dropped.) Then calculate what percentage of the words in `news_words` got a sentiment score.

```{r}
# Write your code here
```

Make a bar graph of the 15 most common positive words and the 15 most common negative words, with a separate panel for positive and negative words. (Hint: This is very similar to the graphs you made for categories and years!)

```{r}
# Write your code here
```

The dictionary labels the word "trump" as positive, "vice" as negative, and "supreme" as positive. In news headlines, these are usually part of names and titles (Donald Trump, vice president, Supreme Court). Use `filter()` to create a new dictionary called `bing_news`, which is `bing` without these three words. Then redo the `inner_join()` with the new dictionary.

```{r}
# Write your code here
```

Now use AI to help you calculate a sentiment score for each article: the number of positive words minus the number of negative words. Then make a horizontal bar graph of the average score in each category, with the bars in order. (Hint: After you count the positive and negative words for each article, `pivot_wider()` can put them in separate columns, so you can subtract them. Use `values_fill = 0` so that articles with no negative words get a 0 instead of `NA`.)

```{r}
# Write your code here
```

## Practice

Now it's your turn. Use SPARCK to answer your own question about the HuffPost articles. Your analysis must:

- Use at least one `stringr` function **or** one `tidytext` function.

- Include a visualization.

For example, you could track how often a word of your choice appeared in headlines over time, compare the most common bigrams in two different categories, compare how often headlines in different categories end with a question mark (`str_ends()`), or try a different sentiment dictionary, like "afinn" (Nielsen, 2011), which scores words from -5 to +5, or "nrc" (Mohammad & Turney, 2013), which labels words with emotions like fear, anger, and trust. (The first time you load "afinn" or "nrc," R will ask you in the console whether you want to download it. Type `1` and press enter.) **Show us something interesting!**

**S:**

**P:**

**A:**

**R:**

```{r}
# Write your code here
```

**C:**

**K:**

## References

Hu, M., & Liu, B. (2004). Mining and summarizing customer reviews. *Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, 168–177. <https://doi.org/10.1145/1014052.1014073>

Misra, R. (2022). *News category dataset*. arXiv. <https://doi.org/10.48550/arXiv.2209.11429>

Mohammad, S. M., & Turney, P. D. (2013). Crowdsourcing a word-emotion association lexicon. *Computational Intelligence, 29*(3), 436–465. <https://doi.org/10.1111/j.1467-8640.2012.00460.x>

Nielsen, F. Å. (2011). A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. *Proceedings of the ESWC2011 Workshop on "Making Sense of Microposts,"* 93–98. <https://arxiv.org/abs/1103.2903>

Silge, J., & Robinson, D. (2016). tidytext: Text mining and analysis using tidy data principles in R. *Journal of Open Source Software, 1*(3), 37. <https://doi.org/10.21105/joss.00037>

Silge, J., & Robinson, D. (2017). *Text mining with R: A tidy approach*. O'Reilly. <https://www.tidytextmining.com/>

Wickham, H., Çetinkaya-Rundel, M., & Grolemund, G. (2023). *R for data science* (2nd ed.). O'Reilly. <https://r4ds.hadley.nz/>