Text Sentiment Analysis

First edition dust cover jacket of East of Eden (1952) by American author John Steinbeck. Photo retrieved from Wikipedia page.

Overview

In this report, I analyze John Steinbeck’s classic novel, “East of Eden”, to determine the most frequently used words and overall sentiments throughout the book. Figure 1 displays the most frequently used words in each chapter for Chapters 1 - 16, and Figure 2 visualizes a cloud map of most frequenlty used word for just Chapter 1. Figures 3 and 4 portrays the sentiment analysis for Chapters 1-16.

Text Analysis

• Load libraries needed for analysis
• Load in the pdf of “East of Eden”
• Convert the pdf text into a data frame
• Wrangle the data to get tokens into tidy format and remove stop words
• Determine word counts by chapter
• Plot it in a bar graph

### Load Packages

library(tidyverse)
library(tidytext)
library(pdftools)
library(ggwordcloud)
library(textdata)

### Get East of Eden Script loaded in

eoe_text <- pdftools::pdf_text(here::here('posts', '2024-02-10-sentiment-analysis','data', 'East-of-Eden.pdf'))

### Get PDF script text into a data frame

eoe_lines <- data.frame(eoe_text) %>% 
  mutate(page = 1:n()) %>%
  mutate(text_full = str_split(eoe_text, pattern = '\\n')) %>% 
  unnest(text_full) %>% 
  mutate(text_full = str_trim(text_full)) 

eoe_chapts <- eoe_lines %>% 
  slice(-(1:247)) %>% 
  mutate(chapter = ifelse(str_detect(text_full, "Chapter"), text_full, NA)) %>% 
  fill(chapter, .direction = 'down') %>% 
  separate(col = chapter, into = c("ch", "num"), sep = " ") %>% 
  mutate(chapter = as.numeric(as.roman(num)))

eoe_words <- eoe_chapts %>% 
  unnest_tokens(word, text_full) %>% 
  select(-eoe_text)

eoe_wordcount <- eoe_words %>% 
  count(chapter, word)

wordcount_clean <- eoe_wordcount %>% 
  anti_join(stop_words, by = 'word')

top_5_words <- wordcount_clean %>% 
  filter(chapter == 1:16) %>% 
  group_by(chapter) %>% 
  arrange(-n) %>% 
  slice(1:5) %>%
  ungroup()

# Make some graphs: 
ggplot(data = top_5_words, aes(x = n, y = word)) +
  geom_col(fill = "darkgreen") +
  facet_wrap(~chapter, scales = "free")+
  theme_minimal()+
  labs(x = " ", 
       y = " ")

Figure 1: Top 5 Words per Chapter. Figure 1 portrays the top 5 most frequently used words per chapter for chapters 1 through 16. The most commonly used word is ‘Adam’ in Chapter 10.

Word Cloud

Create a visualization for most frequently used words in Chapter 1:

• Filter for only Chapter 1
• Determine the top 100 most used words
• Plot it

ch1_top100 <- wordcount_clean %>% 
  filter(chapter == 1) %>% 
  arrange(-n) %>% 
  slice(1:100)

ch1_cloud <- ggplot(data = ch1_top100, aes(label = word)) +
  geom_text_wordcloud(aes(color = n, size = n), shape = "diamond") +
  scale_size_area(max_size = 6) +
  scale_color_gradientn(colors = c("darkgreen","blue","purple")) +
  theme_minimal()

ch1_cloud

Figure 2: East of Eden Chapter 1 Word Cloud. Fig 2 visualzes the most frequently used words from Chapter 1. As the novel starts out with a thorough description of the setting, the Salinas Valley, there are a lot of words describing the geographical location and physical attributes of Central California.

Sentiment Analysis

Determine the overall sentiment (positive or negative) for each chapter:

• Load in the affin lexicon
• Bind the affin words to EOE words
• Determine positive and negative word counts
• Plot it by chapter
• Consider the log ratios and plot it

afinn_lex <- get_sentiments(lexicon = "afinn")
### you may be prompted to download an updated lexicon - say yes!

# Let's look at the pretty positive words:
afinn_pos <- get_sentiments("afinn") %>% 
  filter(value %in% c(3,4,5))

eoe_afinn <- eoe_words %>% 
  filter(chapter == 1:16) %>% 
  inner_join(afinn_lex, by = 'word') ### why inner_join?

bing_lex <- get_sentiments(lexicon = "bing")

nrc_lex <- get_sentiments(lexicon = "nrc")

eoe_bing <- eoe_words %>% 
  filter(chapter == 1:16) %>% 
  inner_join(bing_lex, by = 'word')

bing_counts <- eoe_bing %>% 
  group_by(chapter, sentiment) %>%
  summarize(n = n())

# Plot them: 
ggplot(data = bing_counts, aes(x = sentiment, y = n, fill = sentiment)) +
  geom_col() +
  facet_wrap(~chapter) + 
  labs(
    x = " ", 
    y = "Count", 
    fill = "Sentiment")+
  scale_fill_manual(values = c("positive" = "slateblue", "negative" = "darkred"))+
  theme_minimal()

Figure 3: East of Eden Sentiment Analysis. Fig 3 portrays the number of words per chapter that are associated with positive or negative connotations. Chapter 3 has the most negative words and Chapter 7 has the most positive words.

# find log ratio score overall:
bing_log_ratio_book <- eoe_bing %>% 
  summarize(n_pos = sum(sentiment == 'positive'),
            n_neg = sum(sentiment == 'negative'),
            log_ratio = log(n_pos / n_neg))

# Find the log ratio score by chapter: 
bing_log_ratio_ch <- eoe_bing %>% 
  group_by(chapter) %>% 
  summarize(n_pos = sum(sentiment == 'positive'),
            n_neg = sum(sentiment == 'negative'),
            log_ratio = log(n_pos / n_neg)) %>%
  mutate(log_ratio_adjust = log_ratio - bing_log_ratio_book$log_ratio) %>%
  mutate(pos_neg = ifelse(log_ratio_adjust > 0, 'pos', 'neg'))

ggplot(data = bing_log_ratio_ch, 
       aes(x = log_ratio_adjust,
           y = fct_rev(factor(chapter)),
           fill = pos_neg)) +
           # y = fct_rev(as.factor(chapter)))) +
  geom_col() +
  labs(x = 'Adjusted log(positive/negative)',
       y = 'Chapter number') +
  scale_fill_manual(values = c('pos' = 'slateblue', 'neg' = 'darkred')) +
  theme_minimal() +
  theme(legend.position = 'none')

Figure 4: Ratio of Sentiment Analysis. Fig 4 portrays the average sentiment for the whole book and compares each chapter to that average. Based on a log scale, a neutral chapter will equal 0 (ratio = 1), and a super positive chapter will show the same length as a super negative chapter but in the opposite directions. Chapter 2 was the most positive, and Chapters 3 and 9 were very negative.

Works Cited

Steinbeck, John. East of Eden. Penguin Classics, 2000. Retreived from: https://hitalki.org/blog/wp-content/uploads/2023/05/East-of-Eden.pdf