Understanding Text Differences by Category Using BERTopic: A Case Study on Donald Trump’s Twitter…

Understanding Text Differences by Category Using BERTopic: A Case Study on Donald Trump’s Twitter Conversations

Introduction

Former U.S. President Donald Trump was shot while campaigning in Butler, Pennsylvania, on Saturday (July 13), local time. The Republican presidential candidate was seen bleeding from his right ear as he was surrounded by security officers who escorted him off the stage.

This incident sparked extensive discussions on social media platform Twitter/X, making it the number one trending topic. With a vast number of conversations on Twitter, this article delves into the use of BERTopic for analyzing and categorizing textual data. By focusing on a case study of Donald Trump’s tweets, we aim to showcase how BERTopic can identify distinct topics and themes in political discussions. Donald Trump’s tweets, known for their impact and high engagement, offer valuable data for this analysis.

However, if we segment the discussions by different user groups, we might discover more detailed insights. For instance, 34.8% of tweets from users in Pennsylvania might focus on the shooting incident, while only 3.2% of tweets from users in California do. This discrepancy indicates a need to investigate why the topic resonates more with certain demographics or regions.

Therefore, this article not only explains how to build a topic model using BERTopic but also demonstrates how to compare topics across different categories. By the end, we will generate insightful graphs for each topic, illustrating the diverse perspectives and engagement levels across various segments.

Twitter Data Collection

The data for this analysis comes from Twitter, specifically focusing on tweets related to Donald Trump. Given the vast number of conversations on this platform, Twitter provides a rich source of real-time, unstructured text data that reflects public opinion and engagement on various topics. For this case study, we collected tweets over a specified period, capturing a wide range of discussions surrounding Donald Trump, particularly following the recent incident in Butler, Pennsylvania.

For this time crawling twitter data using the Tweet Harvest Library. The following is the use of the library:

https://medium.com/media/d880ef18d0b68fc9f3a4e49a34323c4d/href

Before starting the text analysis, it’s important to get an overview of our data. In total, we have 10,000 tweets. The tweets were collected using the keywords “Trump” and “Donald Trump” over the period from July 1, 2024, to July 14, 2024. You can download the data in this link: Donald Trump Twitter (kaggle.com)

BERTopic

We now possess data and can utilize our new sophisticated tool Topic Modeling to extract insights from it. As previously mentioned, we will implement Topic Modelling along with the robust and user-friendly BERTopic package (documentation) for analyzing this text.

You may be curious about what Topic Modelling entails. It is a machine learning technique that falls under the realm of Natural Language Processing. This technique enables the discovery of concealed semantic patterns in texts (often referred to as documents) and the allocation of “topics” to them. There is no requirement to have a predefined list of topics. The algorithm will automatically identify them — typically in the shape of a collection of the most significant words (tokens) or N-grams.

BERTopic is a tool for Topic Modelling that utilizes HuggingFace transformers and class-based TF-IDF. BERTopic is an incredibly adaptable modular package, allowing you to customize it according to your requirements.

If you want to understand how it works better, I advise you to watch this video from the author of the library.

https://medium.com/media/2c8f366f26d5a76255ecc3a1acd0eaa8/href

Preprocessing

According to the documentation, we typically don’t need to preprocess data unless there is a lot of noise, for example, HTML tags or other markdowns that don’t add meaning to the documents. It’s a significant advantage of BERTopic because, for many NLP methods, there is a lot of boilerplate to preprocess your data. If you are interested in how it could look like, see this guide for Topic Modelling using LDA.

You can use BERTopic with data in multiple languages specifying BERTopic(language= "multilingual"). However, from my experience, the model works a bit better with texts translated into one language. So, I will translate all comments into English.

For translation, we will use deep-translator package (you can install it from PyPI).

Also, it could be interesting to see distribution by languages, for that we could use langdetect package.

from langdetect import detect
from deep_translator import GoogleTranslator

def detect_language(text):
    try:
        return detect(text)
    except KeyboardInterrupt as e:
        raise e
    except Exception:
        return '<-- ERROR -->'

def translate_text(text):
    try:
        return GoogleTranslator(source='auto', target='en').translate(str(text))
    except KeyboardInterrupt as e:
        raise e
    except Exception:
        return '<-- ERROR -->'

df['language'] = df['full_text'].apply(detect_language)
df['tweets_transl'] = df['full_text'].apply(translate_text)

In our case, 99+% of comments are already in English. Because we are crawling the data only English tweets.

To understand our data better, let’s look at the distribution of tweets length. It shows that there are a lot of extremely short (and most likely not meaningful comments) — around 5% of reviews are less than 20 symbols.

We can look at the most common examples to ensure that there’s not much information in such comments.

The simpliest topic model

We shall now proceed to construct our inaugural topic model. Let us initiate the process with a straightforward approach, beginning with the fundamental concepts to comprehend the functionality of the library, following which enhancements will be made. A topic model can be trained using a concise set of code lines, comprehensible to individuals accustomed to utilizing at least one machine learning package previously.

from bertopic import BERTopic
docs = list(df.full_text.values)
topic_model = BERTopic()
topics, probs = topic_model.fit_transform(docs)

The default model generated 171 topics. We can look at top topics.

topic_model.get_topic_info().head(7).set_index('Topic')[
   ['Count', 'Name', 'Representation']]

Topic -1 is the largest group, corresponding to outliers. HDBSCAN is the default clustering method in BERTopic, allowing data points to not be forced into clusters. In our dataset, 3817 tweets are outliers, representing about 40% of all tweets. This group accounts for almost half of our data, so we will address it later.

A topic representation typically consists of key words that are specific to that topic. To comprehend a topic better, it is advisable to focus on the primary terms. BERTopic utilizes a class-based TF-IDF score to prioritize and rank words.

topic_model.visualize_barchart(top_n_topics = 16, n_words = 10)

Yet, we completed the initial phase in under 10 lines of code. It is remarkable, but there is still some space for enhancement.

Dealing with the outliers

As we saw earlier, almost 50% of data points are considered outliers. It’s quite a lot, let’s see what we could do with it.

The documentation provides four different strategies to deal with the outliers:

based on topic-document probabilities,
based on topic distributions,
based on c-TF-IFD representations,
based on document and topic embeddings.

You can try different strategies and see which one fits your data the best.

BERTopic applies clustering for topic definition, ensuring that only one topic is attributed to each document. Typically, texts can encompass a blend of various topics in practical scenarios. The challenge arises when documents contain multiple topics, making it difficult to assign a specific one to them.

Luckily, there’s a solution for it — use Topic Distributions. This method involves breaking each document into tokens, then creating subsentences using a sliding window and stride approach, and assigning a topic to each subsentence. Let’s give this method a try and see if it helps in decreasing the number of outliers that lack topics.

Let’s try this approach and see whether we will be able to reduce the number of outliers without topics.

Improving the topic model

However, Topic Distributions are based on the fitted topic model, so let’s enhance it.

First of all, we can use CountVectorizer. It defines how a document will be split into tokens. Also, it can help us to get rid of meaningless words like to, not or the (there are a lot of such words in our first model).

Also, we could improve topics’ representations and even try a couple of different models. I used the KeyBERTInspired model (more details), but you could try other options (for example, LLMs).

from sklearn.feature_extraction.text import CountVectorizer
from bertopic import BERTopic
from bertopic.representation import KeyBERTInspired, PartOfSpeech, MaximalMarginalRelevance

# Define representation models
main_representation_model = KeyBERTInspired()
aspect_representation_model1 = PartOfSpeech("en_core_web_sm")
aspect_representation_model2 = [
    KeyBERTInspired(top_n_words=30),
    MaximalMarginalRelevance(diversity=0.5)
]

# Combine representation models into a dictionary
representation_model = {
    "Main": main_representation_model,
    "Aspect1": aspect_representation_model1,
    "Aspect2": aspect_representation_model2 
}

# Define the vectorizer model
vectorizer_model = CountVectorizer(min_df=5, stop_words='english')

# Initialize the topic model
topic_model = BERTopic(
    nr_topics='auto',
    vectorizer_model=vectorizer_model,
    representation_model=representation_model
)

# Fit and transform the documents
topics, ini_probs = topic_model.fit_transform(docs)

I specified nr_topics = 'auto' to reduce the number of topics. Then, all topics with a similarity over threshold will be merged automatically. With this feature, we got 125 topics.

I’ve created a function to get top topics and their shares so we could analyse it easier. Let’s look at the new set of topics

def get_topic_stats(topic_model, extra_cols = []):
    topics_info_df = topic_model.get_topic_info().sort_values('Count', ascending = False)
    topics_info_df['Share'] = 100.*topics_info_df['Count']/topics_info_df['Count'].sum()
    topics_info_df['CumulativeShare'] = 100.*topics_info_df['Count'].cumsum()/topics_info_df['Count'].sum()
    return topics_info_df[['Topic', 'Count', 'Share', 'CumulativeShare', 
                           'Name', 'Representation'] + extra_cols]

get_topic_stats(topic_model, ['Aspect1', 'Aspect2']).head(10)\
    .set_index('Topic')

We can also look at the Interoptic distance map to better understand our clusters, for example, which are close to each other. You can also use it to define some parent topics and subtopics. It’s called Hierarchical Topic Modelling and you can use other tools for it.

topic_model.visualize_topics()

Another insightful way to better understand your topics is to look at visualize_documents graph (documentation).

We can see that the number of topics has reduced significantly. Also, there are no meaningless stop words in topics’ representations.

Reducing the number of topics

However, we still see similar topics in the results. We can investigate and merge such topics manually.

For this, we can draw a Similarity matrix. I specified n_clusters, and our topics were clustered to visualise them better.

topic_model.visualize_heatmap(n_clusters = 20)

There are some pretty close topics. Let’s calculate the pair distances and look at the top topics.

import numpy as np
import pandas as pd
from sklearn.metrics.pairwise import cosine_similarity

# Calculate the cosine similarity distance matrix
distance_matrix = cosine_similarity(np.array(topic_model.topic_embeddings_))

# Create a DataFrame for the distance matrix
dist_df = pd.DataFrame(
    distance_matrix, 
    columns=topic_model.topic_labels_.values(), 
    index=topic_model.topic_labels_.values()
)

# Convert the distance matrix into a list of dictionaries
tmp = []
dist_dict = dist_df.reset_index().to_dict('records')
for rec in dist_dict:
    t1 = rec['index']
    for t2, distance in rec.items():
        if t2 != 'index':
            tmp.append({'topic1': t1, 'topic2': t2, 'distance': distance})

# Create a DataFrame from the list of dictionaries
pair_dist_df = pd.DataFrame(tmp)

# Filter out the invalid topics and redundant pairs
pair_dist_df = pair_dist_df[
    (pair_dist_df.topic1.map(lambda x: not x.startswith('-1'))) & 
    (pair_dist_df.topic2.map(lambda x: not x.startswith('-1')))
]
pair_dist_df = pair_dist_df[pair_dist_df.topic1 < pair_dist_df.topic2]

# Sort the DataFrame by distance and display the top 20 pairs
top_pairs = pair_dist_df.sort_values('distance', ascending=False).head(20)
print(top_pairs)

I found guidance on how to get the distance matrix from GitHub discussions. We can now see the top pairs of topics by cosine similarity. There are topics with close meanings

With real-life tasks, it’s worth spending more time on merging topics and trying different approaches to representation and clustering to get the best results.

The other potential idea is splitting reviews into separate sentences because comments are rather long.

Topic Distribution

Let’s calculate topics’ and tokens’ distributions. I’ve used a window equal to 4 (the author advised using 4–8 tokens) and stride equal 1.

topic_distr, topic_token_distr = topic_model.approximate_distribution(
      docs, window = 4, calculate_tokens=True)

For example, this comment will be split into subsentences (or sets of four tokens), and the closest of existing topics will be assigned to each. Then, these topics will be aggregated to calculate probabilities for the whole sentence. You can find more details in the documentation.

image from Topics per Class Using BERTopic. How to understand the differences in… | by Mariya Mansurova | Towards Data Science

Using this data, we can get the probabilities of different topics for each review.

topic_model.visualize_distribution(topic_distr[doc_id], min_probability=0.05)

We can even see the distribution of terms for each topic and understand why we got this result. For our sentence, we can look at the terms associated with each topic and analyze how they contribute to the overall probability distribution. By examining the key terms, we can gain insights into the context and themes that are most relevant to our data. This helps in interpreting the model’s output and understanding the underlying topics in our dataset.

vis_df = topic_model.visualize_approximate_distribution(docs[doc_id], 
  topic_token_distr[doc_id])
vis_df

This example also shows that we might have spent more time merging topics because there are still pretty similar ones.

Now, we have probabilities for each topic and review. The next task is to select a threshold to filter irrelevant topics with too low probability.

We can do it as usual using data. Let’s calculate the distribution of selected topics per review for different threshold levels.

tmp_dfs = []

# iterating through different threshold levels
for thr in tqdm.tqdm(np.arange(0, 0.35, 0.001)):
    # calculating number of topics with probability > threshold for each document
    tmp_df = pd.DataFrame(list(map(lambda x: len(list(filter(lambda y: y >= thr, x))), topic_distr))).rename(
        columns = {0: 'num_topics'}
    )
    tmp_df['num_docs'] = 1

    tmp_df['num_topics_group'] = tmp_df['num_topics']\
        .map(lambda x: str(x) if x < 5 else '5+')

    # aggregating stats
    tmp_df_aggr = tmp_df.groupby('num_topics_group', as_index = False).num_docs.sum()
    tmp_df_aggr['threshold'] = thr

    tmp_dfs.append(tmp_df_aggr)

num_topics_stats_df = pd.concat(tmp_dfs).pivot(index = 'threshold', 
                              values = 'num_docs',
                              columns = 'num_topics_group').fillna(0)

num_topics_stats_df = num_topics_stats_df.apply(lambda x: 100.*x/num_topics_stats_df.sum(axis = 1))

# visualisation
colormap = px.colors.sequential.YlGnBu
px.area(num_topics_stats_df, 
       title = 'Distribution of number of topics',
       labels = {'num_topics_group': 'number of topics',
                'value': 'share of tweets, %'},
       color_discrete_map = {
          '0': colormap[0],
          '1': colormap[3],
          '2': colormap[4],
          '3': colormap[5],
          '4': colormap[6],
          '5+': colormap[7]
      })

threshold = 0.05 looks like a good candidate because, with this level, the share of tweets without any topic is still low enough (less than 6%), while the percentage of comments with 4+ topics is also not so high.

This approach has helped us to reduce the number of outliers from 40% to 2%. So, assigning multiple topics could be an effective way to handle outliers.

Let’s look at the topic that twitter users are talking about, the 1st topic. To see it, you can enter the following code:

topic_model.get_topic(0)

# the ouput
[('shooting', 0.42058077),
 ('gunfire', 0.39199635),
 ('shooter', 0.3826013),
 ('assassination', 0.37523785),
 ('gunman', 0.37245923),
 ('trump', 0.3536642),
 ('pennsylvania', 0.34694427),
 ('shooters', 0.3438816),
 ('pa', 0.3433311),
 ('suspect', 0.3360313)]

On this topic, it appears that issues related to shootings and gun violence are being discussed by Twitter users. Words such as shooting, gunfire, shooter, and assassination suggest that these discussions may centre on the latest violent incident experienced by the former American president.

topic_model.get_topic(1)

[('biden', 0.67931235),
 ('vice', 0.5260574),
 ('vicepresident', 0.51568705),
 ('bidens', 0.5026252),
 ('presidential', 0.4854238),
 ('donald', 0.46988484),
 ('trump', 0.4616424),
 ('president', 0.43506712),
 ('vp', 0.42558372),
 ('presidency', 0.4070525)]

As with the second topic, it can be seen that the current president, Joe Biden, is the most discussed topic on twitter. The high weight of words such as biden and vice suggests that the conversation may focus on recent actions, policies, or news involving Joe Biden in his role as President or Vice President. Discussions of presidential, president, and presidency suggest that the topic being discussed is closely related to issues of the presidency and high politics.

Summary

Today, we’ve done an end-to-end Topic Modelling analysis:

Build a basic topic model using the BERTopic library.
Then, we’ve handled outliers, so only 5.8% of our reviews don’t have a topic assigned.
Reduced the number of topics both automatically and manually to have a concise list.
Learned how to assign multiple topics to each document because, in most cases, your text will have a mixture of topics.

Finally, we were able to compare reviews for different courses, create inspiring graphs and get some insights.

Next article we will discuss about topic modelling representation using LLM. See you at the next meeting

If you want to dive deeper into BERTopic

Article “Interactive Topic Modelling with BERTopic” by Maarten Grootendorst (BERTopic author)
Article “Topic Modelling with BERT” by Maarten Grootendorst
Paper “BERTopic: Neural topic modeling with a class-based TF-IDF procedure” by Maarten Grootendorst

from langdetect import detect
from deep_translator import GoogleTranslator

def detect_language(text):
try:
return detect(text)
except KeyboardInterrupt as e:
raise e
except Exception:
return '<-- ERROR -->'

def translate_text(text):
try:
return GoogleTranslator(source='auto', target='en').translate(str(text))
except KeyboardInterrupt as e:
raise e
except Exception:
return '<-- ERROR -->'

df['language'] = df['full_text'].apply(detect_language)
df['tweets_transl'] = df['full_text'].apply(translate_text)

from sklearn.feature_extraction.text import CountVectorizer
from bertopic import BERTopic
from bertopic.representation import KeyBERTInspired, PartOfSpeech, MaximalMarginalRelevance

# Define representation models
main_representation_model = KeyBERTInspired()
aspect_representation_model1 = PartOfSpeech("en_core_web_sm")
aspect_representation_model2 = [
KeyBERTInspired(top_n_words=30),
MaximalMarginalRelevance(diversity=0.5)
]

# Combine representation models into a dictionary
representation_model = {
"Main": main_representation_model,
"Aspect1": aspect_representation_model1,
"Aspect2": aspect_representation_model2
}

# Define the vectorizer model
vectorizer_model = CountVectorizer(min_df=5, stop_words='english')

# Initialize the topic model
topic_model = BERTopic(
nr_topics='auto',
vectorizer_model=vectorizer_model,
representation_model=representation_model
)

# Fit and transform the documents
topics, ini_probs = topic_model.fit_transform(docs)

def get_topic_stats(topic_model, extra_cols = []):
topics_info_df = topic_model.get_topic_info().sort_values('Count', ascending = False)
topics_info_df['Share'] = 100.*topics_info_df['Count']/topics_info_df['Count'].sum()
topics_info_df['CumulativeShare'] = 100.*topics_info_df['Count'].cumsum()/topics_info_df['Count'].sum()
return topics_info_df[['Topic', 'Count', 'Share', 'CumulativeShare',
'Name', 'Representation'] + extra_cols]

get_topic_stats(topic_model, ['Aspect1', 'Aspect2']).head(10)\
.set_index('Topic')

import numpy as np
import pandas as pd
from sklearn.metrics.pairwise import cosine_similarity

# Calculate the cosine similarity distance matrix
distance_matrix = cosine_similarity(np.array(topic_model.topic_embeddings_))

# Create a DataFrame for the distance matrix
dist_df = pd.DataFrame(
distance_matrix,
columns=topic_model.topic_labels_.values(),
index=topic_model.topic_labels_.values()
)

# Convert the distance matrix into a list of dictionaries
tmp = []
dist_dict = dist_df.reset_index().to_dict('records')
for rec in dist_dict:
t1 = rec['index']
for t2, distance in rec.items():
if t2 != 'index':
tmp.append({'topic1': t1, 'topic2': t2, 'distance': distance})

# Create a DataFrame from the list of dictionaries
pair_dist_df = pd.DataFrame(tmp)

# Filter out the invalid topics and redundant pairs
pair_dist_df = pair_dist_df[
(pair_dist_df.topic1.map(lambda x: not x.startswith('-1'))) &
(pair_dist_df.topic2.map(lambda x: not x.startswith('-1')))
]
pair_dist_df = pair_dist_df[pair_dist_df.topic1 < pair_dist_df.topic2]

# Sort the DataFrame by distance and display the top 20 pairs
top_pairs = pair_dist_df.sort_values('distance', ascending=False).head(20)
print(top_pairs)

tmp_dfs = []

# iterating through different threshold levels
for thr in tqdm.tqdm(np.arange(0, 0.35, 0.001)):
# calculating number of topics with probability > threshold for each document
tmp_df = pd.DataFrame(list(map(lambda x: len(list(filter(lambda y: y >= thr, x))), topic_distr))).rename(
columns = {0: 'num_topics'}
)
tmp_df['num_docs'] = 1

tmp_df['num_topics_group'] = tmp_df['num_topics']\
.map(lambda x: str(x) if x < 5 else '5+')

# aggregating stats
tmp_df_aggr = tmp_df.groupby('num_topics_group', as_index = False).num_docs.sum()
tmp_df_aggr['threshold'] = thr

tmp_dfs.append(tmp_df_aggr)

num_topics_stats_df = pd.concat(tmp_dfs).pivot(index = 'threshold',
values = 'num_docs',
columns = 'num_topics_group').fillna(0)

num_topics_stats_df = num_topics_stats_df.apply(lambda x: 100.*x/num_topics_stats_df.sum(axis = 1))

# visualisation
colormap = px.colors.sequential.YlGnBu
px.area(num_topics_stats_df,
title = 'Distribution of number of topics',
labels = {'num_topics_group': 'number of topics',
'value': 'share of tweets, %'},
color_discrete_map = {
'0': colormap[0],
'1': colormap[3],
'2': colormap[4],
'3': colormap[5],
'4': colormap[6],
'5+': colormap[7]
})

topic_model.get_topic(0)

# the ouput
[('shooting', 0.42058077),
('gunfire', 0.39199635),
('shooter', 0.3826013),
('assassination', 0.37523785),
('gunman', 0.37245923),
('trump', 0.3536642),
('pennsylvania', 0.34694427),
('shooters', 0.3438816),
('pa', 0.3433311),
('suspect', 0.3360313)]

topic_model.get_topic(1)

[('biden', 0.67931235),
('vice', 0.5260574),
('vicepresident', 0.51568705),
('bidens', 0.5026252),
('presidential', 0.4854238),
('donald', 0.46988484),
('trump', 0.4616424),
('president', 0.43506712),
('vp', 0.42558372),
('presidency', 0.4070525)]