How to Use Large Language Model for Topic Modeling: A Comprehensive Guide

In the previous article, I used BERTopic for Topic Modeling. The task was to find out what topics were being talked about on social media twitter about the former president of the United States, Donald Trump. This approach with BERTopic worked out, and we got some insights from the data. For example, in a large dataset of tweets, we can find that there is a lot of talk about the violent incident experienced by the former American president.

As time goes by, the technology used in text analysis is LLMs (Large Language Models).

LLMs changed the process of building Machine Learning (ML) applications. Before LLMs, if we wanted to do sentiment analysis or build a chatbot, we would spend a long time getting the data labeled and doing model training. Then we would apply it in production (That would also take a long time). With LLMs, we can solve those problems in a short time. Here is an illustration that I got when watching a video from Andrew NG.

You can watch that video in here below:

https://medium.com/media/854f97cf5f46a53fe7e7958632688dac/href

With the development of these LLMs, let’s see if they can help us in accomplishing our previous task of finding hidden topics on social media twitter or on a large collection of tweets.

Introduction LLM

Before jumping into our task, let’s discuss the basics of LLMs and how they could be used.

Large language models (LLMs) are AI models that are usually (but not neccessarily) derived from the Transformers architecture and are designed to understand and generate human language, code, and much more. These models are trained on vast amounts of text data, allowing them to capture the complexities and nuances of human language.

In most business applications, we require a specialized model that can address issues, not a general one. Standard LLMs may not be suitable for these tasks as they are designed to forecast the most probable succeeding word. However, online, there is a multitude of texts where the subsequent word does not constitute a correct response, like jokes or a mere series of exam preparation questions.

Consequently, in today’s world, Instruction Tuned LLMs have gained significant popularity in business scenarios. These models, which are essentially LLMs, are fine-tuned on datasets containing instructions and correct responses (such as the OpenOrca dataset). Additionally, the RLHF (Reinforcement Learning with Human Feedback) technique is commonly employed to train these models.

Another key aspect of Instruction Tuned LLMs is their aim to be supportive, truthful, and non-threatening, which is vital for models that will interact with clients (especially those who are vulnerable).

What are the primary tasks for LLMs

Large Language Models (LLMs) are primarily utilized for tasks involving unstructured data, rather than handling tabular data with numerous numerical values. Here are some common applications for textual data:

• Summarization: Providing a brief overview of the text.
• Text Analysis: Performing tasks like sentiment analysis or extracting specific features, such as identifying labels in tweets
• Text Transformations: Converting text to different languages, adjusting tone, or reformatting from HTML to JSON.
• Generation: Creating content from a prompt, responding to customer inquiries, or assisting in brainstorming solutions to problems.

Our task of topic modeling fits well within the realm of text analysis, making LLMs particularly beneficial for this purpose.

Prompt Engineering

To get the answer from LLM, a Prompt is required. A prompt is a set of words or instructions given to the model to generate a response. You can think of a Large Language Model (LLM) as a highly motivated and knowledgeable junior specialist who is eager to assist but requires clear instructions to do so effectively. Therefore, providing a well-crafted prompt is essential.
Based on this course, you can check it yourself, because it’s free. There are a few key principles that you should keep in mind when creating a prompt.

Principal 1: Provide clear and specific instructions

Here are some tips for implementing principal 1.

1. Use delimiters to separate different sections of your prompt, such as dividing steps in the instructions or framing a user message. Common delimiters include: """, --- , ### , <> , or XML tags.
2. Define the format for the output. For example, you could use JSON or HTML and even specify a list of possible values. It will make response parsing much easier for you.
3. Show a couple of input & output examples to the model so it can see what you expect as separate messages. Such an approach is called few-shot prompting
4. Also, it could be helpful to instruct the model to check assumptions and conditions. For example, to ensure that the output format is JSON and returned values are from the specified list.

Principal 2: Push the model to think about the answer

Here are some tips for implementing principal 2

1. Instruct the model to look for solutions before rushing to conclusions
2. Another method is to break your complex task into smaller tasks and use different prompts for each simple step. This approach has several benefits: it makes the code easier to maintain (similar to the difference between spaghetti code and modular code); it can be more cost-effective (as you don’t need to write lengthy instructions for every possible scenario); and it allows you to integrate external tools at specific stages of the workflow or involve human intervention when necessary.

Principal 3: Beware hallucinations

A common issue with large language models (LLMs) is hallucinations. This occurs when the model provides information that seems credible but is actually false.

To minimize hallucinations, consider these approaches:

1. Ask the model to connect its answer to relevant information from the context, then answer the question based on that data.
2. At the end, request the model to validate its response using the provided factual information.

Prompt engineering is often an iterative process, involving trial and error as you refine and adjust your prompts to achieve the best results. Experiment with different prompt structures, guidance levels, and context to identify the optimal approach for your specific application.

OpenAI API (ChatGPT)

ChatGPT (Chat Generative Pretrained Transformer) is a chatbot that produces human-like AI-generated content based on the input it is given by a user. It was developed by Open AI and released in November 2022 and became the most popular AI tools.

At the moment, many models can be used with this API. gpt-4o, gpt-4o-mini, gpt-4-turbo, gpt-4 and gpt-3.5-turbo refer to the latest model versions.

Quickstart Open AI API

First, create an OpenAI account or sign in. Next, navigate to the API key page and “Create new secret key”, optionally naming the key. Make sure to save this somewhere safe and do not share it with anyone. Keep in mind that ChatGPT API access is not related to the ChatGPT Plus subscription you might have. After registering, you need to pay $5 to be able to use the API. Payment can only be made using a credit card.

What needs to be considered when using this API is about pricing. We will pay for each API we call. For details on pricing for using the API, you can check the following link: Pricing | OpenAI

The price charged depends on the model we use and the number of tokens we request. The more complex (for example gpt-4o or gpt4), the more expensive the price will be. In addition, we need to pay attention to the number of tokens we input (prompt) and output (model response).

Apa itu token? You can think of tokens as pieces of words used for natural language processing. For English text, 1 token is approximately 4 characters or 0.75 words. As a point of reference, the collected works of Shakespeare are about 900,000 words or 1.2M tokens. To get an intuition about tokens, you can try using the following tools.

Let’s see how the tokens in 1 tweet about donald trump and how GPT splits the word into tokens.

You can find the exact number of tokens for your model using tiktoken python library. tiktoken is a library developed by OpenAI for tokenizing text. It's designed to be fast and efficient, specifically for use with OpenAI's GPT models.

!pip install tiktoken

import tiktoken

tokenizer = tiktoken.encoding_for_model("gpt-3.5-turbo")

text = 'breaking in a surprising turn at a pa rally donald trump was swiftly escorted away by the secret service leaving the crowd in confusion more details to come'

tokens = tokenizer.encode(text)
print('Tokens: ', tokens)

decode = tokenizer.decode_tokens_bytes(tokens)
print('Decode: ', decode)

Sending API Request

Let’s start with a simple function that will get a message and return a response.

import os 
from openai import OpenAI

client = OpenAI(api_key="<YOUR API KEY>")

def get_completion(text):
    completion = client.chat.completions.create(
        model="gpt-3.5-turbo",
        temperature=0.0,
        max_tokens=100,
        messages=[
            {"role": "system", "content": "You are an AI Assistant"},
            {"role": "user", "content": text}
        ]
    )
    return completion.choices[0].message.content


print(get_completion('Who is Donald Trump?'))

Below is an explanation of some of the parameters that are commonly used when using this API.

• Temperature: This parameter controls the degree of creativity or variation in the response generated by the model. The temperature value ranges from 0 to 1. The lower the temperature value, the more deterministic and repetitive the response. Conversely, the higher the temperature, the more varied and creative the response. For example, a temperature value of 0.0 will produce very definite and consistent responses, while a value of 1.0 will produce more varied and possibly more creative responses.
• max_tokens: This parameter specifies the maximum number of tokens (words or parts of words) that can be generated in an answer. This is useful for limiting the length of responses generated by the model. For example, if max_tokens is set to 100, the generated response will not exceed 100 tokens. This is useful for controlling the length of responses and preventing overly long responses
• Messages: This parameter is a list of messages to be used as context or input for generating responses. Each message has role and content attributes. The role can be ‘system’, ‘user’ or ‘assistant’. “System” provides general instructions or context, “User” is the input from the user and “Assistant” is the response from the model. This sequence of messages provides the conversational context used by the model to generate appropriate responses.

These parameters help control how the model generates responses and allow users to tailor the model’s behaviour to their needs.

Let’s try to use the fire in our case. let’s try to convert tweets into Indonesian and define the main topics.

Since the model does not store the state for each question in the session, we need to send the entire context. So, in this case, our message argument should be something like this.

system_prompt = """You are an assistant that review tweets \
and identifies the main topics mentioned"""

example_tweets = """Trump adalah presiden yang sangat baik. Saya sangat suka dia."""

translation_prompt = f"""
Please, translate the following customer review separated by triple backticks into English. 
In the result return only translation.

```
{tweets}
```
"""

user_topic_prompt = """Please,define the main topics mentioned in the translated review"""

messages = [
  {'role': 'system', 'content': system_prompt},
  {'role': 'user', 'content': translation_prompt},
  {'role': 'user', 'content': user_topic_prompt}
]

print(get_completion(messages))

Model Evaluation

Evaluation is the process of validating and testing the output generated by your LLM application. Robust evaluations (“evals”) will result in a more stable and reliable application, which is resistant to code and model changes. Eval is a task used to measure the quality of output from an LLM or LLM system. By providing input prompts, output is generated. We evaluate this output against a set of ideal answers and determine the quality of that LLM system.

There are no right and wrong answers in the case of topic modeling (chatbot use cases too). There are several ways to measure the answers generated by LLM.

• BLUE Score: BLEU (bilingual evaluation understudy) is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another.
• Comparing the responses generated with the answers generated by expert in the field. This takes a bit of effort but is effective.
• Using LLM as a judge to assess the answers produced. This has been used several times in studies on LLM. There are several prompts that can be used that can make this LLM model a judge to assess the answers or prompts produced.

In our case, we don’t have one correct answer untuk tweet yang dihasilkan, so we will need to compare results with expert answers or use another prompt to assess the quality of results.

Fine-tune BERTopic using ChatGPT API

The most sensible improvement to the previous method involves utilizing a large language model (LLM) to determine the topics we previously identified with BERTopic. This can be achieved by employing the OpenAI representation model alongside a summarization prompt.

from bertopic.representation import OpenAI
from sklearn.feature_extraction.text import CountVectorizer
from bertopic import BERTopic
import pandas as pd


df = pd.read_csv('./data/cleaned_data.csv', sep='\t')

docs = list(df.clean_text)


summarization_prompt = """
I have a topic that is described by the following keywords: [KEYWORDS]
In this topic, the following documents are a small but representative subset of all documents in the topic:
[DOCUMENTS]

Based on the information above, please give a description of this topic in a one statement in the following format:
topic: <description>
"""

representation_model = OpenAI(client, model="gpt-3.5-turbo", chat=True, prompt=summarization_prompt, 
                              nr_docs=5, delay_in_seconds=3)

vectorizer_model = CountVectorizer(min_df=5, stop_words = 'english')

topic_model = BERTopic(nr_topics = 30, vectorizer_model = vectorizer_model,
                      representation_model = representation_model)

For the dataset, I have provided it at the following link. After running the above code, the next step is to transform it into the Bertopic Model.

topics, ini_probs = topic_model.fit_transform(docs)
topic_model.get_topic_info()[['Count', 'Name']].head(7)

Next, BERTopic sends a request to the ChatGPT API for each topic, including keywords and a selection of representative documents. The response from the ChatGPT API serves as the model representation. The output generated from the above code is as follow:

 Count Name
0 3501 -1_Discussion surrounding Donald Trump, Presid...
1 1897 0_Shooting incident at a Donald Trump rally in...
2 1157 1_Donald Trump's presidency, support, and elec...
3 1097 2_The discussion centers around the use of vio...
4 834 3_Confusion surrounding Joe Biden's references...
5 502 4_Prayers for protection and blessings for Pre...
6 309 5_Donald Trump's involvement in a project rela...

You can find more details in the BERTopic documentation.

We’ve got the topic, but it still relies on using BERTopic to categorise documents using embedding. Could we get rid of it and use our initial texts as the source of truth?

Topic Modelling using ChatGPT

We can utilize ChatGPT for this task by dividing it into two steps: creating a list of topics and then assigning one or multiple topics to each customer review. Let’s give it a try.

Creating a list of topics First, we need to establish a list of topics. After that, we can use this list to classify reviews.

Ideally, we could send all texts to ChatGPT and ask it to identify the main topics. However, this might be expensive and not very straightforward. The entire dataset of hotel reviews contains over 2.5 million tokens. Thus, we cannot process all comments in one go (as ChatGPT-4 currently supports only 32K tokens in a single context).

To address this limitation, we can select a representative subset of documents that fit within the context size. BERTopic provides a set of the most representative documents for each topic, allowing us to create a basic BERTopic model.

from bertopic.representation import KeyBERTInspired
from bertopic import BERTopic

representation_model = KeyBERTInspired()

vectorizer_model = CountVectorizer(min_df=5, stop_words = 'english')
topic_model = BERTopic(nr_topics='auto', vectorizer_model=vectorizer_model, representation_model=representation_model)

topics, ini_probs = topic_model.fit_transform(docs)

topic_stats_df = topic_model.get_topic_info()

representative_docs = topic_stats_df.Representative_Docs.sum()

The topics_stats_df output:

Now, we can use these documents to define a list of relevant topics.

system_prompt = """You are a helpful assistant. Your task is to analyze the tweets"""
prompt = f"""
Below is a representative set of customer reviews delimited with triple backticks. 
Please, identify the main topics mentioned in these comments. 

Return a list of 10-20 topics. 
Output is a JSON list with the following format
[
    {{"topic_name": "<topic1>", "topic_description": "<topic_description1>"}}, 
    {{"topic_name": "<topic2>", "topic_description": "<topic_description2>"}},
    ...
]

Tweets:
```
{representative_docs}
```
"""

messages = [
    {'role': 'system', 'content': system_prompt},
    {'role': 'user', 'content': prompt}
    ]

topic_response = get_completion(messages)

Then, the generated topic is loaded and stored into the DataFrame. An example is as follows.

# Remove the code block notation
cleaned_json_str = topic_response.strip('```json\n').strip('\n```')

# Parse the cleaned JSON string
try:
    topics_list = json.loads(cleaned_json_str)
    # Convert to DataFrame
    topic_list_df = pd.DataFrame(topics_list)
    print(topic_list_df)
except json.JSONDecodeError as e:
    print(f"JSON decode error: {e}")

As a result, we got a list of relevant topics, and it looks pretty reasonable.

Classifying tweets by topics

The next step is to assign one or several topics for each tweets. Let’s create a prompt to do that.

topics_list_str = '\n'.join(map(lambda x: x['topic_name'], topics_list))

tweets = """breaking in a surprising turn at a pa rally donald trump was swiftly escorted away by the secret service leaving the crowd in confusion more details to come"""

system_prompt = """You are a helpful assistant. Your task is to analyze the tweets"""
prompt = f"""
Below is a tweets about donald trump delimited with triple backticks.
Please, identify the main topics mentioned in these comments from the list topic below.

return a list of the relevant topics for the tweets.

Output is a JSON list with the following format
["<topic1>", "<topic2>", ...]

If the topic are not relevant to the tweets, return an empty list ([]).
Include only topics from the provided below list.

List of topics:
{topics_list_str}

Tweets:
```
{tweets}
```

"""


messages =  [  
        {'role':'system', 
         'content': system_prompt},    
        {'role':'user', 
         'content': f"{prompt}"},  
] 


topics_response = get_completion(messages)

topics_response

The following are the results against the tweets. For example, here we only use 3 tweets.

This method yields quite good results and can even process comments in other languages, such as Indonesian (bahasa). For example, we will take a tweet from the @ernestprakarsa account.

https://medium.com/media/b9cf3d3c783159912871f97dc5096fe9/href

topics_list_str = '\n'.join(map(lambda x: x['topic_name'], topics_list))

#using indonesia language
tweets = """Gila. Donald Trump ditembak, kena kupingnya. Makin kalah telak ini mah Joe Biden."""

system_prompt = """You are a helpful assistant. Your task is to analyze the tweets"""
prompt = f"""
Below is a tweets about donald trump delimited with triple backticks.
Please, identify the main topics mentioned in these comments from the list topic below.

return a list of the relevant topics for the tweets.

Output is a JSON list with the following format
["<topic1>", "<topic2>", ...]

If the topic are not relevant to the tweets, return an empty list ([]).
Include only topics from the provided below list.

List of topics:
{topics_list_str}

Tweets:
```
{tweets}
```

"""

messages =  [  
        {'role':'system', 
         'content': system_prompt},    
        {'role':'user', 
         'content': f"{prompt}"},  
] 

topics_response = get_completion(messages)

topics_response

Summary

In this article, we have covered the key aspects concerning the practical use of LLMs: their functionality, primary applications, and operational methods.

A prototype has been developed for Topic Modelling utilizing the ChatGPT API. Through a limited set of instances, it has demonstrated remarkable performance, providing easily understandable outcomes.

The primary drawback of employing the ChatGPT method is its associated expenses. To process all texts within our copilation of tweet (comprising 2.5M tokens) using GPT-3.5, the cost would exceed 75 USD. Despite ChatGPT’s current status as the most effective model, exploring open-source options could be beneficial when handling extensive datasets.

Thank you for taking the time to read my article. We appreciate your support and interest. See you in the next one!