Exploring Homestuck

Main | XML Homestuck | Character Statistics | Profanity | Topic Modeling

Topic Modeling

Mallet is software created by Andrew McCallum at the University of Massachusetts at Amherst. It carries out topic modeling, which is a way to find what topics in what proportion compose a document. In the case of our project, one document is all of the speech spoken by one character. Therefore, we have eighteen documents (one document each for the narrator and seventeen characters), and when we ran the Mallet topic modeling software on the documents, Mallet gave us output in two different forms.

One piece of output that Mallet gives us was the "topic keys" for the documents. A topic in the Mallet sense is a group of words that tend to occur together. Therefore, three different topics to Mallet would look like this:

ruck rugby ball tackle try backs forwards

cheese Mozzarella France milk macaroni goat Swiss

gallop hooves saddle horsemanship Jack broke

To Mallet, these words have no semantic meaning, but since they are statistically related to each other in the text of the documents, they are a "topic". It is the job of a human to determine that the first topic relates to rugby, the second to cheese, and the third to horses.

Naturally, a document is composed of multiple topics, and this is true whether the document is the speech of a character in Homestuck, a journal entry, or a scientific article. Therefore, the second piece of output that Mallet gives us is the composition of topics for each document. For example, for the AT.txt document that contained all of AT’s speech, topic 3 from the "topic keys" composed 29.7% of the document, topic 6 composed 16.4% of the document, and composed 7.8% of the document, etc. Every word from the document belongs to one of the seven topics that Mallet has designated except for the "stop words," which are common prepositions, articles, etc. that the Mallet software is programmed to ignore.

When we ran Mallet on the documents containing each character's speech, the output surprised us. When Mallet returned "topics", or groups of words that tended to occur together in the text of the documents, the list of words in a topic did not have any coherent theme. When we tried to make sense of a topic, the words did not point in any singular direction, and we could not assign any of the topics intelligible semantic meaning.

We may have gotten these results because Mallet does not suit our data well. The webcomic's text is in a conversational style, so often there are no enduring, clearly defined topics that there would be in a newspaper article about a robbery, for example. Also, because of the conversational style of the data, there is a relatively low number of content words. Phatic expressions and words like 'yeah' and 'uh oh' naturally occur often in a conversation.

However, there are other ways we can run Mallet over the webcomic. We created a separate document for each character so that there were eighteen documents. It may have been a mistake to divide the text this way considering that the webcomic is a conversation, which by nature takes place with two or more participants. We mangled conversations by taking each participant's contribution and putting it into a different document. When we read the .txt file created for a character that was part of the Mallet input, the semantic flow was not coherent, because the other half of the conversation was missing. This may the reason that Mallet did not output meaningful topics. With the current data from Mallet, we can only make the conclusion that characters are distinct in some way, because their speech varies significantly in composition of topics.

Before Mallet iterates over the text of the documents, its user must determine how many topics she wants Mallet to create out of the data. We tried a range in number of topics from five to thirty. Mallet’s output from pulling eight topics from the documents seems to be most coherent. The raw data is shown below.

First, we present the topic keys, which is the eight topics that Mallet has found to compose the documents. Remember that a topic is a list of words that are statistically related to each other in the text.

Second, we present the composition proportions for each document. Mallet shows what percentage of words in the document came from the first topic, the second topic, etc. This way, we can see what the narrator and each of the characters talks about the most.

Tavros

Calsprite

Karkat

Davesprite

John

Kanaya

Terezi

Jade

Jaspersprite

Nannasprite

Sollux

Dave

Rose

Narrator

We welcome feedback on our results from the Mallet software. Any observations about a topic that seems to be coherent, patterns in the composition of the documents, or suggestions about using the Mallet software would be appreciated!