Ranking Star Wars Characters with an Adaptive Form of MaxDiff

It’s fun and educational to apply marketing science techniques to topics in pop culture and entertainment. Previously on LinkedIn, I showed how MaxDiff can establish ranking and preference scores for Pixar films: Your Definitive Ranking of all 28 Pixar Movies.  

Back for another round of fun, here I use an adaptive form of MaxDiff (“Bandit” MaxDiff) to discover who the top few Star Wars characters are among a hefty list of 75 candidates. My inspiration was an article in USA Today that I woke up to on May 4 (Star Wars day, “May the 4th"). There, Brian Truitt gave his ranking of the top 75 characters.

I naturally wondered if Star Wars fans in the wild would agree with Truitt’s ranking, and I knew of an excellent way to check that! So, I copied the names of the 75 characters from Truitt’s article and quickly created a Sawtooth survey that included a MaxDiff section for choosing favorite and least favorite characters. After only 45 minutes of effort, I had posted a link to the survey on a LinkedIn post and was collecting data (heavily skewed toward colleagues who follow me and market research topics, obviously). But when you think about it, you folks include a lot of Star Wars fans…some of you are Star Wars fanatics. (Looking at you Walter Williams ).

The 75 Candidates for Best Star Wars Characters

As listed by Brian Truitt in the USA Today article, here is a ranking of the top 75 characters, leading off with Darth Vader (favorite) and ending with Admiral Motti (least favorite among the top 75). 

That’s quite the list! Even though I consider myself a casual Star Wars fan, I don’t recognize 40 of these 75 characters without seeing a picture of them! (And, many I wouldn’t know if I saw their pictures.)

With 40 or more items, it starts to become difficult to administer standard MaxDiff studies to respondents. And I’d rather kiss a Wookiee than ask each respondent to evaluate 120, or especially 300+ items, as it would push both MaxDiff and respondents to the limit! But an adaptive form of MaxDiff (Bandit MaxDiff) makes this possible with reasonable sample sizes—nearly as easy as bullseyeing womp rats in Beggar's Canyon.

Names Only Instead of Graphics

Admittedly, my MaxDiff survey would have been better if pictures accompanied the names of each character (but I took the shortcut of just listing their names). For example, Brian Truitt just referred to “Din Djarin” in his top-75 list, which is the real name of The Mandalorian, or "Mando" for short. So, my survey also only referred to Din Djarin, which a lot of my respondents probably didn’t recognize as The Mandolorian. Similarly, a lot of casual fans refer to a recent favorite character as “Baby Yoda,” but his real name (as represented in the survey) is Grogu.

So, my MaxDiff survey likely understates preference for recent characters like Mando. Though, Grogu’s name seems better known as he cracked the top 10. If I field this survey next Star Wars day, I should take time to include pictures. Instead, my quickly rendered MaxDiff survey using Sawtooth’s survey platform looked like this:

Each respondent completed just eight screens like this, covering 24 of the total 75 characters. (Across respondents, all 75 are evaluated.) Bandit MaxDiff learns from past respondents taking the survey, so next respondents are more likely to see characters in their draw of 24 that tended to be preferred by the previous respondents.

These Are Not The Items You’re Looking For…

With standard MaxDiff, each character would be shown an equal number of times across all respondents taking the survey. If the goal of the research is to learn which are the top 5 or 10 characters from our 75-character list, we’d waste a lot of respondent effort in evaluating characters that don’t have a porg’s chance in the Sarlacc pit of rising to the level of top 10.

Bandit MaxDiff oversamples items that previous respondents are tending to like best. After just one or two dozen respondents have finished the survey, new respondents are tending to see the more preferred characters. At that point, we’re mostly asking respondents to compare winners vs. winners; we’re not wasting much time with the losers. Those aren’t the droids we’re looking for. How does Bandit MaxDiff do this? Using the force, of course—OK, not really…it uses Thompson Sampling.

With adaptive Bandit MaxDiff, we can achieve confidence about the top 5 or 10 characters with only about 1/4 to 1/5 of the sample size that non-adaptive MaxDiff would need to tackle the same question with the same precision.

When we’re using Bandit MaxDiff, we’re assuming we just want to find the top few items on average for the population. We analyze the preferences with pooled (aggregate) analysis, such as pooled MNL estimation.

Drumroll Please…Your Top 10 Star Wars Characters!

1.       Obi-Wan Kenobi

2.       Yoda

3.       Han Solo

4.       Darth Vader / Anakin Skywalker

5.       Chewbacca

6.       Luke Skywalker

7.       R2-D2

8.       Princess Leia

9.       Grogu

10.  Boba Fett

For comparison, Ranker.com has more than 15000 votes currently from website visitors who upvote or downvote the top Star Wars characters. And, son of a bantha, their rankings are eerily similar!

On Ranker.com, Obi-Wan is currently #2, Yoda #3, Han #4, Vader #1, Chewbacca #7. Luke is #5, R2 #6, Leia #8, Grogu #17, and Boba Fett #13. So, our small convenience sample of 136 respondents leveraging Bandit MaxDiff leads to results very similar to Ranker.com that has the benefit of much broader coverage and sample size. Pitting this n=136 survey against Ranker.com’s coverage of this same question is like a Jawa going up against the Death Star. Obviously, for real-world applications, you’re going to want to step up sample size and proper sampling procedures compared to the fun we’ve had here with a smallish convenience sample.

Winning Characters Seen More Times than Losing Characters

Bandit MaxDiff learns from past respondents to oversample items that are tending to be preferred for next respondents. To illustrate, we’ve plotted the number of respondents seeing each character in their Bandit MaxDiff survey. (Remember, each respondent only evaluates 24 of the total 75 characters.) Obi-Wan, our #1 preferred character, was seen by 131 of the 136 respondents. The least preferred bunch of about 40 characters was only seen by around 20 respondents. Bandit MaxDiff learns and updates quickly!

The frequency of respondents seeing characters in their survey by preference pretty closely follows an S-shape curve, as we’ve drawn with the gold line.

Generative AI and Bandit MaxDiff

Marketing departments are increasingly leveraging Gen AI to brainstorm both text and graphical elements. When they ask if survey research can identify the top 10 or so phrases or graphical designs among hundreds, you can say, “Yes, we can do that using Sawtooth’s Bandit MaxDiff”.

More Reading on Bandit MaxDiff…

Why is it called Bandit MaxDiff? When should you use it instead of standard MaxDiff? What sample size is recommended for large Bandit MaxDiff problems? See this white paper on Sawtooth’s website: https://sawtoothsoftware.com/resources/technical-papers/bandit-maxdiff-when-to-use-it-and-why-it-can-be-a-better-choice-than-standard-maxdiff