On December 10, 2011 I had the privilege of joining Dana Chisnell of UsabilityWorks in conducting the first usability study on ranked choice ballots. The goal of the study was to get inside voters heads while they were voting. Before this study, no one had actually watched voters fill out the ballot or listened while they talked through their thought process.
The study, designed by Dana, aimed to answer three questions:
- How well do voters understand ranked choice voting?
- What are voters’ intentions when they mark their rankings?
- Are there particular types of voters who mark their rankings a particular way?
To gather this data, we asked participants to fill out 2 different types of ballots, one from Alameda county and the other from Portland, Maine. We asked them to speak aloud while they were voting and explain their thought process. Additionally, we asked people if we could film their hands while they filled out ballots.
I rounded up willing participants one by one from the Western Addition branch Library in San Francisco and asked them to sit for a few minutes and help us test the design of a couple ballots. Rick and I took turns leading the sessions and taking notes. His critiques of my technique were helpful in catching myself when asking a leading question, or helping a participant finish a sentence.
In this clip, the participant explains a mistake he made while filling out the ballot and the consequences it would have when his ballot was counted.
This study raised questions about the effectiveness of ranked choice ballots in major political elections. Using the Alameda and Portland, Maine RCV ballots many participants made mistakes on their ballots that would disqualify their vote. Often participants did not realize these mistakes and would not have taken the time to correct them.
Ranked Choice Voting was not a popular issue among our survey participants. Of those who had heard of it, few could explain how it worked and some reacted very negatively about the method.
Also apparent during the study was the ineffectiveness of directions on the ballot. Often participants would begin voting, reach a point of confusion and then read the directions — after having already made a disqualifying mistake.