Usability testing for Wikimedia... right now!

(Worrying about) Usability seems to be all the rage in Wikimedia at the moment. I think it’s an important thing to worry about, but we have this problem that we are unable to tell and often even guess, how our projects are perceived by people not already familiar with them.

One quick answer might be Feedback Army, a recently created little service that employs Amazon’s Mechanical Turk workers.

I decided to give it a spin for some aspects of Wikimedia Commons. For US$7 for 10 responses, why not? I sent people to look at Image:Easter_Island_map-en.svg, which was yesterday’s Picture of the Day. Here’s the instructions I gave them:

This is testing the usability of the site, not your ability. If you can’t find the answer to any of the questions, that is itself a valid answer.

  1. What is this a picture of?
  2. Who created this picture?
  3. What file format is this picture in? Is there anything special about it?
  4. Has this picture been judged to be a good one or not? How can you tell or not tell?
  5. How many versions of this picture have been uploaded? By which users?
  6. Would you be allowed to use this picture to create a paper map for sale? How can you tell or not tell?
  7. What date was this page last edited?

And a few questions about you:

  1. Are you male or female?
  2. What is your age?
  3. Have you ever edited a wiki before?

#1 is a gimme to make sure they’re sensible people. #2 is asking people to find the “Author” field in the description below the image. #3 was partially just curiosity about people’s awareness of SVGs, although most respondents picked up on the template:translation possible.

#4 is about the FP (featured picture) and POTY (picture of the year) markers. #5, as a couple of respondents noticed, was ambiguous — unintentionally on my part. I was actually referring to different uploaded versions present under the “File history”, but different translated images are listed in a section called “Other versions” so that was poorly worded on my part. 5 people responded about the translated images, 3 about the file history section, 1 pointed about the ambiguity and 1 didn’t respond.

#6 was about the Creative Commons BY-SA license. Partly it tests if they can locate that information, and partly it tests if they know (or can quickly find out) what that means. So partly it is a test of Creative Commons’ marketing I think. :)

#7 is one that is intentionally a bit confusing. The last version of the image uploaded was at 23:43, 22 October 2008, which is available directly on the page, but the time the page was last edited has to be found from the History tab. (It is 22:11, 24 November 2008.) Exactly half of respondents picked each answer, October or November.

#8, 9 and 10 are just basic demographics.

You can read all the responses on Feedback Army, or better read responses as a spreadsheet that I put together in Google Docs.

So these are just some very quick checks. Other interesting things to check would be if people can navigate and understand history pages, navigating categories, and then obviously editing and uploading.

What questions would you ask? Would you change any of the above questions?

04 December, 2008 • ,



This is cool, though my obvious reservation would be: What kind of person signs up for this service as a reviewer? I think it’s very likely that technical users are highly overrepresented in the feedback group.

Erik Möller · 4. December 2008, 11:42


There is a study of Mechanical Turk demographics here. I mean they are obviously tech-savvy enough to use the Amazon system. But that doesn’t mean they are familiar with wikis or Wikimedia. It is at least better than asking Wikimedia visitors directly.

Half the respondents had never edited a wiki before. It is possible in Mechanical Turk to put certain requirements for people to pick up tasks; I wrote to the Feedback Army guy to incorporate that feature into FA. Then you could ask for only respondents who had never edited a wiki or only rarely edited etc.

pfctdayelise · 4. December 2008, 12:17

Elsewhere on the web...

Commenting is closed for this article.

list of all posts, ever

find articles by tag

monthly archive

most popular articles

  1. [guest] Rethinking the Top Ten
  2. An alternative term for "User-generated content"
  3. How to use Gmail to manage high-traffic mailing lists
  4. NLA Innovative Ideas Forum audio/video now available
  5. Write API enabled on Wikimedia sites!
  6. Free as in Freedom miniconf recap + slides
  7. Templatology, an essay
  8. Top 10 software extensions Wikimedia Commons needs in 2008
  9. Is mass collaboration all it's cracked up to be?
  10. GLAM-WIKI, day one

(from the last 30 days)