Every so often we are asked about website usability, what works, what doesn’t and more importantly, how to get at the information. We typically suggest measuring the usability of the site with a simple questionnaire.

Here are some general guidelines for drafting a solid usability questionnaire:

1. Pre-qualify participants. They may seem obvious, but it’s key. By pre-qualification I mean, the first question should be something to the nature of “How often do you use the system?” And the answer scale should be very often to never. This statement right off the bat validates the users scores, and reinforces data integrity. If they answer “never” the subsequent questions should attempt to address why.

2. Avoid asking open ended/leading questions. This is important for two reasons. One, open ended answers are statically meaningless (if you intend to eventually normalize your data). The goal should be to get hard quantitative data. Doing so will allow you to back into the problems based on low performing categories (I’ll talk a bit more about categories in item #3). Two, the aim of the survey should be to analyze the overall usability of the system and not validate suspected flaws. Chances are if you suspect certain flaws, they exist, asking users directly about these flaws only draws attention to them, where it may not have been an issue. Hence it’s not a true usability issue. Conversely, by analyzing the overall usability of the system in a methodical manner, it will inadvertently flesh out suspected flaws, thus truly validating your assertions.

Side note - Traditionally open-ended questionnaires are good if you are in an exploratory phase of your research / system design and you are looking for user insights on needs and wants.

3. Provide logically grouped statements on various aspects of the system with an answer scale of strongly agree to strongly disagree (plus N/A). I’d suggest considering the following groupings:

• Usefulness
• Ease of Use
• Ease of Learning
• Satisfaction

Regardless of the subject manner of the grouping, the key is to group your statements. Meaning, once you’ve gathered all the data, you can pull meaningful quantitative information for each of the group. For example, if overall statements for ‘Ease of Learning’ received the worst scores, this suggests that education/tutorials/help/support channels need improvement. On the other hand, if ‘usefulness’ receives the lowest ratings, chances are the system itself needs to be reconsidered.

Here are some sample statements for usefulness:

• It helps me be more effective
• It helps me be more productive
• It meets my needs
• It makes the things I want to accomplish easier to get done. (Note how this statement is designed to check against the 1st statement)

4. In instances where the user inputs ’strongly disagree,’ consider leaving a blank text box to provide a reason. This will support the quantitative aspect of the questionnaire where it’s needed the most. Again, we are testing an existing system, not surveying for a features grid.

5. Segment your participants. You should devise different versions of the questionnaire, with some of the same questions, but for different user groups and compare results. This will help you detect bugs/flaws in the system at different levels of the sale and user cycle. For example, ‘Ease of Learning’, if scored very low for trial users, would suggest a very low conversion rate to paid subscribers.

Use this as a starting point to develop your own questionnaire.