Leaving Questions Unforced

It's pretty common practice to force respondents to answer some questions, and leave others unforced. When you do this, there is one very important question to ask yourself: Can the respondents tell which ones are forced and which ones are optional? If they can not, the question becomes: What will your respondents do if they assume a question is forced?

If a respondent has already been prompted to answer a skipped question or two, and they get to an un-forced question, chances are they are going to assume that question is forced too. Unless your survey wording communicates that the question is optional, you've given them no reason to believe that they won't be prompted, should they skip the optional question. If they don't know the answer, or have no opinion on the matter, the respondent is likely to randomly select an answer, leaving you with useless data.

Sadly, there is no "one size fits all" solution to this dilemma.

The (seemingly) obvious solution is to communicate to the respondents which questions are not forced, simply by adding the word "optional" to the question somewhere. The problem with this is that you're going to end up with a lot of people skipping the optional questions, even if they have valid opinions on them, simply to get through the survey quicker. This is especially problematic if you have a lot of consecutive optional questions.

More often than not, a good way to go is to force every question, but be certain that your questionnaire is well-constructed. In other words, don't ask them to rate the hamburgers at XYZ Restaurant, unless you've already determined that the respondent has eaten a hamburger at XYZ Restaurant.

Strategically-placed "don't know / no opinion / does not apply" options can also help you get the most accurate data.

Most surveys are too long, and suffer from fatigue and/or race-through bias. So it's a good idea to look at every question in your survey with great scrutiny. If you're okay with respondents not answering this question, do you really need this question at all?

In a nutshell, it's very important to evaluate each and every "optional" question situation with from a fresh perspective, and use the method that works best for each instance.

Are Your Matrix Questions Too Tall?

I love Grid/matrix questions! They are such a great time-saver when compared to asking respondents to rate each item as a separate question on a separate page. But do your respondents know what answers they are giving? Maybe... Maybe not!

Let's look at a lengthy grid:


Doesn't look too bad, eh? The problem lies in grids that get really long, like the one above. A respondent stopping by, who is answering your questions from a computer with a windowed browser (common among Mac users) or a low-resolution display setting might see this when they scroll down to answer the questions that appear lower on the grid:


Clearly the problem here is that the respondents can't see the column headers, therefore are marking radio buttons without knowing what answers they are giving.

One possible solution is to repeat your column headers at the bottom of the matrix, like so:


Better yet, why not toss one in the middle too?:


Repeat the scale throughout the grid as many times as you need to in order to ensure that respondents can see what answers they are giving.

Suspend and Resume Feature

Many survey applications offer the ability to allow respondents to start an interview, quit in the middle, then come back and pick up right where they left off. Recently I participated in a survey that presented this the wrong way.

The survey I completed actually had a "Quit" button on every page. It looked a little bit like this:

I couldn't believe my eyes. Our industry is facing all-time low response rates, and here I was, being presented with an advertisement to quit the survey on every single page.

Would you train a telephone interviewer to ask respondents "would you like to quit now and finish this later" after every question? Of course not. Your dropout rates would be through the roof.

Using a suspend/resume feature is a great idea in most online survey situations. Mention this option in the invitation, but do not remind them on every page that it's okay to stop the interview. Instead, program your survey in such a way that if a respondent quits by any means (such as closing their browser window) and returns to the survey at a later time, they're automatically directed to the spot in the questionnaire where they left off.

Button Placement

For many, many years, I recommended very specific survey navigation button placement:

The "Next" button was always in the bottom-right corner of the browser window, while the survey "Back" button was always in the bottom-left corner.

My reasoning:

  • I've read several studies about banner placement, and the bottom-right corner freqently achieved the best click-through rates.

  • Because of all the vertical scrolling required to read Web pages, it is very natural for a respondent's cursor to gravitate toward the bottom-right corner of the window.

  • These positions seemed to emulate the natural flow of how we read traditional media. To advance to the next page of a book or magazine, you grab the page on the right. To return to the previous page, you grab the page on the left.

When I set up these guidelines, 17" monitors were considered large, very few people had widescreen displays, and 1024*726 was a high, rare resolution setting. While the same reasoning is still applicable today, the instruments on which our surveys are displayed has changed.

Nowadays it is not uncommon for a respondent to stop by and attept to complete your survey using a 24" widescreen monitor set to 1920*1200. Using my old recommendations, and this configuration for comparison, the radio buttons for a Yes/No question would be approximately 15" away from the "Next" button in a maximized browser window. This is way too far to require a respondent to move their mouse between pages.

In keeping up with the spirit of my previous suggestions, but adjusting for newer, wider displays, I now recommend centering your buttons at the bottom of the page, with the "Back" button on the left, and the "Next" button on the right.

 
BlogCFC was created by Raymond Camden. This blog is running version 5.9.001.