Pimp my option choices!

In my mind one of the biggest turnoffs for respondents completing online surveys is the utterly uninspiring language we use to write them with.

I have talked in this blog about the impact that taking a more creative approach to how we ask questions in online surveys and how this can improve the quality of responses.

I would also like to challenge the way we think about using option ranges in surveys too. Particularly the over reliance on tired and unimaginative 5 point likert scales.

To me phrases "like how much do you agree or disagree with this statement” or”” how appealing or unappealing is this and “on a scale of 1 to 10” come across like clichés and do nothing to stimulate our imagination of respondents when they are answering questions, lack any real context, often don't fit the range requirements of the question.


These are the 5 questions you need to ask yourself if you want to design more effective range options:

1. What is the emotional impact of the range choice you are offering?

Take this very personal example.  I often ask my son on a scale of 1 to 10 how much did he enjoy his day as school.  The question does not inspire him to really think about the answer and so invariably I get either a 1 or a 5 and sometimes a 10 as the answer.  He gives me lip service to answering a question because he has heard several times before and so answers with a cliché  response.

Now one day I came home and asked on a scale of 1 to 10 how much did you enjoy your day at school but change the range choice.

10 = was the best day ever, they cancelled lessons and you all went to Disney world and had free ice creams 

1 = the worse day, all you did was Maths, the teacher kept you in at break times and for lunch all there was nothing to eat but cabbage.  

Now this was the first time I ever got a 6 out of him. 

By adding some emotional relevance or resonance to the option ranges it forced him to properly think and process the question.  This is what a good set of option ranges should do, they are not just there to benchmark, they have an important role at encouraging people to think.

2. What is the context?

Standard range choices can often be intangible choices for respondents to answer as they are so general and un-contextualized. When constructing the range choices you need to think about what actually you want to know.

Take the example the question how much do you like chocolate? Are you asking this in the context of other people or other things? My sister for example absolutely loves chocolate, a lot more than me, and I like chocolate certainly a great deal more than I like Brussels sprouts, but then again I would not like chocolate with my Christmas dinner and I probably don’t like chocolate as much as I like ice cream.

i.e. it's  really difficult to say how much you like something unless it is the context of something else.

Take another example the question “how much do you like watching golf on TV” the easy option would probably opt for a standard 5 point likert liking scale (like a lot, like a little, neither like/dislike, dislike a little, dislike a lot etc) but again with this you are offering no context a better solution might be…

How much do you like watching golf on TV?

I Would turn off the TV if that was all there was to watch

I would watch it if nothing else was on

I would watch in preference to some other programmes

I would watch in preference to most other programmes

I would be really annoyed if had to watch something else

Now I would not say this is a perfect set of answers either because for example your view point on golf may vary if it were say the Ryder cup or a local tournament, but I hope it illustrates my point that this is a range respondent I think I would find this easier to answer than a standard liking question and I think as a researcher you could argue that the answers you go from this would probably be of more value. 

Simply put the liking has been contextualised.

3. What are the natural range limits?

What researchers normally want is a good natural spread of opinion from a question to enable them to differentiate the answers. One of the other issues with offering a standard 5 point likert range question is that the options often do not naturally spread across the range, they tend to get clustered.

Think about the question “how much do you like chocolate?” Nearly everyone likes chocolate so to offer respondents a standard 5 point liker scales leaves the average respondent with only 2 choice which is a bit of a limited range, like a little and like a lot.

Or take the example of how films get rated.  The 1 to 5 star ratings are nearly all top weighted and clustered around 3 4 and 5 to the extent that if a film has a 3 star review we assume it is pretty poor.  Have you ever seen a 1 star review, they are very rare.

We are a bit stuck with the 5 star review process but it is not tremendously useful it really needs to be more exponential range set.


So it is important when thinking about a set of range choices to think about how the answers may get spread and adjust them accordingly.

4. Does the range cross cultures?

My 4^th point is that some standard ranges have a lot of cultural issues to contend with, for example the Asians prefer not to outwardly say they don’t like things so on 5 point range scales the answers get closeted around the top 3 choices, so this is a pretty useless question to use in these market.

5. What visuals can you use to support the options choices

The most effective means of  communicating and helping to emotionalise option choices is with the use of imagery.  Imagery really help respondents make choices more easily and are particularly useful if it is a repetitive question.   It is important that the images chosen accurately reflect the sentiment of each option choice, if they don't they it may be counter productive to use them but they really can add value if you are able to find a way of visualising a range option.   Here is an example below of how imagery can make answering a question a lot easier.

 Issue with adopting a more creative approach

You may instinctively worry about mucking around with using more emotional ranges, as to the impact it will have on the answers.

The answer will almost certainly be slightly different from what you would achieve with neutral/standard question ranges, but I would argue that in most case they will be better, richer, formed out of a greater amount of thinking.

Our opinions are 3 dimensional all an answer to a question is, is a 2d snapshot of that opinion and  a lot of standard likert scales over a very narrow'oblique perspectives on 3d opinions.

In my mind if by tweaking the question you do get a different answer this is an indication that the topic is more complex than you think and sticking to the standard constrained portfolio of question options will perhaps be a safe option for you but you may well end up with a very narrow perspective on a subject.  By switching to a different way of asking the question you can open up greater depth of thought and opinion and if it does throw up unusual answers, well this may be a sign that you need to do more research generally on the topic you are investigating.

Its not to say that you can pluck any whacky range set out of your head and think it will do, there is a lot of thought required.  The basic idea though is to humanize and use descriptive terms that trigger the imagination and contextualize the answers and you will get better data.

Questionnaire piloting: a user guide

I were to offer one piece of advice to the market research industry on my death bed it would be "do more piloting!".

We have found that effective piloting really can lead to more efficiently executed research so it amazes me that more research companies have not latched on to the value of doing this.

The reasons for this are I think historical. In the days of pen and paper interviews it was simply not a practical option to pilot a piece of research as it would potentially add weeks to the turn-around time of a project and could end up being prohibitively costly, so it never became an established part of survey design protocols.

However, today with the speed of online research, sketching out a survey and piloting it with just 25-50 respondents is something that can be done in a matter of just a few hours and will add nothing to the overall cost of a project. In fact, it could very well save you money in the long run.

This blog post is a short guide to how you can best use piloting to improve your market research...

With the right understanding of the techniques involved, piloting can have a very useful role in shaping the content and structure of a survey; helping to define the value of questions, perfect the question wording, assessing what options should be included or omitted, and also to help work out exactly what sample sizes you need to employ. The impact of piloting can often be completely transformational.

Through a process of piloting we have been able to double, and even treble in some cases, the volume of feedback obtained, seen significantly reduced drop-out levels from surveys, enabled us to assess whether surveys have even been worth conducting in the first place and validated theories before the main wave of the research has even been carried out. Used effectively, piloting really can make your research budget go further.

So how do you go about effectively piloting a MR study and what are the main things that can be achieved through this process? Here is a guide to some of the tricks we have learnt about how to effectively do it:

Firstly what can you use piloting to do? 

1. Editing back questions

One of the biggest problems online surveys suffer from is verbosity! They are often too long which can lead to respondents disengaging and dropping out before the end of the survey. The main reason for this is that so often the questions being asked are "speculative" in nature, and so to ensure that you have shed enough light on an issue it is often the case that questions need to be asked about the same topic in a variety of different ways to cover all bases.

Well, anyone who has ever done a survey will, I am sure, have experienced that feeling of seeing the results from a particular question in the survey and thinking, “hm, that's not very interesting”.

We too often find ourselves throwing away the answers from questions in a survey once we see the data. Analyse how many questions in a typical survey actually make it to the report. In my experience, it can often be less than half. There is nothing inherently wrong in this - research is often an exploratory process meaning that it cannot be helped that you end up heading down some blind alleys. It is a part of the process.

The great thing is, is that piloting can really help with this. You can ask a question in a pilot and if it sheds light on an issue keep it and if not, then remove or adapt it.

In my experience you can easily trim back a typical survey upwards of 20% of the questions this way. In fact, I have been known to have thrown away whole surveys in the bin on the back of a pilot because the results were of no real benefit. I have also completely rewritten others in the light of the insight that the pilot survey delivered.

2. Refine option choices

On a more detailed basis you can also use piloting to trim back redundant option choices. How often have you found yourself building a tick list of choices, guessing which options you need to include and finally ending up with a number of options that only a very few % of people actually click on? Conducting a pilot will enable you to see early on how many people click each option allowing you to remove those that are either rarely selected or that significantly overlap with other options.

If you are really clever and can afford to, do a larger scale pilot so that you can look at the statistical relationship between answers to different questions and weed out ones that significantly overlap or can be modelled from related questions. 

3. Refining range options

At an even deeper level you can use pilot data to refine option ranges. Say, for example, if everyone appears to be opting for the top 2 boxes on a likert scale you may well decide to reword the range options to deliver a wider spread of responses.

4. Comparing answers to the same question asked in different ways

The way a question is framed can often have a major bearing on how it is answered. Which is something you can often find yourself getting locked into big debates with your clients and colleagues about. The beauty of piloting means you can test out different wordings and see how it affects the answers. You can play about with different approaches. You don’t have to speculate - you can test. And therefore you can present your clients and colleagues with the facts and end those frustrating debates!

5. Designing more intelligent answer code frames

The code frames used for many questions, particularly word associations, are very often created by the flights of imagination of the researcher or client or have been adapted from another survey. As a result some of the words or option choices you select may do little more than confuse respondents and make it harder for them to make their selections due to the options simply not being relevant.

An example of this was a question I saw recently about what motivates your decision to buy a soft drink. There were over 20 option choices including things such as "I research online before I make my decision" and "I would seek advice from a sales person" which completely missed the mark, having no bearing on the original question itself.

Well, you can use piloting to help you design the most beneficial code frames. You do this simply by asking it as an open ended question: "what motivates your decision to buy a soft drink?". You can then pick out the most popular responses and use them to create the optimum option list within your main study.

This is such a simple technique that we have used with successful results on several occasions and would recommend it highly as a tool to help with creating a far more efficient and engaging survey.

6. More clearly understanding sample size requirements

Without piloting, planning out what sample size to use in a survey is often a question of licking your finger and holding it up in the air. You can only really guess. The only way you can effectively estimate sample size requirements for a survey is by getting some people to do the survey and see how the answers pan out.

For example if you ask 50 people a question and 24 choose option A and 26 choose option B you will need a far larger sample to differentiate which choice is actually better. But if the answers came back whereby 10 people chose A and 40 people B, in this instance you would know that you don’t need any more sample at all; you have already answered the question.

Often the sample size requirement will vary by question, but it isn’t difficult these days to adjust on a question by question basis to determine the required number of people per question. This really can be a very smart way of optimising the length of a survey.

We recently employed this technique in one of our surveys and were able to reduce the overall length of it by 30% having discovered that only 20% of people need to answer this question, 50% that question and 100% those questions and so on.

7. Working out how to encourage the best response to an open ended question

One of the most successful uses for piloting is to experiment with ways to encourage people to give you more open ended feedback. This is often very sensitive to methods used and the gains can be enormous. We recently changed a question in a survey we were working on with Sony music from "Tell us about your interests in music" to "Imagine you were being interviewed by a magazine, how would you describe your interest in music" . As you can sense by seeing this question set out here in their differing ways, this can completely change the quality of the feedback. We found this worked so much so, that we adopted the technique throughout the rest of the questionnaire and the overall word count of the survey increased from 230 to over 400 words per respondent. 

 
8. Working out how to effectively engage respondents

On a broader level you can use a pilot to test out different intro messages and see what effect this has on survey completion times, experiment with different images and styles of communication all these seemingly small things can have a real impact.

9. Error trapping

The last thing is that piloting is great for error trapping. It’s often so difficult to spot all the errors there may be in a survey. Particularly things like broader logic routing problems. Some surveys may have 20+ different paths making it very difficult to check. Again piloting really helps with ensuring you don't run into problems when the real survey hits the field.

Piloting techniques

OK so you want to do some piloting now. How do you go about piloting? Here are some tips and suggestions:

1. Be open minded about how many responses you need for your pilot.  

20 may be enough but you may need 100 if some of the answers are not clear. It’s easy for sample providers to send out small batches of invites and close off surveys. So use small batch samples aiming to get 20 to 50 completes at a time, then review the data and if more clarity is needed ask the panel provider to send out more invites.

2. Use micro test and control cells of 20 to compare question technique.

May sound small but in my experience there its enough to roughly get the picture. Particularly when you are auditing things like the volume of free text this should be enough sample to tell if one version is working more significantly better than another.

3. Don't bother testing out the whole survey

 Pick the parts that you think need developing, but obviously try and make it as close to the real experience as possible

4. Don't fret over things like typo's or conditionality

 Focus on the bigger picture it is, after all, only a sketch.

5. Don't forget to ask respondents for feedback

 It sometimes can really be useful to ask respondents their experience of doing the survey. What they thought about answering specific questions.

6. Suggest, if you can, doing the pilot before you talk to the client. 

Obviously depending on the client and the brief, some will offer invaluable feedback and it is useful that they are involved. But with others it is like taking 2 people to the video shop you can never make a decision.  Most of the things you do in a piloting are quite technical, to decide what range choices to use for example.  Once you do the pilot you can then go back to the client armed with incite as to how to effectively write the final questionnaire. 

7. Build the piloting process into your established methodology and cost for it. 

 Yes it will add to the cost of an individual survey but in the long run, over the rolling cycle of say 10 surveys it will almost certainly pay for itself.

8. Use your company/friend/family as free panellists! 

 It depends on the project but if the demographics are not important, why not?

9. If you are working with a panel provider who is going to do the scripting of the pilot for you, brief them about this at the quote stage. 

And don't muck them around with too much tweaking the pilot! The one issue I would say I have had with conducting pilots is clients getting wrapped up in them and not being able to differentiate the protocols of signing off a pilot versus a main study. I have had instances where I get a long list of minor wording corrections for example that have no real bearing on the results and insisting on loads of unnecessary tweaks and changes.

Why not think about piloting your next online survey you will find that...

• Your survey will be shorter and more efficient
• Will deliver back better data to your clients 
• You will not be wasting so much of your budget on asking unnecessary questions 
• Your survey is less likely to have errors in it and you will sleep better at night while the survey is in the field 

Having seen the benefits of piloting I don't think I personally ever consider launching an online survey without doing a pilot first, even for the smallest of studies.