Where can we inject more creativity into survey design

Here are my thoughts on some examples of the areas where I feel we need to inject a bit more creativity into the design of surveys. This content is taken from one of my presentations on the topic.

1. Think about how we motivate respondents to answer questions and take part in surveys

We so often jump straight into surveys with questions that look like this...

In other words, we jump straight into the nitty gritty of things we want to know.

I ask, has anyone in market research ever heard of foreplay?

If you want to encourage people to think, which is essentially the task we are asking respondents to do, it really does help if you warm them up and provoke their curiosity and interest in a topic first, before wading in with the boring questions.

There are lots of ways this can be done but one of simplest techniques is to try to think of a question respondents might actually want to answer and use it as your opening gambit.

Say for example you were doing a survey about toothpaste. You could jump straight in with questions about what brands of toothpaste they are aware of, or you could ask questions like which of these celebrities has the nicest teeth? 

i.e. You get them thinking about the topic in a more rounded way. We have found that this can stimulate them to take a lot more interest in answering the subsequent questions.

This approach to considering respondents’ motivation is at the heart of effective survey design. You should be thinking about this with every question you write in a survey.

2. Improve skills at copy writing

I think we need to make a fresh start in the way we think about writing questions in online surveys. We are locked into dry research speak that doesn’t connect with respondents. We are obsessed with the platitudes of 5 point, 7 point, 9 point scales and concepts like ‘strongly agree’ or ‘disagree’, which are often meaningless and ineffective in emotionally engaging respondents. Go out and listen to consumers describing brands, and I don't think you ever will hear the words ‘very appealing’ or ‘I strongly like this brand’.

If you want to connect with respondents, it helps if you use their language. Tell me, what does ‘strongly agree’ actually mean to most people? You either agree or disagree with something.

We have a general over reliance on range scales, which to me is just a lazy approach to research. Respondents in turn are extremely lazy in the way they answer them. If you roll up the answers to a cross section of say 10 point range scale questions, you will find that nearly 80% of answers are clustered between 5 and 8. Respondents are not being encouraged to think. Scales mean nothing to most people, or rather, we have learnt to use them in a certain way.

Sweep away verbosity

We need to clear away the verbosity of instructions which may be essential in a face to face interview, but are not needed in questions which have visual cues. I would recommend applying a twitter rule of no more than 140 characters for a question. Believe me, no matter how important you think the instructions are, a huge proportion of respondents do not read past the first sentence.*

* roughly speaking 50%!

Note: if you do have a more complex idea to communicate in a survey, then this needs to be broken down into clear parts and presented presentation style, one thought at a time. Injecting imagery will be a powerful tool to help communicate the message.

Write questions with a sense of humanity

We are habitually autistically precise in our phrasing, which can be completely alienating to respondents. I would recommend writing from the point of view of an advertising agency trying to sell the question to the respondent, or a TV interviewer questioning a famous celebrity. That’s not to recommend adopting the vernacular or colloquial, but question wording needs to be clear. Try to use what might be described as ‘natural language’.

Think more conceptually about how to ask questions

And realise it’s not just about the wording, it’s about the underlying concept. For example, you could ask people to rank their first, second and third choices of toothpaste, or you could say to respondents: imagine you are a judge in an award ceremony and you have to give out 1st, 2nd and 3rd prizes in the Toothpaste of the Year competition. A conceptual shift of emphasis like this that can make surveys far more interesting for respondents to answer. We too often rely on a cliched armoury of standard questions and approaches to problems, and these are habits we need to break!

3. Storyboard the flow of surveys

My next point is the lack of storyboarding in surveys. We throw respondents a jumbled mess of questions, which for the respondent has no sense of flow or structure. We ask them to think of one topic, then another, then the first again. We deliver these horrible never-ending loops of questions that make the respondent give up all hope that they will get to the end of a survey. I feel that much more thought is needed in the way we organise and deliver questions and signpost their order so respondents can grasp where they are and where they are going. It may not be important to you; it is to respondents.

4. Ask fewer dumb questions

All the research we have conducted over the last few years into what makes surveys interesting to respondents leads to one clear observation: respondents like nothing more than to actually think! Yet so often we bombard them with dumb questions that don't require more than momentary thought: how much do you agree with this, do you like this, or pick some words. Very rarely do we actually ask them to think. I encourage you to think of respondents as consultants; try to bring them into the problems you are trying to solve. Ask them more intelligent questions and I think you will be surprised at the quality of feedback you will receive. 

5. Up our game on the visual side

I don't think many people have grasped the impact that quality imagery can have at improving the experience of taking a survey, how well-chosen images can be used to communicate ideas more efficiently or the impact that an image can have on stimulating the imagination. I very rarely see imagery properly used in surveys.

More often what I see is badly chosen clip art slapped to the top of a range question, and imagery that looks like something the intern has been tasked with finding.

6. Rethink the use of likert scale grid questions

I think the industry needs to have a total rethink on the use of likert scale grid questions in surveys. They are simply not working. Respondents hate them and then deliver, at best, watered down data, and at worst, heavily corrupted data.  They are officially my number 1 bugbear with surveys.

If you take out of the equation reading times the average respondent spends 4.3 seconds thinking about the answer to an average question, but as low as 1.7 seconds considering the answer to a likert scale question in a bank of grids. If you look in detail at the data coming back from a grid question, upwards of 80% of respondents show some signs of speeding after the 20th repetition, and I see surveys with repetitions like this in their hundreds.

Part of the problem is our obsession with agreement and liking and 5 point scoring scales. Respondents are so tired of these we have an almost pavlovian response to answering them - we mostly say that we slightly agree, slightly like everything!

What we see if you offer up more creative scales can be radical improvements in attention. To my mind an agree scale is just a really lazy way of asking a question.

How much do you like watching these sports on TV on a scale of 1 to 5? Every sport will score between 2.5 and 3.5! Surely the point of most questions like this is to draw out comparisons. Just look at the behavioural reality of our attitude towards watching different sports on TV; Football would score 100 and most other sports under 10.

The solution is to think about how you ask the question more creatively, understanding the real objective of the question, using more relevant anchor points for these types of judgements and using more bespoke ranges for each measure.

A guide to writing open ended feedback questions

There are various goals to an open ended question but in most cases it is not about the volume of feedback but about the quality. Whether you are trying to get respondents to be analytical, creative or spontaneous, the biggest challenge you face is encouraging people to think and think in the right sort of ways.

The average respondent spend 15 seconds answering the average open ended question, and you get on average of 5 words. Five words might be enough if it constitutes meaningful feedback, but often it is humdrum verbiage, a bit of a nightmare to analyse. If you were to ask people to watch an ad and write down what they thought of it, the most common response would be "it was OK" . The second and third most common remarks would be, "I liked it" and "I didn't like it" , the 4th would be "I don't know" . These responses are clearly not a great deal of value, so that you might as well have asked a yes no question, “did you like it or not”.

This is a guide to show you how to write more effective open ended questions to improve the quality of feedback and to encourage respondents to:

- Put more thought into their answers
- Be more creative
- Be more analytical
- Be more reductive
- Be more free form in their thinking

Techniques for Motivating respondents to give you the type of answer you want.

The first challenge you face is to motivate respondents to answer the question you pose. Often respondents have little or no reason to write something down in the first place. So the question you should ask yourself is, what would make me want to answer this question?

1. Explain the reason why you want to know the answer: It may not always be possible or relevant to do this, but explaining why you want to know can help motivate people to give an answer. Instead of asking "what do you like about this product" you might ask it like this: "This is a very popular product but we are not quite sure why, what do you like about it?".

2. Bring respondents into the problem: I would also advocate bringing respondents into the problem you are trying to solve if you an. So instead of saying "why do you like this brand" you might say "This brand is a lot more popular amongst some people than others and we are trying to understand why."

3. Emotionalise your appeal for feedback: The more you can emotionalise the appeal, the better. Below is an example of the impact this can have. We asked one group of mothers to talk about their experience of using baby wipes and to another group we explained that the reason we wanted them to talk about their experience was to try to improve baby wipes for future generations of parents, and we supported it with this visual. We got over 100% more feedback from this group.

4. Use visuals: Using a well chosen visual to re-enforce questions, as the example above demonstrates, can have a measurable impact on improving the volume of feedback (see my other full blog poste on this). Take the example below, which was part of an international experiment we conducted looking at the general effects of taking a more creative approach to survey design. We asked one group to imagine that they were in charge of the TV coverage for the Olympics games and had to build up a list of sports they would want to cover. We visually re-enforced this request to a second group with a picture of the Olympic logo and a camera. We have tested this out in 15 countries. In every one the presence of the image increased the feedback by upwards of 20%.

5. Challenge them: Using language that challenges respondents to do something can be an extremely powerful weapon to encourage feedback. Instead of asking "what brands come to mind", you might re-phrase that to say "how many brands can you guess". Or instead of saying "what words come to mind when you think of this brand" , you might say "can you think of the most popular words people associate with this brand". This type of approach can sometimes double the level of feedback, we have discovered.

6. Use reward mechanics: Have you ever thought about offering a little prize for the best answer? I can tell you, it is an extremely powerful approach. In experiments we have run we have seen up to 6 fold improvements in feedback. It’s obviously a bit complicated to administrate, but is a somewhat overlooked technique to my mind.

Set expectations on type of feedback you want: Most people use their own standard to assess how much time and effort they should put into answering an open ended question. We have found that establishing some expectations without making them actually conditional can help improve the volume of feedback. There are several different ways you can do this.

7. Use word counters: Asking people to write X number of words on a topic and using a word counter so they can see how much they have written is a great way to extend the volume of open ended feedback. The x limit might be 3 words or 300 depending on the circumstances.

8. Set time limits: Likewise telling them how long they might spend answering a question is another clever technique. If you say "please spend no more than 3 minutes telling us what you think about this topic" this may sound like you are asking someone not to spend too long. In fact you are doing the opposite and establishing a benchmark, that if they come anywhere near to adhering to, they will have spent far than the 15 seconds they normally would spend answering the question.

9.Show examples: By showing examples of what other people have written you can subtly socially condition people to write more. If they see that other people have written several sentences it will encourage them to write a similar amount.

Understand the barriers for respondents to give feedback

10. Deal with spelling and literacy concerns: One of the reasons why some people are reluctant to give open ended feedback in surveys is because they are worried about their spelling and how well they can write. There are a couple of ways of dealing with this. First, make it clear that you are not worried about this. Secondly we have found that showing a range of informal feedback, with not necessarily perfect spelling, encourages respondents to be less concerned about their personal style of delivering feedback.

11. Foreplay - now's not the right time to get intimate! People are often reluctant to answer open ended questions at the very start of a survey; in fact, this type of question can be the cause of considerable drop-out. You need to warm people up and get them in the right frame of mind to answer open ended questions first.

12. Tackle boredom thresholds: People put far less effort into answering open ended questions at the end of a survey and there are often trigger points you will notice wherein people tend to give up, and their desire to get to the end of a survey cuts in so strongly that their responses to open ended questions dry up.

13. Being overwhelmed by choice of what do or say: Sometimes people are overwhelmed by the choice of things they could say in an open ended question and it leads them to say just one thing. If I ask you, for example, what your favourite bands are, this is a highly complex question to answer as someone may like lots of different bands and music artists and with all that choice your mind implodes and you can’t think of any that stand out. This issue has been observed by social scientists in a famous experiment in which people were asked to name something they might find in a fridge. The average person got stuck after naming 7 items. Yet if they were asked to name white things, or things with sell by dates, or bottled items, people were able to list a lot more. The lesson to learn from this is that when wording questions, you need to think of respondents like oysters: you need to feed them the grains of sand to stimulate the pearls of thinking and help them focus. So you might ask “what was the first band you fell in love with?” or “what was the last?”.

How to stimulate more thinking

14. Break out into thinking chunks: If you are asking what do you like or dislike about something you will find that if you break this out into 2 open end text boxes ("likes" and "dislikes" ), you will get more responses. I often see open ended questions that try to ask several questions at once all in one question. My advice would always be to chunk it up and separate out to unique questions the separate things you want to know. You can cluster them together so it comes across as one specific topic you are asking about, but you will get much better feedback if you do it this way.

15. Use random stimulation: Giving respondents some sort of random stimulus can be an extremely powerful means of unlocking ideas. Say you are wanting people to think of an idea for a new type of ice cream; you could create maybe 20 or so random thoughts to help people consider things from different angles. This could be anything such as "if Lady Gaga were creating an ice cream what would it be like", "invent an ice cream to eat on cold days", "create an ice cream you think football fans would like" etc - the more abstract the better and to each respondent you present a random set of these stimuli, one at a time.

16. Set rules: I have talked about this quite extensively in some of my work on the gamification of research, but adding a rule to a question can make it a whole lot more interesting and fun for respondents to answer and thus trigger more thoughtful feedback. The easiest way to do this is by placing word limits - e.g. review this film in 3 words.

17. Make it personal: Always remember you are one human being talking to another and most people's favourite topic is themselves, so try wherever possible to frame the question in a personal context. Instead of saying “why do you like this brand?” you might say "if a friend asked you what you thought of this brand what would you say?".

18. Realise the power of the word "imagine": The word imagine is a magical word to use as a means of getting people to think. We did an experiment in which we asked one group of people to make a list of their favourite shops and to another group said "imagine you could design your perfect shopping centre, what shops would you have in it" - this increased the feedback 4 fold.

19. Apply NLP techniques: I don’t pretend to be an expert in this but the concept of the embedded command in a question can be extremely powerful. I could say to you "tell me what you do in 7 words" or I could say "tell me what you do in EXACTLY 7 words". Respondents seem to treat the word EXACTLY as more of a command, so it tend to solicit more responses.

20. Study the form:  It is interesting to see which discussion topics posted on linkedin trigger the most comment and involvement. By studying this you can start to get a feel of how to pitch a question that solicits a strong response.

What can you achieve if you get it all right?

In experiments the biggest uplift in feedback we have ever achieved as a result of redesigning the way we asked an open ended question has been a 6 fold increase from 17 words to 103 worlds. This used a combination of techniques listed above and might be described as my current PB. With the right thought and approach you can easily achieve double or treble the volume of feedback by re-engineering the wording of a question. But note that, as I said at the start, it is often not about volume of words so much as about the clarity of thinking these words stimulate. We find that with increases in volume feedback is also linked to more thoughtful feedback.

So what to do with all that open ended data?

All this begs the question about what you do with all this open ended feedback, which can be overwhelming. Some clever ways of automatically processing open ended feedback have emerged with social media text analytic software and natural language computational techniques, and there are some very interesting things you can do with respondent self coding feedback, but this I think is a topic for a future post....