One market researchers viewpoint on the World Cup

So England got dumped out in the first round of the World Cup and everyone in our country feels disappointed, an emotion we are quite used to feeling. So begins a round of postmortems that we all probably secretly enjoy as much as the competition itself, working out who to blame for the team’s failure.  In past World Cups this has been quite easy: for example David Beckham kicking a player and getting sent off, having a turnip head as a manager or a lack of goal line technology.  But this year we are all fairly universally perplexed. I have read a lot of overfit analysis, none of which is particularly convincing because, well, in the scheme of things we all thought we played quite well, we had a sparky young team. It seems like we were just a bit unlucky this time round.

The role of randomness

Its quite hard to accept the role that randomness plays in the outcome of world cup matches.  Every nation when they get kicked out or fail to even qualify probably believes their teams were "unlucky" and that their teams are better than they actually are.  So what is the relative importance of  luck v skill when it come to winning the world cup?

Unlike the premiership where there are 38 games over which time the performance of the teams is largely correlated to the quality of the squads (take a read of the Number Game* by Chris Anderson and David Sally)  performance of a world cup squad cannot be calculated by the aggregated skill value of the squad there is a lot more randomness involved.  Imagine if the premiership only lasted 3 games: in two out of the last four seasons the team that one the premiership might have been relegated.

*a must read if you are a market researcher and like football!

There is another factor too, in the premiership the best players get sucked up into the best teams hence the much higher win ratios between the top and bottom performing sides compared to the world cup where the best players are distributed more randomly and is proportional to the size of each footballing nation.  This in tern makes the outcome of international matches even more random.

Who influences the outcome of a match?

If you look at who has goal scoring influence across a team you will notice that the negative effects of causing goals a pretty well distributed across a team but the positive effects of scoring goals are a lot more clustered amongst some individuals. See chart below showing statistics from an imaginary team based on typical performance data taken from the premier league.

 

The potential performance of a world cup team must be measured not by the overall skill value of the team but the value of a smaller network of attacking based players who can make the most game changing contributions. In the case of players like Lionel Messi a single player can carry the whole goal scoring burden of a squad.  It only takes one or two randomly allocated star players in a world cup team to elevate its performance chances (think of Pele or Maradonna).

The performance of defence is more a case of luck. You might have one or two unreliable defenders who you may not want in your premier league squad because you know over the course of a season they may cost you a match or two, but at the individual game level and a world cup is based on the outcome of  three or four key individual games, the chances are a poor defender might well run their luck.   The other two important factors defenders have to contend with are the extra stress and lack of team playing experience of a world cup team compared to a premiership squad.  Without doubt stress plays a big part, players are really hyped up and there is probably an order of magnitude increase in tension which is the root cause of many errors in world cup matches. If you look at the defensive mistakes that cost us goals in recent world cups some of the biggest mistakes were caused by effectively our most reliable players, John Terry and Steven Gerrard and Phil Neville.  There is also a lack of formation practice to contend which is particularly critical for defence. How many hours of playing together does it take for a defence to gel? Most world cup squads have days rather than months to prepare.

A team like England might well have a higher aggregated skill performance average compared to other teams, but this does not result in the same reliable performance ratios that you see in the premiership. This is because over half the value is based on their defensive skill which can be completely undermined by bad luck and we don’t have a cluster of super-skilled players to elevate the team out of bad luck matches by scoring more goals than we let in.

The influence of the Ref

To win world cup matches you are much more reliant on the manager’s structural approach, the contributions from clusters of individuals who might form good attacking combinations and one other person – the REF!  Or rather, the ref in conjunction with the crowd and the linesmen.

If you analyse a typical game you will find that the number of major goal scoring decisions that are in the hands of the referee and linesmen are actually enormous compared to any individual player. It’s difficult to put a figure on it but let’s say on average there are about 6 decisions that could have affected a goal one way or another by the referee* its instantly obvious the relative influence they have on a match.

*That is a wisdom of the crowd estimate by asking a collection of football fans how many goal-affecting decisions are made in the match by the referee and linesmen, six was the median average estimate.

Now in nine times out of ten these decisions balance themselves out but refs are only human and so it’s no wonder why there is such a big home team advantage – with 50,000 fans screaming penalty it must be extremely difficult for refs not to be influenced by the crowd.  In fact you can almost put a figure on the influence of the crowd by comparing home and away goal scoring averages the home side gains an average 0.35 of a goal per game net advantage if you examine premiership games,  which can only be really down to the net contribution of the crowd/ref decision effects.

It’s no wonder as a result that there is such a disproportionate home nation advantage.  Effectively every home nation team is starting with a 0.35 goal lead, this advantage aggregated up over the course of a tournament  has means that nearly 30% of all world cups have been won by the home nation that is 10 times higher than chance.

Am I likely to ever see England win another world cup in my lifetime?

Is probably a question most England fans ask themselves. What does it take to win a world cup – how good do you have to be to override luck?  We have taken a look at this and run some calculations.

The chart below take a little explaining but it maps out a team’s skill level v the number of times it’s likely to win a world cup over the course of someone’s average football supporter’s lifetime of 72 years = 18 world cups.  If there are 32 teams in a word cup and you are an average team and your team qualifies for every world cup final the chances are you will win 1.1 world cups over your lifetime. If in you are England and only qualify roughly 80% of the time the changes will drop to 0.96.  If your team is twice as good as average, you are likely to win roughly 2 world cups and 4 times better 6 world cups.

 England have one one world cup, Germany three and Brazil five so does that mean we are average team and Germany are three times better than us and Brazil four times better than us?

Well essentially yes, if you look at the average game-win ratios of all the teams that have played the most regularly in World Cups v the number of World Cups they have won its pretty closely correlated at 0.91.    Germany has a three times higher win ratio than us and Brazil four times higher.

Now I appreciate there is some self-selection involved here – this chart should really be based on first round matches only for a totally fair comparison, but we don’t have that data. I think it’s reasonable to say though that England has not really been done out of its fair share of World Cups.  I think we have won as many as our teams aggregated performance deserves.  You might argue that some teams have been luckier than most: Italy certainly and others unlucky, Mexico should have won it twice by now based on their aggregated performance.

Roughly speaking that means for me there is only about a 40% chance I will witness England lift a world cup in my remaining lifetime but almost certainly I will have to endure another series of victories for Germany and Brazil.  Oh well better come to terms with it but I live in hope.

But lets fantasise for a minute, how many world Cups could we have won?

Imagine we lived in an infinite number of universes where for the last 70 years we had been playing an infinite series of world cups with a team with the same skill level.

Well on average 32% of the time we would not have won a single world cup by now, in 21% of cases  we might have picked up two and in 7% of case the same number as Germany.  There is one universe in 5,000 where Germany would not have won a single world up and England would have won 4!  Anyone fancy moving there?

Baby steps into the wearable era of research: ESOMAR DD 2014 Roundup

Compared to other global research events the ESOMAR Digital Dimensions conference is by no means the biggest, it faces competition, without doubt, from more ‘ideas’ driven events, but never the less it is by far and away still my favourite market research event on the global calendar. Now I have to say that because I was chairman this year, but I do feel that despite all the competition, it has reliably proved to be one of the most fruitful sources of new thinking and new trends for the market research industry - I consistently learn so much more at this event compared to the others I attend and this year it was particularly fruitful.

I think that part of its success is down to the consistently high standards ESOMAR sets on paper submission, only 1 in 5 papers get selected and it also demands a lot more robust thinking from its participants. What you get as a result from this conference is a really thoughtful mixture of new ideas, philosophy and argued out science.

This year was one of the strongest collections of papers ever assembled, so much so that the selection committee asked to extend the prizes beyond 1st place. There were 6 major themes that emerged and 1 paper that I think could go on to have a major impact well beyond the boundaries of market research and I returned home with 23 new buzzwords and phrases to add to my growing collection (see other post).

The big themes

1. The Physiological data age: At this conference we witness some of the baby steps being taken into the world of wearable technology; and a prostration by Gawain Morrison from SENSUM who were one of the stars of the event, that we are about to enter the physiological data age.  They showed us a galvanic skin response recording of a 7 hour train journey which revealed the insight that the highest stress point on the journey was not caused by any delays or anxiety to reach the station but when the on-board internet service went down!  IPSOS are one of many MR companies to start experimenting with google glasses and showed us how they were using them to conduct some ethnographic research amongst new parents for Kimberly Clarke. We saw some wonderful footage of  a father interacting with his new born child in such a natural and intimate way it does not take much of a leap of the imagination to realise wearable technology is going to be a big topic in future MR events.

2. The Big Privacy issues looming over these new techniques:   With the rise of wearable devices raises a whole range of new issues surrounding data privacy that was widely discussed at this conference,  Alex Johnson highlighted in his award winning work Exploring the Practical Use of Wearable Video Devices, which won best paper, - the central emerging dilemma - it’s almost impossible to avoid gathering accidental data from people and companies who have not given their consent to take part in the research when doing wearable research. It’s critical for the research industry to take stock of.

3. Developing the new skills needed to process massive quantities of data:  The second big focus of this conference, that Alex Johnson’s paper also highlighted, was the enormity of the data evaluation tasks researchers face in the future, for example processing hundreds of hours of video and meta data generated from wearable devices.  Image processing software is a long way from being able to efficiently process high volumes of content right now. He had some good ideas, to process this type of data. He proposed a whole new methodological approach which centres around building taxonomies and short cuts for what a computer should look for and a more iterative analytical approach.  In one of the most impressive papers at the conference TNS & Absolute Data provided an analytical guide to how they deconstructed 20 million hours of mobile phone data to build a detailed story about our mobile phone usage, that could be utilised as a media planning platform for the phone – the research battle ground of the future is surely going to be fought on who has the best data processing skills.

4. De-siloed research techniques: I wish I could think of a better simple phrase to describe this idea as it was probably the strongest message coming out of the ESOMAR DD conference - the emergence of a next generation class of more de-siloed research methodologies, that combined a much richer range of less conventional techniques and a more intelligent use of research participants. Hall & Partners described a new multi-channel research approach that involved a more longitudinal relationship with a carefully selected small sample of participants where across 4 stages of activity they engaged them in a mix of mobile diary, forum discussion and conventional online research - challenging them to not just answer questions but help solve real marketing problems; Millward brown described a collaboration with Facebook where they mixed qual and mobile intercept research and task based exercises to understand more about how mobiles are used as part of the shopping experience;  Mesh Planning described how they integrated live research data with fluid data analysis to help a media agency dynamically adjust their advertising activity; IPSOS showed us some amazing work for Kimberly-Clarke that spanned the use of Facebook to do preliminary qual, social media analysis, traditional home based ethography, and a new technique of glassnoraphy. What all these research companies demonstrated was that decoupled from the constraints of convention, given a good open brief from a client and access to not just the research data that the research company can generate but the data the client has themselves we saw some research companies doing some amazing things!

5. Mining more insights from open ended feedback:  Text analytics in infancy focussed on basic understanding of sentiment but 3 great papers at the event showed how much more sophisticated we are becoming at deciphering open ended feedback.  Examining search queries seems to be a big underutilised area for market researcher right now and KOS Research and Clustaar elegantly outline how you could gather really deep understanding of people’s buying motivations by statistically analysing the search queries around a topic.  Annie Pettit from Peanut Labs, looking at the same issue from the other end of the telescope, showed how the suggestions to improve brands and new product development opportunities could be extracted from social media chatter by the careful deconstruction of the language they used to express these ideas.  And Alex Wheatley, in my team at GMI, who I am proud to say won a silver prize for his paper, highlighted just how powerful open ended feedback from traditional market research surveys could be when subjected to quant scale statistical analysis, rivalling and often surpassing the quality of feedback from banks of closed questions.

6. Better understanding the role of mobile phones & tablets in our lives: We learnt  a whole lot more about the role of mobile phones and tablets in our lives at the conference, some of it quite scary.  We had expansive looks at this topic from Google, Yahoo and Facebook.  AOL provided some useful “Shapely value” analysis to highlight the value of different devices for different tasks and activities, it demonstrated how the tablet is emerging as such an important “evening device” , its role in the kitchen and bedroom and how the combination of these devices opens up our access to brands.  We learn how significant the smart phone is when we go retail shopping for a combination of social and investigative research reasons. We learn about the emergence of the “Google shop assistant” many people preferring to use google in shops to search for their shopping queries than actually ask the shop assistants and how we use the phone to seek shopping advice from our friends and how many of us post our trophy purchases on social media.

The impact of technology on our memory

The paper that had the single most impact at the conference was some research by Nick Drew from Yahoo! and Olga Churkina from Fresh Intelligence Research showing how our use of smart phone devices is really impacting on our short term memory – we are subcontracting so many reminder tasks to the technology we carry around with us that we are not using our memory so actively and this was demonstrated by a range of simple short term memory test correlated with mobile phone usage found the heavier smart phone users performing less well. The smart phone is becoming part of our brain!  This obviously has much bigger implications outside of the world of market research and so I am sure we are going to hear a lot more about this topic in the future.

Scary thought, which made the great end session by Alex Debnovsky from BBDO about going on a digital detox all the more salient.  I am going to be taking one soon!

23 Buzzwords coined and used at the 2014 ESOMAR Digital Dimensions Conference

This years Digital Dimensions conference produced a particularly good harvest of buzzwords, some of which you may have heard before but some I can guarantee are 100% new!

The conference buzzword award has to go to John Humphrey Kimberly Clark & Joost Poolman Simons from IPSOS who in one presentation delivered more new additions to the market research vocabulary pool than I think I have heard in one hit. 

1. Glassnography: The new term for ethnography using google glasses Coined in presentation by: John Humphrey Kimberly Clark & Joost Poolman Simons Ipsos

2. Privacy:  The word probably mentioned more often at this conference than any other (source: various)

3. Fanqual:  Doing qualitative research amongst your Facebook fans by posting ideas and getting their feedback  Coined in presentation by: John Humphrey Kimberly Clark & Joost Poolman Simons Ipsos

4. Spammyness index = How Impersonal x how intentionally manipulative a piece of marketing communication is. Coined by Jacob White VisualDNA

5. Cupcake research: A derogatory term for the type of research that looks great, but is too sweet to eat and has little or no healthy substance.  Coined in presentation by: John Humphrey Kimberly Clark & Joost Poolman Simons Ipsos

6. Phylogenetics: A new very nerdy means of analysing social networks devised and discussed in a paper by OMD (don’t ask me to explain it!)

7. Amazonification: The observation of how some people are using Amazon as first point of call for researching and  purchasing so many different things (Note Ebay is used in a similar way by another sub strand of the population but ebayification doesn’t sound quite right!) Coined in presentation by: John Humphrey Kimberly Clark & Joost Poolman Simons Ipsos

8.Data exhaust: The data pollution that pours out systems that we cannot effectively use (source: various)

9. Shapely value:  The game theory technique devised buy Lloyd Shapely he won a Nobel Prize is emerging as a great way of segmenting activity usage in market research which was ably demonstrated by the team from AOL

10. The Big 5: The five key personality traits to understanding human behaviour: openness, conscientiousness, extraversion, agreeableness, and neuroticism. These are becoming the cornerstone of many standard research measurement techniques so they get a collective noun the big 5  Coined by Jacob White VisualDNA

11. The physiological data era:  Wearable technology is going to be the dominant new source of market research data over the next decade and a prediction that physiological data is going to become one of the key metric of market research coined by Gawain Morrison from SENSUM

12. Why analytics: Analysing big data to gain deeper insights into customers wants and needs (source various)

13. The army of influencers:  An observation of what it feels like to be an expectant mum for the first time and the bewildering array of advice you are offered  in the digital age.  Coined in presentation by: John Humphrey Kimberly Clark & Joost Poolman Simons Ipsos

14. Corporate purpose: In the social media age, companies are realising the collective power of consumers – business that simply pursue the need to make money and don’t  address  the actual needs of their consumers in the decisions can easily become a cropper.  To address this there is a growing trend amongst business to try and define their “Corporate purpose”  Coined by Jacob White VisualDNA

15. Share of experience:  Share of voice is a fairly meaningless idea in a wold where we are bombarded by messages in multi-dimensions.   We should focus on share of experience which is more about measuring what is cutting through.  Coined by Chris Wallbridge Mesh Planning

16.Renegade professionals:  “The future is in the hands of the “renegade professionals” So much innovation is happening outside conventional businesses and it’s the outsiders, the renegades that are inventing the future – very much on their terms!

17. Microboredom:  Those moments waiting in the queue when we noodle on our mobile phones. Mentioned by Alex Drozdovsky BBDO Worldwide

18. Technotots & digikids:  Terms to describe the proliferation and impact of technology on young children Mentioned by Alex Drozdovsky BBDO Worldwide

19.The google shop assistant: The phenomenon that many people out shopping would prefer to look up their query on their smart phone than ask the shop assistant. Google is becoming the shop assistant!

20. Selection over sample: New research techniques are going to be more reliant on having the right type of active participants than larger numbers of balanced sample.  Coined by Grant Bird Hall & Partners

21. The Zero moment of memory: We are using our smartphones more and more to replace our own memory. Coined by Nick Drew from Yahoo! and Olga Churkina from Fresh Intelligence Research

22. Digital Detox: The idea of decoupling ourselves from our digitial devices to allow us to unwind coined by Alex Drozdovsky BBDO Worldwide

23. Turning off is the new emotion: Another idea coined by Alex Drozdovsky BBDO Worldwide we are being so bombarded by data and information in  the new digital age that turning off is the new emotion

A Network of Brains

I would like to put forward a new viewpoint on research participants.

I would like you to move away from thinking of participants as binary bits of data that you patch together to build a picture of a market and behavior, to start thinking of them of a network of brains that you can use to help solve marketing problems.