by Matt Dusig, Co-Founder & CEO and Daniel Ross, SVP of Product & Technology, uSamp
When Google launches something new, it’s always a big deal-no matter how big or small. Within a matter of minutes, blogs are abuzz with speculation (“What is Google Up To?“). Spectators are overrun with whirlwind of emotions: excitement, fear, resentment, and praise for the powerhouse. Its latest innovation (or disruption-depending on which way you read it), Consumer Surveys, is not a surprising move for the company that holds the key to what Leonard Murphy identifies as the “ ‘Big Data’ aspect.” No doubt, Google has made its way into nearly every aspect of Internet life, and this move into the market research industry brings good visibility to on-demand SaaS insights. But if they want to be a true player in this space, there are a few key considerations to keep in mind.
- Google’s primary focus is to monetize premium content for publishers. In this vein, the solution is more polling than full surveying capabilities.
- Pricing is somewhat elusive. For demo targeting, it’s $.50 per response. But they don’t clearly tell you that a response is a single question. A 20 question survey with a screener or demo targeting will be a $10 CPI.
- Analysis of the relationship between survey questions is difficult and sometimes not even possible because of the question number limitation. Each person only answers 1 of the 20, and Google aggregates the data, which makes deep analysis harder.
- Targeting is limited to just age, gender and census region.
- Timeliness constraints such as completing the survey in a timely manner, publisher inventory at the time and competition with other surveys.
Their solution lacks many features that brands, insight directors and market research professionals would need and expect. But as we know, Google usually launches skeleton applications and builds upon them based on client feedback. And there are two elephants in the room – one which plays to their advantage: mobile (Android and the world of apps); and one which is potentially a bumpy road: data mining and privacy concerns.
Daniel Ross, SVP of Product and Technology at uSamp decided to take Google’s Consumer Survey insights program for a test drive. The following deconstructs the insights program from survey set-up to pricing to analysis. Ross researches a general population of individuals in the US that have gone on a cruise at least once in the past year. The sample size for this study is N=200.
Users have the option to:
- Tag surveys with keywords, so they can be referenced later in the “Surveys” section
- Access a small set of pre-canned basic survey templates
- Copy from an existing survey
Audience Selection and Screening
Targeting options are pretty broad, which proves useful when looking for a general population audience. Users have the ability to survey the US represented population, specify gender, age, or geography (by 1 of 4 regions) based on inferred data or can decide to use a screener question.
Given the niche audience (“cruise goers”), it seemed appropriate to use the “custom audience using a screening question” (see Figure 1). At first glance, it appears that users have the ability to screen people.
To test this observation, the following question was posed:
“Which of the following activities have you done in the past 12 months?” Should users not select “Cruise Vacation,” they would be terminated from the survey.
However, Google doesn’t present the option to list of several activities, and instead survey creators must rephrase their question to accomodate a “Yes/No” answer selection. In this case, the question was revised to:
“Have you ever been on a cruise vacation?”
Based on this test alone, the “custom audience” selection is a bit misleading as one can’t really target custom audiences through detailed user profiles. Rather it’s a way to target by asking the general population and then weeding through to find the needle in a haystack.
Google offers 3 different pricing selects.
Pricing is variable based on the number of questions and targeting options. For example, a 3 question survey with a screening question will cost $1.50 per response.
- General population: $0.10 per response (per question) with a $100 cost minimum.
- Inferred demographic targeting*(gender, age, geo region): $0.50 per response (per question)
- *Inferred demographic targeting is NOT self-reported data by the survey taker. This is inferred based on the user’s IP address and cookie data
- Custom audience (Target a specific population like dog owners, moms, or golfers using a screening question (e.g. “Do you own a dog?”): $0.50 per response (per question).
Once you’ve determined the audience selection, the survey building begins! 3 questions were added on top of the screening question configured during the pricing stage.
- You can build a survey with as many questions as you’d like with single select, multi-select, rating and image options.
- For question types with lists, you are forced to randomize the list if you have more than 5 answer choices.
- There’s no ability to set standard survey workflow logic (e.g. skips or branching). The best way to understand what’s possible is to think of a 1 question poll. Your survey (or poll for that matter) is sent out to users in the form of 1 question-which means that those who complete the “survey” are only going to see 1 of the 3 questions.
Google is presenting only 1 question to users, which means you get a unique set of individuals answering only 1 of the 3 questions. As you can see in Figure 2, there is not an equal number of people that have completed the survey questions because it’s a different group of individuals answering each one.
Analysis and Presentation
This makes for an extremely difficult analysis should you want to do a comparative analysis (cross-tab) between questions. For example, if you want to see how important the price of a cruise is to people relative to how someone purchased their last cruise vacation, the analytics are one-off and only allow you to see how people responded to a single question.
Figure 2 shows survey progress 15 hours after the survey was launched–just about half of where we need to be at 200 responses. 70% of respondents were dropped after the screening question—so it is taking a bit longer to complete. Since it’s still early, Google likely only has their polling widget (which they refer to as “Survey Wall”) on a few key publisher sites.
Google’s strength in all of this is their graphing/analytics solution, which uses Google’s charting APIs, but the previous mention of not being able to compare question answers is a huge drawback. Figure 3 shows the results for 1 question compared with the inferred demo/geographic data (per IP location and cookie data).
In summary, here are key findings from this preliminary trial:
- graphing/analytics solution
- statistical relevance measures
- tagging functionality
- custom audience misleading
- screeners ineffective
- inability to compare question answers
- delayed response rate
It is no surprise that Google has come to play in the consumer intelligence space. Whether they can execute a survey platform properly is another challenge that might require the search engine behemoth to take a page from the MR industry.