uSamp Blog

The Answer Network

Product Review: Google’s Consumer Surveys from a Market Research Perspective

« Back to Main Page

by Matt Dusig, Co-Founder & CEO and Daniel Ross, SVP of Product & Technology, uSamp

When Google launches something new, it’s always a big deal-no matter how big or small. Within a matter of minutes, blogs are abuzz with speculation (“What is Google Up To?“). Spectators are overrun with whirlwind of emotions: excitement, fear, resentment, and praise for the powerhouse. Its latest innovation (or disruption-depending on which way you read it), Consumer Surveys, is not a surprising move for the company that holds the key to what Leonard Murphy identifies as the “ ‘Big Data’ aspect.” No doubt, Google has made its way into nearly every aspect of Internet life, and this move into the market research industry brings good visibility to on-demand SaaS insights. But if they want to be a true player in this space, there are a few key considerations to keep in mind.

  • Google’s primary focus is to monetize premium content for publishers. In this vein, the solution is more polling than full surveying capabilities.
  • Pricing is somewhat elusive. For demo targeting, it’s $.50 per response. But they don’t clearly tell you that a response is a single question. A 20 question survey with a screener or demo targeting will be a $10 CPI.
  • Analysis of the relationship between survey questions is difficult and sometimes not even possible because of the question number limitation. Each person only answers 1 of the 20, and Google aggregates the data, which makes deep analysis harder.
  • Targeting is limited to just age, gender and census region.
  • Timeliness constraints such as completing the survey in a timely manner, publisher inventory at the time and competition with other surveys.

Their solution lacks many features that brands, insight directors and market research professionals would need and expect. But as we know, Google usually launches skeleton applications and builds upon them based on client feedback. And there are two elephants in the room  – one which plays to their advantage: mobile (Android and the world of apps); and one which is potentially a bumpy road: data mining and privacy concerns.

Daniel Ross, SVP of Product and Technology at uSamp decided to take Google’s Consumer Survey insights program for a test drive. The following deconstructs the insights program from survey set-up to pricing to analysis. Ross researches a general population of individuals in the US that have gone on a cruise at least once in the past year. The sample size for this study is N=200.

Survey Set-up

Users have the option to:

  • Tag surveys with keywords, so they can be referenced later in the “Surveys” section
  • Access a small set of pre-canned basic survey templates
  • Copy from an existing survey

Audience Selection and Screening

Targeting options are pretty broad, which proves useful when looking for a general population audience. Users have the ability to survey the US represented population, specify gender, age, or geography (by 1 of 4 regions) based on inferred data or can decide to use a screener question.

Figure 1

Given the niche audience (“cruise goers”), it seemed appropriate to use the “custom audience using a screening question” (see Figure 1). At first glance, it appears that users have the ability to screen people.

To test this observation, the following question was posed:

“Which of the following activities have you done in the past 12 months?” Should users not select “Cruise Vacation,” they would be terminated  from the survey.

However, Google doesn’t present the option to list of several activities, and instead survey creators must rephrase their question to accomodate a “Yes/No” answer selection. In this case, the question was revised to:

“Have you ever been on a cruise vacation?”

Based on this test alone, the “custom audience” selection is a bit misleading as one can’t really target custom audiences through detailed user profiles. Rather it’s a way to target by asking the general population and then weeding through to find the needle in a haystack.

Pricing

Google offers 3 different pricing selects.

Pricing is variable based on the number of questions and targeting options. For example, a 3 question survey with a screening question will cost $1.50 per response.

  • General population: $0.10 per response (per question) with a $100 cost minimum.
  • Inferred demographic targeting*(gender, age, geo region): $0.50 per response (per question)
  • *Inferred demographic targeting is NOT self-reported data by the survey taker. This is inferred based on the user’s IP address and cookie data
  • Custom audience (Target a specific population like dog owners, moms, or golfers using a screening question (e.g. “Do you own a dog?”): $0.50 per response (per question).

Survey Creation

Once you’ve determined the audience selection, the survey building begins! 3 questions were added on top of the screening question configured during the pricing stage.

  • You can build a survey with as many questions as you’d like with single select, multi-select, rating and image options.
  • For question types with lists, you are forced to randomize the list if you have more than 5 answer choices.
  • There’s no ability to set standard survey workflow logic (e.g. skips or branching). The best way to understand what’s possible is to think of a 1 question poll. Your survey (or poll for that matter) is sent out to users in the form of 1 question-which means that those who complete the “survey” are only going to see 1 of the 3 questions.

Figure 2

Google is presenting only 1 question to users, which means you get a unique set of individuals answering only 1 of the 3 questions. As you can see in Figure 2, there is not an equal number of people that have completed the survey questions because it’s a different group of individuals answering each one.

Analysis and Presentation

This makes for an extremely difficult analysis should you want to do a comparative analysis (cross-tab) between questions. For example, if you want to see how important the price of a cruise is to people relative to how someone purchased their last cruise vacation, the analytics are one-off and only allow you to see how people responded to a single question.

Figure 2 shows survey progress 15 hours after the survey was launched–just about half of where we need to be at 200 responses. 70% of respondents were dropped after the screening question—so it is taking a bit longer to complete. Since it’s still early, Google likely only has their polling widget (which they refer to as “Survey Wall”) on a few key publisher sites.

Google’s strength in all of this is their graphing/analytics solution, which uses Google’s charting APIs, but the previous mention of not being able to compare question answers is a huge drawback. Figure 3 shows the results for 1 question compared with the inferred demo/geographic data (per IP location and cookie data).

Figure 3

In summary, here are key findings from this preliminary trial:

Strengths:

  • graphing/analytics solution
  • statistical relevance measures
  • tagging functionality

Drawbacks:

  • custom audience misleading
  • screeners ineffective
  • inability to compare question answers
  • delayed response rate

It is no surprise that Google has come to play in the consumer intelligence space. Whether they can execute a survey platform properly is another challenge that might require the search engine behemoth to take a page from the MR industry.

April 4th, 2012

Posted in Uncategorized

with 7 comments

7 Responses to 'Product Review: Google’s Consumer Surveys from a Market Research Perspective'

Subscribe to comments with RSS or TrackBack to 'Product Review: Google’s Consumer Surveys from a Market Research Perspective'.

  1. great work as usual guys. you have answered a lot of my questions about GOOG’s new product. something i’ve come to expect from “the answer network”.

    Matt

    4 Apr 12 at 8:43 pm

  2. i think GOOG will have some profiling done as per user’s input in their google account. As they also have tons of enterprise/business email accounts, it wont be hard for them :)

    deepak iyer

    5 Apr 12 at 6:27 pm

  3. True…Although it seems a bit complicated yet they have huge potential to tap into especially with hundreds of publishers on board, there is no dearth of sample.

    -Vivek (http://www.adtrack.co.in/2012/04/google-launches-online-consumer-survey-platform/)

    Vivek

    6 Apr 12 at 9:51 am

  4. Just to be clear: if you have more than 5 choices in a question, they will show ONLY 5 (randomly selected) of those choices to any individual respondent. Not what most of us expect!

    To me, the primary value of the system is the general population survey to estimate incidence across the U.S. For the $100 charge from Google, I could see running a genpop question to calibrate my next panel survey.

    Jeffrey Henning

    10 Apr 12 at 11:31 am

  5. What’s Going down i am new to this, I stumbled upon this I have discovered It absolutely helpful and it has aided me out loads. I am hoping to contribute & assist other users like its helped me. Good job.

  6. Just desire to say your article is as astonishing. The clearness in your post is simply excellent and i can assume you are an
    expert on this subject. Fine with your permission let me to grab your feed
    to keep updated with forthcoming post. Thanks a
    million and please continue the gratifying work.

  7. There is definately a great deal to know about this issue.

    I love all of the points you made.

Leave a Reply