AdWords is now Google Ads. Our new name reflects the full range of advertising options we offer across Search, Display, YouTube, and more. Learn more

Ads
1.9K members online now
1.9K members online now
Dive into advanced features like Remarketing, Flexible Bid Strategies, AdWords Editor, and AdWords Scripts
Guide Me
star_border
Reply

How do I measure the true ROI - control group testing?

Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

Hi All

 

I'm relatively new to AdWords but I believe the issue of control group testing to be an advanced topic in AdWords. I would like to construct an AdWords campaign so that the true ROI can be evaluated considering any indirect impact and seasonality. Is it possible to set-up an AdWords campaign with simultaneous target and control groups, so that if a search would usually result in an ad impression it doesn't do so for a specified proportion of searches with the same keywords (the control group, all else being equal on average)?

 

Thanks in advance,

 

Pritpal

1 Expert replyverified_user
1 ACCEPTED SOLUTION

Accepted Solutions
Marked as Best Answer.
Solution
Accepted by topic author Pritpal W
September 2015

Re: How do I measure the true ROI - control group testing?

Rising Star
# 6
Rising Star

I would caution you not to try to create any advertising structure that is designed to avoid creating activity. There are long-term performance and quality metrics associated with your website's domain name that you risk damaging if you deliberately include low-quality elements in your AdWords campaign(s).

 

In the end, though, the problem is that ACE is not designed to be used the way you describe. It's designed to be used to measure the potential impact of a change in your AdWords advertising. There is no truly controlled way to use it to measure the impact of advertising vs. not advertising. 

 

Your first post said you were new to AdWords, but did not say that your product/service was a new offering. If you've had your website up for a while, you should already have some baseline interest and sales data, which will make evaluating the impact of advertising easier?

 

 

If that isn't a good solution, then it seems to me that the geographic targeting is going to be your best shot at figuring out how much an advertising campaign will affect sales. 

 

 


Theresa
Google AdWords Top Contributor
*~*~*~*~*~*~*~*

View solution in original post

Re: How do I measure the true ROI - control group testing?

Rising Star
# 2
Rising Star

Good morning.

 

First, I would strongly encourage you to read the documentation on ACE (AdWords Campaign Experiments) before you start, so that you understand that the program can and cannot do.

 

Second, yes, the program is designed to allow you to "test" different items in your campaign, to see which perform better.

 

I do, however, question your perception that you cannot measure "true ROI" without ACE and I'm not sure what you mean by the "indirect impact" of any features.

 

Nor will ACE provide you with any insight into seasonality. If there is more/less demand for your products/services at some times, ACE cannot magically produce more demand than exists or "even out" the demand over time.

 

You said, "...so that if a search would usually result in an ad impression it doesn't do so for a specified proportion of searches..."

 

I'm afraid I don't quite understand what you're saying here. The wording suggests that you want to advertise in a way that avoids getting your ads triggered?

 

If you can add some additional detail or clarification, we'll be happy to try and offer some ideas.


Theresa
Google AdWords Top Contributor
*~*~*~*~*~*~*~*

Re: How do I measure the true ROI - control group testing?

Visitor ✭ ✭ ✭
# 3
Visitor ✭ ✭ ✭
Hi Theresa

Thank you for your response and your offer of assistance. I've read that ACE documentation and believe that I can answer my 'what is the true ROI' question by identifying two cities with similar demographics, targeting one with an adwords campaign while using the other as the 'do nothing control group' and comparing the financials between the two. I've explained the problem and my reasoning in more detail below - if you would like to correct or confirm my thinking please feel free (but not obligated) to do so.

The question that needs to be answered is 'What is the incremental financial impact of doing an adwords campaign, all else being equal'. Incremental meaning what is the difference between two scenarios which happen at the same time (like in a drug trial) - 1) running the campaign, 2) not running the campaign. It is important to evaluate the incremental impact as adwords-driven sales will substitute sales from organic search to an extent. To determine the extent of substitution and seasonality (Christmas etc) you to concurrently 1) run the campaign to customers in your target segment and 2) not run the campaign to a randomly selected sub-segment of these customers - this is the 'do nothing control group'.

I'll clarify further what I mean by 'do-nothing control group' in this instance using a non-AdWords example to explain the concept. Imagine that you've identified (through financial modelling) a segment of 100, 000 customers who you believe would be an attractive segment (would increase lifetime value/profits from) if they bought Online Subscription Service A (OSSA) which the company currently communicates on its website. The prudent thing to do here before communicating OSSA to the 100,000 would be a test. So I'd take 1000 of them at random and send 800 (the 'target group') an SMS/email sometime (say Christmas) informing them about the benefits of the OSSA and don't do so for the remaining 200. The remaining 200 are the control group, and some of these will buy the new product because they hear about it through another comms channel - word-of-mouth, increased organic search because of Christmas (seasonality) etc. Of the 800 in the target group, some will buy OSSA as a complementary product, some will use it to replace an existing service which is a more profitable service for the company, and there will be an incremental acquisition impact through people in the target group telling other people about OSSA. The difference between the per-customer revenue/margin/customer lifetime value impact of the target and control groups gives you true impact of the campaign.

To translate this into the online domain, imagine that you have a business which sells 2 models of car and that you want to increase sales revenue of model A via an AdWords campaign. So you design an AdWords campaign to increase sales revenue of model A and evaluate the conversions, CPA, model A sales revenue and headline ROI of the AdWords campaign. BUT the impact of any campaign is the total sales revenue with the campaign minus what the total sales revenue would be without the campaign. 'Total sales revenue with the campaign' is conceptually simple - it's the total revenues from sales of models A and B. Some people who click the ad will have a look around your website and see that model B better suits their needs (the indirect 'brand halo effect'), some will buy in January, and some people who see the ad will tell other people who will then go on to buy model A or B ('indirect word-of-mouth' effect). It is my understanding that these incremental and substitutional sales can be determined using Google Analytics. Evaluating the 'what the total sales revenue would be without the campaign' requires the do-nothing control group (to determine the organic search substitution impact) - this can be modelled but I would prefer to do a live experiment.

Regards,

Pritpal

Re: How do I measure the true ROI - control group testing?

Rising Star
# 4
Rising Star

Good morning.

 

I've read your additional information (thank you for providing it) and have a couple of thoughts.

 

Of your examples, I think, the auto sales one is a good one. You can measure the value of sales for Car A, the one you advertised for, against the sales of Car B, which you did no advertising for. Although there will always be blurring of the lines (people who clicked an ad because they thought they wanted A but, after reviewing your website, decided to buy B), with enough data you should be able to draw some conclusions.

 

For your second example, the use that comes to mind is geo-targeting--you could geo-target to some specific geographic areas and compare "sales" there to sales in the areas where you are not serving ads. The trick,of course, is to choose your geographic areas so that the populations and demographics are, as much as possible, "apples-to-apples."

 

Either of these two situations-advertising some products but not others or advertising only in specific regions--should provide you with data for comparision of the impact of an AdWords campaign.

 

This is not a situation where you need ACE. There is no way to set up an experiment to show what might have happened if you had served ads or what happened when you did not serve ads. There is no setting for "do not serve ads for 50% of the traffic but tell me about that traffic anyhow." Make sense?

 

 


Theresa
Google AdWords Top Contributor
*~*~*~*~*~*~*~*

Re: How do I measure the true ROI - control group testing?

Visitor ✭ ✭ ✭
# 5
Visitor ✭ ✭ ✭
Hi Theresa

Thank you for your prompt response - my apologies for the delay in getting back to you.

Yes, I agree that careful geographic targeting is probably the best way forward. Another option for a control group would be to 1) create an ACE with two ads which are based on the same keywords - one which is relevant to the search keywords used and one which is completely irrelevant and so won't get clicked even if it results in an impression) and 2) evaluating the impact of both ads. But I'm not sure if it is possible to evaluate the orders from the control group ...

Regards

Marked as Best Answer.
Solution
Accepted by topic author Pritpal W
September 2015

Re: How do I measure the true ROI - control group testing?

Rising Star
# 6
Rising Star

I would caution you not to try to create any advertising structure that is designed to avoid creating activity. There are long-term performance and quality metrics associated with your website's domain name that you risk damaging if you deliberately include low-quality elements in your AdWords campaign(s).

 

In the end, though, the problem is that ACE is not designed to be used the way you describe. It's designed to be used to measure the potential impact of a change in your AdWords advertising. There is no truly controlled way to use it to measure the impact of advertising vs. not advertising. 

 

Your first post said you were new to AdWords, but did not say that your product/service was a new offering. If you've had your website up for a while, you should already have some baseline interest and sales data, which will make evaluating the impact of advertising easier?

 

 

If that isn't a good solution, then it seems to me that the geographic targeting is going to be your best shot at figuring out how much an advertising campaign will affect sales. 

 

 


Theresa
Google AdWords Top Contributor
*~*~*~*~*~*~*~*

Re: How do I measure the true ROI - control group testing?

Visitor ✭ ✭ ✭
# 7
Visitor ✭ ✭ ✭
Good point about the risk involved in introducing low-quality elements. I'll stick to the geographical targeting. I'm on a rapid learning curve at the moment and want to get appropriate measurement techniques in place before I assume P&L responsibility.

Regards