AdWords
4.5K members online now
4.5K members online now
Dive into advanced features like Remarketing, Flexible Bid Strategies, AdWords Editor, and AdWords Scripts
Guide Me
star_border
Reply

statistical significance when running tests?

Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

Hi

 

I´m making a campaign for a conference and have a budget for 2000 dollars.

 

I am going to run a test campaign to try to find the campaigns, ad groups and keywords that convert the best.

 

How do I find out if my test results is statistically significant? How many clicks do I need? 

 

I have made 15 campaigns with 15 adgroups in each campaign and with 4 ads in each adgroup. 

 

I am pretty new to this so all feedback is welcome.

 

/Tobias

3 Expert replyverified_user

Re: statistical significance when running tests?

[ Edited ]
Collaborator ✭ ✭ ✭
# 2
Collaborator ✭ ✭ ✭

I make tests until they spend 20% of my budget and than predict the other 80% of the behaviour of the campaign based on that, because budget limits is a thing to be controled and stoped.

 

In some cases when amount paid for a click is too high and the budget too low, or the time frame is less than 1 week in order to get high and low traffic days, the tests can't gather enough data to be significant.

 

If you get 1000 clicks at 4 cents and spend 40 $ in a day to test some ideeas, that day is the only test you need.

 

Write more details about what you are trying to achieve and what optimisations have you made so far.

Re: statistical significance when running tests?

Top Contributor
# 3
Top Contributor

Hello Tobias;

Great question;

After the experiment has started when the system has enough data that is statistically significant, it will be shown under "experiment" in the segment menu;

 

Read more:

Monitor your experiments

Moshe, AdWords Top Contributor , Twitter | Linkedin | Community Profile | Ad-Globe
Did you find any helpful responses or answers to your query? If yes, please mark it as the ‘Best Answer’

Re: statistical significance when running tests?

Top Contributor
# 4
Top Contributor

If you have 15 campaigns each with 15 ad groups - that's 225 ad groups. If there are then 4 ads per group then you've got 900 ads.

 

Depending upon your sector your clicks could cost anything - but the average global cpc published by Google is 85 cents - so your $2,000 will get you 2,352 (average) clicks. That's 2.61 clicks per ad... I hate to be the one to say this - but nothing is going to be statistically significant on a 2 click sample.

 

You don't mention how many keywords there are per ad group - but that would simply dilute the sample further.... even having just two keywords would give you 1800 ad versions....

Re: statistical significance when running tests?

[ Edited ]
Collaborator ✭ ✭ ✭
# 5
Collaborator ✭ ✭ ✭

Well now, if it is a conference, it would make sense that the campaigns and ad groups are by country and conference topics, so there are only 15 items in the test baseline, the ad groups that are repeated for every country. It doesn't matter that much which ads will bring conversions, as she can use the automatic ad selection for that to always display the better ad.

 

It only matters what keywords are better. So let's hope he can tells us more about the keywords and the scope of her campaign.

Re: statistical significance when running tests?

Top Contributor
# 6
Top Contributor

I don't think I'd be comfortable making those assumptions - and even if I were - there are many more factors that would come into play. I would expect someone looking at a conference 5 miles from their home to respond very differently from someone looking to travel 4,000 miles even if the original keyword and ad were the same and the only thing that differed were their country.... in my opinion there are just too many variables here to make an apples to apples comparison - and if you cannot reasonably compare then you cannot find significant differences.

Re: statistical significance when running tests?

Top Contributor
# 7
Top Contributor

Hi Tobias, interesting question, one that deserves a considered and thought out answer.  Broad generalisations about costs and percentages really aren't much use since each Account is different and performance factors are so hugely variable.

 

As Steve has already said, I think the primary problem with your structure is simply the number of Keywords.  Even if we assume you only have one Keyword in each Group (and I imagine you have many more than that), $2000 is simply far too small a budget to run any kind of reasonable test on an Account structure of this size.  I am, by the way, assuming that the $2000 is for the test not the entire Account budget for "live" advertising as well.  I'm also assuming that we're talking of a "search only" Campaign here.

 

The problem between your budget and your structure is based upon how Google AdWords works, and I'll try to explain.

 

As you'll know, you set a daily budget for AdWords.  Google will attempt to get as close to that budget as possible; sometimes you may spend a little less, sometimes a little more.  If your Campaign settings are to show the Ads evenly throughout your scheduled period (which they will be by default) Google has to ensure your Ad is only clicked as many times in a day as is enough to use up your daily budget.  Let's look at a simple example:

 

Let's say you have just one Keyword, and that clicks on Ads triggered by that Keyword always cost $1.  Let's also assume your CTR (Click through Rate) is 5%.  Let's then say that your Keyword could potentially trigger the display of an Ad 1000 times per day (1000 times per day, people in your targets search on a term that triggers your Keyword).  If your CTR is 5% you could potentially get 50 clicks in a day, costing you $50.  OK, fine, you have a budget of $2000, $50 a day will last you 40 days, more than enough time and plenty of click data.  But you haven't got just one Keyword, you have hundreds.

 


If you keep the daily budget at $50 and add just one more Keyword with the same figures ($1 CPC, 5% CTR, 1000 impressions per day), Google can no longer show your Ad(s) each time it would be possible to do so because to do this would likely induce 100 clicks, costing $100.  So Google can only show your Ad(s) 50% of the potential times it could.  Still with me?

 

Keep adding Keywords and you'll pretty soon find - long before you get anywhere near the number you have - that the figures get silly.  

 

Now, this is not real life.  In real life you'll find that a fairly small percentage of your Keywords will be responsible for a fairly large percentage of your spend so you will get a reasonable number of clicks on a few Keywords.  But what if these Keywords are not the ones that lead to sign-ups?  What if it's all the other Keywords, which Google can't afford to show Ads for, that would lead to conversions?

 

In short, in my opinion, what you're doing - while logical and thought out - is the wrong way around.  What I'd do in your situation is look to start small.  I'd probably have only one Campaign, perhaps only 2 or 3 Groups each with only a handful of Keywords.  I'd pick these Keywords to be a compromise between tightly focused to your theme and "loose" enough to gather a decent number of clicks.  I'd assign a reasonable daily budget - enough that should allow all of these Keywords to show for at least 80% - 90% of possible impressions and I'd run that Campaign for several days.

 

Over those days I'd be looking to tune the Ads to improve CTR and I'd be looking for opportunities to add more tightly focused Keywords based upon search queries.  Over time I'd probably add another Group, and another, as I see trends in the search data.  I might then launch another Campaign to target a different audience, but using the best practices learned from the first.  And so on.

 

This, of course, takes time. Depending upon when your Conference is, you may not have much space left to move in.  But an approach of this nature is going to give you the potential for much better results than a scatter-gun approach with a low budget for a high number of Keywords.

 

Jon

AdWords Top Contributor Google+ Profile | Partner Profile | AdWords Audits

Re: statistical significance when running tests?

[ Edited ]
Collaborator ✭ ✭ ✭
# 8
Collaborator ✭ ✭ ✭

If you want to maintain a big campaign structure, and you have a larger budget , perhaps it's best to contact one of the local adwords partners, who can provide you with the best campaign setup. You can find them via this link

 

https://adwords.google.com/partnersearch/

 

No forum interaction can replace a face to face help

Re: statistical significance when running tests?

Visitor ✭ ✭ ✭
# 9
Visitor ✭ ✭ ✭

Thanks for all the answers!
 
The reason for that I wanted many campaigns and adgroups was because I wanted to have few keywords in every adgroup that was directly connected to my ad. This would then give me a good quality score which will affect my ad positions on the page as well as my prices (correct?).
 
The conference is about bring together even organizers and presenting the latest event technology.
 
My plan was therefor to have the following words on campaign level: 
 
---Technology, design, management, training, it, science, logistics, tech, Solutions, Planning, 
 
I would then for every campaign use the following words on ad group level:
 
----Conference, Event, Summit, Meeting, happening, experience, convention, exhibition, exposition, fair, festival, fest, seminar, forum, workshop
 
and for every adgroup I would only have a one keyword phrase combined with location. So for example:
 
one ad in the Campaign "Technology" and adgroup "Conference" could for example be
 
(name of conference) 2013
Conference Technology displayed 7th
June in Stockholm. Get your tickets here!
(Link)
 
This adgroup would have these keywords: Conference Technology, Conference Technology Sweden
 
But your advice against the "scatter-gun approach" makes sense to me now. So I guess what you are telling me is to not care so much about quality score in the start and put more of my keywords in fewer campaigns and adgroups and then over time start making more campaigns and adgroups. Correct?
 
Extra info: It is a small Conference and the 2000USD is for the whole adwords campaign

Re: statistical significance when running tests?

Top Contributor
# 10
Top Contributor

Hi Tobias, I think my overall advice - in virtually any situation - is start small.   It may not be that you actually have more Keywords per Group - a small number of Keywords is a good idea - it may simply be you don't use all the Groups you're planning initially.  

 

I'd try - if possible - to identify a small number of Groups, say only four or five, that seem to have the greatest potential and run with them initially.  Yes, Quality Score is important but it may not be good from the beginning whatever you do and, depending upon your actual CPCs and clicks, may have only a relatively small impact on your performance.

 

Are you planning to use the Display network?

 

Jon

AdWords Top Contributor Google+ Profile | Partner Profile | AdWords Audits