AdWords
6.5K members online now
6.5K members online now
Use AdWords conversion tracking and reporting to measure your results. Have a question about Google Analytics? Post it here, on the Google Analytics Community!
Guide Me
star_border
Reply

Normalize Split Test results

Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

Hi to everyone,

 

i've a question, i would like to improve some of my ads but my ad groups are pretty vertical and so the result is the my ads are really focused but the volume is low and try to optimize the ads text take weeks.

 

The current structure of my ads is pretty similar, what change between one and the other is actually the "city".

So here's an example:

 

Los Angeles + Keyword

Line1

Line2

 

New York + Keyword

Line1

Line2

 

San Francisco + Keywords

Line1

Line2

 

What i would like to achive is to run ad test on all the ad groups at the same time (so i can get results faster) but the major problem is that every ad is having a different avg position so i cannot just merge all the results to take a decision... what i should do is do a sort of weightened calculation that takes the avg position into consideration making the CTR comparable between the different ad groups.

 

Some one knows how to do that? or have a formula or an excel sheets which can help me handle this?

 

I hope i've been clear because my english is not perfect.. if somehting is not clear just ask, i'll try to do my best to explain it better.

 

Thanks to everyone

 

 

1 Expert replyverified_user
1 ACCEPTED SOLUTION

Accepted Solutions
Marked as Best Answer.
Solution
Accepted by topic author Ale84
September 2015

Re: Normalize Split Test results

[ Edited ]
Top Contributor
# 2
Top Contributor

I don't know if I understand exactly what you mean, but assuming that you want to experiment by varying Line 1 and Line 2 in all ad groups (LA, NY, SF) and gauge the aggregate performance, such as "variation 1 for LA + NY + SF" versus "variation 2 for LA + NY + SF", then you can do the following:

 

  • create a campaign experiment
  • have one ad variation per ad group
  • duplicate all ad groups and have, in each ad group, its ad variation
  • make the group of old ads run only in the "control" segment of the experiment
  • make the group of new ads run only in the "variation" segment of the experiment
  • judge the performance by looking at the "Campaign" tab, and enabling the "Experiment" segment

 

If the "control" segment did better, old ads win, as a group. If the "variation" segment did better, new ads win, as a group.

 

Hope it helps. If this is not exactly what you want, please take a bit more time to describe what you want to test against what.

 

P.S. It goes without saying that the AdWords editor is your best friend.

Calin Sandici, AdWords Top Contributor | Find me on: Google+ | Twitter | LinkedIn | myBlog
Was my response helpful? If yes, please mark it as the ‘Best Answer.’ Learn how here.

View solution in original post

Marked as Best Answer.
Solution
Accepted by topic author Ale84
September 2015

Re: Normalize Split Test results

[ Edited ]
Top Contributor
# 2
Top Contributor

I don't know if I understand exactly what you mean, but assuming that you want to experiment by varying Line 1 and Line 2 in all ad groups (LA, NY, SF) and gauge the aggregate performance, such as "variation 1 for LA + NY + SF" versus "variation 2 for LA + NY + SF", then you can do the following:

 

  • create a campaign experiment
  • have one ad variation per ad group
  • duplicate all ad groups and have, in each ad group, its ad variation
  • make the group of old ads run only in the "control" segment of the experiment
  • make the group of new ads run only in the "variation" segment of the experiment
  • judge the performance by looking at the "Campaign" tab, and enabling the "Experiment" segment

 

If the "control" segment did better, old ads win, as a group. If the "variation" segment did better, new ads win, as a group.

 

Hope it helps. If this is not exactly what you want, please take a bit more time to describe what you want to test against what.

 

P.S. It goes without saying that the AdWords editor is your best friend.

Calin Sandici, AdWords Top Contributor | Find me on: Google+ | Twitter | LinkedIn | myBlog
Was my response helpful? If yes, please mark it as the ‘Best Answer.’ Learn how here.

Re: Normalize Split Test results

Visitor ✭ ✭ ✭
# 3
Visitor ✭ ✭ ✭

Actually you understood exactly what i need to do, i do not completely understand your answer cause i never used the "Campaign Experiment", actually you just le me discover it.

 

I'll get some "know how" on the "Campaign experiment" and i'll try your suggesytion.

If you have some good tutorial that you want to suggest me it would be greatly appreciated, otherwise i'll find them!

 

Thank you very much for you collaboration

Re: Normalize Split Test results

Top Contributor
# 4
Top Contributor

You're welcome, I'm glad I understood the situation.

 

Here is pretty much everything you need: http://support.google.com/adwords/answer/2385204?hl=en

 

It's actually rather simple. You create an experiment, decide how much of the traffic goes to each segment, name your experiment, decide which ad group is part of which segment, and start it.

 

If the above document is not clear enough, please get back to our community and tell us where you're stuck. Or just get back to us and tell us it works Smiley Happy.

Calin Sandici, AdWords Top Contributor | Find me on: Google+ | Twitter | LinkedIn | myBlog
Was my response helpful? If yes, please mark it as the ‘Best Answer.’ Learn how here.

Re: Normalize Split Test results

Visitor ✭ ✭ ✭
# 5
Visitor ✭ ✭ ✭

Perfect,

 

thank you very much.

 

I'll come back once done or stucked!

 

Have a nice day

Re: Normalize Split Test results

Visitor ✭ ✭ ✭
# 6
Visitor ✭ ✭ ✭

Hi, i'm here again cause i'm a little bit confused on a thing, i was trying to implement your solution, but i got stuck in another complication that i didn't atually mentioned in the previus posts.

 

My account is strctured in this way

 

State (campaign) -> City (ad group)

 

I've done this cause i've to manage daily budgets based on state.

Said that, i've to take into consideration that there is only 1 big volume city for each campaign (in my case) so this means that in order to get faster response i've to work at an upper level so not at ad groups level but at campaigns level.

 

The Campaign experiment seems to work at campaign level so i cannot compare results from two different campaigns (also if they are actually pretty simlar because the difference remain the same as i explained in the first post).

 

So in the end what i would like to do is try two different ad on two or more different campaign like this:

 

California (Campaign) -> Los Angeles (ad Group) -> Original AD

New York State (Campaign) -> NY (Ad Group) -> Original AD (same as previuos only city name change)

.....

 

VS

 

California (Campaign) -> Los Angeles (ad Group) -> Variation AD

New York State (Campaign) -> NY (Ad Group) -> Variation AD (same as previuos only city name change)

....

 

Do you think that this is possible?

 

Thank you very much for your help!

 

 

 

Re: Normalize Split Test results

Top Contributor
# 7
Top Contributor

Hello, Ale84.

 

I'm afraid that pan-campaign experiments are not possible. If you want to do it like that, you'll have to create per-campaign experiments, and have one campaign and two ad groups per state. And hopefully results will be similar, and not be state-dependent.

 

But maybe for the experiment's sake you can forget about balancing the budgets until you get some statistically significant data and do the balancing afterwards.

 

 

Calin Sandici, AdWords Top Contributor | Find me on: Google+ | Twitter | LinkedIn | myBlog
Was my response helpful? If yes, please mark it as the ‘Best Answer.’ Learn how here.

Re: Normalize Split Test results

Top Contributor
# 8
Top Contributor
Then again, you can do the following:

- one campaign and budget per state, with an experiment running
- control and variation ad groups inside each campaign, split 50-50

Then you download your report in excel, label your control and variation ad groups and sum columns by label (impressions, clicks, conversions) and judge their performance.

AdWords splits your traffic and keeps it pretty even, Excel/Google spreadsheets do the math.

If you want to make sure when the results are are statistically significant, use a calculator you can find online and a confidence interval you're happy with (95-99%).

What do you think?
Calin Sandici, AdWords Top Contributor | Find me on: Google+ | Twitter | LinkedIn | myBlog
Was my response helpful? If yes, please mark it as the ‘Best Answer.’ Learn how here.

Re: Normalize Split Test results

Visitor ✭ ✭ ✭
# 9
Visitor ✭ ✭ ✭

Hi,

 

thanks again for your help.

 

For your last answer, this was the first thing i would like to do when i came here but my doubt is about the average position of every ad group.

 

Let's say that in LA i've (imp/click/ctr/avg position):

 

Variation 1 - 1000 / 10 / 1% / 3

Variation 2 - 1000 / 15 / 1.5% / 3

 

and in new york i've:

 

Variation 1 - 10000 / 2 / 0.02% / 6

Variation 2 - 10000 / 2 / 0.02% / 6

 

In this case, i cannot sum impressions,clicks and get results, because the avg pos must be taken into consideration otherwise results could be potentially wrong.

 

I think that what i need here is some kind of "avg ctr expectation" whicht tells me what should i expect for and ad that is shown in 3rd position and onw shown in 6th.

 

That way i could say, ok if i got 0.02 CTR in 6th position,  this means that if it was in 3rd position it would hit a 2% CTR.

 

Having the CTR normalized based on the avg position i could really sum variations from different campaign/groups and check which variation wins.

 

Am i right or i'm missing something?

 

Thank you very much!

 

 

Re: Normalize Split Test results

Top Contributor
# 10
Top Contributor

You're right, but what you want is not so easy. Google AdWords also takes into account the "position-induced" advantage or penalty at CTR level when it computes quality score.

 

However, if everything else is the same in control and variations ad groups, and the system rotates them evenly (BTW, you could even use only one ad group and two ads, and set rotation to be even), then, in each state, you will have both ads show in all possible positions.

 

And if in one state they hover around P2 and in the other P3, they are doing this for both variations (control and experiment). So when adding all impressions (not CTR) and clicks and computing the average CTR per group, you could still say that one segment performed better than the other. You don't have to judge the performance in a theoretical and corrected position.

 

So, you can end up with:

 

- State 1 - control - 1008 impressions, 50 clicks, avg. pos 1.32

- State 1 - variation - 1015 impressions, 45 clicks, avg. pos 1.71

...

- State 3 - control - 1809 impressions, 100 clicks, avg. pos 2.09

- State 3 - variation - 1852 impressions, 140 clicks, avg. pos 2.32

 

You needn't worry about the fact that position varies somewhat, because it may vary precisely because the system detected that a certain search term, when paired with one of the variations, performs better in terms of CTR, and thus awards it a higher QS.

 

However, now that I think more about it, even if you find out that (State 1 + ... + State 3) (variation) performs better than (control), as a group, you may still find out that for State 3, it is (control) that performs better, for whatever reason. And this should be reason enough to approach that State differently.

 

But if they all converge towards the same result (one variation performs better in each of the states), then it is pretty clear (yet not certain) that you should use that variation.

 

Even if the sums are significantly different (from a statistical point of view), when judging individual performance, surprises may arise.

 

Short example:

- out of 1000 impressions in each state, per variation, control gets 80 clicks and experiment gets 65

- the clicks are distributed as follows: 35, 15 and 40 for control, 25, 13 and 27 for variation

 

Maybe 80/3000 is within 95% confidence (I did not calculate) better than 65/3000, but I'm pretty sure that 15/1000 is not clearly better than 13/1000. Maybe if the experiment would continue long enough, in State 2 results would be reversed.

 

Having said that, I think I should focus more on statistics Smiley Happy.

 

Good luck, hope I clarified more than I blurred with my wannabe scientific observations.

 

 

 

Calin Sandici, AdWords Top Contributor | Find me on: Google+ | Twitter | LinkedIn | myBlog
Was my response helpful? If yes, please mark it as the ‘Best Answer.’ Learn how here.