AdWords
4.2K members online now
4.2K members online now
Improve your AdWords performance and boost your ROI, CTR, and Quality Score
Guide Me
star_border
Reply

Do we just measure CTR when doing A/B testing?

[ Edited ]
K S
Visitor ✭ ✭ ✭
# 1
K S
Visitor ✭ ✭ ✭

Dear all expert

 

My agency is helping to develop A/B ad copies testing for branded terms to improve CTR, and thus conversion.

 

Here's the result:

Testing variable:  CTR: 9%;  CPC: $0.60;  Avg pos.: 1.3

Control variable:  CTR: 5%;  CPC: $0.50;  Avg pos:  1.6

*The max. bid prices were set as the same.  Each has delivered about 10K impressions.  Testing period: 2 days.

 

The agency claimed the new copy was more effective as the CTR was higher.

 

I am curious if:

1) the testing period is enough to conclude the result, even the brand campaign.

2) Do we have to consider CPC and Avg pos as well? I'd assume a better ad copy should see a lower CPC yet higher ad rank/CTR.  However, in this case, though the CTR was higher than the control, the CPC was also higher that may also lead to a better avg pos --> better CTR.  Or the increase in CPC is insignificant so we can ignore this?

 

Any advices would be greatly appreciated.  Thank you very much.

 

 

 

2 Expert replyverified_user

Do we just measure CTR when doing A/B testing?

Explorer ✭ ✭ ☆
# 2
Explorer ✭ ✭ ☆

Hi,

 

 

If your goal was to improve the CTR of your adverts then I'd say that a $0.10 CPC improvement isn't worth a 4% drop of CTR.

However saying this, I would recommend that you do run the test for more than 2 days, a week minimum in my opinion.

 

Hope this helps,

Do we just measure CTR when doing A/B testing?

Top Contributor
# 3
Top Contributor

Hi @K S

Well.. It looks that the agency is correct; assuming the max bid in both the testing and the control groups was the same. (e.g. $1)

If the max CPC was not changed, the two metrics to consider were actual CPC and avg position. 

The sampling size is  determined by impressions: 10,000  impressions are a sampling size, which   is sufficient to conclude a statistically significant  result. Though, to eliminate the effects of users'  behavior during the week, (e.g. Users don't search evenly during all week days - if applicable) I would extend the test for at least a week.

Moshe, AdWords Top Contributor , Twitter | Linkedin | Community Profile | Ad-Globe
Did you find any helpful responses or answers to your query? If yes, please mark it as the ‘Best Answer’

Do we just measure CTR when doing A/B testing?

K S
Visitor ✭ ✭ ✭
# 4
K S
Visitor ✭ ✭ ✭

Thanks Matthew, understood.

 

Just curious, why don't we just simply optimize the CPC to test if a better ad rank would deliver a better CTR, as I do see an opportunity to improve the ad rank further (there were 3-10% search lost due to rank), normally the higher ad rank = higher CTR?

Do we just measure CTR when doing A/B testing?

Top Contributor
# 5
Top Contributor

Hello, K S.

 

I'm with Matthew and Moshe on the minimum of one week. I've also seen wild changes in behaviour across the days of the week, across weekends and sometimes huge variations according to the time of the day and also day of the week. So I'd definitely extend the test for one week.

 

As to what to optimize for, it depends on the company and its KPIs. Why not conversions? Or better yet, CPA or ROAS? Clicks, CTR, CPC, they can be KPIs as well, but why not cut to the chase and decide how much money a conversion is worth for the company and how much they are willing to pay for it?

 

The AdWords system can be set to target these KPIs and can also optimize according to them, way better than us humans can do.

 

So if you have a campaign where you don't change things too much (I can't do it in E-Commerce set-ups where 3-4 new ad groups are added every day and maybe more are paused, or bids are changed), you can alter your bidding strategy and let the system do the leg work.

 

Hope it helps.

 

Calin Sandici, AdWords Top Contributor | Find me on: Google+ | Twitter | LinkedIn | myBlog
Was my response helpful? If yes, please mark it as the ‘Best Answer.’ Learn how here.

Do we just measure CTR when doing A/B testing?

K S
Visitor ✭ ✭ ✭
# 6
K S
Visitor ✭ ✭ ✭

Thanks Calin, I concur.  Definitely we see conversions and ROAS the utmost important metrics.

 

We did see a drop in conversions since June this year compared to STLY while we indeed invested more this year. The agency still has no clues about the drop, but advised to do a couple tests to improve the CTR, as they believe the CTR increased by 50% the conversion would be improved by 50% if everything remains unchanged.  I have also inquired why not to wait and see if the tests improve ultimately the conversions but were advised that the conversions would not come in such a short period of time and were told this is the correct way and the current top priorities.  I doubt though and feel like "the blind leading the blind". That is why I am here to ask the experts. 

 

The bidding strategies have just been in place, can't wait to see if their recommended strategies really work!